Posts de ‘Flávio Ribeiro’

[Flávio Ribeiro] Using Open Source to Create a Video Thumbnail Service

Monday, July 31st, 2017

Using Open Source to Create a Video Thumbnail Service

Last week The New York Times hosted the 2017’s edition of Makers Week, an entire week dedicated to working on projects and ideas employees want to test, build and innovate on.

There are no boundaries to projects, nor specific scope requirements. You can use your time to do research on new topics or disciplines, contribute to open source projects, fix bugs or create products from scratch. It’s definitely not a new thing and I believe most companies are doing this now so I will not dwell on it. If you want to know more about how this works here at our office, you can read this article from last year’s edition.

Increasing our click through rate

One of our goals as a video team this year is to increase our click through rate when our users see our video player. We usually do a pretty good job of selecting thumbnails for all of our videos — often using photos taken by our own photo journalists. However, after watching a talk from JWPlayer folks and reading a research article from Netflix, it became clear to me that we can’t just assume a thumbnail is good for a given video. We should actually use some data and run A/B tests to come up with the best one.

In order to try to improve our thumbs, we’d need to be able to create them in a cleverer and faster way. So I thought that creating something to generate and serve thumbnails for any of our videos at any time, on-the-fly (on the time of an HTTP request), would be a great project for my Makers Week. When working at, I saw some amazing open source projects being created and maintained by some wonderful engineers I met over there so the scope of the project would be nothing more than putting some of those projects together.


Before starting, I invited Francisco Souza to help me. He’s a specialist when it comes to application deployment and all things related to Docker/Kubernetes/Google Cloud Service. I’m glad he accepted, things were much easier with his help.

Using Open Source to Create a Video Thumbnail Service

As I said before, Lumberjack is a combination of open source projects that allow thumbnails extraction for videos. It leverages the powers of Thumbor, NGINX, nginx-thumbextractor module (or simply módulo do Wanden as we like to say in Portuguese) and the Lua programming language in order to extract and generate pictures for any given moment of any given video of our video library.


We are deploying three services in different containers in a Kubernetes pod:

  • NGINX + Video Thumbnails Extractor Module: Responsible for scanning the video on a given mountpoint, extracting the frame of a given timecode and returning it on the fly. We used gcsfuse to mount our production GCS bucket where our videos library reside, allowing the module to go there and get the frame requested.

  • Thumbor: Responsible for applying filters, crops and resizing of images. Thumbor relies on different engines including OpenCV to apply functions passed as query strings on the HTTP request. It includes smart cropping, face and assets detections and a bunch of other cool stuff. You should take a look on this powerful project and see what it can do for you. The Docker image we are using for this service is available here.

  • NGINX + Lua Application: A simple Lua script that runs inside NGINX with this Lua Module. Given a VideoID, the script is able to fetch the right MP4 video asset from our internal API and send to the thumbnails extractor. The application is also responsible for getting the parameters passed from the user on the URL such as filters and resolution, send to Thumbor and return the final picture back to the user.

Using Open Source to Create a Video Thumbnail Service

We decided not to open source Lumberjack. At the end of the day the whole project is just some business logics around the open source projects I mentioned on this post. So really all of the credit belongs to the guys who created and maintain Thumbor and the thumbextractor module. If you want something similar to what we did you can just deploy them.

Some Use Cases

Imagine we want to crop a frame from this video at 30 seconds to use as the cover of another vertical video on mobile phones (9×16), in grayscale, with smart cropping. We just need to pass the below as parameters and the service will make it for you:×1600&smartcrop=true

Using Open Source to Create a Video Thumbnail Service

Another great feature supported by the thumbnails extractor is the generation of sprites or tiles. We can, for example, generate a sprite map with thumbnails every one second and use it as a moving cover when the user hovers on it:

Using Open Source to Create a Video Thumbnail Service

See how it looks by placing the pointer over the image below:

We can also use the similar approach to use the sprite map for the thumbnails on the seekbar as you can see on this player:


The Thumbnails Service is now part of our Q4 roadmap and Lumberjack is already in production. We didn’t present to the newsroom yet as we are still facing some performance issues with caching and the GCS bucket. The NGINX locations are not optimized so we’ll need to look back on them as well. To sum up, we’ll need to revisit a lot of stuff that we did in a rush during Makers week (I don’t even need to mention that we have zero tests for the Lua script too).

For the future, we want to be able to detect perfect looping GIF’s for using them on social channels and also detect highlights of a video based on the audio and closed captions. That way we can suggest thumbnails for our newsroom editors within our CMS. Finally, we’d love to integrate thumbnails tests with our A/B framework the same way Netflix did.


[Flávio Ribeiro] Snickers: Open Source HTTP API for Media Encoding

Monday, July 18th, 2016

At the beginning of this year we created a group on our video engineering team to deal with the ingesting, encoding, publishing and syndication of The New York Times videos. The main goal of the team was to build a pipeline that is vendor agnostic, cloud-based, efficient and elastic. We also wanted to build an easier workflow for video producers in the newsroom and third-party partners who distribute our videos. We named this team Media Factory.

After a few months of development, the pipeline is almost ready to be deployed to production. It consists of three different steps: Content Acquisition, Transcoding and Distribution. The content acquisition part is responsible for receiving ProRes/Raw videos, the transcoding API transforms the source files in playable videos by our supported browsers and devices and the distribution step sends the final videos to our CDN.

Probably the most important and complex part of the pipeline is the transcoding process. Instead of using heavy and expensive hardware for that we decided to create clients for different encoding services like, Elastic Transcoder and Elemental. This enabled us to decide which one to use based on a set of variables such as availability, speed, formats supported (HLS, webm, mp4, DASH, etc) and even cost.

While discussing the design of the pipeline and the integration with cloud services, we thought it would be useful (and FUN!) to implement another provider that would be just a tiny wrapper for FFmpeg or GStreamer. By using this tiny wrapper, we would have the freedom to deploy anywhere, implement new features (such as the generation of thumbnails) and perhaps, one day, be used to encode our lower priority jobs.

We have plans to write a blog post about the details of each step and how they work together on the Open Blog. We should work on that as soon as we have people at the company using the pipeline.

Time To Learn Some Go

After having a lot of fun on the journey to remove Flash from our player I got attracted by the chance to be part of this team. I was already willing to work with back-end services again and knowing that the company is adopting Go for API’s turned this opportunity into something even more attractive. Although I was participating on meetings and design decisions, I actually started working on the team in April.

I had never worked with Go before and after a few years plunged on dynamically typed languages I felt a bit of pain during the first steps working with it. The rest of the team was already fluent and I though I’d need something to exercise myself and keep up with the team’s pace.

snickers logo

Creating this dummy provider that is a wrapper for FFmpeg seemed like a good chance for me to evolve my Go skills. A month ago I started writting it, I called the project Snickers. I put this name because it’s my favorite chocolate and I was holding one when I created the GitHub repository.

To be honest, the title of this post doesn’t actually represent the current state of the project. Right now, it looks much more like a proof-of-concept/embryo project than anything else.

It supports the download of sources from HTTP and Amazon S3, encode them in H264/MP4 or VP8/WebM using a FFmpeg binding and uploads the results to Amazon S3. We still need a plethora of features to be able to actually call it a Media Encoding API, but one of the purposes of this post is to broadcast the idea and try to engage people to collaborate. Snickers already has contributors!

Next Steps

I have a bunch of ideas for this project. The next steps include building clients in JavaScript so people can use Snickers directly on the browser (with nice progress bars and stuff) as well as a client in Go, probably as part of the encoding-wrapper project. We also need to write some glue code for the Transcoding API so Snickers can be considered in the decision process of Media Factory pipeline. We use a lot of HLS at the company so HLS support is mandatory and probably the highest priority feature.

I’m also interested in working on the creation of a GStreamer driver for the encoding part so we can have benchmarks using different drivers. I saw that there’s already a Gstreamer binding for Go and it seems that the code is simpler than handling libav calls.

You can find some instructions on how to deploy and use the API as well as some examples. I also labeled a bunch of relatively easy issues in case you want to work on and be part of this project.

[Flávio Ribeiro] Working at the Video Team of The New York Times

Tuesday, December 22nd, 2015

So, after more than 3 years of the publication of this post and almost 9 months working here, I’ll give my two cents and elaborate a bit about my experience on the day-to-day work at the video team of The New York Times.

the new york times


The technology of the video team is split into three teams: Video Players (VHS), Times Video and the brand new Video Publishing API team. We are growing really fast and things are changing in every direction; it’s clear to me that everyone is excited and committed to our upcoming challenges.

We work in a fast-paced environment and always adapting our agile processes. However, I never feel rushed and everyone is committed to do deliveries with meaningful impact. The engineers have the freedom to give input on process, features and backlog. For example, after this quizz I was able to implement and send the thumbs on seekbar feature to production (

We have frequent 1:1’s and occasional brainstorm meetings. Our daily meeting is around 11 AM and takes no longer than 10 minutes, we usually have people working from home and the meeting always happens on Google Hangouts.

video team at nytimes

one of our daily meetings some time ago

Our team consists of people from all around the world such as U.S., China, Brazil, Canada among others. Our cultural differences are always the subject during lunch.

Speaking of lunches, while we don’t have free food we do have a restaurant inside the building that serves typical food from a different country every day (somedays the food is awesome, sometimes not much).

Generally, a new hire rotates between all projects during the first month. This helps with understanding the whole video platform and familiarize with both team members and the codebase.

The Times also hosts a tech blog, 100% days (two days to do whatever you want) and hackweeks. There’s also some events open to the public, you can find the schedule of events here.


We use JIRA to track our projects, milestones and sprints. I personally don’t like it — it’s too slow, hard to glue with the code and full of buttons. We use Github for code and Slack for communication throughout the entire company. Some of my favorite channels include #coffee for caffeine addicts, #conferences-events for events and #snacks for free goodies around the building.

The VHS team uses plain JavaScript for the players. We use Grunt for tasks and Karma + Chai + Mocha + Sinon for tests. One of our targets this year was to get rid of Flash so we needed to integrate the player with VPAID for ads and use hls.js for live streams. This whole journey deserves it’s own post, we’re planning to do that on our tech blog.

The Times Video team is responsible for Times Video product. We are using ES6 with Babel and moving from Backbone to React. The back-end is made with Node + Express + Memcached. For testing, we’re using the same stack as VHS.

The Video Publishing API is still being assembled and will support our encoding pipeline, including the production & editing integration, security, encoding profiles, and syndication among others. The company is in an adoption movement of the Go language and we are willing to use it to build the API’s.


I was hired in April 2015 as a contractor and waited until October for the H1B visa, when I started working as a full-time employee. During this time I went to New York twice to get to know the team. I also used those trips to pick out apartments in different neighborhoods to get a better idea of where I would live.

It was my first experience working remotely and it was tough, I was living in Rio de Janeiro at the time and decided to spend some time with my parents and friends in Campina Grande, where I grew up. My life became a whirlwind during those months and my productivity was well below normal.

flavioribeiro github streak

my github streak proving I’m going back to a good shape

Since I arrived in New York, I feel much better. I have a routine, the commute works and the city surprises me everyday. As a natural Brazilian accustomed to a hot weather, I’m afraid of the upcoming winter. However, I’m lucky the cold is slow to arrive this year.

At the building, every engineer has it’s own cube and it’s huge. More than twice as size as my last table. It fits some books, figure toys, a small whiteboard and sometimes another coworker when pairing. You get a little isolated and is quite different from how are the open tables in startups, but I personally like the privacy that the cube give to me. The office is calm and quiet, I never turned on the noise cancellation of my headphone.

I’m very happy and hope to contribute even more on next year. If you want to be part of this amazing team, we are hiring.

video team

united colors of video team!

Related links:

[Flávio Ribeiro] It's time to move on

Thursday, March 26th, 2015

After almost 4 years immersed in a lot of exciting challenges, it’s time to move on.

Sunset by

Sunset from my balcony by Fany

When I started working at, I realized that every if that I wrote would run in hundreds of thousands of computers around the country. Until then, I’ve only worked on small startups in the Northeast of Brazil where the impact of their products have smaller proportions. Describing this feeling is hard. I would argue that, for a software developer like me, knowing that your code is being used is as rewarding as receiving real money.

During the time at Globo, I faced huge projects. As we’ve worked developing software for the video delivery stack, almost every big transmission - like elections or FIFA World Cup - had their special issues and corner cases. Following the trend, we developed a live video streaming stack from scratch and seeing this new infrastructure beating the concurrent users peak twice was priceless.

United Colors of WebMedia

Part of WebMedia Team during the World Cup. We were still believing we would be Champions.

I could also work with the most awesome engineers I’ve ever met. Sometimes I felt that the lunch time was kind of a graduate class (actually, I should had taken a pencil and paper to it). The discussions ranged from writing tests to how telecommunications works.

Since the beginning of last year I’ve worked full time on an open source project which was the real reason for me to get up out of bed to go to work and sometimes even rise up early to answer one issue or fix a bug that someone dropped on github. I’ve already said this here but…Damn, how awesome is working with FOSS!

About 6 months ago I was catching myself visiting LinkedIn, hypothetically looking for a new job, and this made me think that it was time to move on. I’ve read somewhere that when you’re feeling attracted by other opportunities, it’s a big red signal that you’re not happy anymore and need a change. I’ve applied to some companies and met a lot of great guys during the interviews but the last one was mind changing.

If you get the chance to work overseas, take it. There is never a right time. And we always regret the things we don’t do far more than the things we do. (Shane Rodgers)

I’m joining the Video Team at The New York Times. Next week I’m heading to New York to visit the office and meet my coworkers. I must say that I’m totally anxious and excited about it, but a lot of things need to happen before I move definitively to America. Until there, I’ll work remotely (which will be a challenge apart).

I hope that everything that I lived these years were only the beginning and I’m sure I have a lot of things to learn. I’ll try hard to publish more posts regarding the obstacles I’ll face and I hope you’ll be cheering for me.

Related Links:
Hey Feio, how is working at The Times?
What I whish I knew when starting out as a Software Developer
The career advice I wish I had at 25

[Flávio Ribeiro] Disserting on Clappr: An extensible media player for the web

Tuesday, November 25th, 2014

Motivated by Leandro, today I’m gonna write a bit about Clappr, an open source media player for the web. If you’re Brazilian or know portuguese, there’s also some useful information at Thiago’s talk. Actually, I will inadvertently and explicitly steal some data from it.


During the last 4 years at there was no team focused specifically on the development of the video player. It was a monolitic flash component based on OSMF, mainly designed for playing progressive download videos on on-demand scenarios and also cover live streams through RTMP protocol.

Globo Player 2.0
Previous player

At the beginning things were good, but in the course of time, a lot of new features were added (like support for subtitles, advertisement, DRM, new streaming protocols, picture-in-picture, etc). Since there wasn’t an official owner of the project and maintenance programming is a dirty work, we were building a wall with crooked bricks. Player tasks started to be faced with revulsion and pain.

As the FIFA World Cup 2014 project came with a bunch of new requirements and knowing that our live video streaming stack was rebuilt (HLS in favor of RTMP), we decided to go against Joel and bootstrap a brand new video player in order to support the requirements and improve the overall performance on serving the matches and upcoming transmissions.

Open Source

At we use hundreds of open source projects and when starting a project from scratch, we usually ask ourselves Why not open source it?. First, there’s the motto of give something back to the community, but I personally believe that committing and pushing on Github, put your code available to the world and maybe build a community around your baby acts like a fuel for your brain, pushes you to the limits of productivity.

Open source software acts like a fuel for your brain, pushes you to the limits of productivity.

Since we are in times where video delivery is very heterogeneous between browsers and devices, we choose for a plug-ins based architecture, allowing us to provide almost everything as open source, except the components that interact with our internal infrastructure.

Visit Open Source to see more open source projects maintained by my co-workers.

Clappr Architecture

Clappr Architecture, First Meeting
Designing Clappr, you can see the full image here.

With CBSE in mind we could figure out that there were three different plug-in types:

  • Playback Plugins: Responsible for playing a given source. When embedded, Clappr will search through playback plugins until find one who canPlay() the source.

  • Container Plugins: Every playback is associated with a container. In most cases, only one container and a playback will be instantiated. It is inside containers that, guess what, the container plugins work. Some examples are the built-in stats plugin that listen for playback events (buffering, bufferfull, play/pause, etc), the spinner and the watermark.

  • Core Plugins: These are the ones who control everything. Let’s say you want two videos playing at the same time, the main one and one smaller at the corner, like a picture-in-picture. A core plugin is able to instantiate two containers with a playback plugin each, set the size of the secondary video and put it on a bigger z-index.

There’s also a bunch of built-in components, like the Mediator who is responsible for routing events between scopes (and even make the Flash ↔ JS bridge when necessary) and the Loader who manage to load all built-in and external plugins. Wat? External plugins?

External Plug-ins

It’s not only possible but simple to create your own external plug-in. There’s a Clappr Plug-ins Generator that will ask you the kind of plug-in you want to create and build the structure for you. I’ve used it to create BemTV plugins, the playback and a container plugin to show peer-to-peer statistics. has some proprietary plug-ins such as media control bar, thumb-on-seek and others.

Clappr with Plugins
Clappr with some proprietary plugins used at World Cup


Current Status

Clappr already supports the mainstream formats and has some special features like DVR (travel in time on live streams). We’re heading to 1.0.0 release, but first we need to create a decent documentation and landing page. Both are on the go.

We tought that the architecture was so solid that motivated us to bootstrap mobile versions of Clappr. We already created iOS and Android repositories in case you want to follow our future progress.


We are just three commiters, but there are some easy issues opened. If you felt encouraged in contributing, I’ll be more than glad to help you. You can find us at clappr discussion list and I’m always online as flavioribeiro at freenode.

[Flávio Ribeiro] BemTV: Hybrid CDN/P2P Architecture for HLS Broadcasts

Thursday, August 7th, 2014

During the last three years I’ve been working on the Live Video infrastructure at and I’ve realized that one of the biggest problems we had here in Brazil is related to CDN throughput and telecom infrastructure in general. Looking at what has been happening with online streaming of live events around the world (as The Oscars and True Detective Finale) and the fights involving OTT services and telecom carriers, one can realize that this is a common problem everywhere.

With that in mind, I decided to explore this problem and as I’m attending the last year of my Master’s and my colleagues at work were bootstraping a new team to develop a video player from scratch, it happened to be the perfect timing. I joined this new team and defined my Master’s thesis theme aiming to develop a Hybrid CDN/P2P architecture for online live broadcasts using the popular HLS protocol together with WebRTC.

First, we baptized the new player as Clappr - the name “clapper” comes from clapperboard - and open-sourced the project. You can follow our progress on github and yes, we’re aware that we’ll need to write some documentation on how to deploy the player and write plugins, it is already on our backlog. A nice logo and better UI are also on the go.

After that, I’ve created a clappr plugin that tries to drain video segments from peers in order to decrease the number of requests to CDN servers, reducing the cost of transmission and enhancing system’s scalability. If something goes wrong between the peers, it will get from the CDN without impacting user experience. This project is called BemTV and today I’m releasing its first version.

BemTV in action

BemTV is open source. I must warn you that this is a nightly build and needs some improvements before debut on production environment (I’ve performed just some controlled tests) but I’m inviting you to test it on and also look at the code. The video in there is in loop, emulating a live video streaming behavior.

I’ll be more than happy to answer questions and fix bugs or problems that arise. Call friends near you, look at the statistics box and tell me if the video segments are being exchanged between you guys!

[Flávio Ribeiro] Naming Things

Saturday, December 14th, 2013

There are only two hard things in Computer Science: cache invalidation and naming things.

Phil Karlton

I was thinking about how important naming things is to me. Since the beginning of my undergraduate studies, I paid close attention to the names of every project I’ve done and every company or lab that I’ve worked at.

I’m also catching myself arguing about the name of things on my daily work and I was able to notice how bad I am at it. This post aims on forcing me to improve my naming things skill.

Regarding Project names

The last project we did at had a good name. It’s called Sauron, and it’s main purpose is to collect and show server’s health, QoS/QoE analysis, handle video signals, network throughput, etc. It fits well as Sauron represents himself as an all seeing eye. It’s short, compact, nerdy, and I like it.

But we have a lot of bad names there: Streams-analyzer, EvoHandler and VMine are just some of then that are too vague or don’t really explain what they do (sorry coworkers, don’t take it personally).

My undergraduate project was called Marmota, an acronym for making arduino and mobile devices talk that forms the name of a funny animal in Brazilian Portuguese. I don’t know if anyone has looked at this fact, but I have no doubt that the name helped on its 15-minutes of fame [1] [2] [3] [4] [5].

Now I’m trying to engage myself on, an attempt to scale live video streaming using P2P. Actually, is another joke with a bird name and TV suffix, which refers to video broadcast. It’s small, simple and easy to say.

Just to name other cool project names that aren’t associated to animals but are good ones: HTTPretty, Flask, PyPy, Spdy, Bundle, YouTube.

Who would guess — even amid its barrage of Super Bowl ads — that Go Daddy is the name of an Internet domain registration site?

Paul Farhi

Yes, you must be thinking about the projects whose names make no sense but they’re much bigger and successfull than the ones that I listed. I’m not here judging the formula of success. I’m just concluding that short, solid names that make associations with cool stuff and the project purpose itself is a good formula. For me, a good logo as bower, gruntjs or yeoman also adds value to the project.

Regarding code

Naming classes, functions and modules are even more complicated. At our office we work with code that interacts with multimedia, and it’s common to deal with names that have more than one meaning. Stream, Signal, Media and Bitrate are constantly used in our codes, and those are a constant source of confusion.

For now we don’t have a solution for this, but docstrings on the code or a small meeting followed by a document describing the decisions and conventions should fix it. Since our projects are growing exponentially, perhaps it’s time to schedule this reunion.

Uncle Bob’s Clean Code book have an entire chapter explaining names and how to use and improve it in your code. He defends that names must be pronounceable and searchable, and you as developer should avoid reader’s to mentally translate names (avoid mental mapping).

Length is not a virtue in a name; clarity of expression is.

Rob Pike

I agree that length isn’t directly related to self-explanation, and a good trick is look at your language syntactic sugars to improve the readability on the context at where your variable name or function is being used. I’m sure you’ll find something to enforce the name you chose.

Once in a while, ask people outside your team to review your code before pushing it to master. They will ask questions that were obvious to you and your teammates, but will alert where you are using bad names.

Outsiders have other context in mind, and their questions will probably improve your code design and names. It’s like visual design, information architecture. 4, 6 or 8 eyes is always better than 2.

More References:

[Flávio Ribeiro] Starting a blog again..

Saturday, December 7th, 2013

Hey! I’m just testing my blog and adjusting layout and stuff..

See you.

[Flávio Ribeiro] 2012 annual post

Thursday, March 8th, 2012

It seems we have an annual blog post over here.  :)

One year is a lot of time, time enough to change my mind about certain things. The company I’m working for enabled me to attend to a master’s degree and I had my first classes this week. I’m pretty excited about it.  I will study algorithms analysis and distributed computing this semester, and my first impression was the best possible.

At work, I had the opportunity to participate at some projects that reach a large number of people like Big Brother Brasil, PremiereFCCarnaval 2012 and some others that required a lot of effort. It’s being an awesome experience to work on projects of this size.

In December I usually do some kind of retrospective and compare the current year to the previous one. For about 6 or 7 years I’ve been getting the same conclusion: This year was far better. I don’t know if it’s just because recent memories are fresher and stronger, but I hope to continue with this conclusion in the coming years.


[Flávio Ribeiro] Time to move on

Friday, July 22nd, 2011

Hey there, here I am, after one year without posting anything. During this time away from here, a lot of things have happened in my life.

To summarize, I’ve finished my bachelor degree at IFPB in João Pessoa and changed city twice. The first move was to Campina Grande, to start a masters degree in Computer Science at UFCG, but i’ve completely failed. It seems that I cannot stand more time in classrooms, at least this year.

Then, I’ve moved to Rio de Janeiro, where I’ve started working at I’m here for about 20 days and I’m really enjoying :) . I’m still looking for a place to live, getting used to (strange, not saying ugly) cariocas accent and workin’ a lot. I hope to spend a good time here.

Wish me luck!