Barco Labs Video Motion Smoothing

Company
Barco Labs division of Barco NV
Location
Milpitas, CA
Start
January 2019
End
March 2019

An earlier version of our machine-learning person tracking software was too slow to keep up on every video frame. (This was before our team attempted to use GPU acceleration with more efficient person tracking software.) This resulted in jumpy video transitions while tracking someone. I was given the task of finding a way of applying video motion smoothing so the resulting video framing would be smooth and professional looking.

There are a number of ways to approach this including the use of easing functions or Kalman filtering (which does linear prediction). The problem with easing functions is that they require a known end point and, in our use case, the end point was a moving target. The problem with Kalman filters is that a person's motion was rarely linear so its predictions where often less than ideal. So, I decided to use a PID (Proportional-Integral-Derivative) control systems approach to this problem which continuously calculates the difference between the actual and ideal video framing at any given moment and strives to smoothly reduce the difference (or error). In addition, I added a custom ease-in function of my own design that matches the PID's adjustment slope without causing a discontinuity (which is currently under Barco's trade-secret non-disclosure, so can't reveal any details). PID adjustments tend to be discontinuous where a change in direction occurs and I needed an algorithm that would always be continuous. A number of advanced techniques were used to remove over-shooting and oscillations while keeping the algorithm fairly responsive. Here is a graph of how my modified PID algorithm does smoothing.

Video motion smoothing graph

The graph only shows one of the four variables (x, y, w, h) being smoothed. The green line shows the discontinuous transitions from the machine-learning person tracking software and the blue line shows how this modified PID algorithm smooths out those step transitions. The smoothing algorithm was originally written in Python, but it needed to run with maximum efficiency within a video compositor written in C++, so I recoded the algorithm in C++. Here is a short video showing the result. The input video is on the left. The person tracking output from the machine-learning software is in the upper right. And, the smoothed version using my modified PID algorithm is shown in the lower right.

The smoothness and professional look of the resulting video was an important factor in convincing our stakeholders that this research project should be continued.