Years ago, when digital video — or affordable digital video — was new, I thought it would be fun to turn live-action video into an animated cartoon. A Slice of Time and Space explains why that’s not so easy:
Though others have turned a still image into a cartoon, turning a video into a cartoon is more challenging. ‘Some people say it’s easy,’ said Cohen. ‘They use the technique for still images and apply it frame-by-frame. The problem is, if you do this, the images jump all over the place. The background shakes around a lot, and each frame looks like a different drawing. We want to make the video look like a normal cartoon where the motion is smooth.’
Of course, this is why traditional animated characters don’t have any shading; the shading would crawl around from frame to frame.
How does Cohen’s technique work?
“We use a method called segmentation. We extend 2D segmentation to 3D. This creates shapes inside the video. It’s as if you took a stack of photographs and then cut them with a knife as though they were a solid chunk of color,” said Cohen.Pixels in a video can be thought of as lying in six dimensional space – two image axes, time, and three color components. Pixels close to each other in this “space” form denser regions. The program clusters the pixels into a 3D shape, which can then be ‘sliced’ as though you were taking a slice of time and space.
[...]
To define more meaningful regions, the user outlines the shapes on keyframes in the video, such as the pants on the girl swinging. He does this on several keyframes. “We rely on the user to circle things like the girl’s pants. There’s different shading on the pants, and some stripes. We can’t group them automatically,” said Cohen.The system can then interpolate between the keyframes, maintaining smooth trajectories along the time dimension, without jerky transitions or the need to draw on each frame.
(Hat tip to Slashdot.)