Live animation of freehand strokes

One of the goals of Andiamo is to implement a system where real-time rotosketching and animation can be performed  live using video material as the source. Two technical elements are required in order to accomplish this goal. One, a mathematical representation of free hand strokes that allows to attach the strokes to moving features while preserving their overall shape. Second, an algorithm to extract and track relevant motions in a video feed.

A Bezier representation of free hand strokes seems to be a convenient model, since it allows to easily control the shape of the curves by means of anchor and tangent points, while being computationally simple. The only difficulty consists in finding the right Bezier parameters for a given user-drawn stroke, which can be potentially composed by thousands of individual points. This is however a problem that has been solved in the late 80’s by Philip J. Schneider, who proposed an algorithm to fit a set of 2D points to a multi-segment Bezier curve (see details here and here). This is in fact the algorithm used in the open source vector graphics tool Inkscape for its freehand mode,  and probably in commercial applications such as Adobe Illustrator as well.

The parametrization of Bezier curves with physical-like constraints allows to accurately attach curves to image features such as lines and edges. The Snakes algorithm, which also dates back to the 80’s, represents the first one of the so called active contour models where the minimization of a certain “image energy function” keeps the curves locked to nearby edges.

I’m investigating these techniques in the context of my work with the live performance tool Andiamo. I ported Inkscape’s mulit-segment Bezier fitting algorithm to Processing. As a first step, I used the traer physics library, to consider each control point along the Bezier curves as a particle connected to the rest of the curve’s particles by a set of spring forces. This creates a curve than can be treated as a “solid” object but also has some flexibility. The following applet offers an interactive demo of these experiments:

Multiple-segment Bezier fitting + Physics Demo

If the curve’s points are attached to anchor points by means of additional spring forces, displacements or rotations on the anchor points are transmitted to the whole curve. The idea is to integrate these curve drawing methods, as well as the physics simulations, into a new version of Andiamo. Andiamo already incorporates a robust feature tracking algorithm, KLT, so it will be easy to combine the Bezier curve drawing witht the feature tracking module.

While researching these algorithms I found the work from computer scientist Aseem Agarwala. He investigated automatic rotoscoping techniques by using exactly the same approach I’m currently developing: bezier representation of hand strokes + KLT feature tracking. These two Agarwala’s projects, Keyframe-Based Tracking for Rotoscoping and Animation and SnakeToonz, illustrate the feasibility of these techniques and their potential application in the context of live performance. This video is particularly illustrative.