This part of the reference manual will focus on the drawing layer of Andiamo. This is one of the central elements of the software, at least in terms of live performativity. The goal is to create a system to experiment with real-time drawing which, using hand gestures as the primary input, allows to create spontaneous and dynamic compositions and animations by combining live drawing with looping and by integrating hand-drawn lines and video sources, such as movies or live camera feeds. Part of the inspiration for this integration between video and drawing comes from tools such as rotosketch, created by Zach Lieberman, Scott de la Hunta, and Susan Rethorst, from the use of the technique of rotoscoping in film and animation, as well as from my own past experiences in the theater and performance using moldeo as the software for live interpretation.
The drawing layer has a number of different gesture types: yellowtail, multi-bezier and animation. The first one is a based in the code from Golan Levin, while the second implements the same multi-bezier fitting algorithm used in inkscape. The animation gesture allows to add cel animations to the composition and drag them around. The following video exemplifies the use of some of these gesture types:
This video also shows that the gesture color and transparency can be individually selected for each gesture by using the color selector pop-up menu (while the layer color selector affects the tint of the entire layer at once). The pop-up menu is activated by pressing the pen against the tablet (or clicking the mouse) while pressing the TAB key:
In the current version of Andiamo at the time of this writing (021) there are 6 types of gestures, from which the first 5 were used in the previous example. Each gesture type can be selected with a key, indicated between parethesis below:
- normal yellowtail (Q): the yellowtail gesture as originally implemented by Golan Levin.
- yellowtail rotosketch (W): the first point of the gesture moves toward the last once the gesture is completed.
- yellowtail fadeout (E): the trailing end of the gesture fades out, progresively approaching the leading end.
- normal multi-bezier (R): fits the data points entered by the user with a continuous patch of bezier curves.
- multi-bezier shape (T): uses the same multi-bezier fitting algorithm as the normal multi-bezier, but the curves are rendered as a filled polygon.
- cel animation (Y): the frames of a running cel animation are drawn where the cursor is located. As the tablet/mouse is not released, then animation will follow the motions of the cursor across the screen.
Every gesture, except the multi-bezier shape, can be animated through looping. The idea of looping is basically to repeat the gesture within a specified time interval (here the term to “repeat” means to “re-draw”). The looping interval is opened and closed with the SPACE key, and everything that is drawn during the duration of the loop will be repeated periodically afterwards:
A gesture can be contained in only one loop, but a single loop may include multiple gestures. By combining and synchronizing several loops and gestures, complex animations can be achieved, basically using the gesture itself as the source for the motions. Certain gesture types have additional animation patterns, such as yellowtail rotosketch and fadeout. These motions that occur the first time the gesture is drawn will also be repeated during the subsequent loops.
Synchronizing gesture looping to a video clip is an useful feature inspired by Lieberman’s Rotosketch. This feature, when enabled, triggers the creation a new gesture loop every time a clip in a specific video layer reaches the end and returns to the first frame. In the configuration file of the video layer that is to be used as the generator of gesture loops, the following tag has to be added:
The number is the index (zero-based) of the drawing layer that will use gesture loops generated by the video layer. The generation of loops starts/stops by pressing the SPACE key in the video layer:
For this feature to work properly, only the channel in the video layer that contains the desired clip should be playing. Otherwise, new gesture loops will be created when any of the two channels restart playing.
Gestures can be deleted by selecting them with the eraser tip of the pen (dragging with the eraser will create a bounding box and all the gestures intersecting this bounding box are also removed), by pressing DELETE (which removes all the gestures from the layer) and BACKSPACE (which only removes the last drawn gesture). Deletion doesn’t occur immediately, with the gesture gradually fading out during a pre-determined time. This deletion time can be different for each gesture, and can be set using one of the sliders in the top menu of the drawing layer. It is also possible to enable an auto-delete mode, in which gestures begin to be deleted right after they are completed. This mode can be enabled/disabled with the N key, or with a button in the top interface. These interface elements are shown in the next image:
Gestures can also be hidden, which in principle appears to have an identical effect to deleting them. A gesture is hidden through with the same fade out effect used to delete a gesture, however after becoming completely transparent the gesture still exists. Gesture hiding is enabled/disabled with the ‘M’ key or using a button in the top menu of the drawing layer. The time that takes for a gesture to be hidden is also controllable with a slider in the top menu:
Gesture hiding can be used when looping to make a gesture gradually disappear before the ending of the loop:
Cel (frame-by-frame) animations can be inserted into a drawing composition using the animation gesture (selected with the Y key). A drawing layer can hold many different cel animations, each one defined by a collection of individual frame images. These collections are specified in the drawing layer xml configuration file:
<drawing> <animation name="Sequential by James Patterson" prefix="PT_anim" extension="gif" length="12" fps="10" scale="0.5" playonce="no">animations/sequential</animation> <animation name="Falling" prefix="frame" extension="png" length="50" fps="25" scale="1.0" playonce="yes">animations/falling</animation> </drawing>
Each cel animation is defined by the directory where all the frames are stored (in the first entry of the example above it is /data/animations/sequential, /data/animations/falling in the second). The name of the image files should have the same prefix, and numbered starting from zero: PT_anim0.gif, PT_anim1.gif,…, PT_anim11.gif, for instance. The fps parameter is determines the frame rate with which each animation is played, while scale affects the size of the frames when drawn to the screen. Setting playonce to yes has the effect of stopping the animation after the last frame, otherwise it is looped.
The different cel animation loaded into a drawing layer are selected by using the LEFT/RIGHT arrow keys.
Gesture animation in Andiamo can be achieved also by the use of anchor points. These points are invisible in the final output, but they appear in the preview area. Gestures can be attached to the anchor points so that any motion in the anchors will be transferred to the gestures. Anchor points can be generated manually or by the built-in GPU-KLT tracker. Manual anchor points are blue and tracker anchor points are red. The GPU-KLT tracker (currently supported only on NVidia cards) can be disabled while Andiamo is running with a button in the main interface:
Gestures can be attached to anchors points (either tracker or manual points) with two methods:
- When drawing the gesture, it is attached to any anchor touched by the cursor. In the case of the multi-bezier shapes, all the anchors inside the shape are also used to attach the gesture to.
- After the gesture has been drawn, it can be attached to an anchor point by dragging a line between the gesture and the anchor. This dragging is done while pressing the pen button.
Tracker anchor points cannot be controlled by the user, they are automatically placed and moved around depending on the features detected by the KLT algorithm. The maximum number of tracked points can be set with the NUM_TRACKED_FEATURES global variable in the main pde file in the sketch. Higher number of tracked features would require a faster video card to run a smooth frame rates. The other important element to consider when using the KLT tracker is the video source. The video layer used as the video source for tracking is set in the main layer configuration file, data/layers.xml, by adding the tracked=”yes” parameter to the video tag:
<layers> <video>layers/video.xml</video> <video tracked="yes">layers/camera.xml</video> <drawing>layers/drawing.xml</drawing> <osc>layers/osc.xml</osc> <text>layers/text.xml</text> </layers>
Manual anchor points are added tapping the pen while pressing the CTRL key. Manual anchors can be connected to each other in a hierarchy (parent-child) in order to create large structures that can be controlled as a whole by moving the parent points. These structures can serve as the skeleton for a drawing composed by several gestures:
Manual anchors can be connected to tracker anchors as well, however the reverse is not possible. Also, a number of anchor points entered sequentially can be grouped together so all of them are automatically connected to the last point of the group. This is achieved by pressing the COMMA (“,”). All the anchor points entered afterward will be connected to the last point entered before pressing COMMA again.
Manual anchor points can be animated following an user specified curve path. The curve path is generated by grabbing the anchor point and moving it around while pressing the ALT key. The curve that results from connecting the entered positions defines a path that the anchor point subsequently follows:
Finally, anchor points are deleted by pressing CTRL and DEL simultaneously, while CTRL + BACKSPACE deletes only the last entered point.