I’d like to conclude this reference by briefly discussing a the objectives of Andiamo. Its main goal is to serve as an open experimentation and research platform for:
custom graphical interfaces for live performance.
real-time editing and mixing of video (live and pre-recorded) using GStreamer.
techniques of real-time animation using gestural input through graphic tablets and MIDI controllers.
inter-application communication with the OSC protocol.
OpenGL-accelerated image filters for real-time processing of video.
As an experimentation platform, it doesn’t attempt to ever reach the status of a “finished” or a “complete” application, but it will rather stay as a fluid, open source project available for anyone who finds it useful in any way and might be interested in trying it out, learning and hacking algorithms and techniques, adding his or her own modules and hacks and/or adapting it for specific uses. Code contributions are welcome, and I will consider them to be included in the main development trunk. But since the Artistic License was adopted to release this software, you are free to make your own derivatives as you see fit or extract parts of it to use in your own projects.
Successful ideas and approaches tested and protyped within Andiamo will hopefully find their way into platforms more suitable for real-time, high performance graphics, such as Moldeo or Open Frameworks. On the other hand, Andiamo also serves an example on how the use of the GStreamer and OpenGL libraries allow for a certain degree of real-time capabilities within the context of the Processing language and programming environment. Expanding the applicability of Processing into real-time video and live performance certainly is an useful project, since Processing serves as the first step into programming for many artists, designers and other creative people who might later want to transition into more advanced/efficient platforms.
A first application of Andiamo was for my final MFA project at the Design|Media Arts department. A video excerpt of a live animation performance using Andiamo can be seen below:
The process involved in this project is discussed in detail in this blog.
This part of the reference manual focus on the filter layers. A filter layer applies an arbitrary image transformation to the composition stack, by using the output of the previous layer as its own input for processing. The result of the filtering operation is rendered into the composition stack, and can be further processed by filters located downstream in the stack.
The filters are implemented as GLSL shaders that run on the GPU, this allows for real-time effects to be applied to full-resolution video. There are two types of filters: simple and custom. Simple filters just apply a one-pass image transformation on the input. Examples of this type of these filters are convolution filters (blur, emboss, edge detection). In terms of the implementation, simple filters can be specified as one shader operating on a single texture and generating a single output. Custom filters, such as a bloom effect or a particle system, are the result of multiple shaders running in conjunction in a specific order.
Simple filters are listed in the data/filters.xml file:
Each filter entry contains the config xml file for the filter, and its icon. The config xml follows the format of the texture filters of the GLGraphics library.
Custom filter have to be specified within the code of Andiamo as descendants of the FilterLayer class and then added to CustomFilterLayerLibrary (in FilterLayer.pde):
Filters can be added to the main data/layers.xml configuration file, but also while Andiamo is running using the add/remove filter buttons in the main menu:
A filter is completely transparent right after is added added, so the color selector menu of the layer needs to be used to set the transparency to the desired value:
This part of the reference manual will focus on the drawing layer of Andiamo. This is one of the central elements of the software, at least in terms of live performativity. The goal is to create a system to experiment with real-time drawing which, using hand gestures as the primary input, allows to create spontaneous and dynamic compositions and animations by combining live drawing with looping and by integrating hand-drawn lines and video sources, such as movies or live camera feeds. Part of the inspiration for this integration between video and drawing comes from tools such as rotosketch, created by Zach Lieberman, Scott de la Hunta, and Susan Rethorst, from the use of the technique of rotoscoping in film and animation, as well as from my own past experiences in the theater and performance using moldeo as the software for live interpretation.
The drawing layer has a number of different gesture types: yellowtail, multi-bezier and animation. The first one is a based in the code from Golan Levin, while the second implements the same multi-bezier fitting algorithm used in inkscape. The animation gesture allows to add cel animations to the composition and drag them around. The following video exemplifies the use of some of these gesture types:
This video also shows that the gesture color and transparency can be individually selected for each gesture by using the color selector pop-up menu (while the layer color selector affects the tint of the entire layer at once). The pop-up menu is activated by pressing the pen against the tablet (or clicking the mouse) while pressing the TAB key:
In the current version of Andiamo at the time of this writing (021) there are 6 types of gestures, from which the first 5 were used in the previous example. Each gesture type can be selected with a key, indicated between parethesis below:
normal yellowtail (Q): the yellowtail gesture as originally implemented by Golan Levin.
yellowtail rotosketch (W): the first point of the gesture moves toward the last once the gesture is completed.
yellowtail fadeout (E): the trailing end of the gesture fades out, progresively approaching the leading end.
normal multi-bezier (R): fits the data points entered by the user with a continuous patch of bezier curves.
multi-bezier shape (T): uses the same multi-bezier fitting algorithm as the normal multi-bezier, but the curves are rendered as a filled polygon.
cel animation (Y): the frames of a running cel animation are drawn where the cursor is located. As the tablet/mouse is not released, then animation will follow the motions of the cursor across the screen.
Every gesture, except the multi-bezier shape, can be animated through looping. The idea of looping is basically to repeat the gesture within a specified time interval (here the term to “repeat” means to “re-draw”). The looping interval is opened and closed with the SPACE key, and everything that is drawn during the duration of the loop will be repeated periodically afterwards:
A gesture can be contained in only one loop, but a single loop may include multiple gestures. By combining and synchronizing several loops and gestures, complex animations can be achieved, basically using the gesture itself as the source for the motions. Certain gesture types have additional animation patterns, such as yellowtail rotosketch and fadeout. These motions that occur the first time the gesture is drawn will also be repeated during the subsequent loops.
Synchronizing gesture looping to a video clip is an useful feature inspired by Lieberman’s Rotosketch. This feature, when enabled, triggers the creation a new gesture loop every time a clip in a specific video layer reaches the end and returns to the first frame. In the configuration file of the video layer that is to be used as the generator of gesture loops, the following tag has to be added:
<loop>2</loop>
The number is the index (zero-based) of the drawing layer that will use gesture loops generated by the video layer. The generation of loops starts/stops by pressing the SPACE key in the video layer:
For this feature to work properly, only the channel in the video layer that contains the desired clip should be playing. Otherwise, new gesture loops will be created when any of the two channels restart playing.
Gestures can be deleted by selecting them with the eraser tip of the pen (dragging with the eraser will create a bounding box and all the gestures intersecting this bounding box are also removed), by pressing DELETE (which removes all the gestures from the layer) and BACKSPACE (which only removes the last drawn gesture). Deletion doesn’t occur immediately, with the gesture gradually fading out during a pre-determined time. This deletion time can be different for each gesture, and can be set using one of the sliders in the top menu of the drawing layer. It is also possible to enable an auto-delete mode, in which gestures begin to be deleted right after they are completed. This mode can be enabled/disabled with the N key, or with a button in the top interface. These interface elements are shown in the next image:
Gestures can also be hidden, which in principle appears to have an identical effect to deleting them. A gesture is hidden through with the same fade out effect used to delete a gesture, however after becoming completely transparent the gesture still exists. Gesture hiding is enabled/disabled with the ‘M’ key or using a button in the top menu of the drawing layer. The time that takes for a gesture to be hidden is also controllable with a slider in the top menu:
Gesture hiding can be used when looping to make a gesture gradually disappear before the ending of the loop:
Cel animations
Cel (frame-by-frame) animations can be inserted into a drawing composition using the animation gesture (selected with the Y key). A drawing layer can hold many different cel animations, each one defined by a collection of individual frame images. These collections are specified in the drawing layer xml configuration file:
Each cel animation is defined by the directory where all the frames are stored (in the first entry of the example above it is /data/animations/sequential, /data/animations/falling in the second). The name of the image files should have the same prefix, and numbered starting from zero: PT_anim0.gif, PT_anim1.gif,…, PT_anim11.gif, for instance. The fps parameter is determines the frame rate with which each animation is played, while scale affects the size of the frames when drawn to the screen. Setting playonce to yes has the effect of stopping the animation after the last frame, otherwise it is looped.
The different cel animation loaded into a drawing layer are selected by using the LEFT/RIGHT arrow keys.
Anchor points
Gesture animation in Andiamo can be achieved also by the use of anchor points. These points are invisible in the final output, but they appear in the preview area. Gestures can be attached to the anchor points so that any motion in the anchors will be transferred to the gestures. Anchor points can be generated manually or by the built-in GPU-KLT tracker. Manual anchor points are blue and tracker anchor points are red. The GPU-KLT tracker (currently supported only on NVidia cards) can be disabled while Andiamo is running with a button in the main interface:
Gestures can be attached to anchors points (either tracker or manual points) with two methods:
When drawing the gesture, it is attached to any anchor touched by the cursor. In the case of the multi-bezier shapes, all the anchors inside the shape are also used to attach the gesture to.
After the gesture has been drawn, it can be attached to an anchor point by dragging a line between the gesture and the anchor. This dragging is done while pressing the pen button.
Tracker anchor points cannot be controlled by the user, they are automatically placed and moved around depending on the features detected by the KLT algorithm. The maximum number of tracked points can be set with the NUM_TRACKED_FEATURES global variable in the main pde file in the sketch. Higher number of tracked features would require a faster video card to run a smooth frame rates. The other important element to consider when using the KLT tracker is the video source. The video layer used as the video source for tracking is set in the main layer configuration file, data/layers.xml, by adding the tracked=”yes” parameter to the video tag:
Manual anchor points are added tapping the pen while pressing the CTRL key. Manual anchors can be connected to each other in a hierarchy (parent-child) in order to create large structures that can be controlled as a whole by moving the parent points. These structures can serve as the skeleton for a drawing composed by several gestures:
Manual anchors can be connected to tracker anchors as well, however the reverse is not possible. Also, a number of anchor points entered sequentially can be grouped together so all of them are automatically connected to the last point of the group. This is achieved by pressing the COMMA (”,”). All the anchor points entered afterward will be connected to the last point entered before pressing COMMA again.
Manual anchor points can be animated following an user specified curve path. The curve path is generated by grabbing the anchor point and moving it around while pressing the ALT key. The curve that results from connecting the entered positions defines a path that the anchor point subsequently follows:
Finally, anchor points are deleted by pressing CTRL and DEL simultaneously, while CTRL + BACKSPACE deletes only the last entered point.
The composition in Andiamo consists in a stack of layers, with each layer drawn on top of the previous one. The initial configuration of layers is specified in a xml file (layers.xml) stored in the data folder of the sketch. At this point (version 021 of Andiamo) there is a certain flexibility in the layer arrangement, since filter layers can be added or removed while Andiamo is running. The format of the layer configuration file is as follows:
where the type of layer is specified by the name of each xml tag: video, drawing, osc and text.
Andiamo’s video layer
The video layer is basically a two channel video mixer, which can show movies, live video captured with a camera or the output of any gstreamer pipeline. A video layer can also be used store new clips rendered in real-time during the execution of Andiamo, which will be available as any other video file. The configuration of a specific video layer is contained in the file whose file name is given as the content of the video tag in the layers.xml file. For instance, in the sample layers.xml given above, the configuration of the first video layer is stored in data/layers/video.xml:
This sample shows all the tags recognized in by the video layer (with the exception of the <loop> tag which will be discussed in the next part of the manual in the context of the drawing layer):
<movie> tag is used to specify a video file (is is assumed that the given path will be contained in the sketch data folder).
<camera> tag is used to specify a video capture device available on the system.
<pipeline> tag allows to enter an entire gstreamer pipeline
<recording> tag contains the parameters for real-time rendering movie file mode
All the different elements (movie, camera and pipeline) are loaded during initialization and shown in the video layer top interface:
The “media library” strip allows to navigate the different video sources available and select specific ones to use in the two channels of the layer. If the number of items exceeds the width of the menu, then the list can be scrolled left or right by dragging the pen (just pressing it against the tablet, or dragging with the left button pressed when using a regular mouse). The selected video sources are highlighted with a red border. Also, there is a mixer on the left edge of the menu, where the numbers 1 and 2 are drawn on top of a gray gradient. Dragging up or down the pen or mouse in that area will determine the amount of mixing between the two channels.
The bottom menu in the video layer contains the timelines and play/pause buttons for both channels. It also contains a mixer, which works in the same ways as the mixer in the top menu. The mixing also affects the volume of the videos (if they contain audio at all), so that channel 2 is muted when the only channel 1 is visible, volume is 50% for each channel when the mixer is right at the middle, and channel 1 is muted then the mixer is all the way down:
Real-time recording into a video layer it is possible when the <recording> tag has been specified in the configuration file for that layer. Recording starts and ends by hitting the ENTER key, and this will generate a new entry in the media library for the newly created video file.
The parameters in the <recording> tag control the resolution, codec and quality of the resulting file, among other things. In the example above the parameters are:
name: the prefix given to the filename of the video file, in this case “liverec”. All the recorded video files are saved into data/videos
fps: this factor is used to compute the target frame rate of the recorded file, depending on the current frame rate of Andiamo. For instance, if Andiamo is running at 40 fps, a value of 0.8 means that the fps of the recorded file will be 40 * 0.8 = 32.
resolution: is the width and height of the recorded file.
layer: indicates which layer in the stack will be saved into the recording. The index is zero based, so this means that “0″ represents the first layer, “1″ the second, and so on. If the goal is to record the entire composition then the “final” should be used.
codec: the video codec for the recorded file. Theora, x264 and xvid are available, but only theora is functional at this point
quality: the quality of the video file, which can take the following values: WORST LOW, MEDIUM, HIGH or BEST.
I have noticed that some pipelines could fail to restart playing after they are paused, so I added a parameter the “continuous” parameter which could be used with any video source (movie, camera or pipeline):
After some tests I discovered one strange issue in Andiamo’s video layer: setting at every frame the volume of the video channels was creating a drop in the frame rate of around 10 fps per channel. Not sure what the reason could be for such unexpected behaviour, ultimately the setVolume method sets the volume property in the underlying gstreamer’s playback pipeline. The solution for this problem was simple: to avoid setting the volume at every frame, but only when there are changes in the mixing of the two channels.
This fix, which increases frame rates very substantially, plus the inclusion of version 3 of the traer.physics library were included in a new release of Andiamo, 021. The complete skecth folder and the required libraries are available for download here.
