I finished a new version of gsvideo (0.3.12), which adds the possibility of setting parameters for the input plugin in the GSCapture object. An useful application of this feature is to chose the input device in Linux as follows:


capture = new GSCapture(this, 320, 240, new String[] {"device"}, new String[] {"/dev/video0"});


Download from here.

Realmatik is a little application written in Processing to demo real-time GPU video effects on video. Two iterations after the first version, and now it has the possibility of fading-in and out the effects by using bezier curves:

When clicking on the slide bar representing the effect, a bezier curve editor appears in the bottom part of the screen. Selecting and dragging the control points of the curve allows to determine how the fade factor for the effect changes along the playback of the video.

Download the application from the link below:

realmatik 003

It requires the following libraries installed:

gsvideo

glgraphics

controlP5

sDrop

The GLTexture library mentioned in earlier posts got renamed to GLGraphics in this new release, since it incorporates some new functionality that goes beyond the original scope of the GLTexture class. It introduces a new renderer called GLGRAPHICS that descends from the built-in OPENGL renderer. However, GLGRAPHICS can be created without attaching it to any drawing surface, so it can be used as an off-screen render framebuffer.

The problem is that I had to make some slightly changes to the built-in opengl library in order to create GLGraphics. The binary package includes the modified opengl.jar file, but using it might have (unknown) side effects… At least, exporting opengl applets doesn’t seem to work. If you want to try this library, its advisable that you backup your copy opengl.jar, just in case you need to come back to the previous, unmodified version.

Here is a screenshot of a bloom effect implemented with the new version of the library:

The library class introduces some new features, besides the GLGraphics renderer, like floating point textures and user defined filter parameters. Check the examples included in the binary packages:

GLGraphics download page

Online documentation

I created project pages for the gsvideo and gltexture libraries at Sourceforge.net. Here are the links:

gsvideo downloads

gltexture downloads

At this time there are no releases newer than those already available on this blog, but all future releases will be put in sourceforge from now on.

Using the gsvideo and the gltexture libraries, I wrote a little demo application in Processing which lets you play with video files and realtime glsl filters:

The way it works is fairly simple: you drop a movie file into the main window, and navigate the movie using the “timeline” slider located underneath the video images (the one on the left being the source video, and the one on the right the “filtered” video)

Then you can apply up to 5 filters to the video, by choosing the desired effect options in the “FX” menu on the left. The filters affect a certain segment of the movie, the segment is defined by moving and resizing the red bars that represent the filters.

Click on the link below to download a zip with the PDE files and the glsl shaders:

realmatik 001

Update: The drag-and-drop functionality of the version 001 doesn’t work on Linux. Version 002 address that issue, and also correctly scales the image according to the the ratio of the movie:

realmatik 002

In Windows you have to use the expert version of Processing (without java). Besides gsvideo and gltexture, this demo also requires the excellent libraries by sojamo for creating interfaces and handling drag-and-drop operations in Processing: controlP5 and sDrop.

I was finally able to make gsvideo work on OSX (tested on Leopard with an 20″ iMac)! The problem was not as much as with gsvideo itself, which needed just some small changes in order to set the directory where GStreamer libraries get copied. The difficult part was to have GStreamer actually installed on OSX, for which I used MacPorts. This involves first installing the Apple development tools (XCode) and then getting your hands a little bit dirty in the command line. The good news is that the version of GStreamer available on MacPorts has been recently updated to the latest released one, so almost every plugin and codec is working right now (except camera capture).

This is the new version of GSVideo (0.3.11) that fixes the path issues on OSX:

library

source

Hopefully, in a not too distant future, it will be possible to install GStreamer by means of a binary package. As for now, MacPorts seems to be the only way to do it. Here is the homepage of MacPorts, and here a short tutorial about it.

A full install of GStreamer involves installing the following packages: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-bad, gst-plugins-ugly and gst-ffmpeg. And because of all the dependencies that these packages require, the install ends taking up to 360MB!

This new release of gsvideo (0.3.10) solves an issue that affected Linux users: the GSMovie object wouldn’t loop. It’s fixed now (tested on ubuntu 7.10). It also includes the latest version of gstreamer-java (0.7). Grab the library and source packages from the links below:

library

source

I recently finished a project that involved controlling many LEDs from Processing through the DMX USB Pro adapter:

popMotion project

I wrote a little object that encapsulates the handling of the DMX messages, click on the link below to view/download it:

DMX object

It is based on an example posted in the Processing forum. I simplified the code a little bit. It doesn’t create that many intermediate arrays, so it should be somewhat faster.

Blender is a widely used open source application for 3D modeling and animation. A perhaps not so well known feature of blender is its ability to edit video and audio, thanks to its sequence editor. In fact, it is a full blown non-linear editing module, which allows to apply numerous effects, as well as doing simple tasks as blending clips together, adding transitions, etc.

I have been using it recently to do some basic video editing, and I think it keeps up quite well in comparison with other, more “professional”, video editing tools. Since blender has its own unique interface, which doesn’t follow the conventions adopted by other programs (it is heavily optimized for a keystroke-oriented workflow), it might be a little bit difficult at the beginning, but once you learn the basic elements of the interface, is in fact very intuitive. Here you have two pages at BlenderWiki with detailed information about the sequence editor in blender:

VSE manual

VSE tutorial

Simple clip mixing

I just want to add some brief comments about the task of creating simple videos by mixing short clips, adding titles and transition effects (all of this can be found in the VSE manual linked above).

First of all, there is a predefined layout for video editing, which can be selected by going to the top menu and choosing SR:4-Sequence in the layout list (the default being SR:2-Model). This layout has a timeline, an IPO curve editor and a preview window.

In order to render the videos, you have to select the “Do sequence” option right below the ANIM button in the Scene (F10) menu. The output file(s) will created in the directory set in the first box of the output group. By default, blender will render each frame into a separate jpg file, but the output could also be an avi file. To do this, you chose “AVI codec” in the drop-down list of the format group. In this case, the name of the output avi file will be taken from the text entered in the output box. For instance, if it says “C:\Users\Render\test” the output file will be named test0001-0250.avi and created in C:\Users\Render. 0001-0250 reflects the frame range, which of course can be also modified by just entering the desired values for the start and end frames of the animation.

A simple editing task consists in creating a fade-out/fade-in transition between two clips. This is achieved by using two effects in conjunction: color generator and gamma cross. If we want to do a fade-out to black, we add a color generator by going to “Add|Effect|Color Generator” (in the video sequence editor window). The default color of the color generator is gray, but is easily changed by opening its properties by selecting it and pressing “N”. Then we add a gamma cross between the video clip and the color generator. In order to do this, we have to select both the clip and the color generator (shift key + right mouse button on each), but in that order: first the clip and second the color generator. Once they are selected, we add the gamma cross (”Add|Effect|Gamma Cross”). This generates the fade-out. The fade-in is created in the same way, but selecting first the color generator and then the video clip.

Sound can also be edited in the VSE. In order to render a sound file from the sequence editor, go to Scene(F10) -> Sound block buttons -> MIXDOWN. This will create a wav file using the same convention as for avi files. If the video is rendered to avi after generating the wav, then the wav will be added as an audio track to the avi.

To control the volume of an audio strip, add an IPO curve in the IPO curve editor for the selected audio strip, by pressing ctrl key + left mouse button inside the IPO curve editor window. Tab key switches between curve and point editing, and the “i” key inserts a new control point at current frame location. By changing the height of the IPO curve between 0 and 1, the volume changes between silence and the maximum.

Audio and video can also be mixed with an external program such as avidemux. In such a case, the video has to be rendered without the audio track. This is achieved just by deleting the wav file in the case it has been created before and then rendering the sequence again by pressing the ANIM button.

Here is a new version of the opengl texture library for processing (v0.6.5). In fact, I renamed it to gltexture, and also changed the prefix P* to GL* to avoid conflicts with the core classes of processing. I also fixed a minor bug and added an initial version of the documentation (generated from the code with javadoc). Click on the links below to download the library and source code and to see the documentation (also included in the library download):

library

source

docs

As mentioned in previous posts, the idea of this library is to integrate opengl textures and glsl shaders into processing as seamlessly as possible. For this reason the GLTexture object is a descendant of PImage, so all the basic image manipulation funcionality is still available in the new object. GLTexture encapsulates an opengl texture and you can think of it in the same way as the pixels property of the PImage: you copy the image to the texture by calling the loadTexture() method. Once loaded into the GPU, you can apply a GLTextureFilter to it, and do any other opengl-accelerated operation. Then, you copy the texture back to the pixels by calling updateTexture().

This example shows the basic things you can do with the new library:

BasicUse.pde

Here is available as an online applet:

BasicUse applet

Note that in this applet the xml package is not imported (I copied xml.jar manually into the applet directory), but in general you have to explicitly import processing.xml into the main program in order to load the filters. This is actually the same issue that affects the candy library.

The filter object, GLTextureFilter, wraps a shader program that is used specifically to apply 2D filters to the texture objects. It also defines a 2D point grid which the source texture is mapped onto. This grid can be manipulated at the vertex stage of the shader program, in order to generate arbitrary transformations on the mesh.

Format of the xml files

The filters are initialized from an xml file that contains three things: the filenames of the vertex and fragment shaders, and the parameters of the grid (resolution and spacing). Below there is a sample xml configuration file for a filter:

<filter name=”pulsating emboss”>
<description>Emboss with pulsating grid</description>
<vertex>PulseGrid.glsl</vertex>
<fragment>DynEmboss.glsl</fragment>
<grid>
<resolution nx=”10″ ny=”10″></resolution>
</grid>
</filter>

The three important tags are <vertex>, <fragment> and <grid>. In <vertex> and <fragment> you just specify the filename of the vertex and fragment glsl shaders. If no vertex shader is needed, just don’t add a <vertex> tag.

The <grid> tag is used to define a rectangular grid the input texture is applied onto. The advantage of having a grid instead of just a rectangle is that you can modify the vertices of the grid inside the vertex shader, allowing for interesting distortions of the texture. However, if there is no need for such type of distortions, you can just skip the <grid> tag.

With <resolution nx=”10″ ny=”10″></resolution> you just specify the number of points on the grid, along the x and y directions. In this case, the grid has 10×10 points.

So, if you don’t need to change the vertex stage nor a grid, the xml file would reduce to something like this:

<filter name=”gaussian blur”>
<description>3×3 Gaussian blur filter</description>
<fragment>Blur.glsl</fragment>
</filter>

Uniform parameters in the shaders of the filter

The glsl shaders themselves have to follow certain conventions in order to be used inside a filter. These conventions are basically the way you have to name the uniform parameters inside the shaders, for the filter to be able of recognizing those uniforms and link them to the parameters that can be passed from processing.

The naming conventions are the following:

1) the name of the source/input texture units should be “src_tex_unit” + n, where n = 0, 1, 2, etc. I.e., the first texture unit should be named src_tex_unit0, the second src_tex_unit1, and so on.

2) The offset of the source textures is identified by “src_tex_offset” + n, with n = 0, 1, 2, etc. The offset is, in other words, the inverse of the width and height of the texture, and represents the step size to move from one texel to the next.

At least one src_tex_unit0 uniform should be defined in the fragment shader, since a filter needs at least one source texture to operate on.

Then you can define the following optional uniforms to pass general parameters into the shader:

3) timing_data (of type vec2): the first component of this vector is the frameCount and the second is millis() at the time the shader program is executed. This allows to pass timing information into the shader.

4) par_flt1, par_flt2 and par_flt3 (of type float): these float uniforms can be used to pass float numbers into the shader.

5) par_mat2 (of type mat2), to pass a 2×2 matrix into the shader.

6) par_mat3 (of type mat3), to pass a 3×3 matrix into the shader.

7) par_mat4 (of type mat4), to pass a 4×4 matrix into the shader.

These uniforms don’t need to be defined inside the shader, and also the shader can have other uniforms with different names. However, the advantage of having the uniforms listed here is that they are handled automatically inside the filter class. In order to set the value of these uniforms from processing, you use the GLTextureFilterParams object, inside which you have the parFlt1, parFlt2, parFlt3 float variables and parMat2, parMat3 and parMat4 float arrays. So, if the glsl shader that defines the filter has the float uniform par_flt1, you can set its value with the following code:

GLTextureFilterParams params = new GLTextureFilterParams();
params.parFlt1 = map(mouseX, 20, 640, 1, 30);
tex0.filter(pixelate, tex1, params);