OpenGL in Processing 2.0 alpha 5

The march towards the 2.0 version of Processing reached an important milestone with the alpha 5 release, available for download since last Friday. The detailed list of changes is here, while the wiki offers a more in-depth discussion of the new features and ongoing changes. A major update in alpha 5 is the inclusion of the new OpenGL renderer for the desktop and Android modes, called P3D, which has been rewritten from the ground up in order to offer improved performance. I will describe some important elements of P3D in this post.
Update: With the release of Processing 2.0 final, some of the contents in this post are outdated, please check the new reference, and tutorials – specially those about P3D, PShape and shaders – for a detailed description of the finalized API.

1. Shaders and programmable pipeline

GPUs and graphics APIs like OpenGL have been moving away from the fixed-pipeline model to a fully programmable approach where every rendering operation needs to be coded as a shader program. OpenGL ES 2.0 (GLES2) on mobile devices as well as WebGL ditched the fixed functionality entirely so that drawing any kind of 3D scene on mobile or web requires writing the appropriate GLSL shaders. Following this trend, the new P3D renderer in Processing implements all the drawing operations with GLSL shaders, both on desktop (PC/Mac) and Android. From the perspective of most users, this change won’t be very noticeable, since the new P3D shaders are meant to reproduce as closely as possible the visual output of the older software-based P3D and OPENGL renderers in Processing 1.x. However, there is an important practical consequence of upgrading P3D to a fully programmable pipeline: on the Android side, devices that don’t support GLES2 won’t be able to run P3D sketches, this effectively excludes any Android smartphone or tablet which is not running Gingerbread (Android 2.3) or newer. Froyo (2.2) still makes up for a significant proportion of the devices currently in use (around 25% according to the latest stats from Google) but unfortunately its GLES2 Java bindings are broken, which lead us to define 2.3 as the minimum requirement for OpenGL on Android. Likewise, P3D on the desktop will also fail to run if the video card doesn’t not properly support OpenGL 2.0 and programmable shaders. Fortunately, most GPUs released over the past 3-4 years do support programmable pipelines, including the integrated chipsets from Intel. A significant advantage of moving to programable pipelines across all platforms is the increased performance, since GLSL shaders can be exactly customized to the kind of rendering Processing does.

2. Immediate vs retained mode

Another important trend in the recent evolution of OpenGL consists in the elimination of the immediate mode calls (glBegin/glVertex/glEnd) from the API, in favor of the use of Vertex Buffer Objects (VBOs). Basically, VBOs allow vertex data to be uploaded into buffers in video memory during initialization, from where they can be read very quickly by the GPU in order to render the scene. The main advantage of VBO (or retained mode) rendering is that performance can increase quite significantly, since the overhead of copying the data from the main to the video memory every frame with the glBegin/glEnd functions is gone. This is particularly important for mobile devices, where the bandwidth and frequency of the memory bus is limited in order to increase battery life. Starting with OpenGL 3.1 on the desktop, the immediate mode calls are only available through a backwards-compatible profile, while OpenGL ES and WebGL only offer retained-mode rendering.

However, immediate mode rendering has been the main (and only) drawing methodology in Processing until now. Furthermore, the sequential use of vertex() calls enclosed by the beginShape()/endShape() construct is in general very convenient for teaching and rapid sketching of geometrical shapes, as it is easy to grasp and doesn’t add the additional complexity of initializing buffers and other data structures. The new P3D renderer does preserve this immediate mode drawing, but implements it internally using VBOs. The internal VBOs accumulate vertex data as the user issues vertex() calls from the sketch code, and are copied to the GPU memory only once a maximum size is reached. In this way, performance is enhanced by minimizing memory transfers, while preserving the immediate mode that is characteristic of Processing.

But in addition to the immediate mode, P3D also supports a new retained mode that makes possible to create PShape objects holding large geometries, which are internally stored as VBOs in video memory. In the case these geometries are largely static and don’t need to be updated often, then the rendering can be much faster than the equivalent immediate mode code. The two following code snippets compare the rendering a single triangle in immediate and retained modes:

One triangle, immediate mode:

void setup() {
  size(400, 400, P3D);
}

void draw() {
  background(0);
  translate(mouseX, mouseY);
  beginShape(TRIANGLES);
  noStroke();
  fill(255);
  vertex(0, 0);
  vertex(50, 0);
  vertex(50, 50);
  endShape();
}

 

One triangle, retained mode:

PShape tri;

void setup() {
  size(400, 400, P3D);
  tri = createShape(TRIANGLES);
  tri.noStroke();
  tri.fill(255);
  tri.vertex(0, 0);
  tri.vertex(50, 0);
  tri.vertex(50, 50);
  tri.end();
}

void draw() {
  background(0);
  translate(mouseX, mouseY);
  shape(tri);
}

 

Immediate and retained modes in P3D are complementary: when the geometry is highly dynamic and changes every frame, the former is probably a better approach as the PShape objects would need to be updated often anyways. This will bring the performance down to the levels of the immediate mode, or even lower, since the PShape object has a slightly more complex update mechanism. In contrast, big static geometries are good candidates for retained mode rendering.

3. Line and point rendering

Line strokes with arbitrary weight values and stroke caps and joins have always been an elusive goal to achieve with OpenGL. The default LINE and POINT primitives have a hardware-limited size (often around 10 pixels) and don’t support caps or joins. In order to overcome these limitations, the line and point rendering in P3D has been reimplemented using triangle strips for lines and triangle fans for points, although the later is likely to be replaced by more efficient point rendering algorithms at the fragment shader level. The first implementation of line and point rendering available in alpha 5 allows to draw arbitrary thick lines and arbitrary large points, however is still unfinished and doesn’t implement caps and joins yet. This is not a simple problem, since correct rendering of 2D strokes in 3D space presents tricky issues such as self-occlusion of the stroke lines by the shape geometry itself. Algorithms for stylized line rendering like the ones available in the dpix library could be useful to finish this part of the renderer.

Since lines are rendered as quads in 3D space (although aways screen facing, just like typical bilboards), they can affected the perspective transformation (becoming thinner as they move away from the camera), although this is not the default. However, in some situations this might be the desired behavior, so there is a new hint, ENABLE_PERSPECTIVE_CORRECTED_LINES, which can be used to enable the application of the perspective correction on line strokes.

void setup() {
  size(400, 400, P3D);
  strokeWeight(10);
}

void draw() {
  background(150);
  translate(width/2, height/2);
  rotateY(frameCount * 0.01);

  hint(ENABLE_PERSPECTIVE_CORRECTED_LINES);
  stroke(0, 0, 255);
  line(-width/2, -100, width/2, -100);

  hint(DISABLE_PERSPECTIVE_CORRECTED_LINES);
  stroke(0);
  line(-width/2, 100, width/2, 100);
}

 

However, due to a bug in 2.0a5, both lines appear to be without perspective correction.

4. 2D rendering in P3D

As a consequence of how immediate mode and line rendering are implemented in P3D, issues might show up when rendering geometry that it is restricted to a 2D plane. The source of the problem is that rendering of fill triangles, line quads and point fans occur through different shading pipelines, so that triangles, lines and points are not necessarily drawn in the order they are specified in the code, but flushed to the GPU separately in order to optimize performance. If the geometry is truly three-dimensional, there is no problem since the occlusion from the view is not determined by the order in which the geometry drawn, but by its distance to the camera eye. However, visibility during 2D rendering in Processing is dictated by the order in wich each shape is drawn. In order to solve these issues, P3D offers a new hint called ENABLE_ACCURATE_2D which can be used when compositing 2D elements:

void setup() {
  size(400, 400, P3D);
  strokeWeight(10);
}

void draw() {
  hint(ENABLE_ACCURATE_2D);
  background(0);
  stroke(255, 0, 0);
  rect(0, 0, 200, 200);

  translate(mouseX, mouseY);
  stroke(0, 0, 255);
  rect(0, 0, 200, 200);
  hint(DISABLE_ACCURATE_2D);
}

 

This hint does slowdown rendering, since buffer optimizations are turned off when using it, so it is recommended to disable it when not drawing 2D geometry.

5. Automatic video acceleration

When movies are played back or video is captured using the core video library and P3D is set as the renderer, then the frames are automatically uploaded into OpenGL textures encapsulated by the PImage object. This results in better performance (specially for high resolution videos) and reduced CPU usage, however, the frame pixels are not available through the pixels array of the PImage object.

6. OpenGL applets and Java bindings

P3D uses the JOGL bindings to communicate with the native OpenGL libraries on the desktop platforms (Mac, Linux, Windows). The latest releases of JOGL solve the jar signing problem that affected older JOGL versions (1.x) so the OpenGL applets are working again (see this Menger example on OpenProcessing by Martin Prout).

7. What is missing in P3D in 2.0a5?

Quite a few things, although the major elements of P3D are already in place. At this point, the most notable features from the old 3D renderers that are absent in the alpha 5 are PDF recording, DXF export, OBJ loading, GL-accelerated image masks, filters, pixel get, set and copy, as well as line strokes and caps as mentioned earlier.

8. Known issues

Besides bugs that were introduced in the new renderer (or long standing ones) are currently being ironed out, a couple of issues due to the OpenGL-Java bindings are worth mentioning here:

• OpenGL sketches on Mac OS X will print an error message upon starting (or when changing the smooth level with smooth()/noSmooth()) that looks like: 2012-03-27 16:25:12.317 java[5741:1707] invalid drawable. This is an issue with Apple’s Java plugin which seems to have no effect.
• Smooth doesn’t work on OSX Lion, this appears to be restricted to AMD discrete cards.
• Framerate on OSX for sketches with complex geometries could be inconsistent until this bug in the JOGL bindings is fixed.