OpenGL in Processing 2.0 alpha 5   14 comments

The march towards the 2.0 version of Processing reached an important milestone with the alpha 5 release, available for download since last Friday. The detailed list of changes is here, while the wiki offers a more in-depth discussion of the new features and ongoing changes. A major update in alpha 5 is the inclusion of the new OpenGL renderer for the desktop and Android modes, called P3D, which has been rewritten from the ground up in order to offer improved performance. I will describe some important elements of P3D in this post.
Update: With the release of Processing 2.0 final, some of the contents in this post are outdated, please check the new reference, and tutorials – specially those about P3D, PShape and shaders – for a detailed description of the finalized API.

1. Shaders and programmable pipeline

GPUs and graphics APIs like OpenGL have been moving away from the fixed-pipeline model to a fully programmable approach where every rendering operation needs to be coded as a shader program. OpenGL ES 2.0 (GLES2) on mobile devices as well as WebGL ditched the fixed functionality entirely so that drawing any kind of 3D scene on mobile or web requires writing the appropriate GLSL shaders. Following this trend, the new P3D renderer in Processing implements all the drawing operations with GLSL shaders, both on desktop (PC/Mac) and Android. From the perspective of most users, this change won’t be very noticeable, since the new P3D shaders are meant to reproduce as closely as possible the visual output of the older software-based P3D and OPENGL renderers in Processing 1.x. However, there is an important practical consequence of upgrading P3D to a fully programmable pipeline: on the Android side, devices that don’t support GLES2 won’t be able to run P3D sketches, this effectively excludes any Android smartphone or tablet which is not running Gingerbread (Android 2.3) or newer. Froyo (2.2) still makes up for a significant proportion of the devices currently in use (around 25% according to the latest stats from Google) but unfortunately its GLES2 Java bindings are broken, which lead us to define 2.3 as the minimum requirement for OpenGL on Android. Likewise, P3D on the desktop will also fail to run if the video card doesn’t not properly support OpenGL 2.0 and programmable shaders. Fortunately, most GPUs released over the past 3-4 years do support programmable pipelines, including the integrated chipsets from Intel. A significant advantage of moving to programable pipelines across all platforms is the increased performance, since GLSL shaders can be exactly customized to the kind of rendering Processing does.

2. Immediate vs retained mode

Another important trend in the recent evolution of OpenGL consists in the elimination of the immediate mode calls (glBegin/glVertex/glEnd) from the API, in favor of the use of Vertex Buffer Objects (VBOs). Basically, VBOs allow vertex data to be uploaded into buffers in video memory during initialization, from where they can be read very quickly by the GPU in order to render the scene. The main advantage of VBO (or retained mode) rendering is that performance can increase quite significantly, since the overhead of copying the data from the main to the video memory every frame with the glBegin/glEnd functions is gone. This is particularly important for mobile devices, where the bandwidth and frequency of the memory bus is limited in order to increase battery life. Starting with OpenGL 3.1 on the desktop, the immediate mode calls are only available through a backwards-compatible profile, while OpenGL ES and WebGL only offer retained-mode rendering.

However, immediate mode rendering has been the main (and only) drawing methodology in Processing until now. Furthermore, the sequential use of vertex() calls enclosed by the beginShape()/endShape() construct is in general very convenient for teaching and rapid sketching of geometrical shapes, as it is easy to grasp and doesn’t add the additional complexity of initializing buffers and other data structures. The new P3D renderer does preserve this immediate mode drawing, but implements it internally using VBOs. The internal VBOs accumulate vertex data as the user issues vertex() calls from the sketch code, and are copied to the GPU memory only once a maximum size is reached. In this way, performance is enhanced by minimizing memory transfers, while preserving the immediate mode that is characteristic of Processing.

But in addition to the immediate mode, P3D also supports a new retained mode that makes possible to create PShape objects holding large geometries, which are internally stored as VBOs in video memory. In the case these geometries are largely static and don’t need to be updated often, then the rendering can be much faster than the equivalent immediate mode code. The two following code snippets compare the rendering a single triangle in immediate and retained modes:

One triangle, immediate mode:

void setup() {
  size(400, 400, P3D);
}

void draw() {
  background(0);
  translate(mouseX, mouseY);
  beginShape(TRIANGLES);
  noStroke();
  fill(255);
  vertex(0, 0);
  vertex(50, 0);
  vertex(50, 50);
  endShape();
}

 

One triangle, retained mode:

PShape tri;

void setup() {
  size(400, 400, P3D);
  tri = createShape(TRIANGLES);
  tri.noStroke();
  tri.fill(255);
  tri.vertex(0, 0);
  tri.vertex(50, 0);
  tri.vertex(50, 50);
  tri.end();
}

void draw() {
  background(0);
  translate(mouseX, mouseY);
  shape(tri);
}

 

Immediate and retained modes in P3D are complementary: when the geometry is highly dynamic and changes every frame, the former is probably a better approach as the PShape objects would need to be updated often anyways. This will bring the performance down to the levels of the immediate mode, or even lower, since the PShape object has a slightly more complex update mechanism. In contrast, big static geometries are good candidates for retained mode rendering.

3. Line and point rendering

Line strokes with arbitrary weight values and stroke caps and joins have always been an elusive goal to achieve with OpenGL. The default LINE and POINT primitives have a hardware-limited size (often around 10 pixels) and don’t support caps or joins. In order to overcome these limitations, the line and point rendering in P3D has been reimplemented using triangle strips for lines and triangle fans for points, although the later is likely to be replaced by more efficient point rendering algorithms at the fragment shader level. The first implementation of line and point rendering available in alpha 5 allows to draw arbitrary thick lines and arbitrary large points, however is still unfinished and doesn’t implement caps and joins yet. This is not a simple problem, since correct rendering of 2D strokes in 3D space presents tricky issues such as self-occlusion of the stroke lines by the shape geometry itself. Algorithms for stylized line rendering like the ones available in the dpix library could be useful to finish this part of the renderer.

Since lines are rendered as quads in 3D space (although aways screen facing, just like typical bilboards), they can affected the perspective transformation (becoming thinner as they move away from the camera), although this is not the default. However, in some situations this might be the desired behavior, so there is a new hint, ENABLE_PERSPECTIVE_CORRECTED_LINES, which can be used to enable the application of the perspective correction on line strokes.

void setup() {
  size(400, 400, P3D);
  strokeWeight(10);
}

void draw() {
  background(150);
  translate(width/2, height/2);
  rotateY(frameCount * 0.01);

  hint(ENABLE_PERSPECTIVE_CORRECTED_LINES);
  stroke(0, 0, 255);
  line(-width/2, -100, width/2, -100);

  hint(DISABLE_PERSPECTIVE_CORRECTED_LINES);
  stroke(0);
  line(-width/2, 100, width/2, 100);
}

 

However, due to a bug in 2.0a5, both lines appear to be without perspective correction.

4. 2D rendering in P3D

As a consequence of how immediate mode and line rendering are implemented in P3D, issues might show up when rendering geometry that it is restricted to a 2D plane. The source of the problem is that rendering of fill triangles, line quads and point fans occur through different shading pipelines, so that triangles, lines and points are not necessarily drawn in the order they are specified in the code, but flushed to the GPU separately in order to optimize performance. If the geometry is truly three-dimensional, there is no problem since the occlusion from the view is not determined by the order in which the geometry drawn, but by its distance to the camera eye. However, visibility during 2D rendering in Processing is dictated by the order in wich each shape is drawn. In order to solve these issues, P3D offers a new hint called ENABLE_ACCURATE_2D which can be used when compositing 2D elements:

void setup() {
  size(400, 400, P3D);
  strokeWeight(10);
}

void draw() {
  hint(ENABLE_ACCURATE_2D);
  background(0);
  stroke(255, 0, 0);
  rect(0, 0, 200, 200);

  translate(mouseX, mouseY);
  stroke(0, 0, 255);
  rect(0, 0, 200, 200);
  hint(DISABLE_ACCURATE_2D);
}

 

This hint does slowdown rendering, since buffer optimizations are turned off when using it, so it is recommended to disable it when not drawing 2D geometry.

5. Automatic video acceleration

When movies are played back or video is captured using the core video library and P3D is set as the renderer, then the frames are automatically uploaded into OpenGL textures encapsulated by the PImage object. This results in better performance (specially for high resolution videos) and reduced CPU usage, however, the frame pixels are not available through the pixels array of the PImage object.

6. OpenGL applets and Java bindings

P3D uses the JOGL bindings to communicate with the native OpenGL libraries on the desktop platforms (Mac, Linux, Windows). The latest releases of JOGL solve the jar signing problem that affected older JOGL versions (1.x) so the OpenGL applets are working again (see this Menger example on OpenProcessing by Martin Prout).

7. What is missing in P3D in 2.0a5?

Quite a few things, although the major elements of P3D are already in place. At this point, the most notable features from the old 3D renderers that are absent in the alpha 5 are PDF recording, DXF export, OBJ loading, GL-accelerated image masksfilters, pixel get, set and copy, as well as line strokes and caps as mentioned earlier.

8. Known issues

Besides bugs that were introduced in the new renderer (or long standing ones) are currently being ironed out, a couple of issues due to the OpenGL-Java bindings are worth mentioning here:

  • OpenGL sketches on Mac OS X will print an error message upon starting (or when changing the smooth level with smooth()/noSmooth()) that looks like: 2012-03-27 16:25:12.317 java[5741:1707] invalid drawable. This is an issue with Apple’s Java plugin which seems to have no effect.
  • Smooth doesn’t work on OSX Lion, this appears to be restricted to AMD discrete cards.
  • Framerate on OSX for sketches with complex geometries could be inconsistent until this bug in the JOGL bindings is fixed.

Posted March 26, 2012 by ac in Programming

Tagged with , , , , ,

14 responses to “OpenGL in Processing 2.0 alpha 5

Subscribe to comments with RSS.

  1. Awesome work Andreas! Playing with it this week. :) Some things that I had to work through so far. Not sure if this stuff is documented anywhere…

    The change with PGL and getting the GL & GL2.
    beginGL() -> beginPGL()
    the GL additions to textures, such as texture.enable() -> texutre.enable(gl)

    • Hi, the new API is still under revision so no documentation yet…

      You can retrieve your GL and GL2 objects from PGL pgl = beginPGL(); GL gl= pgl.gl, etc… however, the PGL interface will probably undergo further changes in upcoming alpha releases.

  2. Andres, you may want to check out this link for fast line rendering. It builds on the work of Sebastien Hillaire.
    http://prideout.net/blog/?p=61

  3. Hmmmm This looks very interesting! “Added anti-alias methods so that FSAA can set up properly. The API for these is not set yet.” Does this suggest other methods for anti-alias inclusion or library extension. FXAA or SMAA. :)

  4. Hi Andres. Thanks for all the great work and for keeping us updated via your blog. I have a question. Will Processing 2.0 support tessellation shaders as introduced in OpenGL 4? And the same question with regard to GLGraphics.

    • No, because Processing 2 follows the OpenGL ES 2.0 specification in order to ensure feature parity across desktop and mobile platforms. GLES2 only has vertex and fragment shaders. However, geometry and tessellator shaders would be probably supported at some point by a future version of GLGraphics compatible with Processing 2.

  5. Dear Andreas, some time ago I was making an application for Android which combined live camera visuals with imagery. The way the applications was set-up I needed the “A3D” renderer and got it to work with the camera. See the application here: https://play.google.com/store/apps/details?id=processing.android.test.zoals (It is a dutch application) At the time I did not use Ketai, because it did not work. Now some time later I want to continue to work on a potential app which uses the camera with graphics overlayed. These graphics are 3D and work stand alone on a Android device. The Ketai library works fine on my android device (HTC desire Z) but not if I use ” size(screenWidth, screenHeight, OPENGL)” I think it has some thing to do with image buffer and OPENGL but not sure what can you help ? It might be related to this bug: http://code.google.com/p/processing/issues/detail?id=1019

    My output is:

    FATAL EXCEPTION: GLThread 12
    java.lang.NullPointerException
    at processing.core.PTexture.(Unknown Source)
    at processing.core.PGraphicsAndroid3D.addTexture(Unknown Source)
    at processing.core.PGraphicsAndroid3D.getTexture(Unknown Source)
    at processing.core.PGraphicsAndroid3D$TexCache.getTexture(Unknown Source)
    at processing.core.PGraphicsAndroid3D.renderFill(Unknown Source)
    at processing.core.PGraphicsAndroid3D.flush(Unknown Source)
    at processing.core.PGraphicsAndroid3D.endDraw(Unknown Source)
    at processing.core.PApplet.handleDraw(Unknown Source)
    at processing.core.PGL$AndroidRenderer.onDrawFrame(Unknown Source)
    at android.opengl.GLSurfaceView$GLThread.guardedRun(GLSurfaceView.java:1363)
    at android.opengl.GLSurfaceView$GLThread.run(GLSurfaceView.java:1118)

    By the way if this is the wrong place to post this please let me know. But I figured since you are working on the new OPENGL part.. I might as well try here.

    Thanks in advance!

    • Hello, it is difficult to guess a possible reason for this null pointer exception without looking at the code, at least the section that triggers it in your application. Could you post a brief code snippet? Also, what version of Processing are you using?

  6. “When movies are played back or video is captured using the core video library and P3D is set as the renderer, then the frames are automatically uploaded into OpenGL textures encapsulated by the PImage object. This results in better performance (specially for high resolution videos) and reduced CPU usage, however, the frame pixels are not available through the pixels array of the PImage object.”

    Not trying to claim any credit for your hard here, but is that (partially) inspired by my movie-capsule idea when using GSVideo/GLGraphics integration? :-)

    Anyway, looks like this will remove the need to hack around the standard Processing API for performance gains, which means less complicated code. I like it!

    Job van der Zwan

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 60 other followers

%d bloggers like this: