Processing 2.0 is out! Processing 2.0 is in!

Yesterday was a very important day for the Processing project, as a new stable version, “the 2.0”, has been released. This release is the result of the hard work of a small team of volunteers over the course of the past two years, plus the fundamental support and contributions from the entire Processing community. For me, this release is particularly significant since it includes a major rewrite of the OpenGL and video libraries, which represents my main contribution to the project since I become involved in it almost 5 years ago. After a long period of development, it is very satisfactory to reach a point where the code is good enough to abandon the nebulous territory the of alphas and betas. Of course, a stable release like this is also a compromise between imagination and time. Despite of the standing issues that result from that compromise, Processing 2.0 retains all the functionality that turned it into a widely used tool in computational arts, as well as adding new features and improvements that extends its capabilities and also serve as the starting point for future developments. In what follows, I’d like to describe in more detail some of the technical challenges we faced while working on the new OpenGL library, and the solutions attempted in order to deal with those challenges.

Processing 1.x already included an OpenGL-based renderer which took advantage of hardware acceleration in order to speed-up 3D graphics. However, this renderer had to be rewritten entirely from scratch due to the changes in the OpenGL API from 2.0 to 3.0 on the desktop, and the introduction of OpenGL ES (GLES) 2.0 on mobile platforms. In fact, there are two new OpenGL-based renderers in Processing 2.0: P3D and P2D, with the later further optimized for two-dimensional rendering.

Retained mode rendering

As mentioned in earlier posts, the old and easy-to-learn immediate mode in OpenGL:

glBegin(GL_TRIANGLES);
  glVertex3f( 0.0f, 1.0f, 0.0f);
  glVertex3f(-1.0f,-1.0f, 0.0f);
  glVertex3f( 1.0f,-1.0f, 0.0f);
  glEnd();

is all but gone, replaced by the use of buffer objects throughout the entire OpenGL API. For example, the immediate-mode code for rendering a triangle shown above would need to be replaced by something along the lines of (in C):

const float vertices[] = {
   0.0f, 1.0f, 0.0f, 1.0f,
  -1.0f,-1.0f, 0.0f, 1.0f,
  1.0f,-1.0f, 0.0f, 1.0f,
};

glGenBuffers(1, &buffer);

glBindBuffer(GL_ARRAY_BUFFER, buffer);
glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_STATIC_DRAW);
glBindBuffer(GL_ARRAY_BUFFER, 0);

glBindBuffer(GL_ARRAY_BUFFER, buffer);
glEnableVertexAttribArray(0);
glVertexAttribPointer(0, 4, GL_FLOAT, GL_FALSE, 0, 0);

glDrawArrays(GL_TRIANGLES, 0, 3);

The use of vertex buffer objects, although requiring more complex code, also results in faster rendering because the geometry can be organized in batches that are sent to the GPU with fewer calls. In addition, geometry that remains unchanged can be stored in a “static” buffer that is uploaded to the GPU memory only once, which offers significant performance improvements.

What is the impact that these changes in the OpenGL API have in Processing? First of all, Processing’s drawing API is basically immediate-mode, with functions like beginShape and endShape mapping very closely to the gl-equivalents:

beginShape(TRIANGLES);
vertex( 0, 1, 0);
vertex(-1,-1, 0);
vertex( 1,-1, 0);
endShape();

Therefore, the buffer-based operation in OpenGL 3.0+ and GLES 2.0+ posed two related challenges: the first being how to reimplement the OpenGL renderer so that all the immediate mode API in Processing continues to work in the same way as it did in 1.x, while offering reasonable performance, and the second: how to extend the drawing API in order to incorporate retained mode rendering (i.e.: enabling static geometry to be stored in an object) in a way that is consistent with the existing API.

The solution of the first challenge is purely technical and invisible to the Processing user. It involved a lot of code behind the scenes that tries to batch the geometry from the immediate-mode calls (rect, ellipse, etc) as efficiently as possible. Implementing this code properly is tricky because gl-state changes like disabling/enabling lighting or texturing breaks the batching, and also requires to keep track of the buffers not becoming too large (which is particularly important on mobile).

In fact, continous changes to the gl-state can have a substantial (negative) effect on performance and make the new OpenGL renderer slower than in 1.x, even for relatively simple code. For example, rendering a large number of interleaved textured/non-textured rectangles, like in:

for (int = 1; i < 1000; i++) {
  image(img, 0, 0, 50, 50);
  rect(0, 0, 50, 50);
}

would perform slower than grouping the textured and non-textured rectangles together:

for (int = 1; i < 1000; i++) {
  image(img, 0, 0, 50, 50);
}

for (int = 1; i < 1000; i++) {
  rect(0, 0, 50, 50);
}

although the visual output will be the same in both cases. Here, the automatic batching that the renderer in 2.0 does internally in order to optimize performance is helpless. Even though this seems to be a rather contrived example, it is important to keep this limitation of the new renderer in mind, specially when working with more complex sketches where certain rendering paths might indadvertedly lead to this kind of scenarios.

The second challenge, of integrating retained mode rendering into Processing’s API, was addressed with the introduction (or rather, the re-introduction, since the class already existed in 1.x) of the PShape class. In a nutshell, the PShape class now contains methods that reproduce the immediate-mode functions, which allow to “draw and store” the geometry inside a PShape object:

PShape sh;
float angle;

void setup() {
  size(400, 400, P3D);
  sh = createShape();
  sh.beginShape();
  sh.fill(180);
  sh.vertex(-200, -200);
  sh.vertex(200, -200);
  sh.vertex(200, 200);
  sh.vertex(-200, 200);
  sh.endShape();
}

void draw() {
  background(0);
  translate(200, 200);
  rotateY(radians(angle));
  shape(sh);
  angle += 1;
}

The vertex data is transferred to the GPU only once, which makes rendering much faster, specially for large meshes. For a more detailed description of the PShape functionality, check the excellent tutorial written by Daniel Shiffman.

It should also be noted that a PShape object can be modified after creation, although doing so will cancel out the performance gains (since the vertex buffer will be refreshed every time changes are made). All the drawing methods that can be used between PShape.beginShape() and PShape.endShape() to create the shape, have a couple of associated getters and setters that can be used outside the beginShape/endShape block in order to query the current values and set new ones. A simple example of this functionality is the following:

PShape sh;

void setup() {
  size(400, 400, P3D);
  sh = createShape();
  sh.beginShape();
  sh.fill(180);
  sh.vertex(0, 0);
  sh.vertex(400, 0);
  sh.vertex(400, 400);
  sh.vertex(0, 400);
  sh.endShape();
}

void draw() {
  background(0);
  sh.setVertex(0, mouseX, mouseY);
  shape(sh);
}

GLSL Shaders

The other major change in GL 3.0+ and GLES 2.0+ was the removal of the so-called fixed-function pipeline. With this change, things like the default lighting and matrix calculations are gone, and GLSL shaders are now required to perform any kind of rendering operation, even if it is as simple as drawing a two-dimensional, solid-color quad without any perspective or geometric transformations applied to it.

On the one hand, this removes the artificial limitations of the fixed-function pipeline imposed on modern GPUs – which provide quite advanced programming capabilities – and allows for an amazing degree of flexibility to create real-time graphics with the GPU. On the other hand, writing shaders only for doing quick sketching using default camera and lighting settings can become a hassle, and even a significant barrier for programmers who are not familiar with a low-level language such as GLSL. So the challenge here was to combine these two conflicting aspects of shader programming so that Processing users can still get gl-accelerated drawing without having to learn GLSL, and at the same time giving advanced users the option to integrate their shaders into Processing sketches. The new shader API included in Processing 2.0 is described at length in this tutorial, so I won’t go into details here.

One issue that is worth noting is that giving access to a low-level shading language from a relatively high-level drawing API is not entirely straightforward because they operate at different levels: Processing’s API generates geometry with a very specific structure, while GLSL doesn’t impose many constraints on the input geometry, other than the data types. This mistmatch was handled by imposing a number of naming conventions that need to be followed in the shader code in order to be recognized by Processing. In addition to that, the shader must be belong to one of six available types (point, line, color, texture, light, texlight), so Processing knows what type of geometry can be rendered by the shader. The type is specified by a #define embedded either in the fragment or vertex shader, so it doesn’t affect the validity of the GLSL code. All of this considerations are covered in the tutorial mentioned above.

The PGL interface

With evolution that OpenGL underwent over the last decade, the API has by now several different variants: the deprecated pre-3.0 (fixed-function) version on the desktop, OpenGL 3.0+ and 4.0+ (programmable-pipeline only) also on the desktop, OpenGL ES 1.x (fixed-function) and OpenGL ES 2.x+ (programmable) on mobile.

These different OpenGL API “versions” can sometimes be available on the same system, as OpenGL ES 2.0 is effectively a subset of OpenGL 3.0. JOGL – the OpenGL-Java bindings that Processing uses to talk with the native gl drivers – implements a system of profiles that allows to access a specific subset of GL functions. The OpenGL renderer in Processing provides access to all the JOGL profiles through a “glue” object called pgl:

void draw() {
  // get the PGL object and prepare the renderer for low-level gl calls
  PGL pgl = beginPGL();
  GL gl = pgl.gl; // get the common GL profile from pgl
  GL2 gl2 = gl.getGL2(); // get the GL2 profile from JOGL
  …
  gl2.glEnable(GL2.GL_BLEND); // use your GL code
  …
  endPGL();
}

Alternatively, the PGL object also implements all the GL functions that are part of the OpenGL ES 2.0 specification,
plus a few additional ones that are not part of the GLES 2.0 spec but are necessary on the desktop for multisampled rendering (blitFramebuffer, renderbufferStorageMultisample), color buffer IO (readBuffer, drawBuffer), and buffer mapping (mapBuffer, mapBufferRange, unmapBuffer). Offering this interface in PGL itself is motivated by the decision of using GLES 2.0 as the baseline GL API that ensures compatibility across desktop, mobile and also web with WebGL.

void setup() {
  size(400, 400, P3D);
}

void draw() {
  // Setting the background color with OpenGL:
  PGL pgl = beginPGL();
  pgl.clearColor(1, 0, 0, 1);
  pgl.clear(PGL.COLOR_BUFFER_BIT);
  endPGL();
}

One advantage of using the gl functions from PGL instead of the JOGL profiles is that the former ones are “automatically aware” of the changes done on the drawing surface by the Processing code, whereas using the lower-level JOGL calls would require additional work to properly interact with the Processing calls. For example, the following sketch draws some shapes with Processing, and then uses PGL to read the color buffer at the mouse location. Since the color buffer might be multisampled (with Processing internally handling the buffer swapping and blitting), the readPixels() call does some additional setup behind the scenes to read the pixel color from the right buffer:

import java.nio.ByteBuffer;

void setup() {
  size(400, 400, P3D);
}

void draw() {
  noStroke();
  fill(255, 0, 0);
  triangle(0, 0, width, 0, 0, height);
  fill(0, 0, 255);
  triangle(width, 0, width, height, 0, height);

  if (mousePressed) {
    PGL pgl = beginPGL();
    ByteBuffer buffer = ByteBuffer.allocateDirect(1 * 1 * Integer.SIZE / 8);

    pgl.readPixels(mouseX, height – mouseY, 1, 1, PGL.RGBA, PGL.UNSIGNED_BYTE, buffer);

    // get the first three bytes
    int r = buffer.get() & 0xFF;
    int g = buffer.get() & 0xFF;
    int b = buffer.get() & 0xFF;
    println(mouseX + ” ” + mouseY + ” = ” + r + ” ” + g + ” ” + b);
    buffer.clear();
    endPGL();
  }
}

For another example showing the interaction between Processing calls and the GL calls in PGL take a look at the Demos|Graphics|LowLevelGL, included in the Processing download.