Processing library for feature detection and tracking   33 comments

The KLT (Kanade-Lucas-Tomasi) is an algorithm for computer vision that tracks points in a sequence of images (a video for example). Points are selected by being at the corner pixels of the image (pixels where edges intersect). Depending on how an image in the sequence distorts into the next, the algorithm computes an optical flow that allows to track the detected points in time, until the disappear from the image and are replaced by new ones.

Christopher Zach at the University of North Carolina at Chapel Hill developed a GPU-implementation of the KLT algorithm, available here, written in C++ and Cg. The main advantage of using the GPU for the computations is a greater performance when compared with CPU implementations, like the one that comes with OpenCV.

I ported Christopher’s GPU-KLT code into Java, so now it can be used as a Processing library called proGPUKLT. The computations remain executed on the GPU, so it should be comparable in performance to the original C++ library. For the moment it works only on NVidia Geforce video cards (6×00 or newer), and requires the CG toolkit to be installed on the computer.

This library has a simple interface to use the KLT tracker:

import codeanticode.progpuklt.*;
FeatureTracker tracker;
PImage img;
tracker = new FeatureTracker(this, 200, 200, 100);

The library package can be downloaded from sourceforge, and it includes the API documentation and a couple of examples.


33 responses to “Processing library for feature detection and tracking

Subscribe to comments with RSS.

  1. Hi,

    I’m trying to run the proGPUKLT examples on a MacBookPro with NVidia GeForce 8600M GT, Cg toolkit installed. The sketches require more memory (P55 preferences fixes that) and quit with an error:

    Error: Frame buffer is incomplete (GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT_EXT) at pyramid texture buffer
    Error: Frame buffer is incomplete (GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT_EXT) at pyramid texture buffer

    Invalid memory access of location 0xc eip=0x2432ebab

    Is there a workaround/fix (recompiling the lib/shaders) or am I stuck with using OpenCV? ;)

    Thanks in advance,

    • Hi Miha,

      Same hardware (8600M GT) runs proGPUKLT fine on PC (windows and Linux), I’d say it is some issue with the opengl drivers on OSX. What version of OSX are you using? This week I’ll do testing on a similar macbookpro of a friend. I’ll let you know if I find some workaround or any additional info.


  2. Hi Andres,

    I’m running 10.5.8. with OpenGL framework I would really appreciate if you could help.

    I am currently searching the net for a solution and will keep you posted if there’s any progress.

  3. The problem is that things like static methods are not accepted by processing, and they are used in the classes of proGPUKLT. So one option would be to recompile the whole processing sketch including the KLT classes inside eclipse…

    Also, Processing 1.0.7 comes with Java 1.6, if I’m not mistaken. So I’m recompiling proGPUKLT with Java 1.6, maybe this helps…

  4. Andres,

    Thank you very much for all your help (especially posting at 2:45 in the morning)!

    Yes – I ran into the problem of static methods… I’ll try my best with Eclipse.

    The new library doesn’t work either – it produces the same error. I deleted the Processing and Java cache, even tried invoking the manual Tracker Parameters but to no avail.

    I did, however, find two typos in the “GLSLFragmentProgram” source, which *could* mean something;

    “texUnitMap = new HashMap();”
    should be
    “texUnitMap = new HashMap(String, Integer);”


    “protected HashMap texUnitMap;”
    should be
    “protected HashMap texUnitMap;”

    Right? Or is the synthax completely unrelated to the problem?

    I’ll keep you posted if there’s any progress with Eclipse.

  5. Thank you Andres, but it’s the same error. In the meanwhile I even recompiled jogl, since the error report points to a command in an auto generated library (GLImpl), but to no avail.

    I truly suspect the jogl mac library – the jogl demo “Warp” I ran spat out an error (OpenGL extension “GL_vertex_program” not available), while the HDR demo looks, well, posterized to 4 colours :D

    Now I’m trying my luck with the OpenGL Profiler, whitch lets me run Processing in a closed environment where I can change different OpenGL drivers. No luck so far, but there is some progress in the errors – they are different ones.

    Best of luck, Miha

    • ok, keep me posted, a solution to this problem would be great.

      However, the “GL_vertex_program” error seems strange, given the fact that the 8600 certainly supports vertex shaders.

  6. Yep. That’s why I suspected jogl…

  7. But on the other hand, the Eclipse project you provided generates suspicious warnings at places that generate the error. Does compile, does run, with error, of course.

    Here they are:

    Description Resource Path Location Type

    The import processing.opengl is never used /SpriteTracking Eclipse/src line 2 Java Problem
    The local variable size is never read /SpriteTracking Eclipse/src line 122 Java Problem
    The local variable status is never read /SpriteTracking Eclipse/src line 119 Java Problem
    The local variable status is never read /SpriteTracking Eclipse/src line 120 Java Problem
    The local variable status is never read /SpriteTracking Eclipse/src line 115 Java Problem
    The local variable textureTarget is never read /SpriteTracking Eclipse/src line 179 Java Problem

  8. On my part – the OpenGL profiler wrangling gave nothing. I was only able to load a couple of drivers provided and they’re all to old. Probably there for compatibility purposes. The eroors encoutered were that the shaders wouldn’t compile, that the architecture was unknown or that some GPU routines were unavailable. Figures…

  9. Greatsuccess!

    I got rid of the error message about incomplete framebuffer attachment by changing the GL.GL_LUMINANCE8 to GL.GL_RGBA16F_ARB(and external GL.GL_RGBA) in pyramid derivatives creator and to GL.GL_RGBA8 (external GL.GL_RGBA) in pyramid creator. Now the framebuffers appear to have the right color attachment.

    That doesn’t mean that it was *constructed* wrong, it’s just *registered* ok with this hack. The memory access error (0xc eip=0x1bc9dbab) still persits and I suspect that both the memory access failiure and incomplete attachment are related to jogl. I’ll try the beta 2.0 build.

  10. Hey, great find! I made these changes in my SpriteTracking Eclipse project and, at least in Windows, the tracker still seems to work fine.

    Why did you decide to change GL.GL_LUMINANCE8 to GL.GL_RGBA16F_ARB/GL.GL_RGBA8 and GL.GL_LUMINANCE to GL.GL_RGBA?

  11. As a means of circumventing the issue – the other frambuffers did not produce any errors, so the color attachments were ok – except for the luminance one (pyramid texture buffer) – so I decided to change/test it.

    The fact that “luminance” produced errors in the first place is probably a jogl issue. Unfortunately I haven’t tried it with jogl2.0 yet.

    • I started looking at jogl-2.x and it introduces many API changes with respect to 1.x. In particular, the PGraphicsOpenGL renderer would need lots of changes in order to work again with jogl-2.x

  12. :D I’m in the middle of that right now… There are only two lines of code left that elude me… I even transcribed your proGPUTKL classes. If you’re willing to jump into it I would greatly apreciate it and send you my files. Need your e-mail, tho.

  13. But before going into all that I had another idea; The osx specific section on the jogl user guide states:

    “The Mac OS X port of Jogl, in particular the GL interface and its implementation, can be used either with the provided GLCanvas widget or with the Cocoa NSOpenGLView. In order to use it with Cocoa the following steps should be taken:

    – Create an “external” OpenGL context using the GLDrawableFactory.createExternalGLContext() API. The context object must be created while a real underlying OpenGL context is current.

    – Fetch the GL instance out of the context using getGL() as usual. Only use the GL instance when the OpenGL context from the NSOpenGLView is current.”

    So I’m thinking of a way to change the PGraphicsOpenGL to a “createExternalGLContext() API” and try the original progpuklt without jogl2.0 Or is there a simpler way to override a couple of lines in the PGraphicsOpenGL?

  14. Yes, but how?

  15. Hey Andres,

    Scratch the previous post ;)

    I’ve been working on other stuff recently so I couldn’t get back sooner. Sorry. I managed to play with the GL drawable & context in the .allocate() method, but the only thing it produced was another error. So that’s not the way to go.

    Did you make any progress with jogl2? I’m at a standstill with that on account of my lack of expirience…

    In the meantime I saw that you updated your GLGraphics library which uses its own renderer – could one bypass the PGraphicsOpenGL rendering in proGPUKLT using your (GLGraphics)g? If you are willing to wrangle the code (or at least give me pointers) I would greatly appreciate it.

    BTW – im on OSX 10.5 which still uses Java 1.5.


    • The new release of the GLGraphics library works better in combination with the default camera transformation routines of Processing, but it is still based on PGraphicsOpenGL. I just added a version of GLGraphics compiled with Java 1.5, this should work on OSX 10.5. You can get it from here.

      Regarding jogl2, I haven’t done much either since I got caught up with other things…

  16. Thanks for the recompile, but the original 0.9.2 works just fine on 10.5. I pointed that out in regard to proGPUKLT.

  17. Hi,

    I am again at a dead end. I am trying to track features on my web cam captured with gsvideo.
    I am simply passing the GSCapture object, to the tracker:

    that should be ok as cam is a descendant of PImage.

    tracker.getNPresentFeatures() returns always 0.

    Moreover while the project is running my whole screen flickers every 4 to 5 seconds and the live image from the webcam hangs for one second.

    Any Ideas?



  18. Hi Andreas!
    I’m stuck with the same problem on macosx
    Error: Frame buffer is incomplete (GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT_EXT) at histo pyramid buffer

    I read all comments, but didn’t found the solution. Did you solve this problem?

    • hey, I was away and I couldn’t reply to your comment before…

      One basic question that might help to clarify the problem: what gpu are you using?

  19. Dear Andrea thanks to your wonderfull work.
    Why don’t you add some sample to the other.
    In my case the sprite examples is working properly but you have to use a gf8500 or better(i use a gt9500)but the camera seems to do nothing as overlay on the camera image.
    Thanks in adavance

  20. Hy Andreas i’ve been working with the last release of your examples,that require processing 1.5 due to compatibility prolem with glgraphic(probably not connected with your library).I would like to have more example that use your library,because i think that most of people (not very expert as me) don’t understand the realy capability of your Optical flow library.
    I would like to test the library with object tracking with GSCapture or GSPipeline.
    Thanks Enrico

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: