WebGL Image Processing or How to Get Consistent gl_FragCoord Values

My QR Code Scanner for FirefoxOS now uses WebGL to speed up image preprocessing.


And this is how the preprocessed video stream looks like (requires Browser with WebGL + getUserMedia support):

I'ts running some multi-pass fragment shaders to estimate a threshold for each pixel and then applies it to the gray value. I'm using a simple quad as described in this article. When I tested my shaders on four different devices, I got a a different result on each of them. I tracked it down to the following issue. The xy-coordinate of gl_Fragcoord is handled differently on all of my devices. According to the OpenGL specs, gl_Fragcoord.xy is supposed to be (0.5, 0.5) for the lower-left pixel. However I've also observed values like (0.0, 0.0), (1.0, 1,0) or (0.0, 0.5) on my Android and FirefoxOS devices. When gl_Fragcoord is used to sample from a texture, this leads to slightly inconsistent results across devices.

This might not be a big problem for a single-pass algorithm, but with multiple passes the error induced by this inconsistency can easily screw up the result completely. To work around this, I detect the device dependent glFragCoord offset and neutralize it. Therefore I render 10x10 canvas with the following fragment shader, that just puts gl_FragCoord.xy on the red/green channel:

void main() {
    vec4 result = vec4(1.0);
    result.rg = gl_FragCoord.xy / 10.0;
    gl_FragColor = result;
}

I read the values with readPixels():

    function round(val) {
      return Math.round(100 * val / 255) / 10;
    }
    var imgdata = new Uint8Array(4 * canvas.width * canvas.height);
    gl.readPixels(0, 0, canvas.width, canvas.height, gl.RGBA,
      gl.UNSIGNED_BYTE, imgdata);

    var xOffset = imgdata[0];
    var yOffset = imgdata[1];
    // assume 0.1 steps.
    var fragCoordOffset = [round(xOffset), round(yOffset)];

And pass them to my subsequent shader calls. In my shaders I then use the following function instead of gl_FragCoord.xy:

// Fragment coordinate normalized to pixel centers eg: (0.5, 0.5) for bottom-left pixel.
vec2 getNormalizedFragCoord() {
    return (gl_FragCoord.xy - fragCoordOffset) + 0.5;
}

And voila: same results on all devices.

QR Code Scanner for FirefoxOS

I just published a very first version of the QR Code Scanner for FirefoxOS that I've been working on lately. It's also available on http://ffosqr.w69b.com/ (without offline support). The GUI is till pretty simple but it does the basic jobs. And this is how it looks like:

The App is AngularJS powered, test-driven by Testacular, scaffolded with Yeoman and built with Grunt. It uses a custom port of the ZXing barcode processing library together with iconv.js and utf8.js to decode QR Codes. To grab images, it uses the getUserMedia API if available and falls back a static image picker else. Currently Opera Mobile is the only Mobile Browser that supports getUserMedia (and only the presto version, the new webkit-based beta does not), but you can also try it with a recent Desktop Browser (Firefox 20+, or Chrome 21+).

The captured images are transferred to a web worker where the real work happens. This works quite smooth, even with Opera on my Android-powered mobile devices. Though I could not test it on any low-end device yet.

Besides obvious GUI improvements, in future versions I'll add support for more barcode types and speed up the detection by offloading some image prepossessing work to the GPU using WebGL.