Yesterday we shipped Processing.js 1.3.5. This release was mostly devoid of new code, and instead focused on fixing various issues that came-up in our 1.3.0 release. By the time we finished fixing the "few" things we'd found wrong with 1.3.0, we had about 50 bugs filed and fixed.
And despite all this we still get it wrong sometimes. After shipping 1.3.5, and feeling pretty good about our work, Casey filed a new bug, while 1.3.5 was only 30 minutes old. Here's what he was seeing, followed by what it looked like with 1.3.0 (notice the skew):
Jon and I used git to bisect and find the offending commit, and quickly found the culprit: our camera() code was exposing a call-order bug, where sizes aren't being setup properly in two different variables.
What's most interesting about this bug for me is that it would have been impossible for our current tests to ever catch this bug. Our visual ref tests are currently always 100 x 100 pixels in size, and this bug is only triggered because the canvas is not the same width and height.
After we fix it, and in order to write a proper regression test, we'll have to do something different than we usually do. The test will be easy to write, and we won't ever re-introduce this bug again. But it took an edge case, and a user filing a bug for us to get here.
You can write all the tests you want, and you can run them faithfully, but the absence of test failures will never be the same as the absence of bugs. It highlights why you need a multi-pronged approach to fixing things: active users who push your code to the limit and tell you when it fails, a body of tests that protects you from your past mistakes, a set of processes that force you to do extra checks even when you don't feel like it, and finally, humility to accept that your code will always need improvements and will always contain bugs.
I learned long ago, as a Mozilla developer, that all code sucks, all code hides bugs, and there's nothing you can do about it. It's actually very liberating once you learn this lesson, since it allows you to recover more quickly from failed patches, accept criticism in reviews, and never judge other programmers as though you aren't also subject to the same laws of programming.
Anyway, Processing 1.3.6 is coming soon, featuring less skew and with more regression tests :)