Experiments with audio, part XI: RIP Audio Data API (2009 to 2013)

Mozilla has implemented and shipped the Web Audio API recently, and at the same time begun the work to remove the Audio Data API.  I wanted to say something on the occasion of this transition, and what I learned about the web in making the Audio Data API.

In 2009 Seneca's Centre for Development of Open Technology applied for and received one of the largest research grants available to Canadian educational institutions.  Our grant (which I'm still working on today) was aimed at supporting open source development with various industry partners, one of which is Mozilla.

Some day I need to write about all the things Seneca and Mozilla have been able to do together in that time, but one of the crazy ideas we had was to "create something like canvas for audio scripting."  It was the sort of thing that no one at Mozilla had the cycles to do--if you were working on or with Mozilla at this time, you'll remember that shipping Firefox 4 was a somewhat busy period.  I remember writing about it in the grant proposal and literally having no idea how we'd even begin.  My thought was that we'd tackle this late in the grant.

At about the same time I was working on finishing the port of Processing from Java to JavaScript.  As a result of this work I was meeting a ton of talented visual artists, data visualizers, game developers, and others who seemed to always be talking about mixing animation with generated audio effects.  I was also working closely with Mozilla's JavaScript and graphics teams, filing performance bugs.  Through work Mozilla had done to make JavaScript, Canvas, and WebGL fast enough to run Processing sketches, I'd gained a new kind of confidence (hubris?): the web can be fast enough, if you file the right bugs (I gained a bit of reputation among my students and colleagues who made fun of me for always encouraging them to "file it").

Despite what I was being told by many "experts," I knew that JavaScript was fast enough to manipulate audio, and I wanted to prove it.  All that was necessary was to make the raw audio samples available to the DOM.  So in December of 2009 I started down this road.  Along the way I blogged everything, and tried to turn what I was doing into something I could share with my students in class.  I won't repeat things I've already written, but I wanted to talk about some of what I learned along the way.

First, the Audio Data API was created and built by volunteers.  That's one of the most important aspects of what we did, and why I still work on Mozilla.  We often talk about the web as a democratizing technology and as an open platform for everyone.  However, it's not just that wikis are open, or that you can make your own web page.  The web platform itself is open and malleable, and Mozilla is willing to enable a community of participation all the way down the stack.  We didn't make a web page; we remade the browser, and made a whole new kind of web page possible.  There aren't many other platforms where that's even possible.  The web is a different sort of platform, and Mozilla a different sort of gatekeeper.

Second, I learned that changing the web is really hard, and that without encouragement and support, you'll fail.  I can honestly say that if people like Chris Blizzard, Vlad Vukićević, Benoit Jacob, Olli Pettay, and many others hadn't been so relentless in their cheer-leading, and gone out of their way to help and encourage us, I'd have given up.  I learned a ton about how to mentor through that period by being mentored myself.

Third, Chris Blizzard taught me that demos win the web.  It's not true that a good API is all you need.  In fact, 'worse is better' has taught us that the opposite is often true.  As we did our work we received a ton of media and developer attention, and it was because we paired API changes in the browser with cool demos of it being used.  We had a lot of firsts in that period: first FFT calculated and visualized in JavaScript; first web page to make noise from JS; first text-to-speech in JS; etc.  I was lucky to have people like Corban, CJ, Al, Yury and other uberhackers building things with my stuff as fast as I got it working.  Every time I made the browser do some new thing, they had 5 demos to show people how it could be used.  It was the most exciting and fast paced collaborative development period of my life.  Unless you have a demo, your feature isn't real yet.

Fourth, winning doesn't just mean you get your way.  Today I'm writing a post about code I slaved over for close to a year being deleted from Mozilla, and yet I feel very much like we won.  I always remember Shaver talking about Microsoft starting work on IE again as being a "win" for Mozilla, and wondering why he felt that way.  Now I get it.  The work we did accomplished a number of things.  First, it changed peoples' minds about what the web was capable of doing: the web is fast enough, we proved it.  Second, it kickstarted the standardization of audio programming on the web, see Doug's post here.  Third, as Martin Best told me at MozFest a few weeks ago, it gave Mozilla a way to do early work on gaming, when there was no way to do programmatic audio in a browser at all.

I wanted to write this so I could take a minute to thank those that worked on this with me.  I also wanted to bring attention to the fact that you can influence the web.  I see too many web developers complain and not file bugs, or have ideas for how to evolve the web and not have the courage to try hacking their browser.  There is this idea among some that standards need to happen before you can do any work.  I totally disagree.  You standardize what you know is working, and you only get there by doing experiments.  What the web will do in 5 years needs to start today as some hack by a bunch of friends who have an idea.  Don't be afraid to experiment with the web.