Experiments with audio, conclusion

I’ve been working with an amazing group of web, audio, and Mozilla developers on a project to expose audio data to JavaScript from Firefox’s audio and video elements. Today those experiments are over.

In December a few of us working on processing.js had an idea--what if we could visualize sound data coming out of an <audio> or <video> element?  My colleagues were good at thinking in terms of "how can we make what we have now work?" but I had another idea.  "Let's try and teach Firefox how to do this."  In December I set myself a challenge:

This post marks the beginning of what I expect to be a somewhat regular series of posts in which I will document my thinking, learning, and progress related to some Firefox development.  I say “Experiments” above, because I am going to try a number of things.  First, I’m going to push into a part of the Firefox source code where I haven’t worked before, namely, the DOM implementation (note: it scares me, to be honest).  Second, I’m going to do so in an open and pedagogic way, attempting to cast aside my own ego and hesitation at looking foolish–I don’t actually know how to do what I’m going to try, and will learn and fail as I go–on the way to producing an authentic model of open development for my students.  Third, I’m going to work with a few others who are also interested in extending themselves and extending the web.  The only thing I can assure you of at this early date is that these posts will be an honest account of the attempt.
Yesterday, the end result of that work landed in mozilla-central, on its way to inclusion in Firefox 4.  I'm immensely proud of the work we've done, and thrilled that my peers in the Mozilla community have also accepted it.  I'm also very tired :)

There's lots of things that I could talk about in terms of the code and API, and probably we'll do some of that soon (you can already use the API in a Firefox nightly, read about the API, and try live demos).  But what I wanted to end this series of posts by saying something about how Firefox, the Mozilla community, and the open web, make what we did possible.

The other day my family had some friends over, and we got talking about what I do.  Of course they had heard of Firefox, and used it themselves.  "But can you explain the difference between this and the other one I use, Internet Explorer."  One of the big differences, I explained, is participation, the community of involvement, and the accountability that comes with this.

When we started these experiments, we did so without needing permission.  I didn't have to sign an NDA, go talk to and convince the right people, or get approvals.  I just grabbed the source code and started messing around.  And I did make a mess, at first.  I learned as I went, and we iterated on the API a lot (I have over 80 versions of it here, not to mention the various implementations of those).  We weren't judged for doing it wrong, or for the pace or directions we took.  Instead, we heard of lot of "this is very cool!" and "have you thought about this?".  We were able to take one of the world's premier applications (Firefox) and rework it.  It's hard to overemphasize how significant this is.  We couldn't do what we did in very many other contexts.

I said above that participation is paired with accountability, and this is also very important.  In the early months we built something that worked, but not what we have today.  As much as Mozilla made it possible for me to experiment, they also made sure that what got accepted was of the highest quality.  I haven't blogged about audio much over the past three months, mostly because we've been too busy getting the patch fixed up based on reviews.  Before it could land we had to think about testing, security, JS performance, DOM manipulations, memory allocation, etc.  To get this landed we needed lots of advice from various people, who have been generous with their time and knowledge

What's different about Mozilla?  I think of something Joe Hewitt wrote on twitter back in the spring that struck me:

Bottom line: we can currently only move as fast as employees of browser makers can go, and our imagination is limited by theirs. @joehewitt
Not one of the people who did this work is an employee of the Mozilla corporation.  When we decided to get serious about trying to include this in Firefox 4, one of the people working with us filed a bug, and the response was, "I don't think we have the cycles to get this done in time."  "That's OK, we'll do it then," was the reply.  And we did.

The web is too big and too important to only go as fast and as far as a small group of employees can take it.  Mozilla gets this, and values community involvement like ours.  What we did is not unique--there are other great features and bug fixes coming in Firefox 4 that were done by community members vs. employees.  In fact the distinction between the two is often hard to see when you're working on this stuff--we all work together.

Having said that, let me publicly congratulate my amazing audio peers, without whom this work wouldn't have happened: Al MacDonald, Corban Brook, Yury Delendik, Charles Cliffe, Ricard Marxer, and our cheerleader and supporter, Chris Blizzard.  Also thanks to the dozen others who wrote demos with our stuff in the early days.  Demos are how you win (Chris Blizzard taught me that).

I'll end with a video someone shot of my keynote talk at the recent Mozilla Summit.  I was doing a quick demonstration of what is possible with this API.  I look forward to seeing what the rest of the web will do with it.