Mozilla is showcasing our Flight of the Navigator (FOTN) demo, and I wanted to tell you a little bit about it. FOTN takes advantage of many of the new and improved features of Firefox 4, and is an example of the kind of things you can do with HTML5, WebGL, web audio, and faster JavaScript.
The first thing to say is that this is a web page--HTML, CSS, and JavaScript. Some people we've shown this to have assumed it was a movie until they saw the real-time Twitter and Flickr integration. This is a regular web page, and uses the web to full effect. However, it doesn't look like a regular web page, and therein lies much of the power and potential of the web coming with Firefox 4.
We built the demo in order to integrate and showcase the various JavaScript libraries and browser features we were creating together. As a result the demo exploits a number of features in HTML5, namely, <canvas>, WebGL, <audio>, and <video>. It also uses some of our JavaScript libraries (CubicVR.js, processing.js, dsp.js, beatdetektor.js), as well as the various JavaScript speed and feature improvements in Firefox 4. Finally, the demo is driven by the new Firefox Audio API to synchronize the audio and 2D/3D visuals.
Everything you see in the demo's 3D world is done using <canvas> with a WebGL context. The city and all other 3D structures, the models and animation, were created using Lightwave. We used CubicVR.js to work with the resulting 3D XML data, to build and power the scenes. We texture the buildings and other structures with a combination of images (regular jpg, png), videos (WebM), and 2D canvases. For example, we create various effects (interlacing on billboards, ticker-tape displays, audio visualizations) using Processing.js on 2D canvases, and then texture those onto the 3D objects.
We wanted to build a hybrid of what people are used to seeing in the regular web with the 3D capabilities of WebGL. We did this by pulling in live data from Twitter (see the ticker tape animations for #firefox4 tweets) and Flickr (see photo billboards). This makes the demo different for every run--we've had some interesting moments at conferences running this live and pulling things from Twitter.
FOTN relies on many performance upgrades in Firefox 4 to make things run smoothly. First Hardware Acceleration makes it possible for us to mix 3D an 2D graphics very quickly, and run at high frame rates, even when the browser is fullscreen. Next, we use many of the new features in Firefox 4's JavaScript engine, for example Tracing to give us a speed boost on our complex audio and graphics code, and Typed JavaScript Arrays to make our audio analysis work in real time.
As the music plays we use the audio's signal data from the Audio Data API to calculate FFT and beat detection. This data is used to trigger animations, transitions, and visualizations. We're doing this all in JavaScript--the same JavaScript that's being used to simultaneously render our 2D and 3D canvases. It's amazing how far you can push JavaScript now.
Speaking of the music, we're using an Ogg Theora version of Shpongle's "Nothing is Something Worth Doing" (Copyright Twisted Records, UK, used with permission). Grab headphones or turn up your speakers to enjoy this awesome track. We listened to it hundreds of times building this, and it's never gotten old.
We built this demo in our spare time over a 4 week period with nightly builds of what is now becoming Firefox 4. We've been working with these technologies for a while now, and can tell you that the potential of Firefox 4, with all the new graphics and media features it's shipping, is amazing. The web platform is capable of an amazing amount, and it's exciting to think about where people will take it next. I know we're working on a new, larger demo already :)
FOTN was built by the same people who have been working on the Audio Data API and demos with me, and I'll end by naming them all. They are an amazing group of developers to work with: Charles Cliffe, Corban Brook, Al MacDonald, Andor Salga, Maciej Adwent, Boaz Sender.