Introducing MakeDrive

David Humphrey |

I've been lax in my blogging for the past number of months (apologies). I've had my head down in a project that's required all of my attention. On Friday we reached a major milestone, and I gave a demo of the work on the weekly Webmaker call. Afterward David Ascher asked me to blog about it. I've wanted to do so for a while, so I put together a proper post with screencasts.

I've written previously about our idea of a web filesystem, and the initial work to make it possible. Since then we've greatly expanded the idea and implementation into MakeDrive, which I'll describe and show you now.

MakeDrive is a JavaScript library and server (node.js) that provides an offline-first, always available, syncing filesystem for the web. If you've used services like Dropbox or Google Drive, you already know what it does. MakeDrive allows users to work with files and folders locally, then sync that data to the cloud and other browsers or devices. However, unlike Dropbox or other similar services, MakeDrive is based purely on JavaScript and HTML5, and runs on the web. You don't install it; rather, a web application includes it as a script, and the filesystem gets created or loaded as part of the web page or app.

Because MakeDrive is a lower-level service, the best way to demonstrate it is by integrating it into a web app that relies on a full filesystem. To that end, I've made a series of short videos demonstrating aspects of MakeDrive integrated into a modified version of the Brackets code editor. I actually started this work because I want to make Brackets work in the browser, and one of the biggest pieces it is missing in browser is a full featured filesystem (side-note: Brackets can run in a browser just fine :). This post isn't specifically about Brackets, but I'll return to it in future posts to discuss how we plan to use it in Webmaker. MakeDrive started as a shim for Brackets-in-a-browser, but Simon Wex encouraged me to see that it could and should be a separate service, usable by many applications.

In the first video I demonstrate how MakeDrive provides a full "local," offline-first filesystem in the browser to a web app:

The code to provide a filesystem to the web page is as simple as var fs = MakeDrive.fs();. Applications can then use the same API as node.js' fs module. MakeDrive uses another of our projects, Filer, to provide the low-level filesystem API in the browser. Filer is a full POSIX filesystem (or wants to be, file bugs if you find them!), so you can read and write utf8 or binary data, work with files, directories, links, watches, and other fun things. Want to write a text file? it's done like so:

  var data = '<html>...';
  fs.writeFile('/path/to/index.html', data, function(err) {
    if(err) return handleError();
    // data is now written to disk
  });

The docs for Filer are lovingly maintained, and will show you the rest, so I won't repeat it here.

MakeDrive is offline-first, so you can read/write data, close your browser or reload the page, and it will still be there. Obviously having access to your filesystem outside the current web page is also desirable. Our solution was to rework Filer so it could be used in both the browser and node.js, allowing us to mirror filesystems over the network using Web Sockets). We use a rolling-checksum and differential algorithm (i.e., only sending the bits of a file that have changed) inspired by rsync; Dropbox does the same.

In this video I demonstrate syncing the browser filesystem to the server:

Applications and users work with the local browser filesystem (i.e., you read and write data locally, always), and syncing happens in the background. That means you can always work with your data locally, and MakeDrive tries to sync it to/from the server automatically. MakeDrive also makes a user's mirrored filesystem available remotely via a number of authenticated HTTP end points on the server:

  • GET /p/path/into/filesystem - serve the path from the filesystem provided like a regular web server would
  • GET /j/path/into/filesystem - serve the path as JSON (for APIs to consume)
  • GET /z/path/into/filesystem - export the path as export.zip (e.g., zip and send user data)

This means that a user can work on files in one app, sync them, and then consume them in another app that requires URLs. For example: edit a web component in one app and include and use it in another. When I started web development in the 1990s, you worked on files locally, FTP'ed them to a server, then loaded them via your web server and browser. Today we use services like gh-pages and github.io. Both require manual steps. MakeDrive automates the same sort of process, and targets new developers and those learning web development, making it a seamless experience to work on web content: your files are always "on the web."

MakeDrive supports multiple, simultaneous connections for a user. I might have a laptop, desktop, and tablet all sharing the same filesystem via a web app. This app can be running in any HTML5 compatible browser, app, or device. In this video I demonstrate syncing changes between different HTML5 browsers (Chrome, Firefox, and Opera):

Like Dropbox, each client will have its own "local" version of the filesystem, with one authoritative copy on the server. The server manages syncing to/from this filesystem so that multiple clients don't try to sync different changes to the same data at once. After one client syncs new changes, the server informs other clients that they can sync as well, which eventually propagates the changes across all connected clients. Changes can include updates to a file's data blocks, but also any change to the filesystem nodes themselves: renames, deleting a file, making a new directory, etc.

The code to make this syncing happen is very simple. As long as there is network, a MakeDrive filesystem can be connected to the server and synced. This can be a one-time thing, or the connection can be left open and incremental syncs can take place over the lifetime of the app: offline first, always syncing, always available.

Because MakeDrive allows the same user to connect multiple apps/devices at once, we have to be careful not to corrupt data or accidentally overwrite data when syncing. MakeDrive implements something similar to Dropbox's Conflicted Copy mechanism: if two clients change the same data in different ways, MakeDrive syncs the server's authoritative version, but also creates a new file with the local changes, and lets the user decide how to proceed.

This video demonstrates the circumstances by which a conflicted copy would be created, and how to deal with it:

Internally, MakeDrive uses extended attributes on filesystem nodes to determine automatically what has and hasn't been synced, and what is in a conflicted state. Conflicted copies are not synced back to the server, but remain in the local filesystem. The user decides how to resolve conflicts by deleting or renaming the conflicted file (i.e., renaming clears the conflict attribute).

MakeDrive works today, but isn't ready for production quite yet. On Friday we reached the end of our summer work, where we tried hard to follow Pike's 5 rules of programming; specifically, we made it work before we made it fast or scalable. In the fall we're switching gears to start focusing on performance, memory footprint, deployment, and scalability. We want to be able to stand this up and start using as part of Webmaker in the coming months.

Whenever I can I like to do work like this with my students, and MakeDrive was no different. I'd like to thank Ali Al Dallal, Gideon Thomas, Kieran Sedgwick, and Yoav Gurevich. I'm really impressed with how far they came in such a short period of time. Starting in September I'll be getting more students involved as I begin teaching the Open Source courses again at Seneca, and focus on MakeDrive and a few web apps that use it.

There are a lot of different directions we could take this work, and some interesting things happening with the web that we could leverage. I'll close with a list of things I'm thinking about as I plan the next iterations of MakeDrive:

  • Matthew Willse has done some really interesting work thinking about the user end of a web filesystem. I hope he'll blog soon too, but even his initial mockups are very cool. If you have a web-first filesystem, you can do some interesting things that might not make sense in a traditional filesystem (i.e., when the scope of your files is limited to web content).
  • Having a filesystem in a web page naturally got me wanting to host web pages from web pages. I wrote nohost to experiment with this idea, an httpd in browser that uses Blob URLs. It's really easy to load DOM elements from a web filesystem:
var img = document.createElement('img');
fs.readFile('/path/into/filesystem/image.png', function(err, data) {
  if(err) return handleError();

  // Create a Blob and wrap in URL Object.
  var blob = new Blob([data], {type: 'image/png'})
  var url = URL.createObjectURL(blob);
  img.src = url;
});
  • Using this technique, we could create a small bootloader and store entire web apps in the filesystem. For example, all of Brackets loading from disk, with a tiny bootloader web page to get to the filesystem in appcache. This idea has been discussed elsewhere, and adding the filesystem makes it much more natural.
  • The current work on the W3C stream spec is really exciting, since we need a way to implement streaming data in and out of a filesystem, and therefore IndexedDB.
  • Having the ability to move IndexedDB to worker threads for background syncs (bug 701634), and into third-party iframes with postMessage to share a single filesystem instance across origins (bug 912202) would be amazing
  • Mobile! Being able to sync filesystems in and out of mobile web apps is really exciting. We're going to help get MakeDrive working in Mobile Appmaker this fall.

If any of this interests you, please get in touch (@humphd) and help us. The next 6 months should be a lot of fun. I'll try to blog again before that, though ;)

Show Comments