Watching pymake get built

I just finished reading Benjamin Smedberg's fantastic post about the method he's using to write his Python based GNU Make replacement, pymake.  The post offers a number of useful jumping off points for me to talk about various things.  Where to begin?

Well, let's start with the obvious question: why on earth would one do this?  Mozilla uses Make and Makefiles to build on all its platforms.  Some cross-platform systems favour separate native build systems, in order to take advantage of things on each target OS.  Mozilla prefers to have a standard build system, and thus chooses something of a lowest common denominator approach: autoconf + make.  This is the "right" thing on Linux, also easy on Mac, and for Windows, well, there's msys or cygwin, which gets you where you need to be (Mozilla Build provides an easy setup experience).

Mozilla has a lot of Makefiles, and for every Makefile, there is also a person complaining about some aspect of the build system.  For example, building Mozilla on Windows is slow compared to the same build on Linux, due to I/O, extensive use of nsinstall and shelling out for common filesystem tasks, etc.

When you have a codebase that has this much invested in a particular way of doing things, it's hard to change.  Not impossible, mind you, but difficult.  So back to pymake.  If you realize that you can't easily get rid of all those Makefiles, another option is to get rid of make; or rather, create a drop-in replacement and then improve it in ways that matter for your use cases.  This is exactly what Benjamin has done.

How did he do it?  Well, even before he did it, notice something important:

I started pymake with fear and trepidation. I’ve been working extensively with makefiles for 6 years; makefile parsing and execution still occasionally surprises me.
You have to know Benjamin a bit to appreciate this.  He's one of the best developers I know, and not afraid of much in the Mozilla source tree.  The fact that he can admit to "fear and trepidation" should help students just starting out feel a sense of relief: it's normal to not know how you are going to make something work, and to do it anyway.  However, notice what happens next:
This fear was a great motivator: if I had thought this to be an easy job, I might have skipped writing tests until much later in the process. But the testsuite has been absolutely essential
This is the difference between letting fear of the unknown paralyze you and letting it inform your process.  He knew that make was filled with sharp and hidden edge cases.  He also knew that if he assumed this, and wrote tests first, he'd be able to cope.  Notice how an expert programmer works: he doesn't trust his own skill, but puts tools in place to allow himself to maneuver.  The lesson here is to do things that make you afraid, and let that fear guide your work.

What does it mean to cope with something as complex as this project?

  • start small
  • make it easy to write new tests;
  • make it easy to run the tests;
  • don’t waste time writing fancy test apparatus
    The test suite you create enables you to work with confidence, both now and in the future.  As you find bugs, you can write tests to keep yourself from regressing.  As you find that you've made incorrect design choices (note: you will), you'll have the safety to refactor your code.

There's another point I wanted to discuss, namely, the potential for Python hackers to get involved with Mozilla.  I've had a number of good conversations lately with professors who know Python really well, and are also interested in what I'm doing with Mozilla.  From the inside of a project like Mozilla, you realize that it is an ecosystem, with all sorts of parallel work going on, and thus opportunities for a wide variety of involvement.  However, I think from the outside, Mozilla looks like a "web browser project," and if you're not a "web browser developer" then you aren't really needed.  The truth is that there is a lot you can do with just about any technical background.

Maybe you're good at python, interested in build systems, and want to work on profiling a parser.  Turns out this is part of "developing web browsers."  There are so many sub-projects within something the size of Mozilla, many of which might not occur to you if you haven't been working with us in the past.  I'd be happy to try and connect you with some of the work that's going on, and introduce you to the right people.  It turns out that "building web browsers" is a team sport.