SnowyOwls.ca

by David Humphrey

tldr; I made you a little birding web app for Christmas with Begin.com and Next.js to help you find Snowy Owls in Canada. The code is here.

On the Healing Powers of a Side Project

I've finally finished the semester, and am ready for a holiday.  The past three weeks have been non-stop marking: labs, assignments, quizzes, tests, projects, you name it.

I find that these long marking periods go better for me if I also work on a side project in parallel.  When you teach programming, and your marking involves reading and reviewing a million lines of code, you start to yearn for opportunities to write some code of your own.  I always need a project, so while I work through my marking pile, I give myself little breaks to implement bits of my chosen side project.  It keeps me moving forward and happy.

Owls as Inspiration

I've been obsessed with owls throughout the pandemic.  I wrote previously about our woods' new Barred Owl, and how we installed a nest box for it (UPDATE: we've had a second Barred Owl move in, so everything seems to be going to plan).  Our Great Horned Owls have started hooting to each other every evening, which is amazing to listen to as we go for our walks.

But as the snow begins to fall each December, my attention turns to another owl: the Snowy Owl.  Normally at this time of year I'm seeing Snowy Owls on my long commutes to and from work.  With COVID, I'm not out driving anymore, and as such, I'm not having as easy a time finding them.

I decided that this year's marking-side-project would be a tool to help people find Snowy Owls near where they live.  I've long wanted to play with eBird and the eBird API, and hoped that I could get recent sighting data this way.  To use the eBird API, you have to create an account and then request an API key.  After that you can do all sorts of interesting queries to get current or historical data about sightings by species, region, or location.

My goals for the project were these:

  • Had to be shipped at the end of my marking period
  • Include a write-up with some general info on Snowy Owls and advice on how to find them.  I would need to do research for this part.
  • Include some historical data (i.e., charts) from eBird of sightings in Canada to show when Snowy Owls are most often seen
  • Create an interactive, live map of current sightings for the past month
  • Create a sortable tabular view of the same data, with locations and how old the sightings are

Serverless with Begin + Next.js

I also wanted to use this project to learn a few new technologies.  I decided to write the back-end as AWS Lambda functions using Begin, and build the front-end in React with Next.js.

First, I've been meaning to try Begin for over a year.  I've been following along with Brian LeRoux's work on arc.codes and Begin, and every time it comes up in my feed, I tell myself "next time, I'll try this."  Well, this is the time!

Second, I have to teach a bunch of web and front-end React courses next term.  I also need to learn Next.js for a few of the projects.  Picking React and Next.js seemed like an easy way to refresh myself.

Thoughts on Begin.com

Begin is a platform for deploying modern web apps (CDN, data, lambda functions, etc).  I've worked a lot with static hosting platforms like GitHub Pages, Vercel, and Netlify, but never with Begin.  One of my goals for 2021 (first time I've written that, goodbye 2020!) is to learn more about AWS.  My institution has become part of AWS Educate, which means that my students and I can get access to AWS services for our courses.

I thought that Begin would be nice, because it lets me try AWS, but with training wheels on.  I'm not on the hook for configuring or securing anything directly.  Begin gives me a code-based, declarative pipeline from GitHub to AWS, including S3, CloudFront, Lambda, API Gateway, DynamoDB, and probably other stuff I don't know about.  I push to my main branch, and my code is linted, tested, built, and deployed to staging.  Similarly, if I push a tag, everything goes to production.  They have a very generous free tier, which blows away what you get from the other JAM stack providers I've used.

Things that I like about Begin:

  • I bring a git repo, and they connect everything else within AWS.  I'm not ready or interested (yet) in twisting all the dials in AWS, so this is a fantastic option for me.  For the most part, I never felt like complexity of the underlying AWS infrastructure or configs "leaked" into my project.
  • Begin is technology/framework agnostic.  I could have written this web app 50 different ways, and they'd all be workable with Begin.  You aren't forced into a particular framework or set of tooling.
  • Almost everything is configured via files in the repo vs. enormous web consoles.  Begin does have a nice web console where you can see what's happening and change how certain things work.  But the bulk of what you do is done declaratively.  I also like that you work at a high enough level that you aren't stuck authoring thousand-line YAML files.
  • The docs are fairly complete and had most of what I needed.  I've read them all multiple times, and often when I thought something hadn't been covered, I'd go back and discover that the details were in fact there.
  • I love that everything you need for a web app is here.  Almost every platform I use is missing something: static hosting I expect, and serverless functions are becoming mandatory, too;  but to also have a database built in is amazing.  Just about every project you build has some need for data, and Begin has it. I used it to build a simple analytics back-end (thanks to Wes for the sample code I used to write my own), and with more time could have done a lot more. It's really flexible.
  • Being able to bring my own domain and have it hooked up automatically with SSL certs.  I didn't have to fiddle with nginx or certbot.  Amazing.
  • The various pieces of what you're building fit together in ways that make sense.  I've used some other JAM stack platforms, and their functions often felt bolted-on as an after-thought vs. part of the initial offering.  With Begin I have a src/http/* directory for all my functions, and my root directory holds my React app, while shared code lives in src/shared. I like working in monorepos, and this feels like a good design, with everything in easy reach.
  • A lot of what Begin's CI/CD pipeline does for you, I can do myself with GitHub Actions.  But, here I didn't have to do any of it.  Having the entire pipeline preconfigured was excellent, and let me work more quickly.

Things that I found confusing or difficult with Begin:

  • The name.  It's impossible to Google for bugs, docs, StackOverflow, etc.  My general search algorithm uses the technology name as a prefix.  But literally every technology you work with has "begin" in their docs, so that doesn't work.  I never really figured out a good workflow for finding things with Google.  Netlify, Vercel, etc. don't have this problem.
  • Edge cases with the tooling.  I ran into a bunch of situations where the docs and tools didn't match my experience.  For example:
  1. the docs claim you can generate boilerplate code for your http functions. I started from one of their sample projects that didn't have any http functions.  As a result, when I wanted to add my functions, nothing was installed or wired up correctly (e.g., no sandbox).  The docs assume this is all done for you, so when you have to do it yourself, you end up having to dig around in other sample projects to see how they do their configs and setup.  This isn't easy, though, because there aren't very many complex apps to use as an example (or I couldn't find them, see naming issues above).  I'm used to working without docs, so I got most things working on my own.  But I think this could be improved.
  2. Getting my custom domain set up with Begin was difficult (for me).  Their docs discuss a number of suggested domain name registrars.  Based on this I chose Namecheap and bought my domain, snowyowls.ca.  Begin wants you to create CNAME records for staging.snowyowls.ca and www.snowyowls.ca.  I followed all the docs carefully but couldn't get it.  Eventually I found out that Namecheap needs me to include the .www and .staging suffixes for my CNAMEs.  I still don't have all the HTTP to HTTPS redirects working the way they should.  Probably this is something that people who do this all the time would know, or could figure out, but it seems like Begin is aiming for devs who do need help with this.  I wasted a lot of time on issues like this.
  • ENOSPC errors during install or build steps. Begin has hard limits on the size of your development directory (~500M), and also each of your functions (~5M).  As a result of these ceilings, I was constantly banging my head.  So many of the packages I normally reach for were suddenly too big.  This includes many dev dependencies I rely on (Prettier, Jest, and all the ESLint plugins I usually use).  I would get everything working locally, push my code, and have Begin fail with ENOSPC.  Sometimes it would be the total size of my repo; other times a particular function was overweight; still other times the error would be wrong, and trying a rebuild would make it work.  In all cases you get an opaque ENOSPC error and that's it.  I had to rewrite a bunch of modules to inline, stripped-down versions of code I needed.  I don't know how Netlify gets around this while Begin can't (or hasn't), but I think it's a major blocker for them, and will hurt adoption.  (As an aside: we joke about how massive node_modules is, but seriously, the fact that I can't fit a reasonable JS dev environment in 500M without a lot of extra work is totally ridiculous. The environmental impact of having to download close to 1G of deps every time I want to write a single line of JS in a new project, or for every CI run, is a problem we could and should solve.)

Overall, I enjoyed working with Begin on this project.  Once I figured out how everything worked, I was able to work quickly with their pipeline.  It was great writing my back-end and front-end together, and being able to include persistent data for analytics.  I have zero concern about scaling, security, cost surprises, etc.

Begin is great and you should try it.  I'll use it again for sure.

Thoughts on Next.js

When I'm writing React code, I usually use create-react-app.  It works well for small projects, but I find its lack of opinion on lots of things means you have to supplement it with all kinds of extra packages.  I find this frustrating, because I want to focus on my project, not on picking winners in the endless front-end arms race.  At the other extreme you have something like Angular, which I also have to teach next term.  It has an opinion on everything, almost none of which I share.  It's too stifling, and I usually want something in between.

Another option is Gatsby, which I spent a lot of time using last year on another project.  However, it doesn't really fit with the kind of apps I like to write (i.e., dynamic client-side apps vs data-driven static sites).  Next.js seems to offer an interesting alternative, with lots of good decisions already made for me and excellent developer experience, but without a strictness that would limit me from customizing the things I have to.  I was excited by what I read about the version 10 release, and wanted to give it a try.

Things that I like about Next.js:

  • Their "zero config" really is exactly that.  I didn't have to spend any time on setup or fighting with configs.
  • The choice between server-side rendering (SSR), pre-rendering at build time (SSG), or client-side rendering.  I use a mix of pre-rendering and client-side rendering, and it works great.  When I'm done I use npm run build and npm run export and I have a statically built out/ directory that Begin can upload to S3.
  • useSWR is perhaps the best data fetching library I've ever used.  Period.  I absolutely love it, and I'll use it on every React app I build in the future.
  • The filesystem routing is easy to use.  Drop a file in pages/ and it's a page in the app.
  • I love that all static assets in public/ are automatically available at the root within the web app (e.g., public/favicon.ico is href="/favicon.ico").

Honestly, there is so little Next.js code in my repo that it's hard to discuss it or even find it as I scroll through the repo!  Everything is just my React app, with a few cli tools I don't have to configure.  This is what I want.

Things that I found confusing or difficult with Next.js:

  • A lot of the awesome things you read about Next.js 10 doing require a server.  For example, the <Image> component and automatic image optimization is something I was excited to try.  In Gatsby I've loved using the automatic build-time image magic it offers.  However, it turns out that you have to render these images on a server, which breaks with my serverless approach.  Another example is asset compression.  The docs claim you can gzip your text assets.  However, dig a little deeper and this is offloaded to your serverless provider.  I'd love for Next.js to embrace a bit more of what Gatsby does for static optimizations.  Side note: why is this not happening automatically for me with Begin?
  • I initially really enjoyed the simplicity of the Next.js Router.  However, a few weeks into the project, I started to run into problems.  For example, scroll position restoration when navigating between pages is horribly broken.  I had to write my own custom scroll restoration code, which was a total pain and waste of time.  If next.js is going to call them "pages" it really needs to track scroll position between navigations like a regular web page does.  I also had problems getting Next.js and Begin to play nice with deep linking into my app.  For example, I have a page called /map but going directly to that page renders the home page.  I'm not sure whose bug this is, Begin, Next.js, or mine, but I'd like it fixed.
  • I had to write my own custom dev server to get Begin's sandbox and Next.js to play nicely together locally.  With create-react-app, proxying a backend is already baked in. It was a bit annoying to have to stop work on my project in order to implement this myself.  Again, I think that Next.js assumes you have a server, and my use case doesn't seem to be their primary one.  It would be good if they put the serverless use case on an equal footing.  There is a serverless build target, but it didn't do what I wanted (or I don't understand how to leverage it, yet).
  • I don't like having to choose between React and HTML.  Next.js assumes you're only working with React components.  Yes, I want to use React components, but I also care a lot about the HTML that surrounds them.  Next.js lets you escape this and override the HTML, but it's not the default.

I'd use Next.js again.  It's closer to what I want than Gatsby or create-react-app.  I love how minimal it is (a few CLI calls, some components, and hooks). But I also think there's still room for another "React Distro" that does some things differently.  Maybe the type of web projects I want to build are the problem, but I really enjoy making web sites and apps at the same time.  I don't want to choose.  I want both.

Conclusion: Stay Safe and Go Find Some Owls

As we enter our tenth month of the pandemic, I wanted to make something for the current moment.  Christmas won't be the same this year: we won't be able to celebrate or visit our parents, siblings, or their families; I can't get together with any friends for a meal; and many of the usual traditions our family has are off the table.  I'm sad at all of it.

I can't fix any of this, but I wanted to do something to give some small bit of joy over the holidays.  While the pandemic forces us to avoid each other, we're still allowed to go outside, to drive in the country, to walk in the park or along the shoreline, and to look for Snowy Owls.

As I was finishing up the app's code, I noticed that a new owl had been spotted 15 minutes from our house.  My wife and I drove off into the falling snow in search of it, creeping along an old fence line stretched across a farmer's field.  It was really beautiful to be out, to be hopeful, and to be focused on what is yet to come.

Merry Christmas.