Telescope 2.0

by David Humphrey

In a few hours I'll be meeting with the team to ship another major release (2.0) of the Telescope project.  It's been a lot of work, full of late night debugging sessions over Slack and Teams, major successes as we've landed big features, and plenty of disappointments as we've experimented and refactored code.

Before we do this final release, I wanted to take a few minutes to look back over a year of development, and specifically at what we accomplished in Telescope during 2021.

I wrote in January about our plans to create a 2.0 release.  At the time I highlighted some of the areas I thought needed focus.  Today it's interesting to consider how we did on each of the various goals I had:

Goal Grade Notes
Migrate User data and Feed storage out of Redis and into a better database A- We have all the right pieces in place, largely thanks to the tireless work of @chrispinkney and @manekenpix, but there's still a bit more to do for it to be usable in production. I'm hopeful that a future dot release will sovle these.
Fix our Redis data corruption when we lose power on production D We researched, but never implemented a solution
Migrate away from the Wiki Feed List as the source of truth, and use the Feeds from the database C We have the beginnings of a parser microservice that works with the Users service, but it has no tests, and isn't production ready yet.
Modify our Authentication/Authorization system to allow token based authentication A We've shipped this now, and authentication/authorization works on staging, production, locally and in Vercel. We have a bit more work to do before it can be an A+, but it's basically done.
Figure out how to integrate user GitHub activity and contributions (PRs) B- We have the foundations for this to be an amazing part of Telescope 3.0, but the scope of it was too big for 2.0
Prototype having Twitch Streams, YouTube Live, and other "Telescope Podcast" experiences. I think this idea is great, but I'm still waiting to see if the students agree. Not everyone is comfortable to stream their work. Maybe we'll do it, maybe not. N/A We never looked into this. I don't think the group shared my interest in this feature, so I didn't push it. Maybe in 3.0?
Make the app installable as a Progressive Web App (PWA) on mobile A+ This was finally accomplished late in the 2.0 cycle thanks to the tireless work of @tonyvugithub
Improve the Admin dashboard(s) experience N/A We never started on this, and it's something we'll have to consider in 3.0
Better Admin tools for content (user, feed, post) management N/A Same issue.
Analytics D We did a bit of reserach and discussion on this, but never really got started on a solution. I think having the GitHub data will help make this a more compelling feature in 3.0
Build out our error/log reporting that we started with Kibana B+ We ended-up coming at this problem from a different direction, using Portainer. I think solving this in a later release, in combination with an enhanced Elasticsearch and Kibana analytics feature would be great
Improve UX for adding new feeds (e.g., automatic RSS feed discovery from blog URL) A+ We made a ton of progress here thanks in large part to the efforts of a small team of front-end wizards: @PedroFonsecaDEV, @DukeManh, and @Meneguini with back-end support from @tonyvugithub for feed discovery.
Improve accessibility A We spent quite a bit of time improving this, led mainly by @yuanLeeMidori
Improve developer experience A+ There was a tremendous amount of work done on DX, much of it thanks to @birtony, @HyperTHD, and @chrispinkney.
More and better automated tests A+ Our tests have never been better.
Improve docs B+ We didn't put as much effort into this as we could have collectively, but the work that was done was usually thanks to @yuanLeeMidori
Refactor and Simplify backend code. Split code into separate pieces and properly Dockerize: REST API, deployment server, feed processer, frontend A- We spent so much time on this, refactoring the backend monolith into nearly a dozen microservices. The work is nearly complete, but there is more we need to do. This wouldn't have happened without days of work by @HyperTHD, @chrispinkney, @tonyvugithub, @manekenpix, @raygervais, @c3h0, and @izhuravlev.
Continue building on the improvements we've started with Search (e.g., autocomplete for author names) B- Search is one of the areas that is so exciting about Telescope. Not only can we see what happened in the past few days, but also we can go back and search for things in the past. We spent quite a bit of time on search related UI updates, but there's still a lot left to do. Much of the improvements we do have are the work of @rjayroso, @izhuravlev and @huynguyez.
Swag? We have talked about having shirts, mugs, and stickers in the past, but it hasn't happened yet. Maybe for the 2.0? C We've had a hard time executing on this for two years. I think a lot of people want to see it happen, but it's very different work from developing the project. @izhuravlev has shouldered the bulk of this work.

Those were my goals in January, but lots of things changed.  Software development requires you to adjust schedules and feature lists to accommodate the realities of implementation.  As a result, our 2.0 is full of many things I didn't anticipate, which is great: it means we found ways to incorporate the interests of the team as a whole.

I spent some time today reading through the PRs we merged in 2021, and took notes on the major areas of interest and emphasis I saw.  Here's some of what I saw:

  • Snapshot, end-to-end, and unit testing
  • Continuous Integration (CI) and Continuous Deployment (CD)
  • SEO
  • Firebase
  • SAML based Authentication, JWT Authorization, and user Sign-up Flows
  • Security
  • UI Design
  • Logo, CSS, and Theming
  • Improved Accessibility and User Experience
  • Microservices
  • Elasticsearch
  • Redis
  • Full port of front-end from GatsbyJS to next.js
  • TypeScript rewrite
  • Dependency Updates and Maintenance, both manual and automated (Dependabot)
  • Paying off Technical Debt
  • Progressive Web App (PWA) and Mobile UI Support
  • Docker
  • Automation and Tooling fixes, updates, and improvements
  • React
  • nginx, Traefik, and routing, caching, certificate management
  • Documentation
  • Improved Developer Experience, including fixes for cross-platform differences
  • Bug fixes

I also dug into our community and project stats a bit:

  • 167 contributors, including students in the course, other current Seneca students not taking the course, and also some talented alumni and external contributors.  80 developers have added new code in the past year.  Our top contributors list has been updated with both new and familiar faces: @manekenpix, @c3ho, @PedroFonsecaDEV, @cindyledev, @HyperTHD, @birtony, and @tonyvugithub (not to mention Dependabot).
  • This community has done a lot of work together.  Since the start of this year: 225 Issues were created and 301 closed by over 400 Pull Requests.
  • During Telescope's history, 97K lines of code were added and 66K lines removed across 1K files, with 50% of that work happening in the last year.
  • As we ship 2.0, there are 20K lines of code, with the majority in JavaScript (67%) and TypeScript (25%).

In addition to these code and community metrics, Josue and I also looked at the state of the data in Telescope, and it's impressive to consider as well:

  • Over 700 people have written open source blogs that we've indexed.  The number is probably closer to 1,000 if we include blogs that no longer work (e.g., have been removed after a decade).
  • We currently have more than 6,000 individual posts indexed.  I don't have the total word count, but I'd love to know what it is.  It's crazy how much we've written about our experiences being involved in open source.  It's also incredible to consider that RSS feeds usually don't include everything by an author (only the most recent work) so the total number of posts we've had is larger still.
  • Our oldest indexed post is from October 2006 by Ben Hearsum.  Fifteen years!  I had a great conversation with Ben on Twitter this week about his post, and was struck by how valuable the open source student experience really is: Ben turned his open source course work and passion into a successful 14 year career as a Release Engineer with the Mozilla Corporation.  I updated Firefox to version 88.0 today using code that he wrote. I couldn't be more proud of him and his work, and I'm still just as excited about helping our students achieve these outcomes using open source contribution.
  • Our youngest post, at the time of writing, is by Tony Vu, and shows a lot of the same spirit Ben had 15 years ago: "As I am still waiting for reviews on my PWA’s PR, I decided to take on  an issue in an area that I have never touched. After looking at the list  of current issues, I assigned myself to an issue helping to add more packages files to Dependabot config." Working on open source at Seneca means looking around for ways to contribute, jumping into new areas of code, working in community, and writing about our successes and failures.

Telescope 2.0 moves us even closer to having the platform I've always wanted: one that can help gather and reflect the collective efforts of the Seneca community as it works on open source.  We still have more to do, and I'm already thinking about the areas I want to work on in the spring and fall.  Until then, I'm looking forward to reading the blog posts of my current team as they wind down after their final release.  Congratulations to the team on what you've accomplished.

Let me end with a note of public thanks to Josue (@manekenpix) for his tremendous efforts supporting the team and Telescope this term.  Josue reviewed most of the PRs I made, for which I'm extremely grateful, spent hundreds of hours mentoring the current students debug and write fixes, attended meetings, and was always on-call to support our deployments.  There would be no Telescope 2.0 without Josue, and I'm extremely grateful for his generosity and commitment.