When I was young and impressionable, I used to think that working on the latest technologies is the only criteria to call a project “technically interesting”. Over the years though, I have become old and impressionable! Working on the Go team has made me realize there are a lot of things that make a project technically challenging. I want to write about some of the reasons why I think Go is very challenging and why I loved working on the Go team.
Go is a packaged software and is not hosted. Being a CD server, most customers have traditionally wanted to use Go behind their corporate firewall. Lets say that currently there are 65 Go customers (in reality, this figure is much more). That essentially means we have 65 production environments!
Here are some of the implications of having so many production environments:
Backward compatibility
When you have that many customers, you realize not everyone upgrades at the same time. That means, when you write code, its not enough if it works well with “trunk”. It needs to be compatible with historical releases as well. For example, some customers may be on 2.1, while others on 2.3 and some others on the latest released version. All these customers should be able to upgrade to the new version without having to migrate to an intermediate version. This can be tricky and sometimes limiting.
Platform compatibility
Even though Go runs on a JVM and JVM promises WORA, it still gets affected by the ecosystem in which it runs. Lets take a look at the ecosystem variations:
- Windows: XP, Vista, 7, 2003, 2008 - all with different service packs
- Linux: Debian, CentOS, Fedora, Suse - all with multiple Kernel versions with different file systems.
- Mac: Mountain Lion & Snow Leopard
- Browsers: Chrome, Firefox, IE and Safari - all with different versions.
To ensure all these work fine is a lot of work. What is more tricky is to figure out the root cause of problems when things go wrong
When things go wrong
With so many different production environments that can be running on any combination of what you see above, it becomes very hard to figure what went wrong. We just can’t afford to say - “Take that one node off the load balancer. We will find out what went wrong.” Go is usually mission critical (the nature of a build and deploy server is that it is the core of your software. If its down, you can’t release the software you have built easily) and if Go is not working, software delivery gets hurt.
This means, you need to be very vigilant about logging, understand the ecosystem and figure out ways to analyse issues on a live server in non-intrusive ways.
Performance Tuning - Deployment Environment & Load Profiles
Different CPU, RAM, Lx caches, hard disk speeds, network speeds, file systems, operating systems, JDK versions, heap settings, JVM ergonomics tweaks - there are just so many things that can impact performance and we have little control over what customers choose for most of these.
Each customer can also belong to a different load profile. Number of pipelines, agents, concurrent users, SCM repositories, dependencies, test reports, feed crawling - each of this is a load dimension and can vary across customers. To be performant, some of these need more heap, some need more CPU whereas some other need faster I/O.
A typical large Go installation is essentially a highly concurrent, distributed work assignment system that needs to be optimized for throughput.
This is what makes it hard when performance issues surface. Along with this, since each customer installation is different, what works on one instance may not work on another.
A developer on Go ends up needing a very deep understanding of the JVM, OS and some core computer science skills to tune Go’s performance.
3rd Party integration
Given that Go is a CD tool, we need to integrate with the tools in the build and deployment ecosystem. SCM, builder, test, analysis, configuration management - there are many different kinds of tools out there that we need to figure out a way to integrate and support. Coming up with a plugin infrastructure to implement this is something that is pretty challenging.
While none of these have any direct relation to how interesting the tech stack is (it is), it definitely is highly challenging. To keep in mind all these while writing code and continuously delivering with a high energy, highly opinionated and a fun team kind of sealed the deal for me. That’s why I loved working on the Go team.
How about you? Would you be interested in working on a team like this?
