From Shipping to Scaling: Mastering Developer Experience Through Observability

When you're a startup, your initial focus is shipping. It doesn't matter if you're working on a physical product or a service. You build your prototype, launch, and begin to grow your customer base. If your business takes off and you go from surviving to thriving, you shift your attention to scaling. And that's where many companies falter. It takes one mindset to get off the ground, and another to grow. Companies that don't pivot are doomed to fail, but how do you change gears if you don't have the right tools?

Goji Investments launched in 2016. Our platform democratises access to real estate, business lending, renewables, and other alternative investments. It allows asset managers and investment firms to offer financial vehicles to global private investors seeking stability and better yields by looking beyond traditional equity markets. Our scalable modular platform is available as a white-label solution and also integrates into existing platforms using our API. 

Creating an Excellent Developer Experience

In the four years since we launched, Goji Investments' developers have worked hard to help us expand our product offering and bring in new customers. In return, we've done everything to keep them happy, and that means supplying them with the right tools to help them transition from building our platform to scaling it. 

Our DevOps pipeline is a complex machine with many moving parts. We are continuously refining it and adding or subtracting tools based on developer feedback and our changing deliverables. 

We realised early on that the best way to ensure a superb end-user experience is to provide an outstanding developer experience. We help our people stay focused on performing at the highest level by automating repetitive low-level tasks, thus freeing them to work on features that add value to the company. To this end, we have fine-tuned our technology stack to give our eight-member developer team everything it needs to succeed. 

Automating Deployments Wasn't Enough

Our applications are written in a modern Java stack, and we are using Dropwizard as our framework. Our developers typically write code and then push it to GitHub for review. If it's approved, the new code is merged into master, which automatically triggers a build pipeline using ThoughtWorks GoCD. We then run automated unit, integration, and acceptance tests and our developers also have the option to run manual tests to encourage exploratory testing.

From there, we can promote new code into our pre-production environment, which mirrors our current production setup. Anything not covered in prior testing that makes its way into our production environment can be easily rolled back using the blue-green deployment functionality of HashiCorp Nomad, our container orchestration platform. We're a small team, and our codebase is relatively compact, so we don't need the complexity of Kubernetes GKE or EKS to deploy and manage our applications.

One of the benefits of our technology stack is extensive automation. Although our deployment workflow was seamless, things were not as rosy when it came to monitoring. We were using New Relic to monitor our infrastructure. When an application was eating up too many CPU cycles, or an EC2 instance crashed and rebooted, it would trigger an alert. But if we wanted to troubleshoot slow response times, latency errors, and bottlenecks at the service level, we had to comb through logs manually, and that was a waste of time and human resources.  

End-to-End Traceability

The last thing you want to do with your day is analyse log files. You want to build the next killer feature, but if you have to dive into the weeds, you want to do it as quickly and as painlessly as possible. That's where observability and application performance monitoring (APM) comes in. APM incorporates infrastructure monitoring, but also provides service-level metrics, including information about database calls and end-user activity. It allows us to locate bottlenecks and other performance issues without having to divert our developers from their primary duties. 

I'd used APM software at my previous position, and when it came to time to adopt a platform at Goji Investments, I went with what I knew. Experience had shown me that Instana is a zero-touch solution that does everything we need straight out of the box. When you run 20 different microservices that cover everything from taxes, payments, and customer notification—and when you're scaling to add even more—you need to optimise every element of your DevOps pipeline. Instana was the final piece in our puzzle, and it allowed us to automate one of the most cumbersome parts of the development cycle. 

Instana gives us full traceability from our front-end to our back-end. We can track user actions on our website and trace subsequent calls across multiple services. It automates alerts and can send an alarm to our developers' Slack channel when it encounters an issue with CPU, container memory usage, or the JVM heap size.

Instana also integrates with Humio, a real-time logging solution we recently adopted. When we spot an issue in Humio, we can jump into Instana and go directly to the Docker container or host that generated the error. We can also view AWS metrics in Instana and no longer have to pull information from the AWS console. Consolidating all of these functions saves time and simplifies troubleshooting. 

Finding Our Bottlenecks

When we're testing a new build or functionality, we can use Instana to view dozens of metrics in real-time, and that is a complete game-changer. Recently, we developed a feature that distributes dividends to investors and, prior to release we wanted to run performance tests to ensure it would scale to meet the demands of our growing client base. Initially, we ran a performance test of several hundred thousands of payments, and it took over 20 hours.

During the performance tests as we iterated on our code, we found ourselves getting halfway through the process in five or ten minutes, hitting a bottleneck, and then taking 10 to 12 hours to complete the remainder.

With Instana, we were able to see where the problem occurred. The instant Java profiling feature, AutoProfile™, showed us hotspots where the code was slow, and we determined that our database queries were the bottleneck. Once we knew where to look, we went back, fixed the code, and redeployed the application until we got it right. 

Instana allowed us to compare test runs. We could pull up a previous trial run and then compare its timeframe with the current version of the code. Using this iterative process, we streamlined the code, eliminated all the bottlenecks, and reduced the dividend processing time from 20 hours to less than 1 hour.

If we didn’t have Instana, it would have easily taken us four to five times longer to get this process right. 

Giving Our Developer the Tools to Shine

If there's one thing developers love, it's rising to the occasion. The best way to keep them happy is to give them the tools to contribute to your bottom line. Treat them like internal customers by relieving their pain points. Automate repetitive tasks and eliminate tedious, time-consuming work like going through error logs. Reward them with creative work and give them every opportunity to shine. 

Instana is helping Goji Investments streamline the DevOps process by giving us visibility into the inner workings of our applications and microservices. We can trace calls and database queries, and we can monitor hosts, Docker containers, and Java Virtual Machines in real time. All of this accelerates testing and deployment. We can do more and scale faster and attract the world's most talented developers. After all, the smartest people want to work for companies that challenge them and also give them the tools to shine.