Making Music: How Ableton Uses Jenkins to Build a Modern CI Pipeline for C++ Coders

Modern music is driven by software. Laptops have replaced racks of synthesizer modules, sound processors, amplifiers, stompboxes, and turntables on concert stages. Computer monitors sit atop mixing consoles in studios, and hard drives have supplanted tape reels as storage. Whether you're playing live or mixing a track, a software glitch or application crash can be catastrophic.

As a member of Ableton's Developer Tools team, I help build the pipelines that allow our software engineers to provide stable error-free releases of our products, including Live, a leading digital audio workstation (DAW). 

Twenty-One Years of Ableton Live

Ableton Live dates back to 2001. It combines multitrack recording and sequencing with virtual instruments and sound manipulation tools that can be used to assemble a track in the studio or in a live performance setting. Musicians around the world compose, record, mix, and edit audio and MIDI data in Ableton Live. Our intuitive arrangement and session views provide a blank canvas for musicians looking to create their next masterpiece. 

There's a lot of competition in this market. Apple bundles GarageBand with every Mac and iOS device. At the other end of the spectrum, professional recording studios have been using Avid's Pro Tools to record, mix, and master audio for over 31 years.

Our other commercial and open-source competitors include Apple's professional-grade Logic Pro X, FL Studio, and Audacity. 

Live is a mature product in a mature market. But we are continually adding new features, integrating new hardware, and making sure our product is compatible with the current and legacy versions of multiple operating systems.

We are now at version 10 of Ableton Live. Every major upgrade and minor maintenance release comes with its share of challenges. That's why we go through a rigorous development and testing cycle to ensure that every time we refresh our application, everything works from day one. This process can be extremely complicated because Live contains millions of lines of C++ code and a sizeable legacy codebase. Compared to newer languages such as Java and Python, C++ can be tricky to develop and debug. 

Learning About Application Testing at Spotify

I came to Ableton from Spotify, where I worked on their Android app, third-party SDKs, and their desktop client. But previously, I've worked with audio apps for years, targeting end users rather than developers. I started here working on Link, Ableton's open-source protocol that allows devices to sync to a tempo wirelessly. 

I didn't have much experience working on backend services when I started here, but I understood how the development cycle worked. Back at Spotify, we used the BVT—build verification test—method to accelerate the release of our desktop player. We'd select perhapsa dozen of the most important tests and run them on every commit and pull request.

On merges to master, we'd run a larger number of acceptance tests, which took longer. These builds rarely failed, but when they did, we knew that the new version was indeed broken, and we could go back and repair our code before merging again.

Migrating to Jenkins Pipelines

We don't have the luxury of doing selective testing with Ableton Live. There are too many variables at play, and so every commit must be fully tested. This puts a tremendous strain on our developers and our CI pipeline. Our CI infrastructure was not designed with flexibility in mind, and we are still in the process of migrating to Jenkins 2 and Groovy pipelines.

That's why Ableton was stuck with an older version of Jenkins. We have been using an outdated but stable build that didn't offer many of the conveniences of the latest releases. Blue Ocean and pipelines came along at the perfect time for us, which was right when we were considering how to replace our older Jenkins installation. 

Finding the Right CI Pipeline for a C++ Environment

Given the difficulty of maintaining a modern CI pipeline in a C++ environment, Ableton considered a third-party solution. Why not farm out testing to a third-party service like Travis or AppVeyor? Ableton may have a big footprint in the music industry, but we're not a big company. Outsourcing made sense from a staffing and a budgetary perspective. The monthly service fees for these solutions are a lot lower than the salary of a full-time engineer.

We tried out both of these solution providers. Their platforms were adequate, but the tools they provided weren't powerful or flexible enough to do what we needed. Another problem was Mac OS X support. Although Travis supports Mac, it would be difficult to test our GUI using Cucumber, since this requires a graphical login. That was problematic as over half of our users have Macs. That meant seeking out another solution. 

We went back and looked at the major players in the CI arena. There were 12 or so contenders, but we narrowed it down to three: Bamboo from Atlassian, JetBrain's TeamCity, and Jenkins.

We ruled out Bamboo and almost went with TeamCity, but we walked away because it didn't meet all of our needs. To start, it didn't allow us to define a build DSL alongside the source code in git. TeamCity has since added that feature, but it is still experimental and the API was not stable enough for our purposes. 

A more serious issue was capacity. Our old Jenkins instance has over 70 build nodes, and we want to at double or triple that number. We knew from talking to colleagues at other companies that TeamCity starts to stumble as you approach 200 nodes, and this might have limited our future growth.

Going with Jenkins Again

This meant that Jenkins truly was the best solution for our team, but we had to revise our approach to the platform. 

The first thing we did was eliminate the use of custom plugins, and to try to minimize the number of plugins we relied on in general. We had painted ourselves into a corner with our previous Jenkins workflow because we couldn't upgrade our CI DevOps environment without risking compatibility issues with these bespoke components.

Next, our team wrote custom middleware that sits between GitHub and Jenkins. It handles creating Jenkins jobs when a pull request is opened, and triggering builds for certain GitHub events. We unfortunately couldn't use the GitHub plugin for this, because for some of our projects have very long build times and automatically triggering builds for every push event would overload our build nodes.

On smaller projects, we can use conventional CI pipelines where everything is automated. However, we are only halfway there. We have not deprecated our old Jenkins environment to build our flagship product, Live, and are still using AppVeyor/Travis for some other projects.

Jenkins Old and New

We have started figuring out how to move the development and testing of Live to our new Jenkins pipeline, and this will allow us to add more features to the application while lowering our maintenance burden.

In our old Jenkins environment, we have shell scripts which the developers must run to create and delete Jenkins jobs, and no integration with GitHub whatsoever. So developers must manually post links to the build pages from their pull requests, which is tedious and labor-intensive. All that time adds up when you consider that a full suite of tests can take several hours to run. Asking a bot to trigger a build and to set a status instead of doing it manually will accelerate our DevOps cycle. 

Our biggest challenge ahead is migrating our acceptance test suite to the new Jenkins pipeline paradigm. Our acceptance test suite takes several hours to run because each test must launch Live, simulate user actions, and then check whether everything went as expected.

We plan to improve the parallelization of these tests, which are run on virtual machines optimized for specific tasks. I am also hoping to expand our test matrix, so it will better ensure that our software runs well on legacy operating systems. Musicians often use older OS versions to remain compatible with older hardware and software, so we try to maintain backward compatibility for as long as possible. 

Moving to the latest Jenkins will also increase our developers' confidence when making new releases. It will make it easier for them to understand unit test and build failures, which is a common developer headache since the compiler output can be quite large. 

The improved ability to see whether things are broken will give our DevOps team the confidence to try adding new and experimental elements to the code base, thus ensuring robust new features in future versions of Ableton Live.  

The Future of Music

Ableton has entered our third decade of operations, and Live continues to serve many of the world's musicians as their production and performance tool of choice. Our engineers work tirelessly to ensure that we ship stable versions of our flagship product.

We recognize that a software glitch can be catastrophic. A computer crash can spell the end of a performance or a DJ set. It can cause a band to miss recording that perfect take or epic jam session. 

Like the violin- and piano-builders of the past, Ableton is making musical instruments. We use C++ instead of hammers, chisels, and saw, and Jenkins is our workbench.