How We Transitioned Our IT to CI/CD for Greater Agility and Faster Testing

If you’re in SaaS, IT is the core of your business. But IT isn’t always in the limelight. For many of us, it’s what we do internally to support the organization that drives us forward. As Director of IT Software Engineering at a large nonprofit, I know this well.

SIL International is a nonprofit organization committed to serving language communities worldwide as they build capacity for sustainable language development. There are around 7,000 languages used in the world today. Some of these have no written form, meaning many community histories are passed down generation to generation through oral storytelling. 

But if fewer people speak a minority language with each generation, that community—and humanity as a whole—runs the risk of losing that heritage. SIL exists to help preserve these minority languages through literacy, multilingual education, and minority language promotion.

Supporting the amazing work of our colleagues in 100 countries, I lead the global IT software development team here in the U.S. Our team, which is primarily internal, focuses on custom software, such as integrating products or developing a new, unique tool. For example, the biggest project we’re maintaining now is our corporate identity management system, a single sign-on solution. We developed it because we couldn’t find a solution on the market that met both our needs and budget.

SIL is not primarily a tech company. But, my team supports our mission of protecting community heritage and improving the lives of people around the world by making everything in IT as seamless as possible, so our folks out on the field can focus on their jobs. When you’re a team of eight people supporting 4,800 of your global colleagues, that's easier said than done.

Seeking the Combo of Quick and Quality

I joined SIL in 2013, when the global IT team was looking for someone who could innovate and not get bogged down by our IT structure. 

In the past, my team had been too loose with its processes. For example, developers had direct access to production servers. There was little testing and things would often break, which added to greater frustration for both our end users and developers. In general, access controls were loose and processes overly manual. 

But in reaction, the pendulum swung too far the other way: An extremely rigid process was introduced where any changes in code had to be tested through multiple tiers.

Overcompensating for the previous looseness created new problems. Developers had to build RPMs for their applications and attach them to JIRA tickets for the operations team to deploy. But development knew next to nothing about the servers their code ran on. 

This made for a tedious and painful build and deployment process since it depended on another team—meaning we couldn’t fix our own issues. There were few guarantees that the environments were identical through these tiers, and even if they were, the testing amounted to a manual smoke test. Nothing was comprehensive. But all these controls did not guarantee improvement in quality. We still wrote and deployed bugs.

All this moved us toward embracing a DevOps mentality, which I see as being ultimately about taking back responsibility. It’s a case the DevOps movement talks about a lot: When you give a developer a pager, they start writing code that doesn’t crash at night. They don’t want to get those calls. When it’s someone else’s responsibility, you just don’t worry about it the same way or with the same sense of urgency and ownership. 

The way our process worked, it was always someone else’s responsibility, because the developers depended on the operations team to deploy. I knew a move to automated testing was about enabling developers so they were responsible for the deployment process again, while still ensuring quality. 

Building the Case for CI, and on to CD

We knew we had to move to a continuous integration (CI) platform, but there were several options in the market. There was CodeShip, Travis CI and CircleCI, and for our purposes they all seemed fairly similar back in 2013. But there was a major difference in their approach: CodeShip came with a free tier of private repositories. That was a significant motivation for choosing CodeShip. When I first joined this team, all our code was closed-source, hosted on an internal version control system. We moved to Bitbucket for its free private hosting and with CodeShip we’d get free private builds.

As a non-profit with a tight budget, this was incredibly helpful for us. We could prove our use case without breaking the bank. Once we could demonstrate the value of automated testing and deployment to our leadership, it would be an easy sell for the paid tier. 

We started a new project with CodeShip and moved from Subversion to Git. We wanted to incorporate not just automated tests, but CI and hopefully someday CD. 

Once we implemented CodeShip, we started to see changes in how we operated. Since there was more automation and responsibility for a developer’s own code, we now had automated, repeatable processes to dramatically reduce testing and deployment. The new system meant better quality and faster response times to the ever-changing needs of our business. If we have to deploy a fix, it now takes five minutes. Before, it took half a day. 

To the larger SIL organization, the value in CodeShip became clear pretty early on and it was obvious what we could do with a CodeShip Pro account.

Up Next, a Move to the Cloud

This is when we began our DevOps journey of bridging our application development and operations process together by transitioning to the cloud with AWS. Moving to AWS enabled us to take advantage of CD as well. Once we had CI/CD in place for a couple of applications, it was a clear win. We quickly started migrating all our apps to CodeShip and AWS.

CodeShip started us on the CI process, and even some CD. But when CodeShip became truly significant for us is when they began to support Docker. We started using Docker in 2015 because it allowed us to adopt hosted CI for a couple of our projects that had dependencies not supported in CodeShip Basic. 

This was in the very early days of Docker, before it became mainstream. CodeShip was on the forefront of Docker support in a CI environment. This was music to our ears, because Docker had become essential to us. At the same time, CodeShip was developing its CodeShip Pro service. While it was only in limited beta, the CodeShip team knew we needed Docker support and gave us access to it about a year before opening it up for general use. We had a lot of opportunity to provide input on how it works.

Docker on CodeShip gave us complete freedom to adopt CI in all sorts of complex models and approaches. To better facilitate deployments of Dockerized apps to Amazon’s ECS service, we wrote a script called ecs-deploy, which we then wrote about and published on CodeShip’s blog. That repository now has almost 1,200 stars on GitHub. It’s pretty exciting to look at the adoption of that script. Of its 34 contributors, only two are from SIL.

There was a point at the beginning of this process when the idea of CI/CD, automation, and working from the cloud all seemed part of a nebulous future, far off in the distance for an organization like SIL. Today, we consider it the absolute bare minimum standard that all our custom apps are automatically built, tested, and deployed through CodeShip. It’s now a core part of how our team operates. 

In fact, thanks to Docker and CodeShip, we also now have a Day 1 Deployment practice on our team where new members are expected to deploy to production on their first day on the team. We welcomed a new team member this year and he was successful with this goal. This would be a ridiculous concept in a less automated environment.

Support Every Step of the Way

It’s now been five years since we started on this journey, and throughout, CodeShip has kept adding value for us. It starts with great customer support. Any time we need advice or have questions, they are there for us, whether on a call or through chat.

They’re extremely friendly. When we first signed up, I got the standard welcome email from their CEO, Moritz Plassnig, saying, “If you have questions, let me know.” I thought: “Actually, I do have a couple questions. Let’s test this out.” So I replied, and Mo took the time to respond personally. And it wasn’t just a basic email. He noticed my name, and said, “Shipley using CodeShip, that’s great. I love your name.”

We have become more than just clients of CodeShip. Before introducing new features, they ask us if we would like to be part of a test. They then have us sit down with a designer and talk through our experience. I’ve even written several articles for the CodeShip blog. 

CodeShip recognizes that, as a nonprofit, we don’t have a huge budget and the organization has to be careful how it balances spending on IT. CodeShip has helped us keep within that budget.

Throughout the last five years, CodeShip has been there at every step as a product, partner and solution. This all started as a long shot vision for us—almost a pipe dream. But now, agility is just second nature to our team. 

Enabling Our Greater Team to Do Their Best Work

CodeShip is now an absolutely critical pipeline service for us, but more than that, it opened our eyes to a world of possibilities. Today, we’re considering what else we can automate. We’ve shown our leadership what automated CI/CD can do for us and we now perform nearly all our production deployments during business hours, which makes our developers much happier and allows for faster feedback loops with our users. 

As a small team, we can now deliver more quality solutions at a faster pace than we could have before. If we didn’t have all that automation, we couldn’t support half of what we do today. All of this helps us better support our users, the people in the field doing such important work with communities and keeping local languages alive. My team doesn’t have a spotlight on us—and we don’t need one. Because with a mission as grand as ours, we all have our roles to play. And we’re proud to play ours.