100% Test Coverage and One-Click Deployments: How City Hive Is Living the Dream of CD

I once read a definition of continuous development (CD) that stuck with me. “With true CD, you can deploy to production with the click of a button.” I’ll add one annotation: You can deploy at the click of a button—and be certain everything will work as expected. That’s every developer’s dream. At City Hive, that’s become our reality.

City Hive is the gateway for small businesses to move into the digital world. Started in 2014, our first stage of the company focuses on wine and spirit stores, and we now have 1,000 customers across the United States. 

A big differentiator between big and small businesses these days is access to data. We take all the data we can glean from a business’s Point of Sale (POS)—their inventory, transactions, and customers—and we put it into a database and an online platform. We then supplement that with our own high-quality images, product descriptions, and rich categorization.

This serves as the foundation on top of which we build mobile apps, websites, engagement platform, loyalty programs, and event management—all those expert-level retail options in an out-of-the-box solution. A 60-year-old business owner doesn’t want to build and manage a website, they just want to be able to reach their customers where they are. We walk into a store that uses a 20-year-old PC that's connected to their POS. After 15 minutes, that owner has a mobile app and a website with their inventory, and everything syncs in near real-time. In a short period of time, we’ve brought in the technology to help them compete with the larger chains, while maintaining the personal touch that makes them unique. 

By focusing on wine and spirits businesses, we can perfect the experience for this industry and then expand to the next small business use case. That’s the long-term vision of how we grow City Hive into a massive company. 

Testing, It Shouldn’t Be That Complicated

We have a growing customer base and big plans for future growth, but like all startups, we’re trying to do this with a small and efficient team. We currently only have five developers (including yours truly). Back in 2015, we ran into a big problem. We had a robust platform, but no tests. Every few days, we would discover something that broke because of a commit from a few deployments ago. It felt like we were always playing catch-up. Finally we said, “That’s it, no more.” If we were to go any further, we needed better test coverage.

At the beginning of 2016, we made testing a priority: every new code that gets written gets tested; any existing code that we touch gets some initial testing suite around it. But once we started integrating testing into our development process, we wanted a better way to run the tests. This began our search for simple, affordable CI/CD solutions. 

We played around with multiple options, but one stood out: CloudBees CodeShip. We loved how easy it was to use. The setup was minimal: we didn’t need to change anything in our projects—not even a single file. All we had to do was connect our BitBucket account and it just ran. 

How We Got to 100% Test Coverage

Fast-forward to today: In the last three years, we’ve moved from zero to almost 100% test coverage. Our back-end has around 11,000 tests and each of our front-ends has around 2,000 tests.

To be fair, our team has grown from three to five people over that time. But we’ve developed the test coverage this quickly due to CodeShip’s parallel test pipelines. This allows us to split up the tests between different test pipelines. 

Without parallel testing, running our full suite of tests would take an hour. But when you’re developing a feature, you want timely feedback. You don’t want to make changes and come back an hour later only to discover something is broken. With CodeShip, we started running 16 tests in parallel using 4 servers and 4 processes on each of them, which shrunk testing time from an hour down to less than 10 minutes. 

But it’s not just that it’s fast. It’s that you never have to cut corners or second-guess yourself out of concern for speed. The goal is to never think of the time-cost associated with testing. It should always feel worth it. With CodeShip, you can afford to run that test every time.

A CI/CD-First Process, Made Automatic

With testing out of the way, the next challenge to overcome was the speed of manual deployments. During our first stage with CodeShip, I was still running our back-end environment from my console and would push the code to AWS Elastic Beanstalk. 

It still required me to have the computer open with the console to deploy to production. I wanted the true CD experience of deployments with the push of a button—my wild dream was to push to production from my phone.

Thanks to our deployment automation, it now takes 20 minutes to get a complete deployment into production. That’s down from an hour before. We start our work on the feature branch. If that goes to green, it gets deployed automatically to our staging environment. Then, if everything looks good, we merge the staging branch into production and that gets deployed once that code passes the tests. Everything is centrally automated and with the click of a button it gets deployed to production. We can deploy a few times a day per environment with just a small team.

We now have a CI/CD-first approach. There’s never a reason not to have a CI/CD process in place for any of our projects. CodeShip’s simplicity means there are almost zero barriers to having that.

You know when you work on a small feature, invest some time in it, but then you have to put it aside? A month later you try to deploy it and it doesn’t work. That’s because you were working locally and in the last month some version of some package has changed. When you have it on CodeShip, the platform becomes the snapshot for how you deploy. You don’t have to worry about versioning, because that’s taken care of. 

All you need to work on is the code. This is critical for us, because we use AWS Lambda function a lot, and since the scope of the work in Lambda is small, we won’t touch it for months. With CI/CD in place, the cost of fixing very small things is zero. There’s no additional cost because the deployment is automatic. You don’t have to waste a lot of time documenting everything. If the tests pass, that’s it. It’s deployed.

Today, we have nine different projects running in CodeShip with a complete CI/CD flow setup. The projects include our API backend, that's written in Ruby on Rails, AngularJS and Angular front-end projects, AWS Lambda functions written in Python, and even some PHP code. They are completely automated, end to end.

Ready to Grow

CodeShip allowed us to move from a zero-test, completely manual deployment, to 100% test coverage and deployment at the click of a button. The dream of true CD is here. A few times now, I’ve realized that dream as I’ve fixed and deployed some small things using BitBucket from my phone. It gets pushed, and if it passes, it’s deployed. If I need to deploy while I’m on the subway? No problem. 

I want to go back to the caveat I mentioned at the beginning: Not only do you need rapid deployment, but you also need confidence that the code will deploy correctly. At City Hive, we’ve already seen that as we grow, quick deployment cycles help improve customer support. That’s critical because our customers’ business depends on us. 

When one of our customers opens a ticket, we want to find the bug immediately, reassure the customer that we’ve located the problem, and then have that issue fixed within half an hour. We can’t tell our customer, “Please wait, because we can’t deploy during peak hours.”

With CodeShip, we can push the bar higher and say, “We’ll deploy right now,” because deploying through our CI/CD pipeline takes mere minutes. And we have the confidence that it will deploy correctly. 

This is the freedom we need to continue to grow City Hive. We’re helping our customers evolve by evolving ourselves.