One Simple Agile Development Strategy Saved $300,000

Picture the following: You're in the checkout line at the grocery store waiting to buy your much-needed groceries. It's a long line—the kind that makes time seem to slow to a crawl. You look at the line of people in front of you and quickly come to the conclusion that your cashier is either brand new, or just slow on the till.

You look to your left—that line is just flying along, regardless of how many people enter it or how packed their carts are. That's how I felt when I made the decision to "switch lines" and join the team at Grailed. 

Grailed is a curated community marketplace for high fashion menswear. We connect fashion enthusiasts so they can buy and sell affordable name brand clothing on a fair and friendly marketplace. 

I've been a developer here for roughly 18 months. Presently, I'm the tech lead of the core engineering team. We handle infrastructure, testing and anything else related to the capability of the core platform. On a daily basis, we're dealing with the nitty gritty of it all—the tactical stuff that guarantees everything is running smoothly, all in an effort to bring the best possible experience to end users.

I was excited about joining the team at Grailed for a lot of reasons, but if I had to single out one above all others, it would be the excellent development culture. Up until that point, I had mostly experienced the culture of large firms that embraced large waterfall projects and more bureaucratic development processes.

The Dangers of Manual Processes in an Automated World

Even when you remove the bureaucracy and add in agile development, however, things weren't perfect when I came to Grailed. As is natural for a company of this size and scope, there were still a number of factors causing slowdowns, causing our proverbial checkout line to take far longer than it should have. Chief among them were our slow continuous integration (CI) times. This is a development practice that requires developers to integrate code into a shared repository multiple times per day. Every time one of those check-ins takes place, it's then verified by an automated build, allowing development teams to detect problems early and fix them before rolling out changes to end users.

CI is a huge benefit to teams like mine, which build internal applications to help our customer-facing teams deliver the high-quality shopping experience our customers have come to expect. But the process at Grailed was far too manual for its own good. When I first got here, it took 20 minutes (and sometimes longer) to integrate a single piece of deployed code. If you're not a developer, that may not seem like much, but it almost immediately creates a negative ripple effect that you cannot afford. 

Once we had committed our code, we'd tell everyone: "My tests are looking good, let me push my changes to production and everyone else can push their changes after me." It was still one change at a time, and it was naturally very slow. If one of the tests failed, suddenly we were looking at another 20+ minutes to rerun the suite. Since our test coverage was very good, I wanted to adopt a continuous deployment strategy (where deployments would happen automatically once the test suite passed for our main branch after code was merged), but we certainly didn't feel comfortable pursuing that when the feedback loop was that long. 

Sometimes I would spend hours sitting on what I thought was a finished piece of code, but I couldn't do anything with it because I would check the main engineering channel and find out that someone was in the middle of trying to deploy something else. Whenever I wanted to push new code, I would have to try again and again as I waited for an opportunity to jump into the deployment queue. That was the most frustrating part for me, and I think a lot of people felt that way.

I can't tell you how many times I would try to deploy, see that I couldn't do it at that exact moment, and then realize hours later that I'd forgotten about it entirely. It caused friction, it lacked efficiency, and it was a slower overall development cycle. It was a lot of things, but "agile" wasn't one of them.

All of this came to a head when one further addition to the testing pipeline caused the process slow to an excruciating 25 minutes. At that point, we knew we needed a new solution.

Executing Change, One Step at a Time

Because change is always an opportunity, I wanted to take a greenfield approach to whatever solution we chose. I was determined to make sure whatever new technology we embraced relieved as many of our pain points as possible. Our search for a new tool, however, was over fairly quickly because, after investigating a number of potential options, one rose above the rest: CodeShip.

I was trying to introduce Docker to our build process, given that it’s a great way to get repeatable and cacheable builds. In the process of looking at CI and Docker, CodeShip kept coming up again and again. All of the documentation I found during my initial searches for solutions to CI/CD problems was CodeShip documentation. Not only was it all very helpful, but it also led me to the conclusion that they were incredibly savvy from a technological perspective. If you're finding that all of the ideas and thought leadership you're interested in are coming from the same place, there must be a reason for that.

After my initial conversation with CodeShip, I was impressed with the speed at which I was able to get everything off the ground for testing. I had an initial set of Rails unit tests running within one day. Our setup didn't require that much massaging and it was fairly straightforward.

Once we decided to move forward, the setup process was a breeze. Using only CodeShip’s documentation, I managed to migrate everything over from our old CI system to CodeShip within just six business days.

We knew we were looking at a time reduction, but we never expected to see such dramatic results in such a short time. With CodeShip, the length of time spent on CI was reduced by an astounding 66%+. Prior to that, our test suite ran in 22–25 minutes. With CodeShip, that number is down to around seven minutes.

It was a change not only to our CI time, but to the entire process of how we ship code. Having such a short test cycle allowed us to confidently pursue a continuous deployment, and the feedback has been extremely positive. Someone can make a change, step away and take a break for a second, come back and see whether it's green or red. They know exactly whether they can push their PR or not. I can't tell you how frustrating it is for a developer to be sitting on a huge pile of code that’s waiting to be QA'd and not even know if it's going to happen this week or not. Going from that to a situation where feedback is basically instantaneous is such a relief. Nobody spends time thinking to themselves, "Is the build going to pass now? I've re-run this three times and it's been over an hour."

To put all this in context, let's assume our previous processes were roughly 20% less efficient than the ones we have now. When we consider that against our overhead, we estimate we have saved between $200,000 and $300,000 in costs associated with wasted time and energy from developers alone. 

Happy Developers Make Better Software

But perhaps the biggest benefit of all comes by way of office morale. I'm a developer, so I can speak to the fact that we tend to get annoyed very easily. Every little piece of friction is annoying to developers. We're always looking for the next thing to optimize or automate, and for a lot of people, this was it. Happy developers make better software. 

Even now, I have people who come up to me and say, "This was life-changing. It's so much nicer than the old system; I don't even have to think about it."

People who aren't developers don't realize how important that simple idea truly is. People aren't spending half of their workday thinking about deploying while they sit there, waiting their turn and spinning their wheels. Because the process is built to support the way they need to work, they're now able to devote the entirety of their attention to the critical task at hand:  making the product better in any and all ways they can.

Grailed was already the most agile organization I had worked at. But now, thanks to CodeShip, we're more agile than ever before. I feel like I'm still in that grocery store checkout line, but my mind is no longer wandering. I'm not checking the other lines anymore, wondering what might happen if I make the jump, because I already have all the actionable information I need. I know that at any given moment, I'm only seven minutes away from being out the door and on my way home.