Goodbye Sleepless Nights: De-Risking Deployments with Feature Flags

There are many things in life that will always be unpredictable. Unfortunately for us, development falls in that category. Sure, from the planning stage to the final patches, software engineers do everything they can to offer control. The reality is, however, that rolling out new features can be incredibly nerve-wracking. Sometimes, making the slightest change in production has unintended consequences. Other times, the features work exactly as you expected, but they negatively change the customer experience in unintended ways.

Despite this ever-present potential for catastrophe, startups have to find a way to improve their offerings. Just as with organisms in nature, any business that fails to adapt to the changing environment may perish. Thriving requires constant innovation, including a consistent pattern of enhancing your products to match customer needs. The most successful companies find a way to balance this constant innovation while reducing the risks inherent in any change.

Built for a Furry Friend in Need

PetDesk was founded on a mission of improving pet health by helping pet parents stay current and pet care providers stay connected. We support local pet care businesses with industry-leading software that simplifies and streamlines communication. This means less time spent on administrative tasks and more time spent doing the things they love: improving the health and wellness of pets.

To date, more than 1,700 pet care facilities agree with our vision. Veterinarian clinics, animal hospitals and pet grooming facilities all over North America speak highly of our tools. Our online appointment feature saves customer and receptionist time by avoiding a needed phone call. The reminders and confirmation functions help providers avoid wasting time on missed appointments. Loyalty programs, online reviews and targeted messages are all revenue-generating marketing instruments that push service providers to new heights of success.

But it isn't just the companies that love our services. Our Pet Health Mobile App runs on more than one million devices. Our app helps users to schedule regular appointments, refill medications and get access to their pet's records. Apple users rank us at 4.8/5 with more than 95,000 votes. Best of all, we’re just getting started.

Success Breeds Change, But Change Comes with Risk

Every startup company begins by building things that don’t scale. Eventually, that catches up with you. At PetDesk, that was our reality last year. One area where we had reached a tipping point was with our SMS provider. 

Like most startups, we started with a provider for our simple, fledgling needs. However, as we began to grow, we started to experience problems with scale. 

By that time, we were sending millions of text messages a month. Included in those messages were reminders of appointments and other critical scheduling events. We knew that our current SMS provider worked for our current volume. However, we also knew that our future growth plans meant it was time to find a more comprehensive solution. The problem was that any transition process carried the threat of some form of failure.

A sudden outage could mean thousands of appointments missed. Each of those unfulfilled sessions means missed revenue for animal service providers and potential health risks for the pets themselves. One of the reasons we started this company was because we know pets can’t communicate when they need to see a doctor. As a result, they often depend on the regularly scheduled appointments of concerned pet parents.

For our company, outages could also impact the perception that PetDesk is reliable. We like to believe that we are exceptional, but if we start missing email notifications, our current clients could look elsewhere. We needed to secure our ability to expand our services while minimizing the risk of catastrophic failure. After some research, we decided that using feature flags was the best mitigating tactic.

Well, That Didn't Work

We all want to believe that we can do it right the first time. But, sometimes, you win by realizing when you've made a mistake. Once we decided to implement feature flags, our mistake was attempting to do everything in-house. At that point, we traditionally managed things of this nature using our own developers. Feature flags seemed like something we could handle ourselves.

Almost immediately, we realized the time commitment involved. As a small company, we are naturally focused on constant product improvements. It felt problematic to continually draw development time away from things that mattered directly to our customers. We knew feature flags would allow us to toggle functions on and off as needed. However, given the amount of labor it would require, it became apparent that we needed outside help.

Once we decided to find an external resource for feature flags, we considered a number of options. Ultimately we decided to use CloudBees Rollout.

In our research, we found Rollout to be incredibly feature-rich. At the time, we were primarily looking for the ability to toggle SMS providers. However, looking to the future, we knew feature flags could help us elsewhere. 

Rollout provided tools to conduct a number of tests and experiments, and allowed detailed, targeted function toggling. In addition, Rollout offered fabulous support. They worked closely with our team to help us technically evaluate their solution. They also had impressive libraries and APIs. But one of the biggest factors was their track record. Seeing they had 99.9% uptime made us feel confident we could update functions regularly with an acceptable margin for risk. We knew we'd be working with a reliable partner. 

Mitigating Our Deployment Risk

Anyone who has endured sleepless nights knows the importance of resting well. Making any change to your production environment can be dangerous territory, but Rollout helped us sleep a bit easier at night.

We were initially afraid of doing a hard cutoff date, where we suddenly change over our SMS provider and hope for the best. Using Rollout, we were able to take a more measured approach to deployments.

Because we could toggle between providers and features, we took our time and ran a series of deployment tests. As part of these tests, we experimented with some novel functions offered by different providers. If we had a day when we were short-staffed, it was easy to turn off the potentially problematic pilot functions. This was especially valuable given this was around the holidays. 

Rollout enabled us to designate specific testing experiments, like deploying a feature to send 50,000 text messages during a specific time of the day. We could run the test for a few hours, and then toggle the function off and examine the results. It was an incredible comfort knowing we could conduct such a short-term test without the risk of full exposure. Suddenly, the risks involved with updating our application services seemed manageable. We were able to switch over our SMS provider and we're thrilled that we're now in a better position to scale. 

Sometimes, Change Is Good

At this point, we have only used Rollout to enhance our SMS integration. PetDesk, however, has many other areas where feature flags could help us. As a result of the successful SMS enhancements, we can’t wait to apply their solution to other experiments. We are very excited to begin beta testing and developing new tools for our customers.

We have already envisioned a continual process of a number of internal teams regularly using Rollout to test new potential features. This more calculated, targeted approach allows us to innovate without losing the integrity of the entire application. If a new feature does work, this could also allow us to determine pricing and server demand before a global release. Perhaps most importantly, the beta testing process maximizes our opportunity to incorporate customer feedback into our release process.

We recognize that risk will always be a factor. However, Rollout allows us to effectively mitigate risk without losing our ability to innovate. For startups, innovation is our lifeblood. 

PetDesk has lofty ambitions. Our goal as a company is to add 10 million more years to the lives of our pets. With a little innovation and the security of Rollout and feature flags, we get a little closer each day.