Want Shorter Release Cycles? See How Weather.com Uses Sauce Labs to Release More, Faster

If you knew whether or not to bring an umbrella today, chances are I had some small part to do with that. Five years ago, I joined The Weather Company (at the time The Weather Channel), home of weather.com and www.wunderground.com. Our web properties have a big reach. Weather.com alone garners more than 100 million visits a month online.

When I say I work with weather.com , usually everyone knows what that is. For the first time in my over 20-year IT career, my friends have some idea of what I do. For me, that’s really important and is one of the main reasons that keeps me going. It’s also why my job as QA Manager is so important. To make sure everything is as accurate as possible for something not only my friends and family use, but millions of people rely on, is a great responsibility. 

At The Weather Company, we talk about our guiding principles of Science and Safety. We want to promote good meteorological science and also make sure people are safe. That’s why I feel it’s so important for me to get my job right.  I want to make sure we are accurately presenting the correct weather data so people can trust the science and use that information to keep them safe.  Because of this, we cannot afford to release untested features.

One Stack to Rule Them All

In 2016, IBM purchased our product and technology businesses. Along with this acquisition came new opportunities. 

The first was to align all our web properties on the same technology stack. This move comes with a lot of practical implications for the QA groups. First of all, it makes supporting our properties easier. If my Weather Underground folks have a problem with their tests, they are not alone in trying to resolve their issues. They know they can work with my weather.com staff and usually someone has seen the same issue or knows of way to resolve it. It’s about having knowledgeable staff to support the various applications. That’s important because in a big organization, we need to stay as lean as possible and people need to support each other. 

The other benefit is the impact it’s having on our testing. Having our applications on the same technology stack allows us to use a common test automation framework.  Going to a common framework has meant we’ve exponentially grown our testing coverage, because now we have multiple groups sharing knowledge and expertise. More tests means we’re not pushing out things that are going to break to the millions of people that use us every day.

The next opportunity was a push from management to release features quicker and get user reaction sooner. Previously, we would make major design decisions like changing the homepage or forecast page. We’d do focus group meetings and come up with a design and release it, only to find our click-through rates or impressions weren’t as we expected. 

Then the product side came back to us and asked if there was a way for us to release more quickly, get a read on things, and see if it’s converting as expected. If it’s not, then let’s swap it out and change it. 

QA worked with Dev-Ops to build a Jenkins pipeline to make deployments faster.  We build our code, dockerize it, then deploy it out to Dev, QA, and then Production. We have steps where QA will run our automated tests on various OS and browser combinations, in Sauce Labs, prior to moving to the next step. Using Jenkins and Sauce Labs has streamlined this process for us. We used to do this manually and it was a nightmare to maintain. 

Breadth of Testing

Before going to the new stack, we had been using Selenium WebDriver tests that ran on AWS private cloud. The problem we had was AWS limits the types of OS/broswer combinations we could test. Since AWS has primarily Windows and Linux machines, it wouldn’t let us test on everything that we needed to. Every week we run reports seeing how everyone is interacting with our properties and it isn’t just Windows — it’s also browsers like Safari and mobile browsers. The only way we’d be able to do that on AWS is with some heavy lifting of manually configuring the environments or through manual testing. 

There are different services that provide automated OS/browser environment provisioning, so to test them out we did a proof of concept. In testing out those different solutions, we went with Sauce Labs because they were the most mature in the market and gave us everything we were looking for.

One challenge with using Selenium is that you almost need to be a developer to write your tests. Going with this new stack and with Sauce Labs, we determined we’d be able to go with keyword-driven testing. Using keyword-driven testing is something we’d wanted to do, but we didn't have the time to do it with our old setup. Switching everything to a new stack gave us the opportunity to change that framework. 

Keyword-driven testing meant we could have frontline QA people write tests by themselves. When we did that, we saw our velocity skyrocket. We’ve increased the number of people who can write tests fivefold and in turn, the number of tests we run.

Move Fast and Don’t Break Things

Usually an increase in the quantity of tests will increase the coverage of your tests. We’ve gone from having 700 automated tests to 3,000 tests in just under a year. But that is not how we only increase our coverage.  With Sauce Labs, we can run those tests across multiple OS/browser combinations in parallel, so we have even more coverage.  That many more tests means we can feel a lot more comfortable releasing features knowing that we’ve covered what our fans use. 

By moving to this model of faster releases and seeing what works, we’ve gone from releasing once every few days to releasing two to three times per day. Moving faster also means we are able to see results more quickly, showing that new features are being adopted — or they aren’t. That helps us stay competitive and give people the quality of an experience they’ve grown to expect from us. 

The world of how our service is being consumed is changing. Monday to Friday, people are checking on their desktops while they’re at work, but on the weekend, they check on their phones. Since I started, we have noticed more users want the responsive browsing experience. That means testing in multiple environments including mobile browsers is crucial to our success. 

The way people consume our weather is changing, but its importance isn’t. That’s why the role of testing is so crucial. If we ain’t testin’, I ain’t restin’.