Human Power or Computer Power? Our Automation Journey with Cisco

The beauty of automation is its ability to free employees from menial tasks to focus their time and energy on more impactful work. To determine what tasks should be automated, one must look at each role and ask if it requires human brain power, or can be done in a better way.

I work for Telefónica International Wholesale Services, a subsidiary of the multi-national Telefónica Group, which manages their global network platforms, such as the telecommunications provider Movistar. My job is to manage Operation Support System and Business Support System infrastructure for our networks and platforms. 

My team set forth to alter the system situation of TIWS to see how we could evolve through automation. For example, we wanted to take a step back and determine what tasks should fall within our system manager’s purview and which tasks should be automated. While we did not want to automate every tiny detail and the unique quirks of an employee’s job, we needed to consider some variables within our work to help us determine our needs. These variables are the human side and the financial side of systems operations.

On the human side, we first asked our staff in operations where they spent most of their time. We strived to know what caused them stress, the problematic situations they would like to avoid, and the core causes of these issues. When we examined the feedback, we found some cases where automation could relieve a lot of unpleasantness and unnecessary time spent. For example, we discovered a great deal of work was done to reconcile the locations in our interfaces, since they often appeared differently: some were called London, some Londres, and others LDN. We could quickly solve this through automation rather than manual fixes each time.

The other aspect we had to consider was budget. We needed to examine and balance what we paid our employees to do in-house compared to what we paid for external services. For those outsourced services, we looked at what we could bring in-house through automation. There were a lot of factors and strategies for us to consider.

Handling Different Perspectives

Once we decided automation was our instrument, we developed an RFP for different providers. We tested Cisco and three other solutions in a live environment. It was important to do these tests on a live network to prove, definitively, that the solution we chose could guarantee network continuity. Cisco was the clear winner. They had the network, technology, equipment, experience, and expertise we needed.

But the implementation process proved to be difficult because it highlighted the different perspectives around our operation guidelines within the team. When it came to Cisco executing our infrastructure deployment, we discovered that we did not have a standard end-to-end view as a team. 

Sorting out the discrepancy between our operations guidelines and the actual day-to-day work was incredibly difficult. It was painful because we were delayed and results were expected sooner. But Cisco was a great ally. They brought in more expertise to help us and challenged us to be rigorous and precise in defining variables until our team was all in agreement. This preliminary work created a reliable and useful set of new rules to put in place.

The ROI of Automation

The results of the new system we developed for automation (implemented by Cisco), has our employees spending more time on high-level work. From our new business support system, we obtain customer orders with the allocation and assignment filled out automatically. We receive a template with all the parameters, loaded in the web interface. We then complete a quick visual inspection and push to the network. 

The time we gain back through automation is very impressive. Some aspects of network configuration used to take four hours, but now take only twenty minutes. Other new interfaces have gone from a forty-five-minute process to two minutes. We have a clear picture of savings based on time saved in-house or shifted from service provider to an automated service. Every week we report our cost reductions by the log of tools use. Previously, it would cost us €75 per set operation, and now our savings are enormous: in just one week at the beginning we saved €2,000 in operations cost. And that is just at the start.

Even automating our specific reporting system has created huge efficiencies. Where it previously took someone 4 hours each Monday to analyze and graph our centralized logs, we now automate and save €500 per report. Overall, we have had a 20% reduction in operational expenses year over year in the last two years.

We have now reduced our Change Operations staff from 60 to 30, shifting some of the former employees into consultant roles and also hiring more interns  This allows us to have the best talent available, and to augment the quality of our employee’s work. We no longer ask employees to droningly copy commands. Instead, we shift them to creative, high-level problem-solving, or programming the tools. Our team see themselves as part of creating something new and exciting, and this is the best way we can say, as a company, that we believe in them. We want our employees to give us their best, and this means providing a workplace that challenges them to grow and increase their skills.

Automation Is Everyone’s Business

Reflecting on our implementation, I have a few thoughts on how other companies should best look at automation. The most crucial aspect is to have every person on your team invested in the project and on the same page. To do this, you need a strong commitment from management to lead the team and spearhead the project. 

You will also require everyone—all internal stakeholders—to be invested. You cannot simply have your end users give some specifications then leave while you attempt to build your new system. They need to be involved throughout the process. Ultimately, it is their tool to use.

It’s essential to have a strong leader who can communicate the merit of the project and get everyone involved. Automation is a collaborative project because it affects the day-to-day realities of multiple employee’s work. It is not a top-down management decision, but a co-created recipe for success that needs every employee’s input to succeed.

In the end, we had a consistent team dedicated to making it happen: the Cisco team, the network staff, and the IT department. Every week we met to explain to Directors the progress of our implementation. There was no finger pointing because we were all together, and that’s why it worked so well for us. But we had to figure out first how to get there.

The best part of our systems operations is that we have a set of tools that can be put to use in the future. It's like hosting the Olympics. A city will spend tons of money and effort to build the infrastructure, and once the games are over it is left to be used for a multitude of purposes for years to come. 

In our case, we ended up with a GitHub repository with debugging, version control, and code testing tools on every PC. We now have a set of DevOps tools. Now other teams, like the voice team, infrastructure, and our cloud team, are starting to use the GitHub themselves. The benefits keep spreading and growing. Now, our teams can focus their energy on being the best, and we’ll take care of everything else.