Answering the Call for Telecom Innovation: Navigating Networking Automation Through Meaningful Collaboration

At Deutsche Telekom Technik GmbH (DT Technik), the technology unit of Deutsche Telekom in Germany, we service around 47 million mobile customers, up to 18 million fixed-network lines, and about 14 million broadband lines throughout Germany. We are committed to offering our customers the best network experience. Thus, our dedicated team is constantly evolving our networks by leveraging leading-edge technologies. In order to successfully implement innovative technologies, we draw on the competencies and skills of other industry experts. At this point, I’d like to share my experiences of a genuinely positive and meaningful example of collaboration, where Telekom and Cisco decided to leverage a joint fund dedicated to driving innovations in various fields of their cooperation.

We’ve invested heavily into evolving our business model from traditional telecommunications to new services, including a comprehensive cloud ecosystem. This, of course, calls for more efficient networks and a streamlined, optimized infrastructure. In order to achieve this, we wanted to automate as much of our data center infrastructure as possible. But the journey that followed involved a bit of a detour that led our team to adopt a DevOps approach to product delivery.

Surveying the Land

Our team is responsible for providing network automation services regarding data center infrastructure including device life-cycle automation and PNF/VNF provisioning (router, switches, load balancers, and firewalls) to interconnect our cloud environments based on OpenStack, Kubernetes, VMware, and more.

A couple of years ago it took quite a long time to get quality configuration into the networks. Office tools were used as a workaround to do some magic to create the configs. There was no centralized database used to store all the information, and the teams we worked with had different approaches on how to design, operate, and manage actions. We wanted to significantly reduce time to market and improve the quality of configuration data. 

One of our first goals was to minimize repetitive tasks to give the highly trained workers the freedom to spend their time on more complex work reflecting their skill level.

Finding the Right Partner: Pursuing Network Automation with Cisco NSO and Itential IAP

Three years ago, our colleague Johannes Peternek was a student who wrote his thesis on network automation. His research focused on the potential capabilities of network automation, and the solutions that would help to transform these possibilities into reality. We were excited about the various approaches he explored. These new insights laid the groundwork for finding the right accomplice. We were looking for a trustful partner who would enable us to get a head start rather than starting the project in a green field—and to turn our vision into reality.

We first began talking with Cisco about the possibilities and a proof of concept implementation in April 2017. Our decision to choose Cisco was mainly driven by two factors. First, they offered the Cisco Network Services Orchestrator (NSO) product, which would be a big step in orchestrating our transformation. Secondly, we also already had a close relationship with Cisco due to an intense collaboration across multiple innovation projects that have been jointly funded. 

Cisco was one of our main suppliers for LAN infrastructure, routing, and switching. But the Cisco world of software services is completely different, and we’d never worked with them in this capacity before. 

I stepped in as the project manager in April 2018, and we kicked off the RFQ phase and finally started the project officially in November 2018. This LAN automation initiative was dubbed “automation@lan” which eventually was shortened to AUTLAN. With the AUTLAN solution stack, consisting of Cisco NSO and Itential IAP, our goal was to establish a single point for self-services to provision standardized services within our data centers.

Unfamiliar Territory: The Hills and Valleys of a DevOps Approach

An initial takeaway from this project was recognizing that highly complex projects like these take a lot of cooperation. We went through several phases to find the right approach that worked for everyone.

In order to bring out the benefits and thus impress our stakeholders with the project’s value add, we had to establish a use case and deliverables, the most complicated of which revolved around connectivity. The complex nature of this deliverable meant that it was also the most difficult to automate. To make life easier for us, we could have outsourced these tasks, but that’s not what we wanted out of this project.

Our main goal was to completely take over the platform and not rely on purchasing ongoing professional services. Focusing on a goal like that takes a lot of time and effort. Doing it our way required both motivation as well as persuasiveness. It definitely took longer than it would have taken otherwise, but the results have shown that all the hard work had been worthwhile as we were able to upskill ourselves in the process learning how to build, code, and maintain the platform on our own. 

Even though it went well, along the way, we had to face some obstacles no one would have expected at the beginning of our journey. As we moved further into our automation journey, we had to deal with significant department shifts within DT Technik. We used to have siloed operations, planning, and engineering departments. The goal of the transformation was to adopt an agile mindset and create powerful and independent small squads, organized in tribes who are responsible for a service.  Knowing that this was on the horizon, our AUTLAN project team got the chance to start directly as an agile SCRUM team bringing team members from their respective departments into a cross-functional agile squad. 

Our new methodology required us to develop code, talk with customers, learn user stories, and much more. Because of this, working in agile squads made sense. This decision not only improved collaboration and communication, but it was necessary for team members to learn more about how to create—and take care of—our own environment. So, as it turns out, the achievements of the transformation played into our hands.

Controlling Our Own Environment: Building Confidence in Our New Operations

Cisco built a reference environment as a blueprint and we set up the production environment on our own. We were able to develop our own processes for CI/CD testing, and we learned to provide solutions on the platform since we didn’t have an out-of-the-box solution. We can now write test cases, set up servers, and make operating system instalments and applications. We operate the complete platform—from the system layer on a VM to the Cisco/IAP software products.

We also are the only team delivering services on the platform. We had to learn this new software skillset from scratch, along with tackling hands-on software projects and learning new programming languages, all with the Cisco team navigating and supporting us along the way. We’ve also relied on other best practices in the company and the industry as a whole to guide our way forward.

People from former operations departments are happy because they can code and perform more complex tasks. The engineering team, on the other hand, was initially nervous about taking on operations tasks such as being on call and troubleshooting. Being a product owner and talking to customers instead of managing a project was completely new, too. It’s been a time of radical professional growth for all team members, and our new squads rose to the task. 

Plotting Our Next Moves: Thinking Critically About Support and Iterations

We leveraged Cisco and their partner Itential to fulfill all use cases. Both products (Cisco NSO and Itential IAP) complemented each other—NSO as the powerful multivendor orchestration component based on netconf and YANG as well as IAP as the capable workflow and third-party interface.

We leaned on both Cisco Support and Focal Engineering to help us through building our production environment. One of the biggest benefits was Focal Engineering’s support in the background. We had a single point of contact for each and every technical question, and who helped us to learn the software development best practices. Our contact also served as an advocate for us within Cisco, which has proven to be very helpful.

We learned how to approach our products from a new perspective, and we’ve taken that a lot further. It's different from network design, which involves configuring a box and then running the network. From our software development practices, our CI/CD approach, and how we think about continuous integration and automated testing, design, and upgrades, we’ve seen constant changes. We now need to think about how to design the features, so they are reusable, how to test, and how to prevent errors during a major platform upgrade. This kind of methodology was very valuable—it was the foundation for our team’s education and growth.

Maintaining Momentum

This new flexibility in our team roles and our new DevOps environment has given us newfound freedom—we’re operating internally like a small start-up within a large company. AUTLAN has given us a new route to navigate through a complex environment in order to grow.

The telecommunications industry is constantly evolving, and each provider is on their own journey to keep up with the ever-increasing demands of our customers. DT Technik is committed to keeping pace through our commitment to innovation and diversification of services. 

My team and I can now rely on a new method for agile collaboration within our company and a solid automation service to focus on delivery. And working with a partner like Cisco through such a paradigm shift enables us to utilise our full potential in order to provide our customers with the stellar service they’ve come to expect from us. Now, we can do this even more reliably and faster.

It certainly helps to have a map as you embark on a new journey, but sometimes all you have is a compass and trusted partners to help you along the way. And even when charting a new path, that’s usually all you need.