Transforming DevOps at Scale with the Jenkins Templating Engine

When you work in DevSecOps, it sometimes feels like you’re constantly reinventing the wheel. Every team needs a pipeline to automate their software delivery processes by building, testing, scanning, and deploying the team's application. The typical approach is to create a bespoke pipeline for each application, but from a structural perspective, many pipelines employ the same building blocks. Why couldn’t we find a way to reuse them?

When you’re supporting dozens of projects—whether internally or for your customers—it pays to scale. If you can set up multiple pipelines in less time, you can take on more work and can deliver more value. You also free your developers to focus on what they do best: building applications. But when you want to change the status quo, it’s hard to know where to begin.

I’m a senior lead technologist at Booz Allen Hamilton, a technology and management consulting firm with expertise in analytics, digital transformation, engineering, and cybersecurity. In the digital transformation domain, we help our clients address their complex mission and business needs through the integration and adoption of data analytics, modern software delivery, and cloud-native microservice architectures. 

A good portion of our customers are federal agencies. These agencies are understandably risk-averse, supporting missions critical to our nation's security and stability. They often have tried-and-true processes in place and they come to us to help with upgrading their systems to meet the rapid pace of innovation in delivering software capabilities to the citizen.

DevSecOps: Easier Said Than Done

Most days, I play the role of subject matter expert for many different projects by helping our clients understand the value of DevSecOps and modern software delivery to their missions. Typically, the technology is the easy part. Successfully implementing these practices requires a cultural transformation alongside the digital one to break down silos and be open-minded to the idea of changing everyday processes.  

Getting our clients to adopt new tech is the easy part. A common scenario would be multiple contractors, geographically distributed, building a microservice-based application with 50 or so services each with their own source code repositories and wanting to leverage a common pipeline. This requires creating 50 common Jenkinsfiles to put in every branch of every repository. As a result, there was a lot of duplicative effort with little governance around the software delivery process, and no unified approach to security. The teams are using different tools so each Jenkinsfile has to be tweaked slightly, and the project was moving at a snail's pace.

Now that you've managed to create pipelines for each of these services, imagine wanting to incorporate some lessons learned a few weeks later?! You'd have to open 50 pull requests and orchestrate a migration to the next version of the pipeline across all of these teams. It's a maintenance nightmare. 

Once your DevOps pipeline is working, though—the value is clear. Internal hierarchies, external mindsets, and years of maintaining the status quo become easier to overcome after seeing a DevSecOps workflow in action.

Reusing Building Blocks

We needed a way to make this situation easier to manage. Regardless of which tools each team was using, the business process to get code from developer's laptops to production was exactly the same. It would have been great if we could have just pulled out one common and tool-agnostic pipeline definition for every team to use and then let the teams tell us which specific build tools they were using for their services.

Jenkins is the industry's de facto CI/CD server, and its wide extensibility providing integrations for hundreds of tools and services made it the perfect candidate to try and create this solution to our problem. 

Every Jenkins pipeline uses a set of building blocks to implement a common skeleton of a workflow, and that got me thinking: What if we created a tool that allows you to create tool-agnostic pipeline templates for each team to leverage instead of managing 50 individualized pipelines?

That was the reasoning behind the Jenkins Templating Engine (JTE). JTE is a Jenkins plugin that lets you assemble a centralized, hierarchical governance structure built on inheritable pipelines that use common configurations. It also allows you to share pipeline code across multiple teams through the use of common libraries. JTE simplifies pipeline maintainability by consolidating the pipeline definition to a centralized location.

Not only are these pipeline workflows largely the same, the pipeline code that implements various tool integrations is also largely undifferentiated.  For example, the pipeline code required to perform static code analysis with SonarQube does not need to change from team to team.  JTE has allowed us to separate the business logic of a DevSecOps pipeline from the technical implementation of the individual steps. These reusable pipeline libraries have dramatically accelerated the development of mature DevSecOps pipelines for our clients while also bringing a new level of organizational governance to the process. What used to take 5 months is now possible in around 5 days, a 97% decrease in pipeline development time! This type of value to our clients is huge; we can now streamline our software delivery processes in days instead of months and begin delivering on our client's missions almost immediately. 

For organizations with multiple software development teams, it’s no longer necessary to train them all on how to integrate automated testing and write Jenkins pipelines. For bigger companies with a pipeline-as-a-service model, a central team can create pipelines that community development teams can then inherit. Small companies can assign one group to work on pipelines, freeing the other developers to focus on coding. JTE is a practical option for any company that is trying to consolidate expertise and scale. 

From Fanboy to Thought Leader

I have been the lead on the Solutions Delivery Platform (SDP), Booz Allen's open source DevSecOps capability, for two years now, but my enthusiasm for Jenkins goes back further than that. A few months before I embarked on this project, I met the creator of Jenkins, Kohsuke Kawaguchi, at Jenkins World in 2017. I went into total fanboy mode and asked for a selfie that I could post to my Instagram account.

A year later, we invited him to talk about DevOps at Booz Allen's distinguished series, and I showed him SDP. It was a nerve-wracking experience to present my work but he loved it, and personally made sure that my team and I spoke about SDP and what would later become the Jenkins Templating Engine at the next Jenkins World conference.

Even though JTE was a product of Booz Allen Hamilton ingenuity, we knew that the only way it could flourish was by sharing it with the open-source community. Once we did that, the project took off at warp speed. Developers in more than 65 countries have now read our JTE documentation to build roadmaps for pipeline governance, and some have even begun contributing to improving the plugin. Contributors include developers for some of the world’s most admired companies. When anyone suggests improvements to JTE, all of us benefit from their expertise.

Building Better Solutions Together

The value of open sourcing is that we can build better solutions together than we can apart. Booz Allen might have kicked things off with the idea of the Jenkins Templating Engine, but it takes perspectives and participation from the open-source community to help JTE realize its fullest potential.

Another advantage of the open-source approach is enhanced quality and security. The more eyes we have on JTE, the easier it becomes to spot issues that may affect performance or compromise security.

Finally, open sourcing JTE has helped to position Booz Allen as a thought leader in the DevSecOps space.  We have had the awesome opportunity to collaborate with industry peers and present what we're working on to thousands of colleagues via conferences like Jenkins World, KubeCon, Red Hat Summit, and DevNation Federal. 

The audience response is always the same: “It just makes sense,” “This is so obvious,” and “Why aren’t we already doing things this way?” Sometimes, the simplest solution is staring you in the face. I like to think that we were all so busy making Jenkins the market-leading CI/CD pipeline tool that we overlooked the obvious.

On several occasions, I’ve been asked if I consider the project a success. I think it’s a constant work in progress, and I will finally be willing to call it a success the day I can walk away knowing it will continue to grow and mature. The day will come, and it will only be possible through the continued adoption within the open-source community. 

Releasing a Solution into the World

Reducing the time it took for Booz Allen to deploy multiple secure CI/CD Jenkins pipelines was a big step forward for our clients, but that wasn’t enough for us. We are sharing our technology with developers worldwide and crowdsourcing the insight of the open-source community to improve governance, pipeline maintainability, and to increase pipeline code reusability.

I'm having the time of my life working on JTE, and encourage anyone to get in touch with me to discuss this powerful tool on our Gitter Channel. I look forward to sharing my expertise, exploring new ideas, and even mentoring promising young developers as long as I remain active in this community.  A great place to get started would be watching our recent Jenkins Online Meetup discussing the Jenkins Templating Engine.

I have a true sense of accomplishment when it comes to JTE, and that will follow me whenever I move on to another project. That’s the beauty of open source—everyone is always giving back, thus feeding a cycle of continuous improvement. JTE belongs to everyone, and every member of the community has the power to influence its evolution in the years to come.