How Sourcegraph Reaches “Inbox Zero” for CVEs

Common Vulnerabilities and Exposures (CVEs) are a long-standing reality in software development. But the days of strictly in-house code bases are long gone. These days, more companies are leveraging open-source components, dependencies, and libraries to accelerate release cycles, create innovative products faster, and respond to changing market demands.

As a result, developers have less control over the security of their release pipelines and artifact sources when introducing and updating their products.

My role at Sourcegraph is to help the business grow quickly in a secure manner. As the Head of Security, I lead a team of seven that integrates GRC (Governance, Risk, and Compliance), security engineering, and operations into a cohesive, agile security framework that supports the company’s strategic goals and growth. I shape and execute our security decisions and strategies, oversee hiring for the security department, and oversee a diverse, high-performing team.

Sourcegraph is a code AI platform that makes it easy to read, write, and fix code–even in big, complex code bases. We support all major languages, and our Cody AI coding assistant empowers developers to write, fix, and ship code using autocomplete and commands. Our clients are typically B2B enterprises and include respected names like Canva, Databricks, Nutanix, and Reddit.

Evolving, Time-Consuming Vulnerability Practices

The software industry is hyperaware of the risk of supply chain attacks, particularly at the third-party library and dependency levels. The industry mindset about reviewing and scanning for vulnerabilities has shifted, and companies are defining best practices about data hygiene and release sequences. Developers are also facing new government regulations about minimizing risk exposure and vulnerabilities. 

Our internal practices at Sourcegraph reflected these industry trends. When we detected a CVE, our engineering developers or the security team reviewed the parts of our product using the compromised component, library, or dependency. That might have taken anywhere from five minutes to several hours. We then created a ticket, triaged the components, determined the severity of the vulnerability, patched the code or mitigated the problem by creating an exception, and documented the issue. Sometimes, we had to update the version of the affected package. It was a time-consuming process—the equivalent of a full-time engineer spending 25% of their time detecting and remedying vulnerabilities.

The complexity of identifying and resolving CVEs presented significant challenges for our engineering and security teams and negatively impacted our sales and customer success divisions. Our reps frequently fielded calls from frustrated customers who couldn’t leverage the latest version of our software because it had known vulnerabilities. These customers had strict IT policies that (rightly) prevented them from deploying software with known vulnerabilities, and they wanted explanations and exceptions that would permit them to use the affected products. There was a lot of extra work in making exceptions, triggering solutions, and writing patches that resolved these issues.

Combining Open-Source and Enterprise Tools to Reduce Complexity

We looked at several scanning solutions to help us detect and eliminate CVEs. Our top priority was leveraging a consciously minimal product that avoided unnecessary packages in dependencies, employing and securing the fewest possible pieces needed to build working images. We explored scratch and distroless image solutions, but they lacked adequate support, and we needed something that combined the wisdom of the open-source community and the stability and resources of an enterprise-level commercial product.

We found that combination in Chainguard Images, a solution based on Wolfi OS. Wolfi is a community Linux OS (un)distribution built with default security measures for the software supply chain, avoiding unnecessary packages and dependencies. Chainguard Images is a collection of container base images that eliminates complexity and reduces security risks by shrinking the number of components needed to compile an image to the bare minimum. A smaller surface limits the exposure to threat vectors and reduces the number of elements that must be scanned and updated for CVEs.

Chainguard Images also gives us the peace of mind of working with a commercial vendor. We were unhappy with the cadence of patches with some of our open-source tools, but Chainguard Images is regularly updated and patched daily.

Zero Known Vulnerabilities Practically Overnight

We started testing Chainguard Images in late 2022 and spent some time learning the platform. Chainguard supplied best practices for creating Wolfi images, managing dependencies with tools like Melange, and integrating these features into our build system, Bazel. All of this is transparent to our engineers and developers. We define and manage the dependencies, and Bazel automates our build pipelines, creating Wolfi images super quickly.

Chainguard Images was an overnight success. My team previously struggled with minimizing and triaging CVEs in our most critical customer-facing images. After adopting Chainguard Images, we reached inbox zero—zero known CVEs—for the first time in two years, eliminating the daily headache of vulnerability maintenance and freeing my engineers and customer success teams to focus on customer innovation. Where detecting and remedying breaches used to take about 25% of the work of a full-time engineer, it’s dropped to less than 5%. That’s a massive block of time we can allocate to developing new security controls and other improvements.

Chainguard Images helps to streamline and improve customer conversations and interactions, creating friction-free deployments for our users. In the past, customers had to wait weeks before they were comfortable using our latest release, and it took 10–15 business days to approve and review exceptions and issue patches. Now, we mostly ship CVE-free containers. When an issue emerges, Chainguard resolves it as part of its daily patching process so there’s never more than a two- or three-day delay. It’s a massive improvement for our business, and our customers can trust us to deploy robust software developed with components free of known vulnerabilities. 

Using OpenVEX and SBOM To Monitor and Remediate Supply Chain Threats

Another Chainguard benefit is its powerful OpenVEX (Vulnerability Exploitation eXchange) tool that generates machine-readable status updates about common CVEs. It allows software supply chain stakeholders to collaborate and automate identifying and remediating CVEs to create security advisories that downstream software users can employ to determine whether a vulnerability has impacted them and how they might need to address it.

OpenVEX eliminates the noise of non-impactful CVEs, allowing us to focus only on those vulnerabilities that will prevent our customers from using our products. When paired with Chainguard Images’ power software bill of materials (SBOM) features, we can quickly scan component structures to arrive at a comprehensive picture of vulnerabilities in our container base images and quickly address the issue we encounter.

The combination of Chainguard Images, OpenVEX, and Wolfi OS makes it easy for Sourcegraph to monitor our supply chain, review SBOMs, and identify immediate threats we must manage and remediate. We can deploy our products faster, protect our customers from CVEs, and assure them they can safely start using the latest versions of our products within hours or days of their release.

These powerful new tools, including technologies like Sigstore, are moving the industry forward, providing better data about common vulnerabilities and speeding the adoption of minimal images to bolster supply chain security. Whether it’s the fully-supported commercial version or its open-source edition, Chainguard Images is giving businesses the tools to better protect their data and serve their customers by combining the best open-source and commercial tools to secure their code bases.