With CI/CD approaches to software development spreading ever more widely, the benefits are stacking up for many organizations. A recent study by the Continuous Delivery Foundation (CDF) shows that developers who use continuous integration/continuous software delivery (CI/CD) tools are more than twice as likely to be top performers in restoring service and also have the edge in both deployment frequency and lead time for code change.
But the automation and complexity of CI/CD pipelines and processes can introduce significant security risks to the development process if organizations don't plan carefully. Not only do organizations need to ensure that security checks are built into the fast-paced workflow of CI/CD processes, but the tools and integrations of the CI/CD pipeline itself must also be protected.
Here are eight best practices for hardening CI/CD build environments and baking security into the software pipeline.
[ Special report: The State of Software Supply Chain Security (SSCS) 2024 | Download the State of SSCS report ]
1. Reminder: CI/CD tooling is part of the supply chain
A critical flaw in the CI/CD tool JetBrains Team City came to light this September was being actively exploited by October. This highlights the importance of securing the CI/CD pipeline.
A key fundamental to CI/CD security is remembering that CI/CD tooling is as much a part of the software supply chain as any open-source component that goes into the code.
The JetBrains flaw, a remote code execution (RCE) vulnerability accessible to nonauthenticated users, garnered a 9.8 CVSS rating. It was actively exploited by multiple North Korean threat actors within weeks of disclosure.
Henrik Plate, security researcher for Endor Labs, said that exploiting the JetBrains vulnerability could provide attackers with access to secrets, source code, and other assets consumed or produced on the respective CI/CD system.
"As such, it gives rise to supply chain attacks that have far-reaching consequences, not only for the organization running a vulnerable version of TeamCity, but for all users that download and execute software built on such a system."
—Henrik Plate
As Plate noted, the latter is the worst-case scenario, where attackers quietly manipulate software created by flawed tooling such as TeamCity to create legitimate-looking but malware-infected software.
"This underlines to what extent the security of today’s software depends on the security of upstream dependencies of all kinds, not only embedded components — those that become part of the actual software product — but also all the other components used throughout the software development lifecycle, from code to cloud."
—Henrik Plate
Harman Singh, director at cybersecurity firm Cyphere, said that means you must keep track of updates, vulnerabilities, and dependency problems within CI/CD tooling just as you would for any other software.
"By keeping dependencies up to date, organizations can mitigate the risk of exploiting known vulnerabilities. By keeping dependencies up to date, organizations can mitigate the risk of attackers exploiting known vulnerabilities."
—Harman Singh
Liav Caspi, CTO of Legit Security, said that organizations should also be verifying the security and integrity of the components, plug-ins, the GitHub Actions, and all of the other code that comprises the pipeline.
"Verify the build components you're using. Make sure you use verified components and not something shady that you don't know what's running in it."
—Liav Caspi
2. Tighten up access control and secrets management
One of the biggest mistakes that organizations make in securing their CI/CD pipeline is neglecting to implement strong access controls, said Singh.
"By failing to restrict access to sensitive data, they expose themselves to potential breaches and unauthorized modifications. To avoid this, organizations should enforce the principle of least privilege, granting only the necessary permissions to individuals or systems involved in the pipeline."
—Harman Singh
Not only should organizations be enforcing the rule of least privilege within CI/CD tooling, but they should also ensure that they're using multifactor authentication wherever available. "By requiring multifactor authentication for all users accessing the pipeline, organizations significantly reduce the risk of unauthorized access and potential data breaches," Singh said. "This simple step adds an extra layer of security to the pipeline."
Part and parcel with all of this is the role of strong secrets management. Viktor Gazdag, managing security consultant for NCC Group, said that some of the biggest exposures and integrity problems within the CI/CD pipeline have to do with insecure handling of secrets, including hardcoded credentials and poorly secured stores of credentials in development environments.
"The scope and storage of the secrets are also often overlooked and misconfigured, allowing everyone or a wider audience to access them."
—Viktor Gazdag
3. Build security monitoring and telemetry into the pipeline
Vladislav Bilay, DevOps engineer for Aquiva Labs, said that for organizations seeking to level up their AppSec chops and maintain the integrity of their CI/CD pipeline, visibility is crucial. Continuous monitoring and feedback mechanisms are paramount, as is establishing a robust feedback loop for security information across the entire pipeline, Bilay said. That allows teams to detect and respond to potential security vulnerabilities in real time.
"However, this requires a shift in mindset from periodic security assessments to an ongoing, integrated monitoring approach, demanding commitment and adaptability from DevSecOps teams."
—Vladislav Bilay
Yash Mestry, DevOps and security associate for Perpetual, said pipeline observability is huge in the era of cloud-native development and deployment.
"To understand the dynamic and dispersed nature of cloud-native systems, powerful observability and monitoring capabilities are needed. To do this, logs, metrics, and traces from multiple application components must be gathered and examined for signs of security threats and anomalies."
—Yash Mestry
Mestry explained that to securely make the transition from traditional application development to CI/CD and cloud-native development environments, organizations have to rethink their security telemetry and tooling to bring build visibility into the pipeline.
"Tooling must be flexible, automated, and capable of interfacing with the DevOps pipeline in order to support a cloud-native approach. In order to do so, security testing tools that connect with container orchestration systems, vulnerability scanners for container images, and security-as-code frameworks for automating security policies may be used."
—Yash Mestry
4. Security as code is key
The practice of building systems and configuration controls into the CI/CD pipeline that make it easier for developers to make sound choices in how they code, deploy, and run software is a no-brainer for AppSec risk reduction — whether you call that "developer guardrails," "security as code," or "Secure by Default." Rather than just teaching developers about security best practices, the best DevOps teams also set and implement security policies through code and configuration that’s built directly into the pipeline and underlying software infrastructure.
Derek Fisher, author of the Application Security Program Handbook, said in a recent interview on the Security Champions Podcast that it is ultimately about ensuring that engineers have the ability to develop code normally but with the confidence that they're secure.
"In terms of being able to deploy, you have your infrastructure as code developed in such a way that it ensures that you don't have things like configuration drift and that it follows best practices and patterns and nobody's allowed to tamper with that. So I think there's ways that we can build that paved road, those guardrails, to ensure that developers are just focusing on making sure that their code gets out the door on time."
—Derek Fisher
And when security as code is developed, the security team should be coming up with the policies in close collaboration with the developers, and in many instances the engineers may end up being the ones who develop the code that enforces guardrails on the daily. Application security is the sidecar to development, and that means it rides along with the engineers as they're building software, Fisher said.
"What I've found is that my team and the application security individuals that I've worked with are not always the ones coming up with the best solution. To be honest, the security teams are not going to know the full technical stack that the engineers are developing their software on. They may know enough to be dangerous, but they're not going to know all of those languages and all the nuances and the exact way to solve the problem in four different languages."
—Derek Fisher
5. Security gates are critical
A part of the process of setting up guardrails is enforcing the mentality of testing early and often — and integrating those assessment processes directly within the CI/CD pipeline, Fisher added. Establishing security gates and embedding them into the integration and deployment workflow is the bedrock for CI/CD security and improved AppSec performance.
"We need paved roads in existence from a security standpoint where your pipeline has appropriate scanning tools built in at the right times and a feedback loop back to development teams. It's not going to clear a gate with either a certain number or certain level of vulnerabilities, whether critical or high. Full stop."
—Derek Fisher
It's not as if this is a new or foreign concept for developers. "This honestly isn't much different than the way that we have the different guardrails in place to make sure that there's quality built into the development," Fisher said.
"You have unit tests, you have system tests, you have integration tests, and those are all there to ensure that defects are caught as quickly as possible and that the engineer or the development team is able to fix those and get them back out the door as soon as possible."
—Derek Fisher
6. Shore up your secure code signing
As threat actors increasingly target code-signing certificates to stealthily circumvent integrity checking of software code, development teams are going to have to do more to shore up the security of their code-signing processes. Christian Simko, vice president at AppViewX, said this is the gist of the new Code Signing Baseline Requirements issued by the CA/B Forum this year, delineating secure hardware for generating and storing private keys to prevent key compromises. These guidelines should be used to update code-signing practices.
Simko is an advocate for centralizing code signing, which has traditionally been distributed across different development teams and which also has not been very DevOps- or CI/CD tooling–friendly.
"This often leads to the use of multiple code-signing tools by different developers and insecure code-signing keys. To overcome this challenge and support the needs of development teams, organizations should implement a centralized code-signing model. This approach can be used to enforce code-signing best practices, integrate code signing into the CI/CD pipeline, and keep code-signing keys secure. It also modernizes code signing for scalability, efficiency, and agility."
—Christian Simko
7. Store software artifacts sanely
As security teams bolster the security testing, monitoring, and dependency tracking of software as it works its way through the CI/CD pipeline, they're going to be generating an increasingly growing mountain of security artifacts. This can become a data management and analysis nightmare without the right systems in place to sanely store the evidence, said Steve Taylor, CTO for DeployHub and a board member of CDF, in a recent presentation.
"We can't keep track of that in our head. So we need to push this information somewhere, and that's where we need to store the evidence."
—Steve Taylor
Taylor said that publishing security artifacts coming out of the build process into aggregated registry tools such as ArtifactHub, DockerHub, Quay, Maven Central, npm, or Pypi is crucial to making sense out of it all.
"Some of these registries are being extended to store any type of artifact. That's going to allow us to have a one-stop shop for managing all of these artifacts and getting the information about them from there."
—Steve Taylor
8. Don't forget to threat-model the pipeline
Threat modeling isn't just for software that runs through the CI/CD pipeline. Build environment components and the entire pipeline as it is deployed should also be put through their paces, said NCC Group's Gazdag. He encourages organizations to adopt a continuous threat modeling approach to CI/CD environments to keep an eye out for insecurities in connection boundaries, connecting components, and data flows.
"This will allow an organization to see the big picture with gaps and blind spots, so they can put in place the correct preventive and security controls. For example, developers can see where a credential is stored, where it is used, where it came from, how it’s stored, and who can access it."
—Viktor Gazdag
The call to action
With attackers increasingly turning their eye to development environments, organizations must take measures to prevent attackers from infiltrating CI/CD infrastructure to keep them from not only breaching the organization but also introducing dangerous flaws into their applications.
