Continuous integration/continuous delivery (CI/CD) is widely embraced by developers because of its ability to deliver code changes more frequently and reliably. Unfortunately, it can deliver those code changes insecurely. When coupled with command-line interfaces (CLIs) offered by cloud service providers, including Google, AWS, and Azure, CI/CD pipelines can pose a serious security threat.
Orca Security senior security researcher Roi Nisimi explained the threat in a blog post. It stems from how environment variables, which can contain secrets, are handled by the CLIs. Certain commands in the CLIs will include the variables in CI/CD logs, where adversaries can exploit them.
"If bad actors get their hands on these environment variables, this could potentially lead to [them viewing] sensitive information, including credentials such as passwords, usernames, and keys, which could allow them to access any resources that the repository owners can."
—Roi Nisimi
Naomi Buckwalter, director of product security at Contrast Security, said the problem is rooted in the fact that CI/CD security has historically been overlooked by busy security teams.
"After all, development and operations teams generally 'own' what goes on within their build pipelines, and security teams don't necessarily want to be overly prescriptive when it comes to how software is built at their organizations. Indeed, the phrase 'staying in your lane' comes to mind when talking about CI/CD security."
—Naomi Buckwalter
With the rise in breaches caused by weak security controls in popular CI/CD tools, there has been a shift to focusing on CI/CD security. Here's what your team needs to know about locking down your development secrets.
[ Learn more in our special report: Secrets Exposed: An Essential Guide for Securing Secrets in Software | Plus: Keep learning in our related report: The State of Software Supply Chain Security 2024 ]
The secret is out on storing secrets in environment variables
Eric Schwake, director of cybersecurity strategy at Salt Security, said that although the CLIs deployed by cloud service providers are essential for cloud automation, some of the commands within them may return sensitive information, including access keys or tokens, as part of their standard output.
"Cloud command-line interfaces, such as those provided by AWS and GCP, play a vital role in many CI/CD pipelines, allowing developers to interact with cloud resources. If this output is not properly handled or masked before being logged, it creates a significant risk of secrets leakage."
—Eric Schwake
Sarah Jones, a cyberthreat intelligence research analyst at Critical Start, said storing secrets in environment variables is a common pitfall.
"While [that's] convenient, environment variables are often included in logs by default. If a CI/CD job inadvertently prints these logs during execution, any secrets they contain — think passwords or API keys — become exposed."
—Sarah Jones
And even some of the tools developers use to hide secrets are not up to the job for modern CI/CD environments. Jones noted that even with masking tools, pre-existing environment variables within the cloud function itself might still leak the secret.
David Lindner, CISO at Contrast Security, said it's not a great idea to store secrets in environment variables because they lack sufficient security controls, and rely solely on access controls to the running machine.
"Environment variables are typically easily accessible by any process running on the same machine, making them vulnerable to exposure if an attacker gains access to the machine. They can leak through accidental logging, inclusion in debugging dumps, or be visible in process listings."
—David Lindner
Storing secrets in environment variables also makes managing and rotating them across different environments more difficult, Lindner added.
Itai Birenshtok, vice president of research and development at DoControl, said security programs frown upon storing credentials in CI/CD tools, but it is widely practiced as a way to enable desired features in programming.
"[It] is a common practice by engineering teams as a way to execute a command automatically."
—Itai Birenshtok
Sources of CI/CD leaks
Critical Start's Jones warned that misconfigured jobs can also be culprits for secrets leaks. "They can accidentally print secrets during execution or store them in unsecured log files," she said.
One common file that's misconfigured is .gitignore, which is used to specify which files and directories should be excluded from version control, preventing them from being tracked or committed.
Joshua Knox, senior technical product marketing manager at ReversingLabs, said it's often just housekeeping that gets missed given the speed of development these days.
"You're supposed to put the names of files that contain secrets in there. Sometimes developers forget to do that."
—Joshua Knox
Hasan Yasar, technical director for the continuous deployment of capability program at Carnegie Mellon University's Software Engineering Institute, said secrets can be leaked through CI/CD tools in several ways: hard coded in source code or as configuration files, deployment scripts, environment provisioning scripts, and base images of targeting host environments.
Secrets can also be leaked, Yasar added, through inadequate secrets management — failure to secure or rotate secrets effectively — and insufficient access control, which allows unauthorized access to sensitive environments, such as cloud space and build, test, and deployment environments.
Secrets that can be leaked in the CI/CD pipeline, Contrast Security's Lindner said, include administrator, root, and sensitive user passwords for services or frameworks; API keys; environment or service tokens; SSH keys; backup codes of any recovery options; any resource keys including cryptographic keys; database credentials' cloud provider credentials; private or embedded certificates; and OAuth tokens.
"Essentially, any piece of data that grants access to systems, networks, or privileged resources could be exposed if improperly handled in a CI/CD pipeline."
—David LindnerCritical Start's Jones stressed that the secrets leaked through CI/CD tools can be "quite damaging."
"With these secrets in the wrong hands, attackers could gain access to sensitive data, manipulate systems, or deploy malicious code."
—Sarah Jones
Command-line risks get real
Carnegie Mellon's Yasar pointed out that usually the cloud CLI interface requires a user to store or keep secrets in plaintext in local settings such as environment variables and logging and history files. "That can be exposed publicly during code integration as well as during the code merge/commit process," he said.
Lindner explained that environment variables are often used to store access keys necessary for CLIs to interact with cloud services. "If CI/CD jobs execute CLI commands that output the contents of these environment variables, secrets become embedded in build logs, potentially leaving them vulnerable to exposure," he said.
ReversingLabs' Knox noted that a developer who knows how to do it right can spin up a virtual machine with permissions that protect secrets.
"But a lot of people don't know that, so they store credentials in the CLI. If a bad guy gets access to that machine, they know exactly which folder the credentials are stored in."
—Joshua Knox
Cloud service providers need to bolster security
Cloud service providers are making efforts to plug CI/CD leaks, but they can do more. Knox said cloud providers are trying to prevent leaks, but when you create a lot of cloud services, you'll get a warning that it isn't necessary to create credentials to access a resource. "But developers will create the credentials anyway," he said, "because it takes more work to do what they want to do without them."
"That's where the friction comes in. Will developers take the time to hide the secrets properly, or are they going to press the easy button, get their credentials, get their job done, call it a day, and throw it over the fence to the next person?"
—Joshua Knox
DoControl's Birenshtok said it is imperative that engineers be educated on the perils of storing credentials and other types of sensitive data within the CI/CD environments.
"While the credentials make it easier to automate workflows, engineers can and should use a secrets manager or password vault to interact with any interface needed in the CI/CD environment."
—Itai Birenshtok
Lindner said cloud providers can definitely take steps to prevent secrets leakage through CI/CD tools, and outlined the baseline for what is needed.
"They should implement secure defaults in their CLIs and SDKs, heavily promote the use of temporary credentials and workflows, offer more fine-grained control over logging, and provide more clear guidance on secure secrets management practices. Additionally, cloud providers should be introducing tools to scan for leaked secrets in public repositories or within their customer environments to help proactively mitigate risks."
—David Lindner
Salt Security's Schwake said cloud providers have a certain degree of responsibility regarding secure secrets management, but he stressed that security is ultimately the responsibility of development teams leveraging CI/CD in the cloud. "Users must take responsibility for configuring these services correctly and avoid hardcoding secrets in their pipelines," he said.
How to lock down your CI/CD in the cloud
There are a number of best practices to address the risks posed to secrets by CI/CD tools in the cloud. DevSecOps practitioners can integrate secrets management practices for specific tool usage, either as part of CI/CD or cloud deployments, Yasar said.
"Vault practices can inject secrets directly into the runtime environment, avoiding hard-coded secrets or insecure storage in any settings."
—Hasan Yasar
Other action items include:
- Performing regular code and repository scans to identify hard-coded secrets and enforce strict access controls for any environment or tool that handles secrets
- Scrambling sensitive data at rest or in transit with strong encryption standards and doing automated auditing and monitoring of access during the CI and CD process
- Rotating secrets outside the CI/CD process using automation, scanning base images regularly, and storing them in an internal registry (artifactory) to pull directly from CI/CD when there is a need
- Developing a common project/code/IaC template to integrate secure secrets management practices and establishing a program for training and awareness of continuous best practices for developers, architects, testers, and other stakeholders
The days of CI/CD being treated as a security afterthought are over, said Contrast Security's Buckwalter. "Attackers are increasingly savvy, targeting the software supply chain as a critical point of entry."
"Security practitioners can no longer afford the 'staying in [our] lane' mentality. Proactive collaboration between security, development, and operations teams is vital to build a secure CI/CD environment."
—Naomi Buckwalter
The time for action is now, Buckwalter said.
"Assess your pipeline's risks, implement robust security controls aligned with frameworks like OSC&R, and continuously audit your CI/CD security posture. Our ability to deliver reliable, trustworthy software depends on it."
ReversingLabs' former field CISO, Matt Rose, wrote recently that after the CircleCI breach, it became clear to many that an end-to-end approach is needed for managing secrets risk.
Pursuing a holistic approach to software supply chain security is the only way to bring the problem of secrets sprawl under control. Through defense-in-depth — addressing all the activities associated with supply chain and application security risk — you can avoid looking at the problem through a single lens and get a true picture of the risks in your supply chain.
Software development teams need to reconsider their processes to include awareness of and protection against secrets leaks. Organizations should look to better understand the epidemic of secrets leaks — and how to best handle them.
—Matt Rose
ReversingLab's Secrets Exposed Special Report covers a lot of ground, including:
- Why organizations need to take a comprehensive approach to secrets security
- How to respond to secrets breaches when they happen
- Best practices to minimize your secrets exposure risk
