A software supply chain exposure was averted recently when security researcher Adnan Khan discovered a serious flaw in the GitHub repository Puppet Forge. The flaw, dubbed RoguePuppet, would allow anyone with a GitHub account to freely push official modules to the repository of a popular open-source software configuration management and deployment tool, Puppet.
Puppet Forge is a repository of reusable modules that can be downloaded and used to automate tasks and configurations in a Puppet environment. It provides a centralized location for users to find and download prebuilt modules, which can save time and effort in managing infrastructure and applications.
Khan explained in his analysis of the RoguePuppet exposure that Puppet is an infrastructure-as-code service used predominantly by large companies and government organizations with significant on-premises infrastructure:
"Many of Puppet’s featured customers are within the healthcare, financial services, and critical infrastructure industries. These organizations are prime targets for ransomware groups. ... In the hands of even a remotely capable attacker, this vulnerability could have caused extreme damage around the world.”
—Adnan Khan
Here are the lessons learned from RoguePuppet, including what organizations need to do to prepare for such potential software supply chain compromises.
[ See RL's new Essential Guide: Software Supply Chain Security for Dummies ]
The scope of the RoguePuppet exposure
ReversingLabs evangelist Joshua Knox said that if a malicious actor had found this vulnerability, they would have been able to modify any module in Puppet Forge.
"They could do a pull request and push it without any approval."
—Joshua Knox
Jeff Williams, CTO and co-founder of Contrast Security, said RoguePuppet was a "disastrous flaw" that an attacker could have used to quietly backdoor Puppet modules used by companies all over the world.
"Puppet isn't as widely used as it once was, but companies still using it are likely to be the least prepared for an attack. A backdoored Puppet module would be able to run as part of the company's apps and do anything they could do — steal data, corrupt data, launch other attacks — all from within the company's own data center."
—Jeff Williams
Mayuresh Dani, threat research manager at Qualys, said Puppet is mostly used in connection with CI/CD workflows. Even if a couple of well-known modules were backdoored and pushed to their repositories, it would mean that any workflow set to use the latest version of the package could download them as a part of their Puppet deployment jobs, he said.
James McQuiggan, a security awareness advocate at KnowBe4, said the scope of the potential attack would rank with the biggest exposures in recent years.
“It would have been as bad as the SolarWinds or the MoveIT attacks."
—James McQuiggan
Khan noted that RoguePuppet was made possible by a systemic GitHub Actions CI/CD misconfiguration within Puppet Lab’s public GitHub repositories. “The vulnerability allowed anyone with a GitHub account to obtain the API key Puppet used to push official modules to Puppet Forge,” he wrote. “The only actions an attacker would need to do is create a pull request, issue a few API calls, and quietly close their pull request minutes later.”
Lessons learned: A software supply chain security reality check
RoguePuppet is part of a larger problem, said ReversingLabs' Knox. "What happens when you crowdsource modules? You have to do your own due diligence. You can't blindly trust things, but we're seeing that more and more."
"We're learning that you can't blindly install software on a machine or in your environment. You've got to do your own testing. The irony is we've been forever taught to update immediately because that's where you get your security patches."
—Josh Knox
Kevin Kirkwood, CISO of Exabeam, said RoguePuppet shows the pitfalls of open-source software for software development teams.
“If you're using open source code, you have to be careful. It's on the developer to get their testing into the CI/CD pipeline as early as humanly possible."
—Kevin Kirkwood
Naomi Buckwalter, senior director of product security at Contrast Security, said RoguePuppet once again exposes what’s become the Achilles' heel of the software industry: open-source supply chain software attacks.
“While open source fuels innovation and collaboration, it also introduces a potentially vast attack surface that malicious actors are increasingly exploiting. It’s long-overdue for the cybersecurity and software engineering communities to adopt a more proactive and comprehensive approach to securing the software supply chain.”
—Naomi Buckwalter
Contrast Security's Williams agreed that organizations should be extremely careful about the third-party code that they adopt, but he stressed that the responsibility for supply chain security organizations should be the team running software pipelines.
“Your security is critical. You must provide the highest level of assurance that the software produced cannot be tampered with, as all of our security depends on it."
—Jeff Williams
Qualys' Dani agreed, saying teams should have a workflow to approve and vet these packages. “That will take care of the package-based supply chain compromises arising out of the use of this attack," she said. Specific to RoguePuppet, Dani said, organizations should perform frequent CI/CD configuration checks to verify that they are resilient and tamper-proof.
Why trust is key to software supply chain security
One key takeaway from the RoguePuppet exposure is how important identification and authorization are to software security, said KnowBe4"s McQuiggan. “It's crucial that organizations secure their GitHub repositories with proper identification and authentication practices to prevent code from being exposed to change via an API call,” he said.
Organizations also need to focus on how software moves through their pipelines, said Exabeam’s Kirkwood. Shift-left, he said, should mean that as soon as a developer touches a piece of something that has been validated as a known good, it should be scanned. “It should be validated as a nonmalicious piece of code, that it's not going to inject some kind of backdoor or security defect into the repository."
“Don't trust anybody. Don't trust any code repository. Don't trust anything that's open source and available. If it's been touched by a human, it's subject to some level of bad behavior. Validate, scan, remediate, mitigate — do everything you can do as far into that pipeline as you can possibly go."
—Kevin Kirkwood
Khan discovered RoguePuppet with a tool he created called Gato-X, which is designed to detect GitHub Actions misconfigurations at scale. After reporting the flaw to Puppet, he said, the company mitigated the vulnerability the following day and revoked the API token. Over the following weeks, it fixed the vulnerability in dozens of repositories and added a new API token to repositories.
Callie Guenther, cyberthreat research senior manager at Critical Start, praised Puppet’s actions, noting how important prompt response are to mitigating the vulnerability.
"[It's] a commendable example of effective incident management."
—Callie Guenther
Kahn said in his analysis that such forced supply chain attacks are a big blind spot for the software industry.
“When combining ease of exploitation, breadth of impact, and the chance a threat actor would succeed, this is by far the worst Public Poisoned Pipeline Execution vulnerability I have ever discovered. It is shocking that this vulnerability made it into dozens of Puppet Labs repositories in the first place, but when you look at the GitHub CI/CD configuration of most companies it is not surprising.”
Companies often do not pay enough attention to CI/CD security, as it is not considered customer facing, Khan added. But he stressed that these systems often have a direct line to customer impact.
