In today's intricately interconnected and complex software development ecosystem, a single compromised component can trigger a cascade of security breaches across thousands of organizations worldwide. And the cautionary tales keep piling up: In just the past month we’ve witnessed the CrowdStrike incident, where a faulty “channel file” that was automatically pushed out to clients shut down millions of Windows computers, and the “RoguePuppet” vulnerability, which an attacker could exploit to add malware to any Puppet Forge module.
The increased complexity and interdependence of software ecosystems makes software supply chain security (SSCS) a critical concern that affects organizations of every size and in every industry. Everyone wants to crowdsource features to save money. Everyone is at risk because of our growing dependence on third-party software and because of the increasing frequency and sophistication of software supply chain attacks.
Unfortunately, most organizations aren’t prepared. In a ReversingLabs survey of 321 security and IT professionals, 98% said software supply chain issues pose a significant business risk but just 60% said their software supply chain defenses are up to the task of warding off such attacks.
There’s a way forward, but most organizations aren’t walking the path yet. With ReversingLabs' new guide, Software Supply Chain Security for Dummies, we lay out that path. Here's why you need to get started — and how to take your first steps toward a modern approach to managing risk from the software you build or buy.
From CrowdStrike to Puppet to 3CX: Three paths to a problem
The security issues surrounding software supply chain security are broad, but most revolve around three issues: misplaced trust in third-party components, the double-edged sword of automated updates, and challenges in visibility across complex software supply chains.
With the CrowdStrike incident, for example, the law of unintended consequences held sway. The endpoint detection and response company issued a problematic update to a “channel file,” which is part of the behavioral protection mechanisms used by CrowdStrike’s Falcon sensor. It was intended to be a hash update that contained signatures of things the sensor should be looking for. Instead, the update, when automatically installed on client machines, caused Windows to crash upon boot-up, resulting in the infamous “blue screen of death.”
While most organizations have remote support tools to help resolve problems like this, in this case the update loaded so early in the boot process that those tools didn’t have a chance to start up. That’s why technicians at airports were ascending 10-foot ladders to reach Windows machines mounted behind arrival-and-departure screens. They had to physically touch every machine to fix them.
Whether you were directly affected by this incident or not, you should be asking yourself: What are the implications of deep system access for my security software — especially when customers don’t have an option to stop the updates?
The Puppet Forge exposure, on the other hand, fell into the category of a CI/CD pipeline exploitation. It was a GitHub Actions workflow misconfiguration that had the potential for widespread compromise of modules. Much of what’s in Puppet Forge is crowdsourced; it’s a place where people create modules that typically don’t receive a thorough vetting.
Adnan Khan discovered the Puppet Forge vulnerability and wrote about it in his July 2 blog post. He has a tool that he uses to check Git repositories, and when working with it he discovered a flaw in how Puppet set up repositories. By exploiting this flaw, he could make changes and pull requests without anyone approving them.
Armed with this knowledge, he could have poisoned any module. In this case, if you don’t have Puppet pinned to a particular version, it will just pull in the module, which could be harvesting data or doing any number of other malicious things. Khan planned to release his tool as open source at Black Hat. Ask yourself: Could my organization have detected this issue?
Perhaps the most stunning incident was the 3CX attack, a software supply chain compromise in 2023 that occurred after North Korean attackers gained access to a 3CX employee’s administrator account. From there, they gained access by way of a virtual private network (VPN) to the company’s network and compromised the organization’s Windows and macOS build environments. Many companies that used 3CX’s voice over IP (VoIP) software had set the software’s auto update function on by default, so once the hackers gained control and pushed the modified update out, organizations just blindly accepted it. The impact was widespread among the company’s more than 600,000 business customers.
We’re told that updates are good, but that’s not always true, as any grizzled Windows administrator can tell you. So you need to change your mindset and ask yourself: Am I allowing auto updates to software without any vetting or testing on my end? Do I trust that the software vendor is performing testing that meets my standards?
The CloudStrike incident was especially pernicious because the architecture didn’t give the users a choice as to whether or not to accept the update. But with 3CX, at least customers could have chosen not to turn on auto updates. Those who waited a few days before accepting might have dodged a bullet.
Want to beat these modern supply chain threats? Change your thinking.
The best way to protect your organization against these types of potential threats is by changing your mindset about software supply chain security. Ask yourself: What am I running in my environment? What am I allowing for updates? Is there something that has low-level control over my systems that could blow up at any moment? What are people or processes putting into my systems, who’s responsible for those updates, and do those entities perform adequate testing before issuing an update?
As a software vendor you have a responsibility to not let your software become a vector for malware. And if you are a software buyer, you have a responsibility to do your homework. You can’t just blindly trust vendors and their updates.
Think about the software you’re using, the modules you’re adding, and the updates your system is accepting. Start thinking holistically about what you’re bringing into your environments, and whether you’ve done the due diligence required to ensure that the software isn't going to compromise your systems.
It’s not easy to stand by this mindset. In the past, people who adopted it were often ridiculed for being overly cautious. People made fun of them because they wouldn’t blindly accept updates, but Windows admins don’t do so, and you shouldn’t either. Fortunately, that kind of pushback is fading, but if it does come up, don’t hesitate to regale the criticism with stories of what happened with CrowdStrike, Puppet Forge, and 3CX.
For more on how to think through all of these risks, develop a new mindset around software supply chain security, and build an effective SSCS strategy, read our free guide, Software Supply Chain Security for Dummies — and drill down into Chapter 3: “Managing Third-Party Commercial Software Risk.”
