Enterprise organizations in recent years have come to recognize that attacks targeting software supply chains are a major threat. But the focus has been on attacks involving open-source software, since commercial software is a black box for many enterprises.
Cybersecurity incidents such as the one that SolarWinds disclosed in December 2020 have become increasingly common — as have vulnerability exploits used against trusted vendors and attacks on organizations handling enterprise data.
Here are five major commercial supply chain security incidents from the past year — and the lessons they offer for security stakeholders.
[ See Special Report: How to Manage Commercial & Third-Party Software Risk ]
1. Sisense serves up a lesson on credential protection
An April 2024 breach involving business intelligence and data analytics services provider Sisense highlighted the risks to organizations from trusted third parties that do not properly secure customer credentials and secrets.
The incident occurred when a threat actor accessed Sisense's self-managed GitLab instance and found a hard-coded token that provided access to the company's Amazon S3 cloud storage buckets, allowing the attacker to steal multiple terabytes of data. The pilfered data included millions of tokens, passwords, and other secrets that customers were using to access Sisense's services and to connect their Sisense instances with third-party cloud providers such as Google, Box, and Salesforce.
The stolen credentials gave attackers potentially extensive access to sensitive data belonging to some Sisense customers, including a few in the critical-infrastructure sector of the United States. The incident prompted the U.S. Cybersecurity and Infrastructure Security Agency (CISA) to issue an advisory that echoed Sisense's advice for affected customers to immediately reset the credentials and secrets associated with their Sisense accounts.
Trusting data within third-party vendors continues to be a business necessity for many organizations, said Malachi Walker, security advisor at DomainTools, but they can take proactive measures to mitigate supply chain risk.
"If the supply chain is a trusted extension of one's environment, it makes sense for threat modeling to encompass what happens if one of those organizations is compromised."
—Malachi Walker
This should include doing security profiles of vendors before agreeing to trust them with data and then regularly monitoring them afterwards, Malachi said.
The lesson here: Do not hardcode keys and other credentials into code repositories, said Paul Bischoff, consumer privacy advocate at Comparitech.
"This is doubly important for public repositories. Our honeypot experiment showed us that it takes attackers less than one minute to find and abuse exposed credentials on public GitHub repositories."
—Paul Bischoff
2. JetBrains vulnerabilities highlight third-party CI/CD risks
The discovery of multiple critical vulnerabilities in JetBrains' TeamCity continuous integration and continuous delivery (CI/CD) server technology over the past year underscored the importance for organizations to ensure proper access control and oversight of their software development environment.
The vulnerabilities included CVE-2023-42793 in October 2023, two vulnerabilities (CVE-2024-27198 and CVE-2024-27199) in March 2024, and another, CVE-2024-37051, in June 2024. Though the vulnerabilities affected different components and had varying impacts, they all exposed enterprise software development environments to serious risks. The potential consequences of vulnerability exploits included credentials and secrets theft, the surreptitious introduction of malicious code into development and build environments as in the SolarWinds case, and attackers establishing a foothold for further compromise of a network.
One example of the risks the vulnerabilities presented was a campaign by APT29 (a.k.a. CozyBear, Midnight Blizzard, and Nobelium), a threat group affiliated with the Russian Foreign Intelligence Service, that targeted CVE-2023-42793. As CISA mentioned in an advisory on the campaign, APT29 exploited CVE-2023-42793 to escalate privileges, move laterally, deploy additional backdoors, and take steps to establish persistent access to compromised network environments. In a sign of just how popular such vulnerabilities are for attackers, Microsoft reported observing multiple North Korean groups trying to exploit the same flaw to try to infiltrate enterprise build environments.
CISA and the National Security Agency issued an advisory last year that offered recommendations and best practices for organizations to secure their CI/CD pipelines, saying:
"CI/CD environments are attractive targets for malicious cyber actors. [The] goals are to compromise information by introducing malicious code into CI/CD applications, gaining access to intellectual property/trade secrets through code theft, or causing denial of service effects against applications."
CISA's recommendations to organizations for mitigating such threats include implementing a zero-trust model for controlling access to CI/CD environments; using strong encryption to protect data, secrets, keys, and APIs; and minimizing the use of long-term credentials.
Other recommendations include using secure code-signing practices within the CI/CD pipeline, requiring two persons for all code updates, and implementing least-privilege access. Suggested development process mitigations include integrating SAST, DAST, and registry scanning; restricting the use of untrusted libraries; and analyzing committed code.
CISA and the NSA wrote in the advisory:
"The CI/CD pipeline is a distinct and separate attack surface from other segments of the software supply chain. [Malicious cyber actors] can multiply impacts severalfold by exploiting the source of software deployed to multiple operational environments."
3. CrowdStrike: Not all supply chain incidents are maliciously motivated
A faulty content update for CrowdStrike's Falcon endpoint sensor product crashed more than 8 million Windows systems worldwide and caused widespread disruptions to airlines, hospitals, stock markets, retailers, government services and others. The July incident highlighted the risks with having highly interconnected infrastructure and startling gaps in incident response and recoverability capabilities on a global scale. Importantly, it also showed how not all supply chain risks have to do with malicious intent.
The CrowdStrike snafu had to do with a mismatch between the content configuration update and what its Falcon sensor technology was expecting. The issue caused Windows systems to go into a Blue Screen of Death state that required a painstaking and time-consuming recovery. The incident generated widespread concern and scrutiny, including from U.S. lawmakers, who wanted to know what exactly happened and how to ensure there would be no repeat.
CrowdStrike itself made multiple changes to its content update process. Instead of automatically deploying updates as it did before the July snafu, the company now gives customers the option of choosing when and how they want to receive updates.
Roger Grimes, data-driven defense evangelist at KnowBe4, said such incidents highlight how computer security is all about trust. "Each big supply chain incident erodes that trust. It doesn't even take a security incident to erode the trust, as the recent CrowdStrike downtime incident revealed."
Grimes said the time is now for the industry to consider the equivalent of a software bill of materials (SBOM) for mitigating supply chain risks. SBOMs give organizations a way to inventory every component of a software or firmware product and theoretically at least enabling better risk management in the process.
"[Supply] chain risks are begging for the same sort of solution, where every vendor you use provides some sort of SBOM so consumers can at least be aware of what risks they are assuming when they use a particular vendor."
—Roger Grimes
4. Okta breach highlights service account credentials risk
A breach at Okta last October hammered home the importance for organizations to properly manage and secure service account credentials used by automated services or applications to access resources and to interact with systems without human intervention. It also highlighted the need for controls such as multifactor authentication, least-privilege access, and identity monitoring to protect against the fallout from a service account-related compromise at a trusted vendor.
The breach at Okta resulted when a threat actor compromised an employee's laptop and installed malware. The laptop contained a service account credential that the malware leveraged to illegally access Okta's support case management system. The system contained HTTP archive (HAR) files holding sensitive information that Okta's customers had uploaded to help the vendor troubleshoot issues. The attacker extracted from the HAR files session tokens belonging to many of Okta's customers, including BeyondTrust (which first reported the issue), CloudFlare, and 1Password. They then attempted to use the tokens to gain access to the compromised customer environments.
As BeyondTrust noted in a post-mortem of the attack, the incident highlighted why organizations need to manage, control, and audit use of their service accounts:
"Because most service accounts are a special type of non-human privileged account, the best approach to tackling their security is two-fold: first, you need to identify and bring all accounts under centralized management."
5. XZ Utils puts spotlight on need for stakeholders to support of OSS projects
A backdoor in the XZ Utils data-compression utility for major commercial Unix distributions served as a sharp reminder that many commercial software products contain risky — and often barely maintained — open-source components.
The backdoor, tracked as CVE-2024-3094, gave attackers a way to inject malicious code to the OpenSSH server on victim machines and bypass password-based authentication or send arbitrary payloads to affected systems.
A researcher from Microsoft discovered the XZ Utils backdoor in March 2024 and reported it to the maintainers of Debian, Fedora, openSUSE, Arch, and Kali, who promptly fixed the issue. Fortunately, the backdoor was present only in two newly updated versions of XZ Utils and therefore only affected beta and unstable versions of the respective operating systems.
So the community as a whole managed to mitigate the threat before the backdoor was widely deployed. Many security researchers believe the consequences would have been significantly higher if the backdoor had found its way into stable releases of the operating systems. Several vendor-supported, Unix-like operation systems, including macOS, include XZ Utils.
The incident was particularly troubling because it was a supposedly authorized maintainer of XZ Utils that introduced the backdoor into the widely used utility. The person or persons behind the caper worked patiently over a three-year period to first earn the trust of the sole — and overworked — maintainer of the XZ Utils project and then acquire the administrative privileges to make changes to the tool. The individual then used those privileges to introduce the backdoor into the utility.
Comparitech's Bischoff said the XZ Utils incident highlighted how organization invest too little into open-source development and code audits. Instead, we take open-source developers for granted and often implicitly trust them, Bischoff said. Fortunately, in this case, the broader community spotted the malicious behavior before it mushroomed into a major incident, he noted.
"XZ Utils highlights the contradiction that many organizations face: Software updates fix security vulnerabilities but can also introduce new ones."
—Paul Bischoff
CISA perceived the incident as highlighting the fragile nature of the open-source ecosystem. The incident, it wrote, was an example of the "very real and ongoing risks created by maintainer burnout, and the enormous benefits realized through open collaboration as demonstrated by the communities’ response," CISA said. "This compromise highlights a fundamental shift needed: every technology manufacturer that profits from open source software must do their part by being responsible consumers of and sustainable contributors to the open source packages they depend on."
Jeff Williams, co-founder and CTO at Contrast Security sees the incident as a reminder to enterprises about the third-party code in the applications and APIs they use.
"Typical projects use hundreds of these libraries, often without even knowing."
–Jeff Williams
Major attacks show commercial software is the principal attack surface
A recent white paper highlights that commercial software platforms pose a much greater risk to organizations than open-source software. That's because of the trust assigned to software publishers by their customers, but also because of the complex nature of third-party software supply chains, at a time when third-party SaaS providers and cloud-based infrastructure are proliferating.
The broad outlines of the software supply chain security problem are visible in the 2024 Verizon Data Breach Investigation Report, which said that breaches stemming from third-party software development organizations played a role in 15% of the more than 10,000 data breaches Verizon documented. That's a 68% jump from last year’s DBIR.
The increase was serious enough that Verizon introduced a new metric for tracking the growth of exploitation of vulnerabilities and software supply chain attacks. Additionally, Verizon called on organizations to “start looking at ways of making better choices” about the software providers they work with “so as to not reward the weakest links in the chain.”
