While the notable software supply chain incidents of 2023 did not reverberate like the watershed attack on SolarWinds in 2020, they had plenty to teach security teams across software development, application security (AppSec), and risk management.
Software supply chain threats have spiked dramatically over the past three years (up 1,300%), and attacks continued to rise in 2023. The analyst firm Gartner found that almost two-thirds (61%) of all U.S. businesses were directly impacted by software supply chain attacks between April 2022 and April 2023.
Here are lessons learned from key software supply chain attacks that took place in 2023. Put these lessons to work to get out in front of risk at your organization in 2024.
[ See RL's Essential Guide: Software Supply Chain Security for Dummies ]
1. CircleCI raises red flags for secrets
In January 2023, software development service CircleCI discovered that malware had been deployed on an engineer’s laptop and was used to open a single-sign-on (SSO) session backed by two-factor authentication, which allowed the attackers to access the company’s internal systems. After discovering the breach, CircleCI began rotating all GitHub OAuth tokens on behalf of its customers.
Guy Rosenthal, vice president for product at DoControl, said one of the biggest lessons learned from the CircleCI incident was that even properly implemented multifactor authentication (MFA) can be defeated if an endpoint such as a laptop isn't secure.
"The takeaway here is that it’s not enough just to do MFA or just to secure endpoints, but that it must be assumed that each of these items can be defeated. Therefore, multiple security points and constant monitoring are critical."
—Guy Rosenthal
James McQuiggan, a security awareness advocate with KnowBe4, said organizations should understand the weak points of MFA and the importance of implementing non-phishable MFA for sensitive or critical systems.
"MFA session keys can be circumvented and stolen by cybercriminals. Using hardware tokens can reduce the risk of an attack and support other cybersecurity monitoring systems."
—James McQuiggan
The CircleCi attack also exposed the risk the human element can bring to software supply chain security, said Lorri Janssen-Anessi, director for external cybersecurity assessments at BlueVoyant.
"Cybercriminals will always try to come up with plausible solutions to get you to click on a link or download an attachment. Ensuring education, training, and then implementing a security program that has mechanisms in place to protect against this type of compromise is critical."
—Lorri Janssen-Anessi
Matt Rose, field CISO for ReversingLabs, said that good digital hygiene is important but that the CircleCI incident illustrates more broadly that organizations have to not only be concerned about malware injection and secrets leaks generally, but also of the tooling used to build software packages.
When assessing supply chain risk, the focus is on the artifact as it moves through the process, but risks can be lurking in the tools that make up the process itself. If all the testing is done on the artifact — whether it's software composition analysis (SCA), static application security testing (SAST), penetration testing, or something else — something will be missed unless the core competencies, such as IDE and CI/CD, are examined, Rose said.
If an organization looks only at the artifact and not the structure that builds it, that organization potentially has a software supply chain risk.
—Matt Rose
2. 3CX: Don't be complacent about tampering
Another lesson came in March, when 3CX notified its customers that the desktop application they were using may have been compromised in a software supply chain attack that let intruders plant, in a software update, malware that steals information from the computers it infects. Damage was minimized by 3CX's security provider, Crowdstrike.
The 3CX case is good example of why organizations need to use multiple tools and adopt a defense-in-depth approach to security, DoControl's Rosenthal said.
"By watching what applications are doing on endpoints and on the network and knowing what 'normal' looks like, we can detect and alert on anomalous behaviors."
—Guy Rosenthal
A ReversingLabs analysis concluded that 3CX was actually the victim of a targeted supply chain attack, not an opportunistic attack that exploited a vulnerability in a shared software library. It found discrepancies in 3CX's versions of two standard libraries used with the Electron open-source framework on which the 3CXDesktopApp client is built: ffmpeg and d3dcompiler_47.
The message for software vendors here: Companies cannot be complacent. In this case there was ample evidence of tampering with the desktop client updates that 3CX sent out. Even in the absence of warnings by endpoint detection and response, that evidence should have been enough to put a pause on distributing the update, wrote Paul Roberts, cyber-content lead at ReversingLabs.
With sophisticated actors increasingly interested in abusing the hard-earned reputation of software teams to distribute malware, vendors need to be on guard for signs that malicious actors are at work within vendors' development and build processes, Roberts wrote. That awareness may not stop compromises, but it will make it less likely that a software vendor's customers end up suffering the consequences of the company's security failings.
Karlo Zanki, reverse engineer at ReversingLabs, wrote that one key lesson is the need for modernizing security tooling. One requirement for detecting supply chain compromises is the ability to track the evolution of software packages through differential analysis of their contents, he wrote. This includes the raw metadata properties of each software component in the release, as well as their respective behaviors.
"Odd or inexplicable changes between builds should be considered a cause to investigate a possible compromise. This becomes even more important when software packages include components that are pre-compiled at offsite locations and, therefore, not subject to review prior to deployment."
—Karlo Zanki
[ See related post: A (partial) history of software supply chain attacks ]
3. NuGet is a nugget of takeaways
Some valuable lessons were learned from another attack in March. This time adversaries used sophisticated typosquatting techniques to target .NET developers using the NuGet repository. The poisoned packages were downloaded 150,000 times before the malware could be removed from the repository. The packages contained a "download and execute" payload, so as soon as they were downloaded, they automatically downloaded a second, more virulent payload.
Sarah Jones, a cyberthreat intelligence research analyst with Critical Start, said the NuGet attack illustrates why we need enhanced code signing, regular vulnerability audits, and efficient incident response. "Collaboration within the .NET community played a crucial role, stressing the significance of collective action, vigilance in dependency management, and prioritizing packages with community support," she said.
DoControl's Rosenthal said trust was compromised. "In DevSecOps, the devil is in the details," he said.
"It’s easy for developers to glance at the names and other metadata of the packages to select the desired code libraries, but, as this attack showed, that information can be misleading and even intentionally falsified to create trust. In the end, developers need to be careful in curating what code they use, taking time to verify the details before downloading and using. It also reminds us that automatic execution is not a great idea."
—Guy Rosenthal
4. MOVEit: A wakeup call
More key lessons were learned in the MOVEit attack in May. The assault on the transfer and cloud applications of the offering by Progress Software leveraged two critical SQL injection vulnerabilities, CVE-2023-34362 and CVE-2023-35036, that allowed the attackers to gain unauthorized access to customers' systems and steal sensitive data, execute malicious code, and manipulate or delete critical information. The attack affected more than 963 organizations worldwide, including prominent companies such as Shell, British Airways, Deloitte, and the BBC, and even government agencies in the United States and the United Kingdom.
Critical Start's Jones said MOVEit illustrated the need for prioritizing regular software updates, transparent communication between vendor and user, and thorough mapping of the software supply chain.
"The MOVEit attack served as a wakeup call for organizations relying on legacy software. The incident highlighted the interconnected nature of the supply chain and the importance of identifying vulnerabilities in all components."
—Sarah Jones
BlueVoyant's Janssen-Anessi said security teams aren't alone in learning from these kinds of attacks.
"In our annual supply chain survey of C-level executives, we found breaches like MOVEit led to 60% of respondents saying their budget would increase to protect against these type of breaches."
—Lorri Janssen-Anessi
More than anything else, the MOVEit attack shows that the full impact of this kind of breach will not be known for months or even years after it initially comes to light, said DoControl's Rosenthal.
5. HuggingFace: AI dataset poisoning gets smart
While many organizations grappled with artificial intelligence (AI) and its large language models (LLM) during the year, those technologies presented adversaries with more supply chains to attack, as the HuggingFace API compromise illustrates. HuggingFace is a popular resource for developers of LLM projects. It contains hundreds of thousands of AI models and datasets in its repository. At the end of 2023, cybersecurity researchers at Lasso discovered more than 1,600 leaked tokens that opened up access to the accounts of 723 organizations, including Google, Meta, Microsoft, and VMware.
Steve Wilson, chief product officer at Exabeam and project manager for the OWASP Top 10 for LLM, said poisoned ML models exposed a bigger software supply chain security concern.
"Hugging Face highlights the risks of tainted ML models as a supply chain vulnerability. This was a wakeup call regarding the potential for malicious actors to manipulate machine-learning models, turning them into vehicles for cyberattacks."
—Steve Wilson
Wilson said the detection of zero-day vulnerabilities in essential MLOps tools such as PyTorch "underscored the fragility of the AI supply chain."
"Such vulnerabilities in widely used tools can have far-reaching impacts, affecting numerous applications and systems built on these platforms."
—Steve Wilson
Put these lessons to work
Wilson called the AI supply chain "a war zone," where threats are constantly evolving, because traditional AppSec testing tools such as SCA and the focus on vulnerabilities using the CVE database, while valuable, "are still adapting to the unique challenges posed by AI supply chain security."
"The taxonomy around AI vulnerabilities is still developing, indicating that we are in the early stages of effectively understanding and combating these threats."
—Steve Wilson
Jeff Williams, CTO and co-founder at Contrast Security, said there were general lessons security teams can learn from 2023's class of software supply chain attacks.
"Hopefully, we learned that software supply chains are complex — an entire ecosystem of tools and technologies that have traditionally been ignored by both security and hackers. But that has all changed as hackers have realized the power of attacking supply chains."
—Jeff Williams
Williams noted that there are four major categories of supply chain targets: software you write, software you import, software you buy, and software you use to build software.
"Organizations should start with the basics. Think through the threat model, limit the breadth of the ecosystem, establish controls to minimize vulnerability, and establish infrastructure for fast detection and response."
—Jeff Williams
With software supply chain risk front and center for CISOs, the lessons learned from 2023's crop of software supply chain attacks need to be learned fast, as even more adversary activity is expected in 2024.
