The attack on SolarWinds may be approaching its second birthday, but it was still a popular topic of conversation at this year’s RSA Conference. That's not surprising, as the SunBurst attack, which was behind the SolarWinds hack, is one of the most significant cyber attacks in history. But the continued focus on SunBurst isn't just because it was a widely publicized attack. The fact is: there is still a great deal that the cybersecurity community has to learn from this major incident.
In particular, the information security industry is using the attack on SolarWinds as a clarion call to "shift left" and better understand and protect the software supply chain, including continuous integration and delivery (CI/CD) systems and processes. For example, a recent survey of 300 IT professionals working at software enterprises found that 87% were aware that software tampering, as occurred in the SolarWinds attack, can lead to enterprise security breaches. Attacks on CI/CD pipelines were also a concern. Around 40% of those surveyed said CI/CD toolchain exposures pose a risk to their organization.
“In recent years we saw major breaches in CI/CD… Lots of companies - thousands of them - were impacted by many of these breaches."
—Omer Gil, Head of Research at Cider Security.
Gil and Cider CTO and co-founder Daniel Krivelevich used a talk at RSA to present the findings from a months-long project to identify the top 10 CI/CD security risks, based on recent supply chain attacks. Companies, he said, were starting to take note of risks to supply chains — an area that had been "completely overlooked ... in recent years by security."
Here are the top lessons from their presentation, which your software team can use to sharpen your security approach with CI/CD threats.
An analysis of major CI/CD breaches
The two told attendees that the conclusions of the project’s 40-page report (PDF) are clear. "The engineering train is moving faster and faster,”" Krivelevich said, pushed along by the embrace of DevOps and continuous integration and delivery within software teams.
Before looking at security risks for CI/CD, security teams need to better understand the engineering ecosystem. Rather than it being a clean, linear production line (in theory), it’s actually quite the opposite, due to its constantly evolving nature, making it complicated and large, Krivelevich said. This is why software practitioners need to “look under the hood” of the entire ecosystem.
This foundational understanding is what CI/CD security should be, Krivelevich said, “building visibility over the engineering ecosystem." That is where software practitioners need to start.
Where do software engineers stand now in making this a reality? "Behind," said Gil. A lack of knowledge and technology protects the CI/CD ecosystem across the industry is one reason for the delay. Also, organizations don’t have the resources to meet these needs. This has direct consequences on the industry and sets the stage for successful attacks on CI/CD pipelines, as Cider's breach case studies show.
Here are five top CI/CD security breaches — and how they happened.
1. PHP Git infrastructure compromise
Cider’s first case study was based on an incident that was discovered in March of 2021: a malicious version of the PHP repository was distributed to customers, causing a software supply chain attack similar to that of SolarWinds.
Here are the risks out of the top 10 associated with this incident:
- CICD-SEC-1: Insufficient Flow Control Mechanisms
- CICD-SEC-7: Insecure System Configuration
- CICD-SEC-9: Improper Artifact Integrity Validation
- CICD-SEC-10: Insufficient Logging and Visibility.
2. Stack overflow breach
Back in 2019, there was a leak of source code as a result of attackers gaining access to a old service account, allowing them to log in and misconfigure code in a Github repository.
Here are the top CI/CD risks:
- CICD-SEC-2: Inadequate Identity Access Management
- CICD-SEC-6: Insufficient Credential Hygiene
- CICD-SEC-7: Insecure System Configuration
3. CodeCov
This incident, which happened in 2021, is also considered to be one of the most famous examples of a software supply chain attack, next to SolarWinds. In this case, data was exfiltrated by attackers from the CI of a build, and thousands of organizations using CodeCov were impacted as a result.
This incident has many risks:
- CICD-SEC-5: Insufficient Pipeline-Based Access Controls (PBAC)
- CICD-SEC-8: Ungoverned Usage of 3rd Party Services
- CICD-SEC-10: Insufficient Logging and Visibility
- CICD-SEC-6: Insufficient Credential Hygiene
- CICD-SEC-9: Improper Artifact Integrity Validation
4. Travis CI secrets exposure
In September of 2021, a Travis CI flaw exposed the secrets of thousands of open source projects. API keys, access tokens, and credentials were leaked, putting organizations that rely on this open source code at risk.
The main risk:
- CICD-SEC-4: Poisoned Pipeline Execution
5. Dependency confusion attacks
Dependency confusion attacks have become a risk for many companies, including big ones like Microsoft and Apple. A white hat hacker published a proof of concept in which NPM repositories can be attacked via the dependency chain.
The big risk:
- CICD-SEC-3: Dependency Chain Abuse
CI/CD breach analysis takeaway: Shift your mindset
After assessing the various breaches, the speakers closed their presentation with a call for the information security industry to shift its mindset on application security - from the writing of code all the way through the production pipeline.
The two also noted that software practitioners need to understand what the full-scale of the attack surface is and could be. This requires constant and continued analysis of the full engineering ecosystem, while also keeping the top CI/CD risks in mind.
