Rampant lapses in software supply chain security don't manifest suddenly. They build up over months and years, one out-of-date component, overly permissive account, or misconfigured API at a time. And over time, these gaps mount up, like bad credit card debt on the ledger of supply chain security.
Each charge against security technical debt may seem inconsequential, and many are incurred on purpose — "just for now" — so the DevOps team can speed up the next minimum viable product release or high-priority feature request; they figure they'll deal with it at some future date. Most of the time, though, teams never get back to those security problems.
Andrew Barratt, vice president at the cybersecurity advisory service Coalfire, said it is arguable that almost all application security lapses come down to technical debt you were either aware of "or didn’t know you didn’t know."
"[Supply chain security technical debt] usually stems from a complete lack of refactoring in a highly agile environment."
—Andrew Barratt
The problem, of course, is not with the individual charges against security technical debt; it's in the accumulation. Just as with a high-interest credit card, there's a compounding factor to security technical debt as it's racked up. For technical debt, compounding interest comes in the form of software supply chain risks.
The potential impact of vulnerabilities grows more serious when each one is combined with other flaws in the supply chain, and the likelihood of exploitation increases as the volume of flawed components mount across a supply chain. And the more that organizations rack up that security technical debt, the harder it becomes just to pay down the interest — let alone the principal.
If organizations want to get a handle on software supply chain risks, they need to understand how security debt grows and what it will take to start paying it off the smart way before they face a "bankruptcy" situation — be it regulatory noncompliance or a costly breach.
[ Learn more: Software Supply Chain Risk Report: Tools Gap Leaves Orgs Exposed ]
How supply chain security debt builds up
Tom Molden, CIO of global executive engagement for Tanium, said technical debt is the result of trade-offs that have been made over time, either intentionally or without choice.
"Weaknesses in the software supply chain are often attributable to such decisions. The way most companies manage technology generally favors the gradual increase in technical debt and, consequently, more exposure in the software supply chain."
—Tom Molden
This is not a blame-the-developer story. Most devs contribute to technical debt because of business decisions made for them. They're just trying to make the best decisions they can under the time and budgetary constraints put in front of them.
Molden shared the example of the decision in a budget cycle to prioritize a business innovation technology over an investment in the underlying infrastructure. Such a business decision can result in an exposure whereby the modern application code does not adequately compensate for something in the legacy infrastructure it is running on.
"Software developers are always having to compensate for suboptimal conditions in which the software is going to run, and it’s easy to overlook something."
—Tom Molden
The old rule of "fast, cheap, good — pick two" applies as strongly as ever to software development today, with security functionality falling under the "good" header. When the business tells them to, though, development teams will consider "good enough" to be known insecure functionality — if they can still tick off the "fast" and "cheap" checkboxes, Barratt said.
"So, you’re shipping product fast, getting releases out ahead of time, but then suddenly you are a victim of your own excessive success in getting product to market and find that you’re not spending the necessary time refactoring code and threat modeling the threats across libraries, and your minimum viable product is now in production and has more Band-Aids than a box of Band-Aids."
—Andrew Barratt
Matt Rose, field CISO for ReversingLabs, said organizations that have built out aggressive DevOps and CI/CD programs without a strong application security component are struggling more than ever to keep up with the volume of security issues that keep rearing their heads. Rose said this "victim of their own success" situation is common on agile teams that depend on and reuse of open-source components as a way to further accelerate their output.
"Applications are getting released faster than ever, and they're more complex than ever, and that speed has given limited time to address the technical debt that you're actually creating."
—Matt Rose
The burden of supply chain security debt is heavy. The recent Software Supply Chain Security Risk Report, based on a survey conducted by Dimensional Research of 321 IT professionals and published by ReversingLabs, found that nearly 90% of practitioners have detected security issues in their supply chain in the last year. And 88% say software supply chain security presents enterprise risk to their organizations.
Rose says software development is like a social media-driven platform, where everyone wants the latest and greatest features, the best look and feel, and the newest capabilities to keep their users and business stakeholders happy and interested in the product. But all of that comes at the cost of technical debt — security or otherwise.
"They're like, 'We'll get to that at some point.' Usually, the feature-function debt gets paid down more than the security stuff, because functional issues are more apparent and likely to be iterated on. And so teams keep pushing down the technical debt associated with security patches, vulnerabilities, or supply chain elements."
—Matt Rose
Pay down security debt principal vs. interest
Most organizations today are unfortunately only barely paying down the interest of software supply chain security debt. They engage in reactive vulnerability management or isolated app sec testing, but they never quite start paying down the principal by addressing underlying issues that create debt. One example Rose gave is a team seeking out a single CVE across the application portfolio over and over without ever recognizing that it keeps appearing because of heavy dependency on a single component that's reused again and again across the supply chain.
"It's a balloon mortgage, and you're only paying the interest — you're not paying the principal at all."
—Matt Rose
Organizations have to get smarter about tackling the problems that incur the debt in the first place. Of course, this starts with underlying fundamentals such as developing with security-by-design principals, such as those advocated by CISA.
David Lindner, CISO for Contrast Security, said that product security teams are a core strategic building block for establishing that long-term security-by-design ethos.
"Security by design emphasizes integrating security considerations into every phase of a product's lifecycle, from its initial design and development to deployment, maintenance, and eventual retirement."
—David Lindner
In immediate, in-the-weeds, tactical terms, there is also the matter of paying down existing security debt, said Coalfire's Barratt. The most dramatic tactical moves to pay down security debt are very specific to the product at hand, but there are some key places to look first.
"This is highly product-specific. I would spend time looking at interface boundaries, typically where data is or can be consumed between libraries or even between systems where there is a risk that manipulation could take place and unintended results happen leading to security vulnerabilities."
—Andrew Barratt
Establishing a mature supply chain security program that pays down security debt on a regular basis also requires taking a holistic and systematic approach to evaluating threats across the entire supply chain, he said.
"Systematic understanding of everything and the ability to systematically evaluate threats across your software supply chain are important. Software bills of materials (SBOMs) are helpful here, but really tools are needed that identify all the threads of an application supply chain so that fixes can be applied and mass analysis can be performed quickly."
—Andrew Barratt
ReversingLabs' Rose said organizations too often are only looking at a few pieces of the 10,000-piece jigsaw puzzle that is today's software supply chain risk.
"I don't think as an industry we're really looking at that whole picture. And that's where the technical debt really comes in — you're just looking at pieces. As an industry, we're still looking at pieces of the puzzle. The alphabet soup of application security testing solutions — SaaS, DaaS, IaaS, SCA, RASP, API scanning — are all looking at a certain area or fraction of the application."
—Matt Rose
The Software Supply Chain Risk Report noted that 74% of IT and security professionals say those fragmented app sec tools — SAST, DAST, and SCA — aren't adequately protecting organizations from supply chain threats.
Rose explained that organizations need to work smarter to pay down debt by focusing on the security issues that are both likely to be exploited and pose the highest impact based on how they're implemented within an application.
"The supply chain risks that you're talking about don't really propagate themselves until the whole package is put together. And that's what I'm really screaming about: How are you testing? You have to be able to take an educated guess based on time and resources available to measure the impact and likelihood to focus the attention on the technical debt that is most important to security risk."
—Matt Rose
