The Past, Present & Future of Software Bills of Materials

In this episode, host Paul Roberts chats with Beau Woods, Founder & CEO of Stratigos Security, about the history of the software bill of materials (SBOM) – from its beginnings, to its modern-day use, to efforts underway to adapt it for the future. 

EPISODE TRANSCRIPT

PAUL ROBERTS: Welcome back to another episode of the ConversingLabs podcast. I am your host, Paul Roberts, the director of editorial and content here at ReversingLabs and, of course, host of the ConversingLabs podcast. Great to have you back with us here in the studio.

I am thrilled to welcome Beau Woods, who is the founder of Stratigos Security and a Cyber Safety Innovation Fellow at the Atlantic Council, a cyber safety advocate group [called] I Am The Cavalry. His work bridges the gap between security research and public policy — critical these days to have somebody who's capable of doing that.

And he works to ensure the connected technology that can impact life and safety is worthy of our trust. Sadly, much of it is not. More recently he served as an Entrepreneur in Residence with the U. S. Food and Drug Administration. Beau has also been among the early and most vocal advocates of software bills of material, or SBOMs.

That's going to be a topic that we are going to talk about today. Beau, welcome.

BEAU WOODS: Yeah. Thank you for having me again. This is great. Good to be back. And especially post-Hacker Summer Samp. When I'm like relaxed, I think I had one of the best and most enjoyable Hacker Summer Camps in the past, I don't know, decade or so. Which was ... it was nice. It was relaxing.

PAUL ROBERTS: Let's talk about Hacker Summer Camp seeing as we're a couple weeks out from it.

How was the show? We're talking about DEF CON and Black Hat out in Vegas, BSides as well.

BEAU WOODS: BSides for those who don't know is I describe it as like the hacker family reunion. It's where you get to go to hang out with your friends, but there's also space there to have different types of conversations. I Am The Calvary has a full two-day track within BSides where we talk about things that are extremely relevant and increasingly relevant around internet of things. This year was a lot on food supply on water, wastewater. On healthcare hospitals and the extant and imminent threats to those from adversaries. So it was deeply substantive, highly interactive.

All of the sessions are recorded. And so you can go and watch them if you would like. But that for me is a place where I go and I put my thinking cap on and I go and engage with people.

So that was a good lead into the week. And then DEF CON is just a massive show. This year it was in a new venue, which for the first time in a very long time, everything, or at least almost everything was in one spot.

So the Las Vegas Convention Center, they finally made the move to the massive space to be able to handle the crowds. And I have to say- The fact that I didn't have to go outside, the fact that there were no bottlenecks and lines, the fact that everything was just there and more or less self contained, that was really nice. I really enjoyed it. And it didn't feel too much like a showroom floor they did have the vendor area, which they always have, which is the little small organizations, typically small businesses that sell directly to the hacker community. And like No Starch Press is there with books.

And you've got [what] one of my friends was selling, cat sniffer, which is like these little open source hardware/ software defined radios. And actually I saw Linus tech tips just did a video on it, which is pretty cool. I'm sure that his site is blowing up right now. So like they've always had the vendor area, but this year they had something called, I think, exhibitors, which is more like Microsoft or Google or, whoever you're going to have that would typically be at Black Hat.

This year they brought some of those over to DEF CON and started that. But it still didn't feel like a showroom floor like CES or something,

PAUL ROBERTS: Yeah, no, it was a good show I thought and some really important talks there. As you and I were talking about before it started, I think a little bit of a smaller audience this year, a little bit of smaller attendance.

But, I thought it went well. So let's just talk about you. You and Josh Corman and I Am The Cavalry were among the first groups really to start talking openly about the need for SBOMs, years ago. Talk about how that came to be and your introduction to this concept of the SBOM and why you think it's valuable.

BEAU WOODS: Yeah, sure. I Am The Cavalry, for the folks on the podcast who aren't aware you can go and look us up, but the very short version is that we say that our dependence on connected technology has grown much faster than our ability to secure them in areas impacting human life, public safety, national and economic security.

So the very shorthand version is where bits and bytes meet flesh and blood, right? We've traditionally in information security, cybersecurity tended to focus on threats to data confidentiality, intellectual property, credit card numbers, social security numbers, personally identifiable information, private health information, protected health information.

But much less so on the consequences of failure that are irreparable, irrecoverable. So things like when hospitals go offline or particularly medical devices are unavailable, so a threat to the system integrity or availability, then you can't treat patients and that has real world outcomes that are very negative.

And that was started by as you mentioned, Josh Corman and Nick Percoco in 2013. I jumped on board really quickly.

But that was the first time I ran across the concept of a software bill of materials. But it was not a new concept even in 2013. The idea of the software bill of materials is that you can list out all the ingredients that go into the software or the hardware product that you're putting out there.

Now, an ingredients list is an oversimplified example. But for people who don't know the concept very well that starts to get you closer to what it is, which is different from a nutrition label, which is a whole separate thing. And where SBOMs had already been well established was in places like the aerospace industry, where they have very low tolerance for any problems and a very high expectation of sound engineering practices. It was also used very frequently in financial services and in some of the first conversations about software bill of materials, some of the large financial services companies talked about how they used SBOMs to ensure that the vendors that were selling them products actually themselves knew what was in the products.

So just by the fact of asking for a software bill of materials, you can cause that market to change. The company to have to respond and say, yes, we actually do know what's in our software. And later on when working with a bunch of medical device makers in private once the Mayo Clinic started requiring them to have software bill of materials and the FDA started pushing them to have software bills of material, some of those medical device makers admitted that they actually didn't know what was in there, that the software that they got was more or less a binary blob that they had from whoever had developed it and moved on to a different team or through whatever acquisition that they had in the past.

And so the act of trying to do some forensics on their products caused them to realize that they had wildly outdated and insecure software that they in fact, some of them end of life products because they could no longer support it. It was just an impossibility to support those products.

All because people started asking for this transparency during the procurement cycle or asking companies to attest the fact that they actually knew what was in their software, including the packages, the dependencies, the versions of those. Any of that information. It's an extremely powerful concept that even if it's not used by the company that's buying the software at all it can be a powerful force to improve security throughout the supply chain and chain of operations.

PAUL ROBERTS: Yeah, this has been, I think, a huge kind of change in thinking in the past decade and a half, starting to ask uncomfortable questions about what's in the software that we're all using and relying upon that we're running critical industries on, critical infrastructure on and so on.

How have you seen the kind of thinking about SBOMs change, just in the time from when you and Josh, I Am The Calvary and those other folks were floating this idea out there. And there was a lot of pushback and a lot of resistance to it, to where we are now, where you've got CISA, NIST actively promoting the idea of SBOMs?

BEAU WOODS: Some of the first conversations around software bill of materials that I was involved with were with congressional staffers, particularly the House Energy and Commerce Committee. Talking to them about this idea of software transparency through a mechanism called Software Bill of Materials.

And in fact one of the, I forget which senator it was or representative, but somebody introduced a bill into Congress to require software bills of materials for things sold to the government. And it was fascinating to watch because for such a fairly straightforward and simple idea. There were a wide variety of reasons why organizations pushed back.

All that this bill said, I think was something like you have to provide transparency and it provided some of the fields like the name, the version number, and maybe a couple of others.

But companies just, many of them were fine with it, but many of them were besides themselves. They said that it was an impossibility- you couldn't possibly track all the software that was used in producing a product. They said that it was intellectual property, that knowing what third party libraries you're importing on open source is somehow secret sauce. They said that it would be prohibitively expensive that in order to know that you'd have to spend millions and millions of dollars extra, just all of these, many of them absurd criticisms of the idea. Now fast forward more than a decade and a lot of the people who were early at the table criticizing and saying that it was impossible, their engineers are at the table saying, oh yeah, we've been doing it for 15 plus years.

I don't know why, you heard why people said that it was impossible. And so it's like this sweep of an idea. And I don't remember who came up with the framework. But somebody said that for any new idea that's sufficiently different, sufficiently advanced you have a few stages before you get to adoption.

One is ignoring it. And once you can no longer ignore it, then you say it's not possible. And then once you get past that you say that it's just hard or unfeasible. And then I think the fourth one is that yes, but it was my idea all along. So we're now at the the stage where most people are at the, yes, it was my idea all along stage or just straight up adoption, which is fine.

It can be scary. It can be hard, particularly if the software that you're making, you have a cycle where you build it once and you think you're never going to have to support it or the support is going to be just small incremental things, where you're going to have to tweak a couple of lines of custom code, but you'll never have to update any of the libraries.

And I think that by-and-large across most of the industries we've seen those practices are no longer considered effective. They're no longer considered best practices. That shouldn't really be surprising. A lot of the principles of this materials transparency, knowing your supply chain, go back to some of the lean manufacturing processes started in the 50s by W. Edwards Deming and his wife. And they were radically transformative for many manufacturing industries, most notably the automotive industry. And not because it was imposed upon them, but because it dramatically improved quality, reduced cost, reduced safety issues on the manufacturing floor, reduced liability in the real world when the products went to market.

And so a lot of these practices are very well founded. They have a long track record. It's just figuring out the little engineering tweaks we need to do to let them apply to software. So they're not impossibilites. They're not even infeasibilities. In fact, I would wager that you would have very few organizations that would come up to you and say openly, we have no idea what's in our software. We don't track that. We don't know how to support it. We can't do any of those things these days.

PAUL ROBERTS: It's true. And on the other hand certainly I think the standard or the status quo of software, the way it's designed, developed, deployed, and implemented is different than other industries, right? And we all make analogies between, automotive industry or food production and stuff and make analogies to the software space.

And there are some real differences. There is a lack of transparency that is hurting companies. We read about it every day. How that manifests itself, that lack of transparency. Story today about Volt Typhoon exploiting an 0-day in some software that's used by major ISPs and in the U.S. and other countries.

And using that for a targeted attack and probably espionage and other things.

BEAU WOODS: Yeah. And you're not always going to catch the zero days with an SBOM. In fact, SBOMs are not meant at all to catch zero days. They're by de facto by default meant to catch things that have a much longer timeline, things that are already known and have fixes available for them by and large, which is the vast majority of the issues that we face, particularly in embedded systems, particularly in IOT, particularly in automotive and aerospace,

in medical devices and in energy sector... because the development, deployment, implementation, operation, maintenance timelines are so much longer in those industries. You tend to have the long tail of vulnerabilities, which makes it really tough to do something about it, right? If you've got power plants, for instance, some of the equipment that they buy, they expect to live for 20 to 40 years.

And they do return on investment calculations over that lifetime. In the face of a changing environment and a face of changing components, because software wasn't always a part of those systems. And in the face of an adaptive adversary, you have to change how you do things.

And there are different models of operation that this environment will cause organizations to have to move to, and that can be scary, but it's also necessary. And I think that a lot of organizations, particularly the younger organizations have realized that. In fact, one of the things I did over the past decade is I worked for the FDA for a year on a pilot project called the Software As A Medical Device Pre Certification Program, and without getting into all of the wonky regulatory details of it, we brought in, I think it was nine companies across the spectrum who make pieces of software that are medical devices, and it was interesting having them in the room where one organization, typically an older organization would say, Oh, this type of software development- it's not possible. It can't be done. And some of the younger ones who were more recently founded, brought up on modern software practices were like, how can you develop software without doing that thing? So there's these diametrically opposed positions that I think, with more learning, education, experience over the last 70 plus years now on computer software we're figuring out better ways to do things and you have to adapt.

PAUL ROBERTS: No, it's a good point. And we do in the cybersecurity industry and the industry generally, we do tend to focus on things like zero days, previously unknown vulnerabilities, when in reality the truth is a lot less dramatic and sexy than zero days. It's about old, known vulnerabilities and old components, things that an SBOM can call out like, Hey, you're using this, 10 year old open source library.

There are a whole bunch of known issues in that and so on. Or just giving organizations an idea with an example people use a lot as Log4j. Where is it in your organization? What applications and services that you use are dependent on this library or not dependent on it.

And so you don't need to worry about them. Very hard for companies to answer that absent something like a software bill of materials.

I kind of dial in occasionally to the SBOM working groups. One of the things that CISA runs, Al Friedman, great job, but one of the things that it seems like we're dealing with now is, as a practical matter, is how to make SBOMs actionable and not merely a kind of compliance checkbox that gets checked, but without any real meaningful impact on cyber resilience or readiness, right?

Okay, we got a PDF of the SBOM and so check, we've fulfilled that requirement. And it's sitting in a folder. Nobody ever looks at it. It's not part of our incident response planning or thinking in any way. I think everybody wants to avoid that outcome and actually have these be useful and actionable. I'm interested in your thoughts on what the best way to do that might be to actually get organizations not just generate SBOMs or consume them, but to actually make them part of their process?

BEAU WOODS: I hear that question a lot about, no one can use it right now, therefore we shouldn't do it. Or, all kinds of objections to SBOMs or like challenges. Here's the one thing we need to unlock SBOMs is the, this thing. And in reality, I think there's a lot of different sets of value that SBOMs bring to the table, before I mentioned that just the act of asking and requiring SBOMs and the legal attestation against the SBOM, yes, this SBOM is accurate to the best of our knowledge, can provide a huge amount of value. Without even the company doing anything with it.

I might reframe the question, which is how can we increase the value that SBOMs bring, or how can we continue to optimize the value or get more from SBOMs than just the act of receiving it as a part of your checklist, putting it in a filing cabinet somewhere and doing nothing else with it.

And I think one of the things that you're getting to is the idea that when there is a new vulnerability discovered and announced that companies have a problem understanding if they're affected, where they're affected and how they're affected. So do we even have this open source closed source library in our production environment or is it something that we've managed to avoid either deliberately or intentionally?

Because one of the ways that you can use SBOMs is to say, we will never have this certain component in any of the software that we buy. And some of the financial services institutions have actually done that. So if you provided a software bill of materials and it says you've got this component in there.

They'll say, can we disable it? Can we turn it off? Can we remove it from this system? Because we have said that we don't want that. That's knowing if you're affected. Where you're affected, secondly, is which of my systems, technical systems, have this piece of software?

That is a fairly easy thing to do through looking at machine readable software bills of materials. So not the PDF that can't be OCR 'd but something more like a JSON file or any of the standardized formats that exist that are mutually translatable amongst each other that can help you understand the technical aspects.

And then how am I affected? What business processes to those map to? Is it my number one revenue and if that system goes down for an hour, I lose millions of dollars or is it, just this weird little site that I can turn off in the meantime? And if you need to go deeper, so let's say it does affect your mission critical systems, the things that are generating millions of dollars an hour or keeping people alive- is it within an exploitable pipeline? Is it an exploitable pathway? You can do that analysis. Now, those things are not part of an SBOM, but without an SBOM, you can't do them at all. So if you want more options, if you want more capabilities, if you want more possibilities in order to stay online or to maximize your revenue and to maximize your security, optimize your security and risk, then you need that transparency.

You need some kind of a software bill of materials. Even if it's a rudimentary one that just lists the product and components, even that can be helpful when you're in a crisis, when you're in a vulnerability investigation, trying to figure out whether, where, and how you're affected.

PAUL ROBERTS: There's a lot of folks who worry that, we're shying away from the hard ask for software producers, particularly those, again, selling into the federal government, but also private sector or local government and so on, of asking for them to produce one of these. The cynical folks, cynical voices out there say the reason behind that is that there's a lot of pushback from industry because there are a lot of skeletons buried in those binaries.

Really old files, vulnerable files, known exploitable vulnerabilities associated with components of these commercial binaries.

What's your thought? And how hard do you think we need to be pushing both in the market, on the part of enterprises, software buyers as well as obviously at the federal level in a regulatory way to make sure that SBOMs are standard fare?

BEAU WOODS: Yeah. There is there's an executive order that requires vendors to provide software bills and materials to the federal government for anything that they buy or anything that they sell. Now, I believe agencies can waive that or can come up with other standards. But it is there. And so the vendors should be providing that anyways, just as a matter of course. But I'm also a big believer that if you want to legally attest to the fact that you don't know what's in your software, then that can be an okay option too. Like it's entirely-

PAUL ROBERTS: Not a good look.

BEAU WOODS: Product out there for you and that you don't know what's in it.

You bought it from someone else back in the old days before we had rigorous methodologies to capture dependencies and libraries. You got it as a binary blob from a third party. They went out of business and so you don't know what's in it. There could be very good reasons for some or all of your code to be unknown to you.

It's the comparison I make is I just bought a house in Washington DC and on the disclosure notification there were two questions. Do you know if there is asbestos or lead paint in the house? And this is a hundred and twenty five year old house. So the answer is no. I don't know.

Unless you've done testing on that, you can't say. And-

PAUL ROBERTS: Almost certainly yes.

BEAU WOODS: The answer is almost certainly yes, right? So then I buy it knowing that, I can make my own risk decision based on the information that I have, which I think is certainly an acceptable thing to do. And so where companies can't provide software bill of materials or can only provide a software bill of materials for a certain part of the code, they can provide those, or they can say, we have no idea.

They can provide the software bill of materials where they know what's in it and then say, there's a certain part of code that does this thing that we just don't have transparency ourselves and therefore we can't possibly provide it to you. Buyer beware. And I think that's a perfectly acceptable thing for manufacturers to do, at least in the short term.

For the next, I don't know, five years until these manufacturers can get up to speed but that's where you need the forcing function. Of telling them that they have to do it because if you don't and there's no incentives for them to change, then there will be no change. If there's no Internal or outside force that makes a thing happen, then it won't happen. So I think that the act of requiring that by the federal government can significantly help improve the quality and availability of these software bills of material as a tool for buyers, for operators of whatever type of equipment they're bringing in.

PAUL ROBERTS: Software bills materials necessary but not sufficient. So they're a critical part of improving the overall, quality integrity is software ecosystem, but they're not going to do the whole job but. So a couple kind of questions forking off of that.

One is how would you like to see software bills and materials evolve from where they are today? Looking forward five years, 10 years, what do we need to add to them to make them more potent, actionable? And second of all, what do we need in addition to software bills and materials to address these other sort of ancillary issues around the security of software supply chain that aren't captured merely by that. Yeah, we know what's in this application.

BEAU WOODS: Yeah. I think the the pieces of the toolkit. Are there we know what needs to go into a software bill of materials. There are some remaining issues on exactly what that looks like, exactly what that means, but they're engineering problems, not science problems. In other words, we don't have to invent new laws of physics.

We just have to apply the things we know to get the final. 2%, 5% of the way with the tooling that we have. But there's a reason why we talk about tools, tactics, procedures, or tools and techniques is because that's one part of it. And we also need to have evolution of the processes within manufacturers to generate these.

To improve as well as buyers to consume them as well as other third parties. For instance if there is an ISAC that can be some form of a, like a SBOM escrow, and they have an expertise to be able to say, these are the. Tools that exist. These are the software bills of material that are predominant or the the software packages that are predominant in our industry.

And we're going to tack track these SBOMs on behalf of the people in our industry. I think those types of things can be immensely beneficial to overall adoption of things like software bill of materials. There's a- any number of analogies that, that you might want to draw, but availability of a capability doesn't mean that it's universally applied and used or that it's universally applied and used correctly.

Look at the TLS SSL on websites. That standard, I think started out in the mid nineties, like 95. It was one of the things that was a bottleneck for e commerce. And banking financial transactions to happen is they wanted to ensure that there was security, the banks wanted to assure security for their customers.

But the number of SSL TLS websites available even today is not a hundred percent. So availability of a tool doesn't make it ubiquitous. And so now I think we're on that leg of making software bills of material universally applied, universally distributed and made available to anyone. And again, the high performing organizations will be the best able to take advantage of those.

Some of the lower performing or lower resourced organizations may not be able to take full advantage and gain full value, but there is value throughout the ecosystem in the mere act of asking for requiring and reviewing those software bills of material. For instance, when the Mayo Clinic started doing it they were doing, they were asking for SBOMs overtly and specifically because.

Not just could they benefit from them, but because they knew that manufacturers were not going to make multiple versions of their products, they were going to make one version that they sold to the Mayo Clinic and to everyone else. And so they recognized that it was a boon to the entire industry to require software bills and material just for themselves.

PAUL ROBERTS: I mean you mentioned medical devices, obviously the patch act is one of the most forward a piece of federal legislation that actually does require SBOMs. Do we need something like that for pretty much every other part of the economy as well to make it happen there?

Or is it gonna be enough to leave it to industry to regulate itself and do the right thing?

BEAU WOODS: I would like to think that we would have forward looking regulators and every other industry, like we have in healthcare and medical devices specifically but not every sector risk management agency has regulatory authority. And of those that do and of those that do, not all of them have the same authorities that the FDA does.

The FDA specifically has the authority to block new devices from coming to market and to pull them off of the market if they're found to be unsafe. So for the Industries that have those types of regulators and have that gating function. I think it would be great to see something similar as long as those regulatory agencies are able to capture those to do something with them, to speak competently on them and not to inadvertently miss out on.

The things that they'll need to do in order to ensure that the industry complies, right? The FDA is savvy on that. They've got people there who know what they're doing, who sit in the SBOM meetings regularly. And so they know what they're looking for and how to make those regulations. In general, I picked up from Josh is change won't happen until the pain of change becomes less than the pain of staying the same. I think that's true all around. Rather than taking more of a, an approach of I'm sure something will happen and out of the goodness of hearts people will change what they do because they see they've seen the light. They know the wisdom of the better way that's just not how people operate and that's fine.

That's just it totally makes sense. Why expend energy on something when you don't know what the gain or the benefit is going to be?

PAUL ROBERTS: Yeah, or we're gonna upgrade our" pretty please" to" pretty, pretty please" and then we'll see what happens and it's "don't get your hopes up too high."

Okay, this is a final question, which is you got a lot of really interesting stuff going on your life Beau and interesting stuff coming up in the next few months.

Hackers on the Hill you mentioned as well as continued work with I Am The Cavalry. Give our listeners just a little sense of some of the stuff that's on your stovetop and that you're working on.

BEAU WOODS: Yeah, sure. Hackers on the Hill is an event that I've run for the past several years. It's really casual informal. It's meant for technical people and public policy people, specifically congressional staffers primarily to be able to come together. And so that the they can build relationships, build bridges, build experience and knowledge about working with the other one before they're forced to in a time of crisis.

So that's coming up in January. We usually do it alongside ShmooCon and post ShmooCon, which is 2025 is going to be ShmooCon's last year.

We'll still continue doing Hackers on the Hill and maybe look to expand it a little bit. I don't know. We will see. In addition, you mentioned I Am The Calvary stuff going to be doing lots and lots of that. Again, go check out the BSides Las Vegas track that we did for a couple of days out there in the desert for a little bit more information on where our mind is at and what we're looking towards.

You invited me to promote myself. I'm not a very good self promoter, but I will mention that I've got a book on practical IOT hacking from No Starch Press, which was a phenomenal organization to work with.

There's lots and lots of other things. But yeah, I'm always happy to chat. Always happy to talk to you, Paul, and happy to be here talking more about software supply chain and keeping things on the up and up for security.

PAUL ROBERTS: Beau Woods, thank you so much for coming on and speaking to us on the ConversingLabs podcast. It's been a pleasure, and I have a feeling that we'll be doing this again sometime soon.

BEAU WOODS: Hopefully. So thank you very much, Paul.