ConversingLabs Cafe: Chris Romeo on the state of application security

In this episode, we interview Chris Romeo, CEO of Kerr Ventures and long-time application security (app sec) practitioner, on the sidelines of RSA Conference 2023. Romeo gives a rundown on the state of app sec and comments on other software threats posed to organizations today. Listen in.

EPISODE TRANSCRIPT

CHRIS ROMEO
Chris Romeo, CEO of Curve Ventures.

PAUL ROBERTS
Okay, we're talking at the RSA Conference, Chris, just yesterday you gave your State of AppSec Security presentation. Do you want to enlighten our listeners, viewers with what 
is the state of AppSec?

CHRIS ROMEO
Sure. Yeah. It's definitely there's a lot of moving pieces to the state of AppSec, but what I shared in this talk, first of all, starting out, thinking about, okay, we have all these different programming languages, we have all these different frameworks, we have containerization, we have orchestration, we have DevOps and building software at a high rate of speed.
And really one of the primary points I wanted to make there is all these things are in their security infancy. AppSec started 20 years ago. Like we can draw back to OWASP beginning and AppSec starting to become a thing. But we're still in our infancy when it comes to all of these different things.
And yes, there are a lot of best practices that I shared from people, process, tools and governance. The standard kind of approach is the way that I think about it. But there, there is a lot more room to go and I really think we're approaching the year of the application. I don't know if it's 2023, I don't know if it's 2024, but we are, we're quickly approaching where security at all, all of our other levels has start, has gotten a lot of focus.
It's gotten a lot of investments. It's got, there's lots of tooling that brings that together, but really the application's still being in its infancy. The year the application's coming, I don't know if it's '23, '24, maybe it's '25, but the year of the application is on the way.

PAUL ROBERTS
It's, in some ways it's challenging cuz you're trying to hit a moving target, right?
Application development changes radically, has changed radically continues to evolve. Deployment, same thing. Shift to microservices and so on. What are some of the challenges that exist today around securing applications that maybe didn't exist 10 years ago?

CHRIS ROMEO
The OWASP Top 10 is something that I always come back to.
Maybe some of it existed 10 years ago, but some of those pieces are new. We've seen some new things make its way to the top of the list. Injection is no longer the number one biggest AppSec problem that we have. It's broken authorization. It's not being able to determine who can access what and having that be very solid.
But when you think about what we've seen in the pipelines, attackers coming after the pipelines, now, it leads me back to another OWASP project, the Top 10 CI/CD controls that need to be considered and challenges. And so that, those are the types of things that when we're thinking about the modern application, these are the places we have to excel.

PAUL ROBERTS
So on the positive side, obviously it could be the year of the application. Do you think that the big shift or change is just the amount of attention and resources this problem is getting or has something else changed?

CHRIS ROMEO
I think we've invested heavily in so many other places, but the application has been the place that hasn't gotten the attention. And so a perfect example, think about Zero Trust. Okay. Zero Trust considers, policy-based access controls and segmentation and all of these things. Then you ask about what about the application. Most zero trust architectures say we didn't really think about the application. We didn't even really play it into our picture.
We have this solid network approach for boundaryless networks and all these other things, but they almost forgot about the application. So there's an example of we need to focus and we need to keep pushing towards bringing the application in line with everything else.

PAUL ROBERTS
One of the sessions you did this year at RSA is a sort of red team blue team steel cage match. 
It's Chatham House rules. You don't talk about what happens in the room. But at a high level, what can you tell us?

CHRIS ROMEO
What we did with this experience is we set teams across eight tables and teams are going head to head against each other, and they're also playing to be the highest score in the room.
And so we gave each team a separate design. And we told them, hey, come up with the 10 best threats you can create based on those. And then after 25 minutes, we swapped them. So team one got team two's list of threats and then vice versa across the room, everybody swapped. And then they had to mitigate a set of threats that somebody else had created against a different document.
And we scored this thing in real time as we were going, and the feedback in the room was just phenomenal. People loved the ability to put their hands on and actually participate in this because a lot of times with threat modeling, I can lecture about it for 40 hours and I'll love it. Like I'll love every minute of it.
But at the end of that time, the people won't be able to threat model. They'll be like, I listened to this guy talk for a long time. And so threat modeling is about putting things into action. And that's what we did there.
And it's a skill. And, but you learn by doing. You can't. You can only learn a little bit of it by listening to other people. You have to get your hands dirty. And that's what we were able to create that experience. And people just loved it. They were raving coming outta the room. So what were some of the models that they were raving? Or given, so we took a, just a, we built a design for an e-commerce application, and one of the designs was the API gateway and further back to microservices databases, third party processors.
And then the other side of it was the mobile app. And so it was and we didn't design this thing with security in mind. We left a few things out to make it more interesting, but we tried to have a realistic architecture that people could find. At, in their businesses today. And so we wanted to that was important though, is to have something real versus just making up something that's not really applicable.
They'll, these folks will be able to take back, go back to work and see things now in their architectures and go, wait I thought about this already. I did an exercise where we pinpointed these types of problems.

PAUL ROBERTS
So one of the big buzzwords on the show floor at RSA and just in general in the industry, is software bills of material that is coming directly from guidance from the federal government and private sector as well.
You actually wrote a really interesting piece for ReversingLabs.com on sort of SBOM skepticism. I guess one way to look at it. Yep. Things to think about. Give us your take on software bills of material, their uses and limitations.

CHRIS ROMEO
Yeah, so I'm perhaps gonna be painted as an SBOM pundit, and I guess that's okay.
I've reached a point in my career where I can be a pundit and it's okay. I can live with it. But when I think about SBOM, so I certainly, I understand the value proposition. Like I want to know what is included in the software that I'm using, whether that's SaaS, whether that is something that's being used onPrem, it doesn't matter.
I want to know what the component, what the ingredients list is. So I don't have any problem with that. I want that capability myself. The challenge I have with SBOM is I think everybody's jumping into the deep end of the pool and saying, we have to do this. We have to create SBOMS for everything on Earth.
But nobody's thought about how to actually manage those things. What are you gonna do if some and nobody's even considered at the speed of DevOps. It's one thing for me to give you one SBOM a month. What if I release 50 to a hundred times a day? Am I giving you a hundred? You take, are you ready to accept a hundred SBOMs a day?
Process them and do some meaningful thing out of it. And so that's why I'm pushing back against this idea of just hitting the gas pedal and going all in with SBOM. And I got lots of friends that are doing very positive things. Steve spring it. But the Cyclone DX for example, they're doing, under OWASP's kind of tutelage there.
They're doing great things. But what I, my, what I'm trying to say is let's make sure we're getting value out of this. So I don't wanna make another compliance activity. I've been in security for 25 years at this point. I've seen a lot of check the box and I've become an anti compliance person cuz I'm gonna make the investment in something.
I want to be able to get the value. There should be a return on it. Don't just do something because we want to sell a product. You can do the same things that are required and make your product more secure. And if you do that, so that, that's really my overall challenge with SBOM is let's
get value out of it for the investment, for the time and money and everything that's gonna go into it. If we can do that, I'll become a fan. I'll wear an SBOM t-shirt, hashtag SBOM, I'll wear it around RSA. I'll wear it next year. We gotta move forward as an industry a little bit here.

PAUL ROBERTS
Yeah, there's been an SBOM T-shirt. So you know, folks who advocate SBOM say, listen, this is, think about the food supply chain, right? Think about how the sausage is made literally right. We need to know what ingredients are in the sausage. And then beyond that, yes, we need to have some standards around how it's manufactured and produced, sanitary conditions and so on.
That's all we're looking for. We just need to know what's in this stuff. And from that, we can start down the long road of, getting it so that the product that comes out is, safe, secure, reliable. So they look at it that way. But when you say we need to make sure that we get, this is productive, this is beneficial for us.
What do you have in mind? What type of processes we have to built around?

CHRIS ROMEO
We have to reach the point where we have technology that can ingest all of these SBOMs and truly provide actionable outcomes from it, whether that is feeding back into the vendor chains to get things updated. Whether it's doing something inside of an enterprise, causing them to do something to make an upgrade or something like that, it's I feel like that's the piece that's missing.
We don't have that actionable glue that can make SBOMs really beneficial. And when you think about, you use the food label example, who's the primary consumer of the food label? People. We look at the thing and we can understand it. We don't get. What if your ketchup bottle sent you 500 labels a day to, to peel off and stick on your bottle because they were changing the components that went like, I get that, trying to draw that conclusion, but it's the illustration starts to break down when you think of the 500 labels being stuck on the ketchup bottle and because the thing, ketchup doesn't change that much.
Ketchup stays the same for, probably hundreds of years, probably we've been making ketchup the same way. And to, to mix that metaphor almost oversimplifies the value prop. And once again I wanna see these things provide value. Like I'm not just, I'm not sitting here saying, I think they're bad and we should never do them.
I'm saying we should gain the value out of them versus going down this road and causing enterprises to have to spend millions of dollars to generate these things that are never gonna be used by anybody for the next couple of years. 
They just wanna sell, oh, I wanna sell to the federal government, so I'm gonna generate all these SBOMs.
I'm gonna do what they say I have to do. But there's not gonna be any value coming out of it. So then they're just balancing their business and saying we can spend 5 million on SBOMs to sell 50 million of product. That's a compliance activity, right? Like why? Why spend 5 million?
Why not spend 5 million to make the product more secure?

PAUL ROBERTS
What in your mind, would be a good application? Of either, SBOM as they exist, or the idea of an SBOM, how could it be optimally useful at this point?

CHRIS ROMEO
I think an SBOM that could encompass everything about a given SaaS service, for example, I think that starts to become even more interesting to me because it's more than just the individual components or libraries and things that are plugging into a web application.
It's now a rundown of what AWS services are included in this thing. Give me a holistic view of everything that's included within this. And, but then I've gotta have a way to act to, to perform, to, to act on it though. And that's the part that I feel like we're still missing in our industry is there's not an easy way to act on these things at the speed that we're developing software.

PAUL ROBERTS:
Yeah, the consumers of software, right? Are, enterprises let's say are, don't really have the systems processes built up to be able to ingest, maybe ingest SBOMs. Yep. And then figure out how to respond. Let's say there's a vulnerable component. Figure out how to respond to that.
So they need there that capability needs to develop. Do you feel like there's enough guidance being given to organizations right now on how they might develop that capability?

CHRIS ROMEO
No, I haven't seen, I haven't seen anything. OWASP has a project called Dependency Tracker, for example, and it's another Steve Spring IT project.
It ingests SBOMs and then gives you an enterprise view of it. Now it's an open source project. I don't know how many enterprises are willing to roll that out now as their entire, into their software supply chain security. So I think there's some room for vendors to step up in this space to give us.
To give us more of these capabilities to make these things actionable.

PAUL ROBERTS
Yeah. What in your mind, a lot of people feel like, listen, just saying we need to get developers to develop more secure code is sort of a boil the ocean solution. We're not gonna do that. There aren't many incentives actually within organizations to emphasize security at the expense of deliverability and features.
So what is in your mind the solution here for applications, application developers, software publishers to balance the demands they have to get code delivered but also address some of these underlying...

CHRIS ROMEO
Yeah. I would disagree a bit with the thought about developers can't do security, they can't keep up with it.
I think we've in our industry, we've, a lot of people have said that over and over again, so people are starting to believe it. I've seen the opposite happen. And when you think about developers in general want to create the best possible thing. They're creative people that write code to, to perform some function.
They don't want to create insecure versions of what they do. If they had the choice to learn about security and apply that to what they do, and now they know security and they could choose to be secure, insecure? They're not gonna, developers are, they want to build great stuff. That's what makes them great developers.
And so I, I don't see this world where, we can't, the developers can't help us. They can't get to where they need to do now. Are they the only answer? No. Once again, coming back to what I started talking about people, process, tools, governance, like that's what you need in an AppSec program.

The people need, the developers need knowledge, experience. They need to be taught because we know they're not learning in the university system. Nobody's teaching secure coding. But we can do that inside the company. We can pair that with a solid security process, secure development life cycle. We add tools such as, ReversingLabs and other tools that exist in the [00:14:00] market to help put together and check the code and ensure we're not having these software supply chain issues.
And then governance is looking across an entire fleet of applications and ensuring we've got metrics that are letting us. See that our program is inching up and maturing and getting better over time because this is not something that we can do in, you're not gonna do this in a quarter.
You're not gonna say oh, we're gonna, we're gonna push security to our developers in one quarter from now. We're gonna, no, this is a multi-year, slow, incremental improvements over time as we go and, but you can turn these ship. I mean, I did it at Cisco. I helped to do this at Cisco. Cisco when I was there, 25,000 engineers.
This is the battleship. This is trying to turn the battleship is what we were doing. We turned it a little tiny bit of a degree every day for five years. And when we got five years into the future, we looked and we said, oh, we've made a bit of a left turn to people thinking more about security.
So it's possible it can be done at scale, just don't think it's gonna happen overnight.

PAUL ROBERTS
So the 3CX hack was the incident that happened. And, before this conference, obviously it overshadows everything going on here. Software supply chain hack, actually maybe a second order software supply chain hack.
As it turns out, what lessons do you take from that incident? What lessons are there out there for, companies like 3CX that are developing software, selling it to enterprises? What do they have to learn?

CHRIS ROMEO
I think the biggest lesson I would like to see, everybody take away in the software supply chain right now is in the world of transitive dependencies.
Just the vulnerabilities that exist. It's not even our first line dependencies that are really being that, that are drawing in all these challenges. It's the fact that, like you think about a node JS application, you can create a node JS application, add five packages. Run an update and all of a sudden there's 278 packages that your project is using,
that your simple application is using. And so I think that what we really need to get a handle on here is this transitive dependency problem. Having the right tooling. And I think there's a future, this is one of the things I talked about in my State of the Union of Application Security.
Like here's my ultimate dream for software security: Think about what Let's Encrypt did to the world of browsers, of web server certificates. When's the last time anybody worried about a web server certificate? I get some people probably still buy them from some vendor, but when you think about what Let's Encrypt has done for the general marketplace, like we don't worry about browser to server, TLS, HTTPS connections anymore cuz Let's Encrypt just took that out of, they took the problem almost away from us, made it easy to refresh those certificates automatically every 45 or 90 days or whatever.
That's what we need for software supply chain. And I get that's only gonna solve part of the problem, but being able to point back and say, this project was signed at build time by this particular developer or group according to something they're listing in GitHub, for example. Having that level of provenance.
That doesn't prevent me from somebody injecting code and messing with the development team and trying to sneak code in. But it does let me, in my build pipeline at least have the state of mind to say, this is a project that was, or a package that was built by that team because it's signed and there's a route of trust all the way back to the top.
I think that's more than half the battle for what we need to do. I don't know how to do, I'm not the guy to do it, but I'm the guy that's gonna sit here and trumpet the fact. I want to encourage other people though. Yeah. If we could get to that level of not having to worry, like a package, someone's creating a package, they could just plug into something like, Lets Encrypt for software packages and have all the tooling to do the signing and everything.
Cuz let's face it, that's not easy. Crypto's hard for a reason. Like I go to, if I go to the crypto sessions here, half at RSA, half the time, I'm like, I don't know what these people, I consider myself to be mildly educated and intelligent. And I'm like, I don't know what they're talking about...
qubits and, quantum and all these other things. I'm like, I don't understand it. And so we know crypto's hard. So why make it difficult for those people that are just trying to create open source  packages and because they wanna make the world a better place, give them the tooling that they need to help all of us in the software supply chain.