Steve Lipner of SAFECODE on Supply Chain Security - Is It Even Possible?

When it comes to secure software development, Steve Lipner is one of those information security industry leaders who was there at the creation, so to speak. Lipner, the current Executive Director of the non-profit SafeCode, served as the Director of the Microsoft Security Response Center (MSRC) and from 2004 to 2013 - a critical period that saw Microsoft launch the now renowned Security Development Lifecycle (SDL) initiative, which Lipner oversaw. As part of SafeCode, Lipner has worked to promote secure development principles more widely in industry. SafeCode provides free resources on secure software development as well as advice and recommendations for development organizations in the form of white papers, blog posts, social media posts, and more.

ConversingLabs host Paul Roberts chatted with Lipner as a part of our ConversingLabs Cafe series of chats at the recent 2022 RSA Conference in San Francisco.

In this conversation, Lipner explains what secure software is, recounts his own experiences on Microsoft’s Software Security Development Lifecycle Team at as the point of the spear in Microsoft’s Trustworthy Computing Initiative. Lipner stresses that secure software must come from within (so to speak). Outside consultants may be able to promote best practices, but they will never be able to grasp what needs fixing as well as members of your own development team. That’s why an organization’s developers need to be trained and motivated to write secure code, which means seeing mistakes as they write code and throughout the entire development process.

Lipner also talks about the Biden Administration’s Executive Order (EO) on Improving the Nation’s Cybersecurity, released in May 2021. Lipner believes that the impact of the EO is still a work in progress. He noted that Safe Code’s member companies have made it a priority to demonstrate that they are meeting the requirements set forth in the EO. He’s particularly a “fan” of Section 4 of the EO, which lists the requirements for a robust software security program.

EPISODE TRANSCRIPT

PAUL ROBERTS
Hey, welcome back to ConversingLabs. This is ConversingLabs Cafe. We're doing a bunch of talks with people who are presenting at this year's RSA Conference out in San Francisco. And I'm very happy to welcome into the ConversingLabs studio Steve Lipner from SAFECode. Steve, welcome and welcome to ConversingLabs. Great to see you again.
 
STEVE LIPNER
Thanks, Paul. Glad to join you. Glad to be back.
 
PAUL ROBERTS
So could you just tell us a little bit for audience members who aren't familiar with you and your work, the work that you do?
 
STEVE LIPNER
I've been doing what we'd now call cybersecurity since late 1970, and for the last 20/22 years I've been focused on secure software development. How do organizations build software that can resist attack? I did that at Microsoft from 1999 through 2015, retired there as the Director of Software Security and leader of the Security Development Lifecycle team. SAFECode is an organization that was created by Microsoft and a number of other industry players to share best practices in secure software development, both among the members and with the community at large. I had been the board chair there before I retired and joined them as Executive Director at the beginning of 2017. It's a small organization, but it's done a lot to help the community, I think.
 
PAUL ROBERTS
And yeah, just tell us a little bit what is SafeCODE's role? Like, how do you work with organizations around some of these issues, around software security?
 
STEVE LIPNER
So we're a member organization. As I said, we're very small, not a lot of paid staff. So when we do things, it's because the members and employees of the members decide to do them or want to do them. And what we've done is released a batch of free guidance, advice, recommendations in the form of white papers, blog posts, occasional tweets. Here are the things that you ought to be thinking about as you attempt to make the software that your organization delivers secure. We've done that primarily ourselves. We've occasionally partnered with other organizations, notably the Cloud Security Alliance and the Center for Internet Security.
 
PAUL ROBERTS
So as you mentioned, you were integral to Microsoft's Trustworthy Computing Initiative and secure software development, embrace of kind of secure software development going back to the early part of just random millennium. I think Microsoft really is one of the companies that has been most successful in embracing these concepts of secure development lifecycle and really actually kind of putting their money where their mouth is in terms of making organizational changes to realize better security of the other end. Can you talk just a little bit about the experience of being within the company during that period and what lessons you took away from that?
 
STEVE LIPNER
I go back to Code Red and Nimda Slammer Blaster if people remember some or all of those.
 
PAUL ROBERTS
I think Slammer came out in my first week as a cybersecurity reporter. Trial by fire.
 
STEVE LIPNER
Good timing.
 
PAUL ROBERTS
Good timing. "I think this job is going to keep me busy."
 
STEVE LIPNER
I have war stories from Slammer. It certainly kept me busy around then.
 
PAUL ROBERTS
Which was a SQL Server vulnerability for those of you who weren't born then. Yeah, go ahead, sorry.
 
STEVE LIPNER
Yeah, I guess that's a risk. People who weren't born then. The things we did, we decided that the security initiative initially came out of the product groups, the organizations that built the software. And what we decided, just because that seemed like the natural thing to do, was that the way we were going to get our software security better was by organizing, training and equipping our developers to build more secure software. And that was the mindset in the Windows Security push in 2002. And that was the mindset in the Security Development Lifecycle team when we created it in 2004. And that I believe it certainly was in place in 2015 when I retired. To the best of my knowledge, it's still the way they do it. And I'm a strong believer that you get secure software not by bringing people in from the outside to try to find all the bugs because they can't, but by having your developers motivated and trained and equipped to write secure software. Find out where they've made security mistakes, fix them as they go, so that's one piece and the other is continuous improvement. You're not going to get perfect software, but you're going to learn lessons when you make mistakes. And the key thing then is to update your process, update your training, update your tools, so that you fix not only what's reported, but all its friends and relations, all the other mistakes that are like that.
 
PAUL ROBERTS
And also, I would say really kind of prioritizing security. I can remember in the early days of that shift when you kind of stood up Secure Development Lifecycle, there were a number of planned releases that slipped because of security issues. Microsoft was saying we were planning on releasing, this is what we told you, but we're actually pushing it back because there's still more to do. We haven't addressed all the security issues. And as a journalist it was like, "oh my God." But I'm not sure that many companies out there right now are adhering to that or clearing that bar, are saying, we've identified some security issues here in what we're pushing out and so we're going to miss it, either miss the Sprint or whatever. But what are your thoughts?
 
STEVE LIPNER
Well, certainly during the SDL days and even the Windows Security push, we took scheduled hits to get on track with secure releases. A lot of the companies I work with, the companies I work with in SAFECode, and a lot of others that I run into that are not SAFECode members have secure development processes and have commitments and do, to the best of their ability, create and release secure code. Not saying everybody's perfect, I'm not saying it's uniform. It's always a risk management decision what you fix and what you ship. But organizations that have been through that generally understand the price of shipping something that you ought to have fixed, and there's a bias toward doing it right. The other thing is that I talked about risk management. Risk management has to be informed. And when you have a secure development team, its role, in addition to the training and tools, is to have somebody at a senior enough level to explain, here's what might happen if we ship with this. And then if you're going to accept the risk, at least you're going to accept it knowingly. And often when you go into somebody and you say, well, blah blah blah blah blah blah blah blah blah risk blah blah blah blah blaster, that will end the conversation.
 
PAUL ROBERTS
So you're out at RSA, you gave a presentation on Monday and supply chain security, well, supply chain security and the security of development pipelines and development processes is really top of mind. It's a big theme if you just sort of count the number of presentations that are about software supply chain or DevSecOps. It's a lot of them. Obviously we can kind of do SolarWinds and CodeCov. There have been a bunch of incidents that have happened in the last couple of years that highlight this incident. What is your sense about why right now this is what everybody is talking about? How do we explain its moment?
 
STEVE LIPNER
Well, I think a lot of the explanation of the moment is in fact because of SolarWinds and then to a considerable extent Log4j. We've seen from time to time other intrusions into development environment and repositories more or less restricted or focused. But I think SolarWinds really got a lot of peoples' and organizations' attention. Of course, there's the US. Government focus in the Executive Order as well. And so, it's part of the process of building and delivering secure software. We had been talking about supply chain security actually, both at Microsoft and at SAFECode back in 2009, 2010, 2011. If you go back to the SAFECode website, you can find a paper on supply chain security and code integrity, couple of papers from 2010. So it's not a new issue. I think Microsoft, we created our supply chain code integrity policy around 2010, maybe 2008 or 2009. And so it's something that you're just aware of. I don't know why we became aware of it initially, but it's just a natural thing to worry about. And so, bigger organizations become more aware of it sooner. The broad awareness is also good because back to the supply chain, everybody depends on lots of stuff they didn't write. And so it's good to have that understanding broadly.
 
PAUL ROBERTS
Right, and that does seem to be one of the issues that we're seeing play out now, is I think maybe one of the things that's changed in the last 20 years, right, is just increased reliance on third party code, whether it's open source or commercial, proprietary third party libraries and modules and so on. And that's kind of the way applications and services get built these days, right?
 
STEVE LIPNER
It is indeed. People integrate components, they integrate commercial components, they integrate open source components, they copy stuff off stack overflow.
 
PAUL ROBERTS
Literally copy, right?
 
STEVE LIPNER
That's the way it is. I was going to say it's fine, it's reality, but the important thing is that people understand that security is part of that equation. And what are you going to do to make sure that the software you deliver to your customers, either on prem or online, is secure and has taken best practices into account?
 
PAUL ROBERTS
What is the advice that you would have? I know one of the concepts that you talk about or that you were talking about in your presentation is this notion of how best to do supply chain security. We talk about supply chain security again, because that's kind of from the industrial world or manufacturing world, that's a notion that we're all familiar with. Make sure all the components aren't compromised or counterfeit and let's supply that to software. Does that model actually work in the modern context of software, agile software development? Is it applicable? And if so, what do companies need to do to get up and running with this if they're not doing it already?
 
STEVE LIPNER
So the concept is pretty good. I mean, the session we did at RSA, Tony Sager from center for Internet Security and I were the speakers and John Pescatore from SANS was an original contributor who we had planned to have as a speaker. And he did a lot of work on the slides. His contribute, he referred to the supply web. There's just a lot of stuff flowing into any modern software project. Okay. And going back to the earliest days of the Microsoft SDL, you said you need to know what the components are that you're using that you don't build. Okay. And that's sort of a law of nature, if you will.
 
PAUL ROBERTS
What other people's code are we relying on?
 
STEVE LIPNER
Yeah, what other people's code are you relying on? Initially at Microsoft, we were concerned about another group in Microsoft, but it still applies. What are you shipping and why do you believe it's secure? And what are you going to do if there's a vulnerability found? There's a SAFECode document, Practices for Using Third Party Components, something that's approximately the title that we released probably four or five years ago still applies. Main thing is knowing having an inventory, having a way to discover if there are vulnerabilities found, having a way to assess the impact of those vulnerabilities on what you shipped, and then having a servicing plan to get well if you need to. That applies just across the board, whether it's a couple of lines in somewhere that you copied, or whether it's a big component that you're shipping or hosting on your service.
 
PAUL ROBERTS
So you mentioned the Executive Order. President Biden issued a Cyber Executive Order earlier back in 2021, basically calling on the government agencies to address supply chain security, among other things, to start looking into using software bills of material. And there was a lot in the Order with the goal obviously of kind of shoring up the defenses of the government agencies. What impact do you think has that had practically? And when may we see the fruits of that Executive Order?
 
STEVE LIPNER
Well, the impact is a work in progress and the way that the government is going to require attestation to the requirements of the Executive Order by its suppliers of software is still not finalized. But I talked to the SAFECode members and they're all already worrying and taking action to make sure they're prepared to demonstrate that they're meeting the requirements of the Executive Order. I'm very much a fan of the Executive Order or part Section Four, which is sort of the software supply chain piece. I'm very much a fan of that because it requires sort of sensible things that you would do if you had a software security program. So if the government's requiring people to do that, I think that's cool. Software BIll of Materials is a piece of that. It's a major manifestation of the supply chain side. The formalism of the bill of materials and the materials format is less important, but requiring people to know what they're shipping is really important. It's primarily for the developer. The developer needs to know what's in their code so if something goes wrong, they can find it and remediate.
 
PAUL ROBERTS
This is what Log4j kind of underscored for us. And we were talking about SQL Slammer. In some ways SQL Slammer actually raised this issue as well because if I remember there were all these products that actually used aspects of like Microsoft Access for example, that initially people didn't think was vulnerable, ended up being vulnerable. Log4J is the latest example. Hugely widely used open source library. But who's using it directly or indirectly really hard to know.
 
STEVE LIPNER
Yeah, actually you just triggered my PTSD and I probably haven't thought of that in 15 years. There was something called MSDE, which was SQL Server by another name for embedding into other applications. Yeah, it turned out to also have the Slammer vulnerability and so we had to go around and build a tool to enable organizations to find where they had MSDE and update it. I could have gone all day without thinking about that, but today similar. You got to know what you got to know what you're shipping, you've got to be prepared to remediate, and that's the focus. A lot of good stuff in the Executive Order, a lot of good stuff in the Secure Software Development Framework, which is NIST's guidance on secure development, including both development practices and supply chain repository protection. Basically it's a brief document, but it's very complete, very actionable, very good guidance for developers.
 
PAUL ROBERTS
Do you think organizations, ReversingLabs did a survey on this of developers, small population, 300/350 respondents, but they were basically sort of like, yeah, SBOM is a great idea, but we don't have the expertise internally, we don't have the budget. I mean, it was all the sort of like, yeah, it's great, it's useful, really help us, but I don't know if we can do it because of all these other contingent issues: staffing, expertise, knowledge, and all these other things. So I mean, is that going to play out at scale here as organizations try and implement this idea that is really more about it's not a technical implementation, it's more of an operational procedural thing?
 
STEVE LIPNER
Yes. So if you're building software, so there are two things, okay? There's SBOM, sort of capital S, capital B, et cetera, as a sort of a specific set of formats and tools and stuff. The SBOM requirements, I think that came out of NIST. I think that's where they came from are just some sort of fundamental things. What's it called and where is it? Maybe a check some date and a check some to make sure you know what version you've got. At some level, if you're building software, you've got to have a configuration management system and a repository. Okay? And so if you're including this component in the bill, you've got to have some way of picking it up and including it. And if you know that and you have a way of querying that, I tend to think that that ought to meet the SBOM requirement. The important thing is to know what you're shipping and ideally you'd get an S bomb from your suppliers so that you know what they've incorporated all the way down. The government working groups on SBOM have sort of talked about how much of that can we do, how long will it take? I think that's still a work in progress. But the focus on knowing what you're shipping is the key thing, not exactly how you do it. And I guess you can do it. You can do it in a way that's expensive, but I'm not sure I agree that you have to do it in a way that's expensive.
 
PAUL ROBERTS
Okay, final question, Steve. How do we lift all boats here with software security? Again, you were there in Microsoft where company made some very substantial changes and improvements in its practices and the outcomes. How do we do that across this huge, now amorphous industry that isn't just born and bred software companies, but actually every company companies making home appliances and everything else.
 
STEVE LIPNER
So that's a great question. I mean, things like the Executive Order and customers, business customers, enterprise customers, probably going to get on board and ask, do you meet the EO requirements? Do you implement the SSD? I think that'll be helpful and that'll get to a lot of organizations that are building and shipping software. I think the other piece of that, that I would love to see is more sort of bottom up. I really would like to see the people who are educating and training software developers. You want your code to be fast. You want your code to be correct. You want your code to be memory efficient. Maybe we ought to say you want your code to be secure. Is that another property that people ought to be taught as they learn the program? That would sort of be a paradigm shift, a culture shift. The education system isn't doing as much of that as it should, and I would love to see some encouragement in the system that got that message out.
 
PAUL ROBERTS
Steve, is there anything I didn't ask you that you wanted to say?
 
STEVE LIPNER
I think you've done a good job of covering the landscape. The RSA conference is back. It's live. It was fun to be here and see people in person after two and a half years.
 
PAUL ROBERTS
Okay, I hope you your mask on when you're outside of your hotel room Steve.
 
STEVE LIPNER
Try to do that.
 
PAUL ROBERTS
Hey, Steve Lipner. Thank you so much. Of SAFECode. Thank you so much for coming out and speaking to us on ConversingLabs. It was a real pleasure to speak with you again, and we hope to have you on again.