The Art of Security Chaos Engineering

In this episode, host Paul Roberts chats with Kelly Shortridge, a Senior Principal at Fastly, Black Hat 2023 speaker, and author on how to foster agility and nimbleness in enterprise security teams.

EPISODE TRANSCRIPT

PAUL ROBERTS
Welcome back to the ConversingLabs podcast. I'm your host, Paul Roberts. I'm the Cyber Content Lead at ReversingLabs, and I am here with the amazing Kelly Shortridge.

KELLY SHORTRIDGE
Hello!

PAUL ROBERTS
Principal Security Engineer at Fastly. 
Kelly, welcome.

KELLY SHORTRIDGE
Thank you.

PAUL ROBERTS
So we are here, the Black Hat Briefings. In the sweltering desert of Las Vegas.

KELLY SHORTRIDGE
Yes.

PAUL ROBERTS
And you gave a really interesting talk yesterday which picks up on a book that you just co authored on chaos engineering which I think is an amazing concept. So I want to talk about that. But before we dig into that, Kelly, tell us just a little bit about yourself and the work that you do at Fastly.

KELLY SHORTRIDGE
Yes. At Fastly I think about the future of security, which is delightfully vague, but very compelling, because obviously Fastly has a CDN heritage, content delivery network, and does a lot in infrastructure because we have a serverless platform. So a lot of actually interesting opportunities for that secure by design buzzword everybody's talking about.
We also have our next gen WAF, so it's interesting also to think about the end to end spectrum of what does it mean to protect an application or a system.

PAUL ROBERTS
Yes, in the cloud.

KELLY SHORTRIDGE
Precisely, yeah.

PAUL ROBERTS
One of the things I really love about your book is like it's the infusion of, in some ways, psychology in some ways
 into DevOps, DevSecOps, and talking about, the desire to make defenders, security engineers, security teams, as nimble as attackers. And this is a huge issue in security, right?

KELLY SHORTRIDGE
Huge, yes.

PAUL ROBERTS
Because attackers are incredibly nimble. They have no allegiances to technology or strategy, whatever works, they will do.
So talk a little bit about what that means practically for DevOps organizations to become more nimble.

KELLY SHORTRIDGE
Yes. So I think with DevOps organizations, a lot of times they are already maybe pretty nimble because they've leveraged things like automation. I talked in my talk about CSV pipelines, things like infrastructure is code are great.
I think what we see though, is the DevOps teams and security teams often have these kinds of like parallel constructs with what they're doing. And it's often the defenders that like you said, aren't as nimble as attackers, they aren't leveraging things like automation. A lot of times they're very focused on what we see here, bolts on tools, blinky boxes, little visualizations and they aren't really thinking about how do we infuse things by design.
How do we make sure we can update the design dynamically as conditions evolve as attackers change their methods?

PAUL ROBERTS
One of the things you pointed out in your talk, and you have this great kind of ice cream cone concept which you can explain is to invert the investment of resources and talent away from what I would call layer eight type
processes, right? Stuff that really rely on humans to do things that humans have proven themselves to be not great at doing, like religiously following security policy, without exception. Not a great...

KELLY SHORTRIDGE
No, it's not.

PAUL ROBERTS
History of people doing things like that. And to invest instead into things that you can automate things that are kind of data driven.
So talk about the ice cream cone concept and what that would mean practically for development organizations in terms of how and where they're investing their time and resources.

KELLY SHORTRIDGE
Definitely. So instead of the typical triangle, you got the ice cream cone here and I love it because it's a great metaphor for, if you have a big base of the cone, you can scoop a lot of resilience ice cream into it.

PAUL ROBERTS
And is the code ice cream? The ice cream is...

KELLY SHORTRIDGE
The ice cream to me conceptually is resilience. It's like how resilient is your system to failure. The idea is basically things at the top rely least on human behavior. So eliminating hazards by design or reducing hazardous methods and materials.
Now for DevOps organizations... Hazardous methods and materials could be manual processes or C code, right? A code that does not have memory safety. So when we think about, okay, how do we improve security? We want to focus on those. So that makes us think about things like standardization, memory, safe languages, isolation I talked in the talk about message bus brokers and cues, how those can be helpful.
There's immutable and femoral infrastructure. None of that, again, is really on this floor, but it's something that's really going to help security. So again, it gets us to think differently about what solutions we should reach for first. And to your point, it does mean you need to be thinking about this earlier. But I think the good thing is that I view security as a subset of software quality.
And you already consider that during the design phase. So it becomes a kind of more natural infusion.

PAUL ROBERTS
So for the past 10 years, we've been talking about this notion of Shifting Left, even though that's a fraught idea these days. Which I think was interpreted as pushing more security responsibility onto development teams in a way that maybe wasn't entirely fair.
So what would building resilience in and nimbleness... what are some basic ways that development teams can do that from where they are now?

KELLY SHORTRIDGE
Yeah, I think you can think of in some sense, safe code is code you can easily change. So even before you think about like resilience against attack, just make sure you can change things on demand, right?
Because in the talk, and definitely in the book, a key theme is that if you can ship code whenever you want, you can ship security changes or fixes right in an emergency, so making sure it's that nimbleness making sure you can change code on demand is super important. I think again, it's that idea that can we keep our architecture flexible enough through things like modularity?
Can we just keep our options open because that's what attackers try to do, right? They pivot to get to their goal. We should be able to as well.

PAUL ROBERTS
Right and the goals here are like things like you said, modularity, isolation, being able to, we're never going to prevent attacks, being able to prevent the lateral movement, the escalation type of thing.

KELLY SHORTRIDGE
The key thing is minimizing impact and this is important because the SEC has now said you have to report to material impact of cyber attacks so if you can keep impact minimal, think about you have your billing service as a serverless function. It's going to be very difficult for attackers to move laterally, even to the database where that's hosting the billing data or to any other services, you've contained the impact quite nicely.
So it's again, it's that is something where you don't have a bolt on tool, but it's something that's really powerful to minimize impact and both the book and the talk kind of talks about how we can minimize that impact because like you said, failure is inevitable.

PAUL ROBERTS
What's so interesting is a lot of what you're talking about really dovetails with, I was just at the CISA Unsafe at Any Speed talk before I came here, they're going to be joining us later as well, Bob and Jack.
This dovetails a lot with what we're hearing from the administration about embracing secure by design, secure by default, established concept. Mindful of the sort of psychology piece of this, how do we do this across, in this case, $21 trillion economy, without the stick, right?
How do we do it with merely carrots or ice cream and not the stick of, like the Patch act in medical devices, the FDA saying, if you don't do these things, we're not going to approve your product for release and sale. In essence, that is a real good motivation for a company.

KELLY SHORTRIDGE
It is, yes.

PAUL ROBERTS
But that doesn't exist in other sectors.
What are your thoughts on how to, again, not rely on, individual companies to embrace these concepts and do it, but actually get results and help companies to do the right thing?

KELLY SHORTRIDGE
Yeah I'll also note that when I was writing the book all 140,000 words over nine months. What I didn't know at the time was that CISA was working on secure by design.
So it was a happy accident. And I love what they're doing. So I will not pretend to be a policy person, but I do think one thing that's interesting is - thank you - is the cross disciplinary angle. So again, a lot of the things that we need to be resilient against attacks are things that again are part of software quality or things that we want for reliability, performance, availability reasons.
And so I really wish there was more collaboration between platform engineering teams or DevOps teams and security teams. A lot of our goals are ultimately the same or the means to those ends are the same. But right now, that's not true. So I think in terms of carrots or ice cream, to get there, it's really thinking about what other goals can we accomplish with these kind of like safer means.
And again, something like isolation. Sure, we think about it in the sense of isolating attack impacts, but that's true for performance failures or bugs too, right? Yes. It's a great way to uphold availability and ensure that there aren't contagion effects, so we need to be thinking more in that way.
mindset. I think I'm certainly not a fan of punitive measures. Though I think there are some nudges that we could potentially use, but I think unless you had an hour, we probably can't cover that.

PAUL ROBERTS
Like you said, you wrote a book with 140, 000 words, worked full time in the Bay Area, high tech firm, full time, plus cat mom. How do you manage to balance all those responsibilities?
I'm particularly interested because I've never written a book and I'd really like to. How do you do that?

KELLY SHORTRIDGE
Yeah. Listen, I live in New York city and I love the museums and everything. I don't do the nightlife when I'm at home on the weekends with my cats, I am reading papers across all sorts of disciplines. I'm trying to pattern match to where we can benefit cybersecurity or software infrastructure.
And really, this book was like pouring out my soul almost. I just had so much I wanted to say to get out. And I was very grateful Aaron contributed chapter nine, the case studies, because he's much more of a social butterfly than I am. So he knows all the people doing chaos experimentation. So it really helped bring, that really brings a lot of like how people are doing in practice in the boat too.
But really it was just, I, all of this was swirling in my head, and the practice of writing it really gets you to think very rigorously about your ideas and get them really solid. I actually love the process, I highly recommend it, but I would not do it unless you feel like this just like burning need where, I would be walking around, yeah, I would be walking around the streets, to the bodega and I'd just be like writing chapters in my head.
And it's okay, you need to write the actual book, right?

PAUL ROBERTS
Final question. So we're here at Black Hat. I know you're running around like crazy. You're doing a talk. You're promoting your book, you're probably doing stuff for your employer as well. However, anything that's interesting to you or caught your eye or seems, yeah, this is an interesting new thing that I'm seeing?

KELLY SHORTRIDGE
Interesting new thing? I'm gonna be honest, the security industry is not the best at innovation.
I am always surprised at the spectacle that vendors put on here and the amount of money that goes into that, which I personally wish would sometimes be more in the products. I think I am glad that some of the AI hype is tempered since RSA. I think people are realizing like it is a tool, but it's a tool that needs to be aware of the context of the problem.
And so I feel like I was dreading kind of the hype around everything is generative AI now, it seems like that's not the case. I'll say I'm pleasantly surprised by that.

PAUL ROBERTS
Yeah, people haven't glommed on to that quite as much. There's a long and not very proud history of companies pivoting to whatever the big term is.

KELLY SHORTRIDGE
At least it's not blockchain anymore.

PAUL ROBERTS
It's interesting because in the CISA presentation, Bob and Jack's presentation. They, so they're obviously, they structured it all around car safety and unsafe at any speed. Ralph Nader's very famous book. And they actually mentioned this report that Ralph Nader did in the 60s on the Corsair which was a car that was involved in a lot of one car accidents, so it would just flip over like in a big breeze.
It was incredibly unstable. And that they actually, Corsair for a while, sold this kind of bolt on thing, oh, if you're concerned about, the back of your car might flip over...

KELLY SHORTRIDGE
That's awful. Oh my gosh.

PAUL ROBERTS
They were like, that's kind of like the modern cybersecurity industry and you're like oh my God they're right.

KELLY SHORTRIDGE
Yeah, they're definitely, I love that analogy. I'm probably going to steal it. Thank you, Bob.

PAUL ROBERTS
Anyway, Kelly Shortridge, thank you so much. And where can people find your book on chaos engineering?

KELLY SHORTRIDGE
Yes, it's called Security Chaos Engineering, Sustaining Resilience in Software and Systems. You can find it at all major retailers online, Amazon, Bookshop, you name it and I hope you enjoy it.

PAUL ROBERTS
Kelly Shortridge, Principal Security Engineer at Fastly. Thank you so much for coming out.

KELLY SHORTRIDGE
Thank you for having me.