Lessons Learned from CI/CD Compromises

We chatted with researchers Iain Smart & Viktor Gazdag of NCC Group about their research into attacks on CI/CD pipelines.

EPISODE TRANSCRIPT

PAUL ROBERTS
Hey, welcome to ConversingLabs. This is ReversingLabs' podcast, where we talk to threat intelligence experts, software assurance experts, generally smart cybersecurity people. And we are here on the show floor at the Black Hat briefings in Las Vegas. And with us, we have two of the presenters from this year's show, Ian Smart and Viktor Gazdag, who are both from NCC Group. And you just did a really interesting presentation on CI/CD Compromises, supply chain compromises. So before we get talking about that, just tell us a little bit about the work that you do and your roles at NCC Group. And I'll start with you, Ian.

IAN SMART
Sure. So I'm the containerization practice lead. I do most of my work in breaking out of containers or pipeline or cloudy things. I've been pentesting for a few years now. And more recently, the jobs that kind of influenced this talk, where customers went, hey, we've got these CI pipeline things. Can you just fraud them a little bit, please? So we did, and we broke out of a few and kind of thought we should tell people.

PAUL ROBERTS
Yeah, I was joking that your title is Containerization and Orchestration Practice Lead.

IAN SMART
Which is lots of words. I'm not sure what they all mean.

PAUL ROBERTS
I don't know if you had to explain to your parents exactly what you're doing.

IAN SMART
They gave up a long time ago.

IAN SMART
Okay. And Viktor. How about you?

VIKTOR GAZDAG
Yeah, I'm a Jenkin Security MVP, and I'm also the cloud research group leader, and I'm recently doing a lot of cloud security assessment.

PAUL ROBERTS
Okay, so your talk is called RCE as a service, RCE (remote code execution). But it's actually really about a report out from your field experience, pentesting development organizations and development pipelines, CI/CD pipelines. And you did this in your talk. You kind of went through what some of the key terms are. But when we talk about development pipelines, what are we really talking about? What does that term mean practically?

IAN SMART
So effectively, it's a string of tasks to run when developers write code to speed up the process. So instead of manually writing code, compiling code, uploading that to production server somewhere, pipelines generally are a bunch of programmatic tasks that just automatically do all of that for you and take some of the grunt work out.

PAUL ROBERTS
So this is something that NCC Group does as a consultative service that you offer. Talk to me about the types of organizations that kind of come to NCC Group and sort of what brings them to you. What types of issues are they having? Is this merely, we're curious about this and we want answers, or are they coming to you after having experienced some kind of an adverse incident?

VIKTOR GAZDAG
Thank you. So all kinds of sectors coming to us and asking for help, and it also depends on if they had any problems before or they want to move their own environment and create a new one, or they want to move into a cloud native environment or create a cloud native pipeline. So it's basically everything. So we do configuration, review or test, like if something was already done, like they created roles and then we can review those roles and see if there is any problem with that. And sometimes it's even not related to a pipeline. Sometimes it was like in our presentation we mentioned that we started off with a web application and then we got access to a CI/CD pipeline as well. And Ian mentioned in his examples that even like a Red Team assessment or scenario so it's like everything.

PAUL ROBERTS
So it may not even really start as a Red Team assessment of the development organization. It might start with something much more narrow. But in the course of your activities you realize that in fact the development pipeline is implicated.

IAN SMART
Yes. It's not unheard of for us to stumble onto a pipeline and then go oh, this looks like a target, let's see what we can do with it. The impact has as we spoke about in the talk being quite bad in some cases.

PAUL ROBERTS
This is something that I think we're talking about a lot. The SolarWinds compromise obviously put the security of development organizations on the radar, certainly even though it's not like a new concept. But generally with development organizations, what are some of the high level security problems that you all have encountered in your work? What's common out there?

IAN SMART
So generally the security principles that have been around for as long as security has been an issue. So things like RBAC, secrets management and just the general threat modeling... 

PAUL ROBERTS
RBAC (role based access control).

IAN SMART
Yes, thank you. So people running pipeline as administrator users rather than restricting the privileges, people allowing components to have full administrative access over the cloud rather than only what they need, we see that not as locked down as it generally should be.

PAUL ROBERTS
So the problem is that this is sort of like the trip inside the sausage factory, right? And in the end of the day, the purpose of the sausage factory is to make sausage. And the purpose of these development organizations is to produce applications and develop code. And my sense is often that encourages a certain relaxed attitude towards security authentication as developers are working to just get stuff going, get code working right. And in many ways trust is an inherent part of what the code that they're developing needs to be is trusted code needs to have access to certain systems and so on and data to work. So how do you balance that? How do you balance security within these groups without sort of breaking the development process altogether? How do you do that?

VIKTOR GAZDAG
We don't do that, the companies do that. So we always recommend something and we also explain why we recommend something and most of the time the developers are actually accepting the recommendations. So they are very welcoming about it and it's based on the company culture and what they really want and how much they accept about the recommendation because we can recommend everything but it will be up to them if they really implement any of the recommendations. Right, but yeah, we will try to keep it in mind what are the existing processes, how can we improve but not really break and make it the same flow that they had before? Try to include that.

PAUL ROBERTS
One of the issues you brought up in your talk was the issue of dependency confusion that you talked about a customer that you were working with and basically the method of compromise that you discovered or that you kind of decided upon was basically just to create your own module and then link it to their own project so that it basically pulled in the malicious code. This is becoming more common. Right. We're seeing on platforms like NPM and Python package index efforts to do this by malicious actors. So talk a little bit about how that works and what companies, what development organizations can do to try and get on top of that particular problem.

IAN SMART
Yeah, so there's two attacks there. There's the first one you mentioned, which was us compromising a build pipeline and being able to specify an external dependency on our own and that was a network filtering problem in terms of the build environment could reach our attacker server and we could just specify whatever we wanted. So that was a network filtering problem effectively.

PAUL ROBERTS
They weren't looking for that outbound connection.

IAN SMART
Exactly. They weren't doing anything to stop their build servers talking out to the wider Internet and we know that dependencies need to come in from the Internet, which leads us onto the NPM and Python type of attacks. Generally what we recommend is pinning your version where possible and as far as possible. And we are aware that it's a hugely difficult problem doing some form of code review or due diligence on each version or at the very least. If you're using an external dependency. Mirroring it internally and making sure that you've got a reproducible build and that should that package have been compromised. You've got a local version to do analysis on because otherwise there's every chance that a package is compromised for a very short time. You ingest that package and by the time you realize something's gone wrong, you've rebuilt your application, the public version has been patched, there's no record and you have no idea what the impact was exactly, especially in the kind of containerization space where things are designed to be replaced as quickly as they can.

PAUL ROBERTS
And then the other is dependency confusion. So you talked about just sort of there being an outbound connection to your attack server that they didn't filter. The other attack is sort of dependency confusion where you've got like an internal module and somebody sets up a public module that sort of looks like that, or his name has the same name and actually that gets pulled in by accident.

IAN SMART
Yes. So that kind of typosquatting and similar standing dependencies. There's also various platforms that have a particular search order. So we saw this with Podman defaulting to key.IO before the docker hub and people doing squatting attacks against the kind of core critical images to run various containerized platforms. And obviously that has quite high impact if no one notices. Again, explicitly defining where you're retrieving things from and as it sounds, just reading what you type in very carefully can go a fairly long way to making sure you're not getting something typosquatted.

VIKTOR GAZDAG
Indeed.

PAUL ROBERTS
Viktor, you talked about just sort of paying attention to the plugins that you're using and what they do and really kind of understanding like running them, understanding how they function kind of before you commit to them, basically. Can you just talk a little bit about that and some of the problems that you discovered in your research just with that?

VIKTOR GAZDAG
Okay, so before the talk, I did some research in Jenkins plugins for about Jenkins plugins and I was kind of focusing three type of vulnerabilities. One was stored credentials that was also about cross site scripting and I forgot what was the server side request. Forgery was the third one. So I went through systematically on the plugins, downloaded them, sometimes look for the source code and look for patterns. And I also tried them and then I did a responsible disclosure. So Jenkins has a very good security team and they were very good partners. So we had a very good conversation and yeah, they were very also grateful that I have them to submit these vulnerabilities because they have a lot of work to do and I just have them so they have less problems. But I just recently checked again the advisory and I started to see a lot of cross site scripting, so they probably started to look for those issues as well. And the big thing is that the main thing that I always say about the plugins, that the plugins are developed by third party people or person or companies and not by Jenkins. So there is a couple of plugins, but most of the plugins are not developed by Jenkins. And that's a huge difference because Jenkins or Cloudbase is a company and they have developers and processes and everything via the third party developed plugins. It's like open source free code that you can use and if the developer says like, okay, I'm no longer interested in maintaining the plugins or I don't have time to fix the vulnerability, then the vulnerability will be there and you will get the warning that there's an issue with the plugin and a lot of companies still using it because they are depending on it.

PAUL ROBERTS
There's a huge kind of layer eight component to all this. Right. Which is developers being in a hurry. Developers being human beings. Seeing the first thing that comes up in a search result and kind of grabbing that or not reading down carefully and understanding what they're using versus what they may be intend to use. It makes me discouraged that there's an easy way to solve this problem because humans are going to continue making avoidable mistakes. But I don't know. You talk a lot, for example, you talk a lot about the need for better monitoring within development environments for some of the suspicious or malicious activities that you all were engaged in as you were doing these assessments. So I guess that's one thing that would help.

IAN SMART
Yes. So logging and monitoring is only as useful as the team actually processing the logs. So we can say enable all the logs till the cows come home and you can have terabytes worth of log data, but if no one's actually reading that, then it's not doing very much.

PAUL ROBERTS
Indeed.

IAN SMART
And obviously that...

PAUL ROBERTS
There are companies here, that just... That's their whole business model.

IAN SMART
How many things are we looking at right now? We will literally do this for you and we will solve all of these problems.

PAUL ROBERTS
Look over there.

IAN SMART
Logging and monitoring is great as long as you've got someone who understands what they're looking at. And again, that kind of comes back to the threat modeling, risk analysis side of things. As you said, these pipeline help developers go faster. They also provide security benefits and reproducibility and in automatic programmatic scanning. So if you've got some tooling that can run on every commit, great, that's more testing that's happening. But as you say, they are also just very capable pieces of software that have permissions to do all sorts of things. And if you're not logging what they're doing and someone does get in, then you probably should have some logs. So you know what they actually did.

PAUL ROBERTS
One of the issues you brought up was just the presence of secrets you mentioned SSH keys and so on, just kind of lying around within your development environment, sometimes for no clear reason. And that in the hands of an adversary, obviously can be very powerful. Is there an easy solution to that problem?

PAUL ROBERTS
Either, Viktor?

VIKTOR GAZDAG
Signature management tools maybe?

PAUL ROBERTS
Yes, there are some of those out here too.

IAN SMART
Yeah, there are a few of those here.

VIKTOR GAZDAG
Limiting the scope of the secret, like who can access what. 

PAUL ROBERTS
I mean, is this about development organizations just buying and deploying some of the stuff that broader IT organizations have brought and deployed to just protect user populations? Is that the issue or is it we actually need more specialized tooling really to address some of these threats and attacks that are specific to CI/CD pipeline?

IAN SMART
Most of the things that we spoke about and bearing in mind this is more focused on companies' internal pipelines rather than the wider problem of internet wide supply chain. Most of the problems we've seen have been misconfigurations rather than fundamental flaws in the software. So a developer or an admin has ticked the wrong box or has put a secret somewhere they shouldn't have, and it's just not been found until we came along and pointed it out. And almost every finding that we've raised has been remediated very quickly and the issues have gone away. So I wouldn't say there's a systemic problem in what we've been looking at, or a need for additional software, more requirements to pay careful attention to the way you're deploying what you've got.

PAUL ROBERTS
Okay, so, final question for development organizations that have been operating under the move fast and break things model, but probably have some nagging concerns that they've got exposed secrets, they've got, who knows, bad dependencies, they've got overprivileged users who are vulnerable. What would your checklist be for those organizations to start getting their security house in order?

VIKTOR GAZDAG
Contact NCC Group.

PAUL ROBERTS
We'll put your contact information up on the screen. Contact NCC Group. Okay, good.

VIKTOR GAZDAG
Yeah, it's let's see, network segmentation. Log access control, signature management, development environment. Yes, exactly. All the success control that we mentioned, and I barely see the auditing and monitoring and other things that we mentioned before, that even if an organization has auditing anywhere, nobody is checking it. And there is no less. Like there is a role change that would indicate some breach or something. So there are these low hanging fruits, I would call them that. It's easier to implement and may or probably not take the flow, but it can help and mitigate in a lot of times.

IAN SMART
Yeah, I would say start this off as a tabletop exercise and say, we've got a pipeline. What can I access? What permissions does it have? What secrets does it use, and who has access into the pipeline? So once you understand what's happening, who can make something happen and what is happening too, and hopefully you have a paper trail and logs of this, you should at least understand the blast radius of a compromise developer or someone who's gone rogue.

PAUL ROBERTS
I don't know if you saw Adam Shostack's "A fully trained Jedi, you are not," but he talks about the same type of threat model, I guess. 

IAN SMART
At least the industry is getting consistent with threat modeling. We can say that.

PAUL ROBERTS
Ian, Viktor, thank you so much for coming over and talking to us. And great job. I really enjoyed your talk today at Black Hat.

IAN SMART
Thank you.

VIKTOR GAZDAG
Thank you.