They say there’s no such thing as bad press but getting name recognition for a data breach never feels good. For Codecov, that sick feeling started on April 1st when a user notified them there was an issue with their Bash Uploader code. Getting told that you may have been hacked is terrible, but having it happen on April Fools Day had to add a layer of confusion. I’ll bet there was at least one conversation asking if this was real or if they were being punked. But Codecov did the right thing by starting an investigation, notifying their users, and building a patch.
Much like SolarWinds before it, this incident shows we need to take a more critical look at the software we are using and trusting to be good. For years the determination that a file or application was “good” was based on knowing that it was not definitively bad. It goes back to the original anti-virus and intrusion detection days where a file, application, or network traffic was compared to a signature, and if there was no match, it was allowed to continue. Over time malware analysis vendors like ReversingLabs began adding other dispositions like “suspicious” or “unknown” and, most importantly, “good” to acknowledge good from bad with the gray in between definitively.
There are many details on the attack, with more to come, but let’s focus on the code added to the Bash Uploader. Codecov most likely scanned their software with some form of AV before it was released. Unfortunately, the line of code added to the Bash Uploader was not by itself “bad,” so a different type of check needs to happen. As you can see from our supply chain analysis below, the original script uploads files to a remote server. In addition to that, backdoored script uploads environment details and credentials to a hardcoded IP address.
I’m not saying it’s good coding practice, but it would be easy to assume a legitimate reason for this code that wasn’t documented. A malware scan and even many application security testing tools wouldn’t flag this. And while it’s true that the hash of the uploader changed and could have been caught if the hash was verified, you still wouldn’t have details on what was changed without some time-consuming effort.
Here at ReversingLabs, we’ve realized that looking for malicious content is insufficient to stop these attacks when dealing with the software supply chain. This is why we created a solution to help organizations verify their software has not been tampered with or had malicious code added. You can see in the above screenshot we analyzed two files which we called Bash-Before and Bash. Our ability to analyze the files for malware and highlight the differences is key to catching these attacks before the software is released to the end-users. Imagine how things could be different if Codecov could automatically analyze the package, compare the results to the last scan, and alert the differences? At the very least, questions would be asked, and the additional code would be removed. Maybe an investigation would be started, but their customers wouldn’t be impacted as severely.
The attack shows it doesn’t take much to have a significant breach, but the headache and cost saved by getting an alert with three green boxes are substantial. Being able to dig into the alert and see in plain text what’s changed is the difference between releasing software on time and the pain and suffering of a breach.
Codecov is another reminder that our trusted software supply chain is under attack, and software manufacturers need to add more in-depth analysis to pinpoint behavioral differences between compiled software versions. If you're building software, it’s an opportunity to take a critical look at the software development lifecycle and align yourself with the organization’s security team to prevent an attack rather than respond to one. ReversingLabs can help by providing the development team clear and accurate information on the build with the same tools the security team uses for malware analysis and threat intelligence. If you want to learn more about how we’re helping companies secure their supply chain, you can visit our solutions page or reach out to us, and we’d love to solve these problems together.
