Tampering with the automated systems that build software is an effective software supply chain attack vector. Code compiled on compromised servers can produce binaries with embedded malware or changes that make the software behave like malware. The complexity of modern build systems exacerbates the problem, since the many moving parts can create several opportunities to embed tiny but malicious changes into the final build. It’s a challenge that SolarWinds had to overcome as it launched its award-winning Next-Generation Build System to detect novel attacks from patient, nation-state backed adversaries.
The Critical Need for Early Detection of Build Environment Tampering
Since it is not a common practice to perform a comprehensive risk assessment on the outputs of build pipelines, these unauthorized changes are invisible to both developers of the original code developers and enterprise buyers. Adding the ability to identify indicators of compromise empowers software producers to stop a software supply chain attack from reaching their production or customer environments. The need for this identification has increased interest in reproducible builds as a software development practice capable of verifying that software tampering has not taken place.
For example, SolarWinds describes its blueprint for secure software development as using “a unique parallel process to develop software builds in multiple secure, duplicate, and ephemeral environments.” However, improving the environment is only one aspect of their process. “By relying on reproducible builds, developers can ensure their software behaves the same way, weeding out disparities in code, identifying anomalies, and preventing intrusions.”
[ Learn more in our guide: Why Complex Binary Analysis is Critical to Software Supply Chain Security ]
How Reproducible Builds and Tampering Detection are Related
Reproducible builds are a set of software development practices that ensure that any party given the same source code, dependencies, build tooling and automation will obtain the same binary output, regardless of where and when the compilation takes place (see Fig 1). The most common way of checking the produced builds is comparing their software hashes, which are the same only when they are bit-by-bit identical.
Figure 1: Reproducibility means the binaries produced from two separate environments are exactly same
If reproducibility is achievable, then comparing final binaries is a powerful way to detect build system tampering by maintaining two build environments that are identical but separate, isolated and configured so that a single compromised account cannot alter both environments. In other words, if an attacker compromises one environment, there is a very low probability of the attack being replicated across the other environment. In this scenario, different binaries are produced only when one environment is compromised and software changes are introduced.
Delivering Exactly Identical Builds Is Not Easy With Complex Pipelines
In practice, completely identical build artifacts with identical file hashes is not easily achieved for organizations with complex build pipelines for a variety of reasons (see Fig 2). For example, there may be differences in timestamps due to time zone settings or configuration that uses the latest commit time instead of build time. Locale settings, file ordering and file paths can also cause differences resulting in build artifacts that are not strictly reproducible.
Figure 2. Many ways for build pipelines to create non-reproducible binaries
This means that the first step for software teams after a failed reproducibility check is an intensive review of the entire build environment for variations — build paths, archive metadata, architecture information, build ID & path, timestamps, file encoding and more — and triage the source of the differences. This additional effort can be significant, even when prior software versions were deemed reproducible.
Instead of expecting bit-by-bit identical build artifacts, some teams look for a degree of equivalence. For example, focusing on the build process itself, where a "repeatable build" is defined as having the same build steps always happen regardless of the build environment. However, the binaries produced do not need to be bit-by-bit identical.
From a software supply chain security perspective, the equivalence that matters is whether the same source built in different environments produces binaries with the same software behaviors. Software behaviors in this context are statically identifiable indicators of the actions that software could perform on the machine where it is deployed. For example, RL uses complex binary analysis to match various binary patterns to actual software behaviors such as:
- Writes to files in Windows system directories
- Deletes a file/directory
- Tampers with keyboard/mouse status
Once static behavioral indicators can be accurately identified from binaries, comparing two builds for behavioral equivalence becomes a dramatically simpler task. Since traditional application security testing tools (e.g. SAST, DAST, SCA) are not designed to perform this type of behavioral analysis on binary outputs, new assessment solutions are required.
How Spectra Assure Identifies Build Tampering With Reproducibility Checks
ReversingLabs Spectra Assure™ identifies indicators of compromised build environments by checking two binaries that should be reproducible for significant behavioral differences. Enterprise DevOps teams that have implemented two separate and isolated build environments can integrate reproducibility checks using Spectra Assure’s CLI or Portal APIs. The binaries produced are automatically analyzed in their entirety – including proprietary, open source, commercial, and third-party developed components, files and other software artifacts. Any behavior differences between the produced binaries are reported as a failed reproducibility check failure (see Figure 3). Automated build pipelines can use this failure status to stop the process from advancing and alert teams that tampering indicators should be investigated to stop potential software supply chain attacks before software is released to customer or production environments.
Figure 3: Spectra Assure Summarizes Differences Causing Reproducibility Check Failure
What makes Spectra Assure stand out against other solutions is its AI-driven complex binary analysis, which assesses binaries rapidly, from small dependencies to large, complex software packages in as little as minutes or hours for very large (multi-gigabyte) packages. Software producers can seamlessly integrate reproducibility checks and tampering detection into existing build environments and security testing gates without significantly impacting delivery timelines.
In addition to reproducibility checks, Spectra Assure simultaneously outputs a comprehensive risk analysis report which identifies other next generation of software supply chain threats by analyzing and assessing:
- Malware: Identify malware and other threats with the world's largest Threat Intelligence database covering 40 billion files with 16 in-house developed malware detection engines to prevent advanced threats from spreading throughout the software supply chain.
- Tampering Identification: Stop the ship for any indication of tampering with dedicated digital signature analysis to validate component integrity, and threat hunting policies to flag software changes that resemble attacks previously researched by RL’s experts.
- Exposed Secrets: Discover secrets that made their way into the final build, and the services they access to protect critical access, information and intellectual property.
- Application Hardening: Leverage the best-in-class analysis to discover coverage gaps and validate remediation to prevent vulnerabilities from being easily exploitable weaknesses.
- Version Differentials: Verify which security issues have been patched or introduced with the latest release or update, and highlight any new issues introduced between software versions.
- Vulnerabilities: Reduce developer fatigue and prioritize remediation efforts by highlighting actively exploited vulnerabilities.
Primary Benefits: Team Productivity and Software Assurance
Spectra Assure’s approach automates build system integrity validation and tampering detection without having enterprise development teams tinker with CI/CD tooling to create bit-by-bit reproducible builds. Team productivity improves by avoiding time spent on triaging and remediating build environment variations that produce different software hashes.
By focusing on build equivalence that matters most from a software supply chain security perspective (confirming that the same code compiled in separate build environments produce the same software behaviors) Spectra Assure makes reproducibility verification achievable for large and complex software deliverables. Identifying tampering indicators earlier in the CI/CD process gives organizations more time to investigate and stop potential software supply chain attacks before their release deadlines and deliver trust in your secure software development strategy to customers.
Learn more about RL Spectra Assure. Plus: Get a 14-day free trial for your organization.
