In July, a botched software update by CrowdStrike led to millions of Windows systems crashing worldwide, resulting in $10 billion in financial damage, by some estimates. Recent guidance released by the U.S. Cybersecurity and Infrastructure Security Agency (CISA), the FBI, and the Australian Signals Directorate aims at preventing another such incident.
The agencies explain in their document, titled "Safe Software Deployment: How Software Manufacturers Can Ensure Reliability for Customers," that many software manufacturers and service providers deploy software and configuration updates as part of their service offerings. Although these updates may enhance features or address security vulnerabilities to provide benefits and security to customers, software and the systems that deploy software are highly complex and continually evolving, making it difficult to deploy updates that are secure.
Here's what your AppSec team needs to know about CISA and other agencies' big secure software push.
[ See Special Report: How to Manage Commercial & Third-Party Software Risk ]
Experts see a solid framework for safe software deployment
All software manufacturers need to implement safe software deployment programs supported by verified processes, including robust testing and measurements, the agencies advised in their document. The program should support and enhance both the security and quality of the product and deployment environment.
Software manufacturers should incorporate safe deployment practices early in the software development lifecycle (SDLC), the authors added. Safe deployment processes do not begin with the first push of code; they start much earlier.
To maintain product quality and reliability, technology leaders should ensure that all code and configuration changes pass through a series of well-defined phases that are supported by a robust testing strategy. These phases are designed to catch any new issues and regressions, reducing the risk that flawed software will impact customers.
Iftach Ian Amit, founder and CEO of Gomboc.ai, said CISA's recommendations provide a solid framework to create safe and secure software deployments, and accurately capture the basic elements that most of the industry is already working with — testing and vulnerability scanning — as well as the broader context of the program, "which speaks of the deployment environment, the processes around it, resilience and ongoing updates.”
James Scobey, CISO of Keeper Security, said CISA’s recommendations are "on point and highly relevant" to today’s development environment, while noting that they are brief and easy to implement.
“They highlight practical ways that organizations can control risk in a process that’s often overlooked. By emphasizing phases like automated testing and canary releases, CISA has shown that safe deployment isn’t just a best practice; it’s essential for any company serious about security."
—James Scobey
The new software deployment guidelines can help transform deployment from a risky phase to one that actively strengthens security, "which is a huge win for both developers and customers,” Scobey said
Meeting the need for structured and repeatable practices
The multi-agency document sets out the key phases of a safe software deployment process — planning, development and testing, internal rollout (dogfood), deployment and canary testing, controlled rollout, and planning again, after receiving feedback. There’s also a lengthy discussion of playbooks, which help ensure that safe software deployment processes are well documented, repeatable, and resilient, the advisory explains. The playbooks provide clear guidelines, best practices, and contingency plans to help teams navigate each phase of deployment.
Playbooks also help software manufacturers reduce risks and respond effectively to any issues that arise, ultimately safeguarding both the software and the customers who rely on it, the documentation added.
John Terrill, CSO of the IoT security company Phosphorus Cybersecurity, said the new recommendations fit into another CISA initiative, Secure by Design, which the agency proposed in April.
“Secure by Design is not just an application security idea. You want secure components, you want best practices around scanning code and testing and production and all that, but you also want to make sure that it's all deployed in a reasonable way."
—John Terrill
Keeper Security’s Scobey said that a safe software deployment program is crucial to Secure by Design, which advocates baking in security at the conception of software, with security functions and parameters designed, architected, and coded into the software at every stage of its lifecycle.
"[It brings] security into every stage of the SDLC, not just after the code is written.”
—James Scobey
CISA’s guidance reinforces the importance of structured and repeatable deployment practices, which give organizations a powerful framework to catch issues before they become real threats, Scobey said. “By embedding thorough testing, phased rollouts, and feedback loops, teams can effectively reduce risks to customers and maintain software reliability. This way, deployment becomes as much a part of the security strategy as the code itself, which is exactly what Secure by Design aims for," he said.
However, integrating a secure deployment program into the SDLC can be challenging, Scobey said. “Time to deployment is a critical metric for any firm producing software — not only to realize customer value, but to drive cybersecurity improvements through patching and bug fixes,” he said.
“Organizations need buy-in across teams and a willingness to invest in automation, tooling, and training — often while managing tight deadlines. Coordinating controlled rollouts and real-world testing takes time and collaboration, which can be hard to maintain in fast-paced environments.”
—James Scobey
The payoff is significant, he said. "By building security directly into the deployment process, teams can prevent costly fixes down the line and ensure that updates enhance, rather than compromise, software reliability. It’s a shift in approach that’s well worth the effort.”
However, Phosphorus' Terrill said achieving these goals is going to be difficult. “I think there are just a lot of best practices that are easy to talk about but hard to pull off because they really require a focus and a lot of investment in things that are unsexy," he said. "Developers like building stuff. They don't like fixing stuff. It's just not a thing that gets them excited most of the time."
Lessons from the CrowdStrike incident
To many in the industry, much of the advice offered by CISA isn’t new. "All of this is super common,” Terrill said. “This is telling people rolling out software that these are things of concern that they should be doing.”
“I think this is a little bit of CYA, where they're just trying to make sure they've got the information out there so somebody can't say that they didn't state it."
—John Terrill
That might especially be the case after the CrowdStrike incident, said Frank Balonis, CISO and senior vice president for operations for Kiteworks. “I believe CrowdStrike is a great example of not following this process. If they had the kind of process in place that's discussed by the CISA guidelines, their problem would never have happened," he said.
Balonis said that before releasing code to any customer, it's best to release it internally to ensure that it's stable and to discover any bugs or other issues. Kiteworks, which has done this for years, then has a group of early adopter customers use the software, and it uses the feedback from those users to further improve the product. The final step: Roll it out, first to a limited number of customers.
"Not only does that allow the customer to understand how the product is being updated and released, but it also allows us to understand that the release that we have is viable, reliable, and secure before we release it for the entire customer base. Before we did this, we would do a release and just release it to everybody. Then there'd be issues and we would end up releasing a minor version just a week or two later to fix them. We would do that again and again, really disrupting our customers and how they're using the product."
—Frank Balonis
Balonis said that when companies that implement these new guidelines from CISA release their software to customers, "there's very little doubt that you've successfully developed a new version with the additional functionality and security that will meet a majority of everyone's needs."
The other benefit is that such a process can let your team prepare for the next release, as opposed to wondering what you have to fix or improve on the version you just released. "You can focus on the next release and be more efficient in your ability to release good software that's secure and reliable," he said.
Securing releases demands new focus, tools and process
Safety and security incidents often result from multiple contributing factors, including people, processes, and technology elements working together in a system that may become misaligned — sometimes in unexpected and unlikely ways, the multi-agency guidance noted.
A safe software deployment process should be integrated with the organization’s SDLC, quality program, risk tolerance, and understanding of the customer’s environment and operations. By adopting a systems-thinking approach, the agencies reason, teams may reduce the likelihood of their deployment process operating outside the boundaries of safety.
Jasmine Noel, senior product marketing manager for ReversingLabs, said that CISA’s guidance highlights that risk assessment should be part of "security in every stage."
"With billions in customer revenue at stake, the metrics, processes and tooling that built up around a 'move fast and break things' culture must change to support a 'rapidly deliver safe software' approach, and that requires strong leadership and a solid framework."
—Jasmine Noel
Noel said it's important for teams to reckon fully with the two sides of safe software deployment: software producers and enterprise buyers. She said that assessing third-party software risk before deployment leads to meaningful collaboration and remediation planning with software producers, which significantly reduces risks.
"A virtuous cycle of transparency benefits everyone."
—Jasmine Noel
