Generative AI applications can be a rich source of opportunity for increased productivity and innovation for organizations. At the same time, they are fast becoming a headache for security teams. In a recent report, titled "The State of Attacks on GenAI," Pillar Security cautioned that "the unchecked proliferation of AI technologies without robust security measures poses significant risks."
Based on an analysis of more than 2,000 real-world, large-language-model (LLM) applications over a three-month period, the report found that AI's integration across enterprise platforms is "rapidly expanding the global attack surface."
"The anticipated ubiquity of local AI models could further exacerbate security challenges, as monitoring and controlling threats across millions of decentralized endpoints becomes increasingly complex."
The report also noted that the introduction of AI agents adds another layer of complexity to the security problem. "These autonomous systems can interact with various environments and make independent decisions," it explained.
GenAI has invaded enterprises and organizations. So what can security teams do to get a handle on threats? Here are five action items from the report — with expert analysis on each.
[ See Special Report: Protect Your Organization Against AI/ML Threats ]
1. Guard against sensitive data leakage
The report found that 90% of successful attacks on GenAI applications resulted in leakage of sensitive data. Michael Lieberman, co-founder and CTO of the security firm Kusari, said there’s still significant work to be done in the area of securing sensitive data in LLMs, so organizations need to limit access.
“We’re beginning to see proxy tools that can inspect prompts for anything suspicious and monitor LLM outputs for sensitive data. However, in the long term, we need to ensure that the LLMs entrusted with sensitive data are fully reliable. Currently, it’s evident that many LLMs are not there yet.”
—Michael Lieberman
However, keeping your sensitive data out of LLMs may be easier said than done, said Marcello Barros, a cybersecurity leader at Hacker Rangers. “This recommendation is almost unrealistic since training AI without real data is pretty much impossible."
Leon Goldberg, CTO at Sola Security, said that that the requirement for access to sensitive data means organizations must treat every interaction as potentially malicious — and with the possibility of negative security consequences.
“With this assumption in mind, the LLM agent should be restricted to only the data and resources that are acceptable for it to access. For instance, database connections should be limited to specific customers, and resource access should be governed by RBAC — Role-based Access Control."
—Leon Goldberg
So by assuming that training data will be leaked, "it's crucial to ensure that the training data is fully anonymized, especially when dealing with customer information," Goldberg said. He suggested a separate model be used for each customer to mitigate the risk of sensitive data leaks.
Another key area of concern for data leaks is conversation histories. Goldberg said they should be completely isolated, and properly authenticated and authorized, so that all interactions are logged for audit purposes.
2. Be aware of weaknesses in your guardrails
Pillar researchers found that 20% of jailbreak attempts on GenAI applications were able to evade guardrails designed to prevent abuse of the apps. James McQuiggan, a security awareness advocate at KnowBe4, said that since GenAI and LLMs are based on algorithms to determine the next word, they can be limited in understanding the context of the words used.
"Since some LLMs utilize keyword-based filtering and do not think like human beings, these vulnerabilities are only fixed reactively and not proactively. Addressing these weaknesses requires more adaptive and context-aware defense mechanisms."
—James McQuiggan
Strengthening these defenses involves using dynamic and context-aware security reviews, and adversarial testing to simulate potential vulnerabilities and address them proactively, he said.
Stephen Kowski, field CTO at SlashNext, said that basic security filters that just look for specific words are failing against attackers who know how to rewrite their requests in clever ways.
"Even the built-in safety features of AI models can be overridden when attackers use advanced techniques to confuse the AI's original security programming. Users can also deliberately craft prompts containing encoded data exfiltration commands, turning the AI system into an unintended data transfer channel."
—Stephen Kowski
3. Be prepared for LLM jailbreak attacks
Three predominant jailbreaks were cited in the Pillar report. One was "ignore previous instructions," which attempts to get an LLM app to disregard its guardrails. Another is the "strong arm attack," where, through persistent and forceful requests, the attacker gets the app to reveal sensitive information or perform unauthorized actions. The third was the use of Base64 encoding to get the LLM to execute malicious code.
KnowBe4's McQuiggan said these techniques can be countered with prompt validation systems that reject invalid or suspicious sequences. He said to also consider utilizing filters that review the context of text inputted, and to conduct adversarial training to improve the models against these styles of attacks.
Another way to address these kinds of attacks is by implementing AI-based anomaly detection systems that analyze patterns across multiple interactions, said Dev Nag, CEO and founder of the chat firm QueryPal.
"Some organizations are using classifier LLMs to examine user inputs before they reach the main application. These systems can be trained on known attack patterns and evolving threat data to flag suspicious behavior."
—Dev Nag
Nag said monitoring the total number of interactions and their patterns may also help identify potential attack sequences. That's because research shows malicious interactions often follow predictable patterns, as well as unusual volume and frequency, he said.
4. Take action to slow down attackers
Vulnerabilities in AI apps can be exploited very quickly. The Pillar report said adversaries require an average of only 42 seconds to complete an attack. It also found that, on average, threat attackers need just five interactions with a GenAI app to complete a successful attack.
McQuiggan said one way to combat such attacks is to build deliberate latency into the app for suspicious requests to slow down attackers. Another tactic: Deploy honeypot-style traps to engage attackers in other interactions, so the models and LLMs can detect and subvert the threat, he added.
To detect malicious intent in a set of interactions, Hacker Rangers' Barros said techniques like behavioral analysis, intent detection, and contextual tracking was effective.
"Monitoring for unusual patterns or repeated requests, along with setting thresholds for suspicious activity, helps identify potential attacks."
—Marcello Barros
5. Be prepared for the proliferation of smaller ML models
The Pillar report explained that the AI industry is witnessing an explosion of smaller, more efficient machine-learning (ML) models. "With OpenAI's advances in model distillation and improvements in hardware capabilities, it is becoming increasingly feasible to run sophisticated AI models locally on personal devices," the report noted:
"This shift towards smaller, decentralized models democratizes access to AI technology but also introduces new security challenges. As these models are deployed across numerous devices, the attack surface expands, making it essential for organizations to address potential vulnerabilities associated with local execution."
The report authors said security measures must evolve to protect data privacy, ensure model integrity, and safeguard against threats in diverse and distributed computing environments. QueryPal's Nag said smaller LLM models introduce significant risks, because they often lack the sophisticated security measures of larger commercial models — but still have access to sensitive enterprise data.
"These models can be vulnerable to prompt injection attacks that could lead to unauthorized data access or system exploitation. There's also increased risk of model theft, since smaller models may be easier to extract or reverse engineer, potentially exposing proprietary information or leading to the creation of malicious clones."
—Dev Nag
To understand the scope of the problem, SlashNext's Kowski advised picturing hundreds of employees using different AI-powered tools across your company — with each interaction becoming a potential point where sensitive data could slip out.
"When users copy-paste company information into various AI apps without proper controls, it's like having data flowing through countless unlocked doors. The rush to adopt new AI tools every month makes tracking where all your sensitive information is going nearly impossible."
—Stephen Kowski
With GenAI, it's a matter of trust
Fostering a deeper understanding of the right use cases for generative AI and LLMs will be critical for organizations using AI, said Nicole Carignan, vice president for Strategic Cyber AI at Darktrace. She said many people see GenAI as a data retrieval vehicle or a search engine. However, most generative AI tools are generating content that is based on the data they were trained on, the data in the prompt, and the question they were asked – not just retrieving data.
One outcome that has made headlines is AI hallucinations. Carignan said that it is critical to understand the appropriate use cases for each ML technique to mitigate the hallucinations. She said organizations should implement AI training for employees to ensure it is being used safely and securely. "Only then can organizations reap the many benefits of AI," she said.
Kusari's Lieberman said the use of GenAI came down to trust, which is tricky for security teams. He said trust depended on rigorous practices such as provenance tracking, thorough testing, fuzzing, and robust security measures. Adversarial training can also help identifying and mitigate vulnerabilities — and bolster trust.
"Advancements are coming, but I suspect we may never get to a place where AI is fully trustworthy."
