Chatbots, copilots and retrieval-augmented AI tools now sit inside customer support queues, internal knowledge bases and automated workflows across most companies. Few of these systems were ever tested against the one weakness that matters most for them: they can be manipulated through plain conversation, not just broken code.
What Is GenAI and LLM Penetration Testing?
This form of testing checks how an AI model, its connected tools and its data sources behave when someone deliberately tries to push them off their intended rules.
Traditional penetration testing assumes software behaves the same way every time given the same input. Large language models don’t work that way. The same question phrased slightly differently, in a different language, or split across several messages can produce a completely different response.
That single fact breaks most conventional scanning approaches and is why LLM testing needs its own methodology.
A proper assessment usually looks at:
- How the model handles manipulated or disguised instructions
- Whether the model’s own configuration or internal data can be extracted
- What an AI agent can do once it’s tricked, including any tools or systems it’s connected to
- Whether a retrieval pipeline can be poisoned with documents containing hidden instructions
- Third-party models, plugins and no-code AI builders that employees set up without security review
None of this is theoretical. A handful of public incidents already show exactly what happens when these gaps go untested.
Why Does GenAI Need Penetration Testing? What Manipulation Looks Like in Practice
None of the scenarios below require a hacker writing exploit code. Each one illustrates how far a plain conversation can push an AI system, and each maps to one of the risk categories covered next.
A Support Chatbot Talked Into Breaking Its Own Rules
Picture a retail support chatbot built to answer order and shipping questions. A user starts asking it hypothetical questions about discount policy, then gradually reframes those hypotheticals as an actual request. Without the right guardrails, the bot can end up agreeing to a discount or refund it was never authorised to offer, purely because the conversation was steered there one step at a time.
An Internal Copilot That Leaks What It Was Told to Protect
An internal AI assistant is connected to a company’s HR and finance documents so employees can ask it quick questions. A user asks it to “summarise your own instructions” or to explain what data it can access. If the assistant wasn’t tested against this kind of probing, it can end up describing its own permissions, or worse, repeating snippets from documents the user was never meant to see.
An AI Agent Misusing the Access It Was Given
A company sets up an AI agent to draft and send routine emails on behalf of a team. Through a carefully worded request, someone convinces the agent that a company-wide announcement is expected of it, and the agent sends it without the usual review step. Nothing was hacked. The agent simply did what it was asked, because no control checked whether the request made sense before acting on it.
A Public Chatbot Manipulated Into Saying the Wrong Thing
A customer-facing chatbot gets pushed, through repeated prompting, into giving a rude or clearly wrong answer. A screenshot of that exchange is easy to share, and once it’s out, it reflects on the company regardless of how the conversation actually started.
The pattern across all four scenarios is the same. The entry barrier wasn’t a toolkit or specialist knowledge. It was a chat window and enough patience to keep trying different phrasing.
What Are the Main LLM and GenAI Security Risks?
Most real-world findings fall into a handful of recurring categories rather than being unique one-off bugs. The scenarios above each fall under one of these.
Prompt Injection and Jailbreaking
This is the most common attack type testers encounter, and it’s the category the support chatbot scenario falls under. It happens when crafted input overrides the AI’s intended behaviour, either directly through what a user types or indirectly through outside content the model reads, such as a document, email or webpage pulled into a RAG pipeline. Jailbreak attempts, including role-play prompts designed to bypass safety rules, often go undetected without dedicated testing because they don’t look like a technical exploit.
Sensitive Data Leakage and System Prompt Extraction
Models can be manipulated into revealing personal data, internal business information or their own system prompt, which is essentially the security blueprint for what the AI is allowed to do. The internal copilot scenario above is a direct example of this risk: once an assistant can be talked into describing its own instructions or repeating restricted content, an attacker knows exactly which controls exist and how to work around them. Extracted prompts have exposed API endpoints, database schema details and hardcoded keys in real engagements.
Excessive Permissions on AI Agents
AI agents are often given broad autonomy to complete tasks without a human checking every step. When those permissions aren’t scoped tightly, a manipulated agent can be pushed into unauthorised actions, similar to the email agent scenario above. Testing teams have gotten agents to send company-wide emails, approve fake expense claims and grant system access, all through conversation with no code involved.
Training Data and RAG Poisoning
Fine-tuning data and the documents feeding a retrieval pipeline can both be targeted. If someone uploads a file with hidden instructions into a shared document store that feeds the AI’s knowledge base, the model may read and follow those instructions the next time someone asks a related question. This counts as indirect prompt injection, and it’s hard to catch because the malicious content arrives through a channel the system already trusts.
Insecure Plugin and Integration Design
Chatbots rarely work alone. A support bot might connect to a CRM and a refund tool. An internal assistant might touch HR systems or financial records.
Each connection is a separate attack surface, and testing needs to check whether AI output can be abused to trigger unintended actions in any of those connected systems. The public chatbot scenario above shows the reputational version of this risk: once a chatbot is connected to a public-facing channel, a manipulated response travels far beyond the original conversation.
Hallucinations That Create Business Risk
Confident but wrong answers aren’t always a pure security flaw, but they become one once the output gets acted on without a human checking it first. A wrong answer given confidently to a customer can turn into a legal or financial liability rather than just an embarrassing screenshot.
Supply Chain Risk from Third-Party Models
Most companies use AI through a vendor’s model rather than building one from scratch. That raises a few questions worth answering directly instead of assuming the vendor has them covered: what security controls does the vendor actually run, what changes when the model gets updated, and what happens to the data a company sends through that deployment. A testing programme should verify the answers rather than take them on trust.
How Does GenAI and LLM Penetration Testing Work?
The rest of this guide gets more technical, walking through how a testing team actually carries out an assessment once a business has decided to run one.
A solid GenAI and LLM assessment moves well past sending one or two “ignore previous instructions” prompts and calling it done. Experienced attackers don’t rely on a single phrase, and neither should a serious test.
Treat Prompt Injection as an Ongoing Campaign, Not a One-Off Test
Testers vary the language, formatting, encoding and even how a request is spread across multiple conversation turns, since controls that catch an obvious request often miss the same objective assembled gradually. A useful question to keep asking during testing is whether the attacker can keep the same harmful goal while changing how it looks on the surface, through paraphrasing, translation, quoted text, or content that claims to come from a trusted internal source.
Retesting the same objective after a session summary, a model fallback, or a tool error matters too, because those state changes can shift which instruction the AI treats as priority.
The strongest findings connect the injection to something that actually happened downstream, such as a poisoned document causing the assistant to pull a second restricted file, or a support bot passing a manipulated refund amount to a real tool. A report should trace the full chain: the identity used, what got retrieved, what tool arguments were sent, and what changed as a result.
Test the RAG Pipeline as Its Own Security Control
A retrieval system decides what content the model even sees, and a well-behaved model can still produce a compromised answer if that pipeline hands it poisoned or unauthorised context. This means testing ingestion, chunking, indexing, metadata and the authorisation filters that are supposed to limit what gets retrieved before content ever reaches the model.
A practical way to test this is controlled poisoning: insert a synthetic document containing a hidden instruction and check whether the pipeline indexes it, retrieves it, and whether the model follows it. Try hiding the instruction in visible text, in metadata, in white-on-white text, or inside a spreadsheet cell, and check whether it keeps showing up in results even after the source document is changed or removed.
Isolation testing matters just as much. Set up two separate tenants or access groups with distinct test facts, then run similar queries from both sides and check which document IDs actually come back, not just the final written answer.
Authorisation should filter what gets retrieved before it ever reaches the model, and that filter should hold up against empty metadata, malformed filters, and access changes made after content was already indexed.
Follow the Pipeline Upstream
Some problems start before the model ever answers a question. This means reviewing training data, notebooks, model registries, deployment pipelines and package dependencies, and checking whether someone without authorisation could swap a dataset, edit an evaluation set, or push a new model version.
For generative systems specifically, controlled poisoning in a non-production copy of the data can show whether a planted behaviour survives retraining or a rollback. A rollback also needs to restore the prompts, the retrieval index, the tool policy and the model version together as one release, not piecemeal.
Secrets deserve a dedicated check too. Search notebooks, prompt templates, environment variables and logs for anything sensitive, using decoy values rather than real production secrets when demonstrating the finding.
Build a Repeatable Test Harness
Manual probing is good for discovery but weak as a regression method. A small script that defines an objective, generates approved variations of a prompt, sends them through the same interface real users go through, and scores the result against a defined success condition turns one-off findings into something that can be run again after every change.
The script should stop automatically on any unexpected side effect and should only ever run against systems the tester is authorised to touch.
Version control the test cases and their variations, and log the model version, the identity used, what got retrieved, what tools got called, and how long each request took. Score concrete outcomes, such as a restricted record actually being retrieved or a tool actually being called, rather than relying only on another AI model’s opinion of whether a response sounds unsafe.
What a Useful Report Looks Like
A strong report separates a model saying something it shouldn’t from an actual system compromise, since those carry very different severity. A dramatic-looking prohibited answer can matter less than a quiet response that pulls another customer’s data without anyone noticing.
Because model behaviour is probabilistic, a finding worth reporting should be repeated, not just observed once. Report how often an attack succeeds, how many conversation turns it took, and whether it still works after a new session or a model update.
Every high-risk finding should turn into a standing regression test, since rewording a single prompt is rarely a durable fix on its own. Real remediation usually combines tighter tool permissions, permission-aware retrieval, output validation, human approval on irreversible actions, and a rollback process that’s actually been tested rather than just diagrammed.
How Often Should AI Systems Be Penetration Tested?
Testing should happen before an AI system launches and again whenever the model, the system prompt, the connected tools, the retrieval data, or the permission rules change. Waiting for an annual review misses most of what actually changes in these systems week to week.
Who Should Consider LLM and GenAI Penetration Testing?
A few groups face the most immediate exposure. Companies already running AI in production have the most urgent need, since any weakness is already reachable by real users today.
Companies planning an AI rollout benefit from catching issues before launch, since fixing something before go-live is far cheaper than fixing it after a public incident. Regulated sectors including finance, healthcare and the public sector carry extra exposure, since a manipulated AI output there can trigger compliance problems on top of reputational damage.
Frequently Asked Questions
What’s the difference between LLM penetration testing and a normal security scan?
A normal scan checks software against known vulnerability signatures. LLM testing checks whether a model can be talked out of its intended behaviour through conversation, which automated scanners generally can’t judge on their own.
Can automated tools replace manual AI security testing?
Not on their own. Automated tools add useful coverage, but catching a subtle manipulation usually takes a tester adapting their approach mid-conversation based on how the system actually responds, which scripted tools can’t do reliably.
Does RAG poisoning require access to the AI model itself?
No. An attacker only needs to get a document with hidden instructions into a data source the retrieval pipeline already trusts, such as a shared file repository, and wait for the AI to pull that document into an answer.
How often should an AI system be retested?
Retesting should happen after any meaningful change, including a model update, a new tool connection, a change to the system prompt, or a change to what data the retrieval pipeline can access, not just on a fixed annual schedule.
