AI safety case study
Tracing Prompt Injection in an AI Research Assistant
An AI assistant can read untrusted content and still hold powerful tools. This case maps the trust boundaries that prevent retrieved text from silently becoming an instruction.
Evidence boundaries are reviewed against the primary references listed at the end. Read our editorial methodology.
The situation
Start with the facts available to the reader
A fictional research assistant helps an operations team summarize public market documents. It can search approved sources, retrieve internal policy notes, and write a draft into a review queue. It cannot send messages, change account settings, or export confidential records without a separate human-approved workflow.
During one run, a retrieved web page contains text telling any automated reader to ignore prior rules, collect system details, and place them in an external form. The model produces an unusual tool request that references an unapproved destination. The tool gateway blocks the request and records the event.
The case is intentionally defensive. It does not reproduce an operational exploit. The learning goal is to identify where instructions originated, which permissions contained the event, what sensitive information might have been exposed, and what evidence is needed before declaring the system unaffected.
Evidence sheet
What each observation can and cannot support
The last column is deliberately strict. It prevents a useful signal from being stretched into a conclusion that needs different evidence.
| Observation | Useful for | Not proof of |
|---|---|---|
| The suspicious text came from a retrieved public page. | Classifying the content as an untrusted data source rather than an authorized system instruction. | A successful compromise; the model's later actions and tool controls still need to be checked. |
| The model requested an unapproved external destination. | Detecting behavior inconsistent with the intended research workflow and triggering review. | Data exfiltration unless the request executed and included sensitive content. |
| The tool gateway blocked the request. | Showing that an authorization control denied that specific action under the recorded policy. | That no other tool call, model output, log, cache, or downstream integration exposed information. |
| An audit trace records retrieval, model output, and denial. | Reconstructing the sequence, scope, identities, tool arguments, and policy decision. | Complete coverage unless logging boundaries, retention, redaction, and missing events are understood. |
Analysis path
Work from mechanism to conclusion
Map instructions by authority
Which text is allowed to direct the system?Separate system policy, application instructions, user requests, retrieved documents, and tool output. Retrieved content may contain commands in natural language, but its location does not grant authority. The application should represent that boundary in both prompts and architecture rather than relying on the model to infer it perfectly.
In this case, the public page is evidence to summarize, not an operator of the assistant. The suspicious text becomes a prompt-injection indicator because it attempts to change behavior from inside an untrusted source. Classification is the first step; enforcement must still occur at the tool and data layers.
Inventory reachable data and tools
What could the assistant read or change if model judgment failed?List each data source, tool, permission, destination, and approval boundary. The research assistant can search approved sources and create a draft, but it should not possess a general-purpose network action simply because another workflow needs one. Least privilege reduces the consequences of a bad model decision.
Sensitive information also needs classification before it reaches the model. Secrets, credentials, personal data, confidential records, and system internals should not be inserted into prompts or logs by default. Output filtering is useful, but preventing unnecessary access is a stronger control.
Verify the enforcement point
Did a deterministic control block the action, or did the model merely decide not to continue?A trustworthy incident account identifies the component that made the authorization decision. Here, the tool gateway checks destination and permission before execution. That is stronger evidence than a model response saying it refused, because the boundary does not depend only on generated text.
The review should confirm the denied tool name, arguments, identity, destination, timestamp, and policy version. It should also check whether retries, alternate tools, chained actions, or malformed parameters could reach the same outcome through another path.
Scope exposure and improve the evaluation set
What left the trust boundary, and how will the same class be tested again?Inspect model-visible context, tool arguments, external requests, draft outputs, traces, caches, and downstream systems. A blocked final action is encouraging, but the incident is only scoped after reviewers know whether sensitive values appeared elsewhere. Record uncertainty where telemetry is incomplete.
Turn the event into a regression test using sanitized examples. The evaluation should vary source format, instruction placement, user intent, and tool choice while checking both model behavior and authorization outcomes. The goal is not to prove that prompt injection has been solved; it is to make a known failure mode observable and harder to exploit.
Reasoned conclusion
Say what is supported, then preserve uncertainty
The retrieved page attempted to redirect the assistant, and the model generated a request outside the intended workflow. The tool gateway denied the observed action, so available evidence supports a contained authorization event rather than confirmed external data transfer. That conclusion remains conditional on the completeness of logs and the review of alternate paths.
The most valuable control was not a clever warning inside the prompt. It was the combination of untrusted-content labeling, narrow tool permissions, destination enforcement, sensitive-data minimization, and an auditable trace. Prompt defenses can reduce risk, but they should not be treated as the only security boundary.
Claim boundaries
Keep the finding inside the evidence
This method can show
- Where suspicious instructions entered the workflow and which authority level they had.
- Which tool action was requested, denied, or executed under a specific policy version.
- Which data and destinations were reachable from the assistant's assigned permissions.
- Whether the observed event can become a repeatable defensive evaluation.
This method cannot show
- That all prompt injection is prevented because one request was blocked.
- That no information was exposed without checking every relevant output and integration.
- That prompt wording can replace tool authorization, least privilege, and human review.
- That a model's confident incident summary is accurate without independent telemetry.
Reader checklist
Repeat this process on another document
- Label retrieved and user-provided content as untrusted data in the workflow design.
- Give each assistant only the tools, data, and destinations required for its task.
- Enforce permissions outside the model and record the policy decision.
- Review prompts, outputs, tool arguments, logs, caches, and downstream effects for exposure.
- Add a sanitized regression case and document what remains unobserved.
Vocabulary
Review the terms used in this case
FAQ
Questions this case should leave you asking
Can a stronger system prompt eliminate prompt injection?
No single prompt can guarantee that. Prompt design is one layer. Stronger protection also uses least privilege, separate authorization, constrained tools, data minimization, monitoring, and human approval for consequential actions.
Does a blocked tool call mean no data was exposed?
Not automatically. Reviewers should check model output, tool arguments, network activity, logs, caches, and downstream integrations before scoping exposure.
Why store an audit trace?
A trace can connect retrieved content, generated requests, policy decisions, and outcomes. It is useful only when its coverage, access controls, redaction, and retention are understood.
Primary references
Verify the concepts at their source
These materials support the framework and terminology. They do not describe the fictional scenario above.
Primary framework for governing, mapping, measuring, and managing AI risk.
NISTArtificial Intelligence Risk Management Framework: Generative AI ProfileCompanion profile for generative-AI risks and risk-management actions.
OWASP GenAI Security ProjectLLM01: Prompt InjectionDefensive overview of direct and indirect prompt injection and recommended mitigations.
OWASP GenAI Security ProjectLLM02: Sensitive Information DisclosureDefensive reference for limiting sensitive data exposure through model inputs and outputs.