Can Your AI Assistant Keep a Secret? Understanding Prompt Injection

01 / AI security

How content becomes an instruction problem

OWASP distinguishes direct prompt injection through user input from indirect injection through external sources such as files and websites. It also notes that retrieval-augmented generation and fine-tuning do not fully mitigate the vulnerability. Impact depends on the system’s business context and the capabilities connected to the model. [1]

The assessment question is concrete: can a lower-trust input cause behavior that the legitimate user did not request or authorize? Include retrieved documents, messages, tool responses, and imported content in the review. A document being relevant to a task does not make its instructions authoritative.

02 / AI security

An illustrative support-assistant scenario

Imagine an employee asks an assistant to summarize a supplier attachment. The assistant can also retrieve internal contract notes and prepare outgoing messages. The attachment contains a request to include confidential notes in a message to a new recipient.

This is a hypothetical example, rather than an incident report. Break the assessment into separate questions: can the assistant access the notes, does it propose sharing them, does a tool allow that recipient, and does the employee see the actual content before approving?

A refusal in one run is useful evidence about that run. It does not establish that every input or future model version preserves the same boundary. Record the configuration and repeat the cases that matter to the workflow.

03 / AI security

Limit what a successful injection could achieve

OWASP’s Excessive Agency guidance identifies excessive functionality, permissions, and autonomy as sources of harm. It recommends narrower tools and permissions, with independent authorization and approval for consequential operations. [2]

In our suggested design review, a document-summary tool begins with access to the document needed for the task. Broader retrieval and message-sending capabilities require an explicit business purpose. The server should check the user, resource, recipient, and action rather than accepting the model’s plan as authorization.

Make approval meaningful. Show the content to be sent and its destination. A generic “continue” button gives the reviewer little evidence on which to decide. If the system cannot safely determine whether an action is allowed, give it a controlled way to stop and return the task for review.

04 / AI security

Use detection as one layer of defense

Microsoft’s Prompt Shields documentation distinguishes attacks in user prompts from attacks embedded in documents. It explicitly warns that a result detecting no user prompt attack does not establish that a request is benign or aligned with intended use. [3]

OWASP’s prevention guidance similarly treats guardrails as part of a layered design, alongside constrained tools, validation, and human approval. A second model used to review a request can itself be susceptible to prompt injection. [4]

For a deployment decision, ask which controls still protect data if the detector misses a case. Consider access checks before retrieval, validation before tool execution, and output handling before display or transmission. Track both missed attacks and blocked legitimate tasks when evaluating a detector.

05 / AI security

Build a small, authorized evaluation

Use a test environment with synthetic documents, restricted test records, and controlled recipients. Define the permitted outcome before introducing adversarial content. Keep the normal business task in the test so success includes usefulness as well as resistance to misuse.

• Retrieval: Does each user receive only the records they are entitled to access?
• Instructions: Can external content change the requested task or claim extra authority?
• Tools: Are proposed actions checked against the user’s permissions and approved scope?
• Output: Could generated links, markup, or messages expose information when rendered or sent?
• Recovery: Can the team stop the integration, inspect the event, and revoke affected access?

Record the model and application versions, tool permissions, input source, observed output, and downstream result. Separate a misleading answer from an unauthorized action; both may need attention, but their business consequences and fixes can differ.

06 / AI security

Make the release decision about a workflow

Begin with a bounded use case and test the boundary that matters most. A read-only assistant and an agent permitted to send messages should not inherit the same approval assumptions. Assign owners to unresolved risks, retain regression cases, and rerun the evaluation after material changes to models, connectors, permissions, or data sources.

The useful assurance is specific: under the tested conditions, the application prevented particular data and actions from crossing an unauthorized boundary. That is a stronger basis for an operational decision than relying on a model’s promise to keep secrets.