When a document tries to become an instruction
A financial assistant may retrieve an invoice, filing or customer message to answer a legitimate question. Prompt injection occurs when adversarial content attempts to redirect that assistant’s behavior. The malicious text might be embedded in material the user wanted analyzed, rather than supplied as the user’s own instruction. That distinction makes indirect injection particularly important for systems connected to search, documents and external tools. NIST’s March 2025 adversarial-machine-learning taxonomy places these attacks within a broader set of model and system threats. [1]
An ordinary hallucination is different. A model can invent an earnings figure without anyone trying to manipulate it. A misleading source can also contain false financial claims without explicitly attempting to redirect the system. These failure modes can interact, but their causes and evidence differ. Calling every wrong answer an injection obscures what actually failed.
A bounded financial example
Consider a hypothetical assistant asked to summarize three supplier invoices and prepare a reconciliation note. One invoice contains adversarial material attempting to influence subsequent actions. The assistant can legitimately extract invoice dates and amounts; the supplier does not thereby acquire authority to change payment instructions or request unrelated records.
The critical boundary is between evidence and authorization. A document may be authoritative about its own quoted terms, yet still have no standing to grant the assistant additional permissions. A genuine counterparty can supply data relevant to a transaction without becoming entitled to control the entire workflow. Authentication of the document’s origin and authorization of an action are separate judgments.
The impact depends on the surrounding application. If the assistant can only generate a private draft, a successful attack may corrupt that draft. If it can retrieve confidential account files, send email and initiate changes, the same confusion can have broader consequences. Identical model behavior can therefore produce very different operational losses in two different deployments.
Why ordinary injection analogies are incomplete
The UK National Cyber Security Centre’s December 8, 2025 discussion warns against treating prompt injection as simply another version of SQL injection. Language models process instructions and contextual material through the same natural-language interface. Delimiters, detection and instruction-priority training can help, but do not create the same hard separation between code and data that a conventional interpreter can enforce. [2]
This does not mean every AI workflow is equally unsafe. It means that a sentence telling a model to ignore malicious documents is not equivalent to a technical permission boundary. The useful security question concerns the actions that remain possible after model behavior has been influenced, as well as the probability of that influence occurring.
A system might resist an adversarial invoice in testing yet still hold credentials that permit unrestricted account exports. Another might occasionally misread the invoice but lack any path to external transmission. The first has a lower observed manipulation rate; the second may have a smaller worst-case consequence. Neither dimension can stand in for the other.
Permissions outside the model
Least privilege limits available operations to the task’s needs. A reconciliation assistant could have read access to a specified invoice folder and permission to save an internal draft, while having no payment tool at all. Even if its output becomes unreliable, the missing capability prevents a direct payment action through that application.
Research on secure agent design describes patterns that separate planning from execution, isolate processing of untrusted material and constrain available actions. These approaches trade flexibility for stronger restrictions. A fixed plan can be easier to protect, for example, but may handle unexpected legitimate workflow changes less gracefully. The 2025 design-patterns paper is research guidance, not evidence that a financial institution has deployed every pattern successfully. [3]
Data boundaries also have to survive intermediate processing. A summary derived from confidential records remains derived from confidential records even if a model rewrites it. Renaming a file or passing text through a second model does not make its contents public. An application that checks only the final tool name can overlook the sensitivity of the material sent through it.
Provenance and policy enforcement
The CaMeL research system separates control and data flow and tracks provenance so that an external enforcement layer can evaluate operations. Its authors report benchmark results and discuss explicit policy restrictions rather than relying solely on the model to behave correctly. They also acknowledge limitations, including policy specification, user burden and potential side channels. The paper does not establish that prompt injection has been eliminated. [4]
In a hypothetical financial deployment, provenance could distinguish an account identifier supplied by the authorized user from one extracted from an outside document. The same identifier string could then be treated differently depending on its origin and intended use. This illustrates an architectural idea rather than a claim about any particular vendor’s implementation.
Policies can be incomplete. A rule that allows sending any draft to any previously contacted recipient might still permit an inappropriate disclosure. A rule that blocks every externally sourced value might prevent normal invoice processing. The hard problem is representing the legitimate workflow narrowly enough to constrain harm while retaining the data transformations it actually needs.
Approval has to concern the actual action
Human confirmation is meaningful only when it exposes what will happen. A vague request to continue does not convey the destination, information being transmitted or effect on a financial record. A review screen generated entirely from the influenced model’s own description can also misrepresent the underlying action.
For the illustrative invoice workflow, a trustworthy approval view would identify the precise operation and its parameters from the execution layer. If the proposed action changes after review, the reviewed action and executed action are no longer the same. Approval fatigue creates another weakness when frequent low-value prompts make consequential ones easier to overlook.
The balance differs by task. An internal summarization job can have a narrow, mostly read-only authorization envelope. A process that modifies payee details has a different consequence profile. Expanding model autonomy without changing those surrounding controls is a system change even if the model itself is unchanged.
Testing the system rather than a slogan
The NCSC’s November 2023 secure-AI-development guidelines address security across design, development, deployment and operation, including threat modeling, monitoring and maintenance. This lifecycle perspective matters because a new connector, tool permission or document pipeline can change exposure after the original model evaluation. [5]
An illustrative test might record whether an adversarial document alters a summary, causes an unauthorized data request or reaches a blocked action. These are different outcomes. A denied transmission is evidence that an enforcement control worked even if the model proposed it; a clean-looking answer does not prove that no inappropriate intermediate access occurred.
No finite test set establishes immunity to every future document or interaction. The defensible claim is narrower: specified controls constrained specified consequences under an explicit threat model and were evaluated against stated cases. Financial AI can gain useful capability from documents and tools, but the authority to read information, interpret it and act on it remains three separate things.
Sources
- NIST, Adversarial Machine Learning taxonomy, March 24, 2025Official sourceBack to text: ↑
- NCSC, Prompt injection is not SQL injection, December 8, 2025SourceBack to text: ↑
- Beurer-Kellner and coauthors, Design Patterns for Securing LLM Agents against Prompt Injections, v3 June 27, 2025Technical reportBack to text: ↑
- Debenedetti and coauthors, Defeating Prompt Injections by Design, v2 June 24, 2025Technical reportBack to text: ↑
- NCSC and international partners, Guidelines for secure AI system development, November 27, 2023SourceBack to text: ↑