Part IV · Proving it works · Chapter 13
Security and how much autonomy to grant
Permissions before context, untrusted text never in charge, and autonomy granted per action by impact and reversibility.
14 min read
AI systems inherit every security problem of ordinary software and add a new one: an input channel, natural language, that can carry instructions. The good news is that the defences are mostly architectural. Get the structure right and a model that is fooled can do very little harm.
What you'll take away
- 01Three rules that prevent most serious incidents.
- 02The threats that matter in practice, beyond the headlines.
- 03Why prompt injection is solved by structure, not by better prompts.
- 04How to set autonomy per action, by impact and reversibility.
- 05How to work with security and data teams so they help rather than block.
Three rules
- Permissions before context. Decide what a user may see before anything reaches the model, in code, on the server. Anything the model sees, assume the user can get it to repeat.
- Untrusted text is never in charge. Documents, emails, web pages and tool results are data. They can be summarised and quoted. They cannot change what the system is allowed to do.
- Autonomy is granted per action. Reading is cheap. Writing, sending and paying are not. Each action gets its own level of independence, set by what it could break.
Assume the model will eventually be fooled, and design so that being fooled is cheap.
What actually goes wrong
| Threat | What it looks like | The structural defence |
|---|---|---|
| Data leakage across users or tenants | A missing filter returns another customer's documents; a shared cache serves one user's answer to another | Server-side filters, an assertion after retrieval, user scope in every cache key |
| Prompt injection | A retrieved document or email says 'ignore your instructions and…' | Untrusted content can't trigger actions; a policy gateway sits in front of tools |
| Excessive agency | An assistant with broad tool access does something nobody intended | Least-privilege tools, per-action approval, caps on rounds and spend |
| Unsafe output handling | Model output rendered as HTML, run as SQL, or used as a shell command | Treat output as untrusted input: escape it, parameterise it, validate it |
| Denial of wallet | Crafted or automated requests that maximise tokens and tool calls | Per-user rate limits, token caps, budgets with alerts |
| Sensitive data in logs | Prompts and responses logged with personal data, kept for years | Redact before logging, set retention, restrict access to traces |
Untrusted text is never in charge
Prompt injection is the number one risk in the OWASP project's Top 10 for LLM applications. It comes in two forms. Direct injection is a user typing instructions to override the system. Indirect injection is the more dangerous one: instructions hidden in content the system reads, such as a web page, a support article, an email or a PDF.
You cannot reliably filter it out with a prompt, because to the model, instructions and data are both just text. Defend with structure instead:
- Enforce what tools may do in a gateway outside the model: allowed actions, allowed recipients, limits. A fooled model asks; the gateway refuses.
- Keep content you haven't vetted away from any step that can take high-impact actions, or strip that step's tools while it handles untrusted text.
- Never let retrieved content raise its own trust: a document can't grant access, change the system prompt, or add a tool.
- Validate outputs before they go anywhere that executes or sends them.
- Log and alert on suspected injections. They are often probes before a real attempt.
The lethal trifecta
Simon Willison describes a useful test. An agent becomes exploitable for data theft when it combines three things: access to private data, exposure to untrusted content, and the ability to send data out (an email, a web request, even an image link). Any two can be managed. All three in one agent, without a hard gate between them, means an attacker who can get text in front of it can probably get your data out. When you review a design, look for that combination first.
Grant autonomy per action
"Should the AI be autonomous?" is the wrong question. Ask it of each action, and answer by two measures: how much damage a mistake could do, and how easily it can be undone.
| Level | The model's role | Typical actions |
|---|---|---|
| Suggest | Recommends; a person acts elsewhere | Account closure, large refunds, medical or legal advice |
| Draft | Prepares; a person reviews and sends | Customer emails, contract wording, claim decisions |
| Auto with a check | Acts after a confirmation step | Rescheduling, small refunds within policy |
| Auto-execute | Acts within strict limits, logged | Tagging, routing, internal notes, read-only lookups |
| Forbidden | Not available to the system at all | Anything the client's policy reserves for named people |
Start conservative and widen autonomy with evidence. When an action class has run at the draft level for a month with almost no corrections, that is the argument for moving it up. Write down who agreed to the change. Separate the model's recommendation from the authority to execute: the model can propose a refund, but a policy check in code decides whether this refund is eligible to proceed automatically.
Data governance in practice
- Classify the data the system touches: public, internal, confidential, regulated (personal, health, financial). Each class has rules about where it may go.
- Know where it travels: which model provider, which region, whether the provider keeps or trains on it, and what the contract says.
- Develop on synthetic or redacted data until production access is approved. Separate the data you evaluate on from the data you run on.
- Log carefully: redact personal data in traces, set retention to match the client's policy, and restrict who can read logs.
- Plan for deletion: if a person asks to be forgotten, their data must leave the index, the caches and the logs too.
Working with security teams
Security reviewers are not obstacles. They are stakeholders with a legitimate fear and the authority to say no. Meet them early, ideally before the design is fixed, and bring a short threat model: what data the system touches, where it goes, what actions it can take, the worst realistic misuse, and the controls for each. A reviewer who helped shape the controls signs them off. One who sees them for the first time in a review starts from scratch.
Expect questions like these, and have the answers written down:
- Which model providers receive our data, in which region, and do they retain it?
- How is access to documents enforced, and how is that tested?
- What can the system do without a person's approval?
- What is logged, for how long, and who can read it?
- How would we detect misuse, and how do we turn the system off?
Try it
Set the autonomy levels
A car rental company wants an assistant for customers and branch staff. It could: answer questions about the rental agreement, extend a booking by a day, add a child seat, waive a late-return fee, charge a customer for damage, and report a car as stolen to the police. It reads customer emails and has access to the bookings and payments systems.
Give each action an autonomy level and justify it. Then check the design against the lethal trifecta.
Write your answer down first, then reveal the worked answerWorked answer
Agreement questions: automatic and read-only, quoting the customer's own agreement.
Add a child seat: automatic. Small, priced, and easy to undo.
Extend by a day: automatic after the customer confirms the new price, within a limit set in code and only if the car is free.
Waive a late fee: a draft for branch staff to approve to begin with; it costs money and sets expectations. Widen later if the data supports it.
Damage charge: suggest only. It is disputed often and has legal and reputational weight, so staff decide with photos in front of them.
Report as stolen: forbidden. A wrong report has serious consequences for a real person.
Trifecta check: it reads untrusted text (customer emails), holds private data (bookings and payments) and can send messages out. All three are present, so outbound messages go only to the customer on the booking, use fixed templates for anything about money, and any session that read an email needs staff approval before sending.