State, memory and agent boundaries

Demos run on one laptop with one user. Production runs on several servers with thousands of users, any of which can fail at any moment. This chapter covers two questions that separate the two: where state lives, and where the lines go when one agent is not enough.

Why assistants forget in production#

In a prototype, the easiest place to keep a conversation is a variable in the running program. Production is different: the same program runs as several copies so it can survive load and failures, and each incoming message is handed to whichever copy is free. Every copy has its own private memory. Message two of a conversation is often handled by a copy that was never shown message one.

It shows up as an assistant with random amnesia, and it is usually blamed on the model. The model was never given the earlier messages. You can't solve it by steering each user to the same copy for ever, because copies are replaced on every deploy and whenever one crashes. The conversation has to live outside the program, somewhere all copies read from.

Three identifiers, kept apart#

Any assistant with sign-in juggles three different identifiers, and mixing them up causes a whole family of bugs. There is the proof of sign-in, which is short-lived and replaced every time the person logs in. There is the person, who stays the same for years. And there is the particular conversation, because one person may have many.

Store conversations under the person and the conversation, never under the proof of sign-in. File them under the sign-in credential and every user starts from a blank page each morning, when their session renews. Translate the credential into a person once, at the door, and use the person from then on.

Choosing where state lives#

Start with whatever database the client's team already runs and backs up, which is often Postgres or SQL Server. A table of messages, indexed by conversation and time, is plenty fast for most assistants. Reach for a key-value cache such as Redis only for a reason you can name: values that should vanish on their own, like temporary state; counters every copy must share, like rate limits; or very hot reads at large scale. An unnecessary extra system is one more thing for the client to operate after you leave.

The question that matters more than the technology is retention. How long may conversations be kept, who may read them, and must they be deletable on request? Those are data-protection decisions for the client, and the answers belong in the design doc before the first message is stored.

Long conversations#

Sending the entire history with every message gets slower and more expensive each turn, until the request hits the model's context limit and fails. Decide early what the model actually needs.

• Recent turns in full, sized to your budget and measured, not guessed.
• A short summary of everything earlier, rewritten as the conversation grows.
• Search over the archive for when someone says "like we discussed last week".
• Facts as data: preferences and account details kept in fields, not buried in old messages.

One design choice catches teams out: whether to store what tools returned. Without it, every follow-up question triggers the same lookups again, which is slower and costlier but never stale. With it, answers are faster but can rest on outdated facts. Pick deliberately per tool, and put a size limit and an expiry on everything you keep.

Before calling an assistant production-ready, be able to say where each piece of state is kept, how it is found again, and when it is deleted.

When one agent becomes several#

A single agent that reads the claim form, checks the policy, decides on a courtesy car and writes to the customer is a good first version. It gets harder to live with as it grows. Improving its letters risks breaking its policy checks. When a customer gets a wrong answer, the logs show one long trajectory and no obvious culprit. And it holds every permission any of its jobs needs, all the time.

Split it where the responsibilities, the permissions or the owners genuinely differ: the part that reads documents doesn't need to send email, and the policy check belongs to a different team than the letters. Don't split for the sake of it. Every extra agent adds a hand-off, a contract to maintain and a place for things to go missing. If you can't describe a proposed agent's job in one plain sentence, it shouldn't exist.

Keep the coordinator thin. Its job is sequencing: who to ask, in what order, and what to do if someone doesn't answer. If it starts judging eligibility itself, you have quietly rebuilt the single large agent.

How agents talk to each other#

Wait, or announce?

If the next step can't start without the answer, ask directly and wait: the letter can't be written until courtesy-car eligibility is known. If nothing is waiting on the answer, post a message that the thing happened and let interested parties pick it up in their own time. A newly registered claim might start an adjuster assignment, a fraud check and an acknowledgement letter, none of which need to hold each other up. Waiting is simpler to reason about; posting messages keeps one slow part from slowing everything.

Contracts and versions

The data passed between agents has a shape: field names, types, required fields, error formats. Write it down as a schema and validate it at both ends. Renaming one field can break a workflow in which every agent is individually correct. When a contract must change, keep the old version working until every consumer has moved.

Containing failure#

Every dependency will fail eventually: a model provider, a CRM, an internal service. Design so the failure stops at its boundary.

• Timeouts on every call. A missing timeout turns a slow dependency into a hung request, and then a hung service.
• Retries with care. Retry reads with increasing delays. Retry writes only with an idempotency key.
• Circuit breakers. After repeated failures, stop calling the dependency for a while and fail fast, then test it again later. The pattern, popularised by Michael Nygard, stops one outage from consuming all your capacity.
• Graceful degradation. Decide what the user sees when part of the system is down. If recommendations fail, the product page still loads without them. If the research step fails, offer a draft labelled as missing that evidence.
• Queues for slow or bursty work, so a spike becomes a delay rather than a failure.