Part IV · Proving it works · Chapter 12
Cost, latency and sizing
Price one unit of work, compare it to what the customer pays today, and size the system on a napkin before you build it.
13 min read
Most AI projects that fail commercially don't fail because the model was wrong. They fail because nobody priced one unit of work against what the customer already pays, or because a system that was fine at 100 requests a day fell over at 10,000. This chapter gives you the arithmetic to avoid both, and a calculator to do it live.
What you'll take away
- 01How to price one unit of work, today and with your system.
- 02Where model costs come from, and the levers that move them.
- 03The costs that don't appear on a pricing page, especially people.
- 04How to size a system on a napkin in two minutes.
- 05How to treat latency as a budget split across stages.
Start from a unit of work
Pick the unit the customer already counts: a ticket, an invoice, a claim, a document reviewed. Find out what one costs today. That is usually minutes of someone's time multiplied by their loaded hourly cost, which includes salary, benefits and overheads. If you can't get an hourly figure, ask for team cost and monthly volume and divide.
This baseline is the most important number in the engagement. It turns "AI will save time" into "each claim costs $3.20 today and would cost $0.60". Sponsors can take that sentence to a finance committee.
Where model cost comes from
Model APIs charge per token, usually more for output than input. Cost per unit is roughly the number of model calls per unit multiplied by the cost of one call, and the cost of one call is its input tokens at the input rate plus its output tokens at the output rate. The levers follow directly.
| Lever | How it saves | Watch out for |
|---|---|---|
| Shorter context | Fewer input tokens: retrieve less, summarise history, trim boilerplate | Removing evidence the answer needed |
| Fewer calls | Merge steps, skip the model where code can decide | Merged prompts that do several jobs badly |
| Smaller model for easy cases | Route routine cases to a cheaper model, hard ones to a larger one | Routing errors; evaluate each route |
| Caching | Reuse answers or prompt prefixes that repeat | Cache keys missing user or permission scope: a data leak |
| Batch processing | Many providers discount work that can wait hours | Only for work nobody is waiting on |
| Shorter outputs | Ask for structured, concise output; cap output tokens | Truncated answers |
Run the numbers
Change the inputs to match your client. The defaults describe a typical document-handling task. Notice how quickly the human review line dominates. Halving the review rate usually matters far more than halving the model price.
Of the new cost, $0.030 is the model and $0.150 is people reviewing.
Saves $36,400 a month
Illustrative defaults. Check your provider's current prices, and add hosting, evaluation and support time before quoting a client.
In most deployments the expensive part is not the model. It is the people correcting what the model got wrong.
The costs nobody puts on the slide
- Review and correction time, which depends on your error rate. Improving accuracy usually saves more than switching models.
- Retries and failures: calls that time out or fail validation and run again.
- Infrastructure: hosting, the vector database, storage, logging and monitoring.
- Evaluation runs: the full test suite on every change, sometimes with an LLM judge.
- Operations: someone owns this system after you leave. Their time is a cost.
- Growth: costs at today's volume and at ten times today's. Some grow linearly, some don't.
Sizing on a napkin
Interviewers ask for this, and clients need it, because it tells you which parts of the design matter. Work from people to requests, then to what each request fans out into downstream.
| Step | Example | Running total |
|---|---|---|
| Employees who could use it | 8,000 staff, 5,000 in relevant roles | 5,000 |
| Daily active | 30% of eligible | 1,500 people |
| Requests per active user | 4 a day | 6,000 a day |
| Average rate over a 9-hour day | 6,000 ÷ 32,400 s | ≈ 0.19 per second |
| Peak | × 4 for the morning rush | ≈ 0.75 per second |
| Fan-out per request | 2 searches, 1 rerank, 2 model calls | ≈ 1.5 searches and 1.5 model calls per second at peak |
The answer is usually humbling: the request rate is modest. The hard problems are rarely raw throughput. They are rate limits on the model provider, latency added by each stage, slow downstream systems, and correct permissions. Say that out loud in an interview. It shows you sized the system in order to make decisions, not to show off arithmetic.
Latency is a budget
Agree a target with the client, ideally as a percentile (95% of answers within 5 seconds, say), then split it across stages. Each stage gets an allowance. A change that blows one stage's allowance has to win it back elsewhere or justify a higher total.
What actually helps
- Stream the answer, so perceived latency is time to first token.
- Run independent steps in parallel: searches, lookups, checks.
- Rerank a short list, not a long one; it is often the slowest step.
- Keep the model in the same region as the data and the users.
- Set timeouts per stage with a defined fallback, so one slow dependency doesn't stall everything.
- Load-test at peak, not at quiet hours. Queues behave differently under load.
Cost guardrails
Treat cost like any other production signal, with an owner, alerts and limits.
- Budget per workflow, tracked daily, with an alert well before the limit.
- Caps on tokens per request and tool rounds per request, set in code.
- Per-user or per-tenant rate limits, so one heavy user or an attacker cannot run up the bill. Security people call this denial of wallet.
- A cost check in the release pipeline: tokens and cost per request on the eval set, compared with the last version.
- Dashboards by workflow, model, prompt version and cache hit rate. That is how you find the cause of a spike in minutes, not days.
Try it
Size and price a pilot
A municipal permits office receives 3,600 applications a month. Staff spend 12 minutes checking each for completeness, at a loaded cost of $40 an hour. Your design makes three model calls per application, about 6,000 input and 600 output tokens each, at $3 and $15 per million tokens. You expect 20% of applications to need a 5-minute human check.
Work out the cost per application today and with the system, the monthly saving, and the request rate at peak if applications arrive over 8-hour working days, 22 days a month, with a 3× peak.
Write your answer down first, then reveal the worked answerWorked answer
Today: 12 minutes at $40 an hour is $8.00 per application.
Model: 3 × (6,000 × $3 + 600 × $15) ÷ 1,000,000 = 3 × $0.027 = about $0.08.
Review: 20% × 5 minutes at $40 an hour = $0.67. Total with the system: about $0.75 per application.
Monthly saving: ($8.00 − $0.75) × 3,600 ≈ $26,100, before hosting, evaluation and support.
Rate: 3,600 ÷ 22 days ≈ 164 a day; over 8 hours that is about 0.006 a second, or about 0.017 at a 3× peak. Throughput is irrelevant here. Accuracy on the completeness check and the review rate decide the business case. A strong answer says so.
Go deeper
Practise the skills
- Ownership & business valueOwn the outcome, not the ticket. Then turn what you learn in the field into product.
- Applied GenAI: RAG, agents & toolsBuild LLM systems that are grounded, bounded and debuggable, not just impressive in a demo.
- Reliability & incident responseBuild systems that fail safely, and stay calm when they break at 2 AM.