AI research at scale

Language models can read a company's website, job ads and news and answer questions about them across thousands of rows, research that used to take an assistant all week. They can also be confidently wrong at the same scale. The craft is asking questions narrow enough to answer reliably, and checking the answers before they reach a prospect.

What AI research is good for#

• Classifying: is this company B2B or B2C? Does it sell physical products? Which of these five industries fits best?
• Extracting: find the number of locations, the name of their payroll provider, whether they mention ISO 27001.
• Summarising: what did they announce in the last three months that matters to us?
• Drafting: one sentence connecting a fact about them to the problem we solve. More on this in chapter 12.

It is weakest at facts that aren't on the pages you give it. If you ask a model for a company's revenue and the website doesn't say, it may produce a plausible number anyway. Give it the source text, and ask it to say when the answer isn't there.

Prompts that survive a thousand rows#

A prompt that works on the three rows you tried will meet hundreds of odd cases in production. Write it for the odd cases.

You are given the text of a company's website.
Question: does this company operate more than one physical location
(clinics, offices, stores or sites)?

Answer in JSON:
{"multiple_locations": "yes" | "no" | "unclear",
 "count": number or null,
 "evidence": "a short quote from the text that supports the answer, or null"}

Rules:
- Use only the text provided. If it doesn't say, answer "unclear".
- Do not guess a count. Use null unless the text states or lists the locations.

• One question per prompt. Combining five questions makes each answer worse and harder to check.
• Fixed answer options, including 'unclear'. Free text is hard to filter and hides uncertainty.
• Ask for evidence: a quote from the source. It makes errors visible and checking fast.
• Feed the source text in rather than asking the model to look things up from memory.

Check before you trust#

1. 1

   Validate automatically

   Reject answers outside the allowed values or with missing evidence. Re-run or mark them unclear.
2. 2

   Check the evidence exists

   A quick text search confirms the quoted evidence actually appears in the source. Invented quotes are a common failure.
3. 3

   Sample by hand

   Read 20 random rows, plus every row that will be used in a message. Count errors.
4. 4

   Set a bar

   If the sample error rate is above what you'd accept, say 1 in 20, tighten the prompt and repeat before using the column.

No evidence, no claim. If the model can't quote its source, don't send what it wrote.

Cost and speed#

• Run AI research only on rows that passed the cheap filters (see chapter 8).
• Trim the input: the relevant page or section, not the whole site.
• Use a smaller, cheaper model for simple classification; save larger models for drafting or hard extraction.
• Cache results by company, and don't re-research a company you looked at last week.

What not to automate#

Anything that would be harmful or embarrassing if wrong deserves a person: claims about a prospect's finances, health, legal trouble or personal life; anything sensitive in regulated industries; and messages to your most valuable accounts. AI can prepare the research. A person should read it before it goes out.

Research this company and tell me if they'd be a good fit for our HR software,
how many employees they have, and write a personalised opening line.