RAG Support Agent

An inbound support ticket arrives by webhook. The agent searches a document knowledge base and either answers from what it found or hands the ticket to a human. It never does both, and it never answers from memory.

Why retrieval rather than a fine-tune

The source document changes — prices, module lists, schedules. A fine-tune goes stale the day the document is edited and cannot be corrected without retraining. Pasting the whole document into the prompt works until the document outgrows the context window, and it pays for every token on every ticket regardless of relevance.

Retrieval keeps the answer tied to a document a non-engineer can edit. Re-run the indexing trigger and the agent is current.

Chunking, and why these numbers

Documents are split at 800 tokens with 80 tokens of overlap, and the top 8 passages are retrieved per query.

800 is small enough that a retrieved passage is mostly about the thing that matched, and large enough to carry a complete answer such as a pricing table. At 300 the retrieved text kept ending mid-sentence and the model filled the gap by guessing. At 2000 a single chunk spanned three unrelated sections and the relevant sentence got diluted.

The overlap exists because facts straddle boundaries. A course name at the end of one chunk and its price at the start of the next are useless apart. Ten percent overlap was enough that no answer in testing needed two adjacent chunks retrieval had not already pulled together.

Retrieving 8 passages is a recall decision. Precision is cheap here because the model discards irrelevant passages, but a missing passage becomes "I do not have that information" and an unnecessary escalation.

The escalation contract

The agent does not return prose. It returns a typed object with two fields: the reply text, and a boolean saying whether a human is needed.

This matters more than it looks. If escalation were inferred from the reply text — searching for "I am not sure", say — the routing would depend on phrasing, and phrasing is exactly what a language model varies. Making the escalation flag a separate boolean the model must set turns the decision into data, and a conditional node routes on it deterministically.

The system prompt names the categories that always escalate regardless of what retrieval returned: refunds, complaints, payments, and account-specific questions. These are cases where a fluent, well-grounded answer is still the wrong outcome, because the customer needs a person with authority to act.

Three constraints, because one is not enough

The tool description tells the agent to search before answering. The system prompt says to answer strictly from what the tool returned. And the prompt names the failure mode explicitly — never invent prices, products or facts.

The third exists because the first two still leave a well-behaved model happy to produce a plausible price when retrieval comes back thin. Naming the specific category of hallucination that would be most damaging works better than a general instruction to be accurate.

What escalation actually sends

The mail to the team carries the agent's draft reply alongside the customer's question. The retrieval work is already done, so a human edits rather than starting from nothing. That is the difference between an escalation that saves time and one that just moves the ticket.