Two levels: the business one, then the engineering one
A mail room with one very fast clerk.
Everything that arrives in a shared inbox gets read once, sorted into one of eight trays, stamped with how urgent it is, stripped of the few facts worth keeping, and passed to the team that owns it — with a reply already written. What used to be a person's first hour of the day happens in about four seconds and costs a fifth of a cent.
The business case
Where the money actually is
Not a chatbot
A chatbot talks to one customer at a time and needs supervision. This reads the whole stream and hands off. Nobody has to be in the conversation for it to be useful.
The urgent ones surface first
A chargeback threat sitting behind forty newsletter pitches is the expensive failure. Priority is decided on arrival, so the escalation reaches management the same minute rather than the next morning.
Honest about doubt
Roughly a quarter of the stream is deliberately not auto-routed. A message in a review queue costs a minute of attention. A message routed to the wrong team costs a day.
The pipeline
One model call, five outputs
Splitting this into five calls would be five times the cost and five times the latency for no measurable gain. The schema forces every field to come back in one response, so the whole decision is made with the whole message in view.
message arrives ──► keyword baseline (local, free, 1 ms)
│
├──────────────► used as a second opinion
▼
one model call ──► category · priority · sentiment
(structured entities · summary · reply draft
output, streamed) confidence
│
▼
┌── agree, confident ────► routed to the team, SLA clock starts
└── disagree or unsure ──► human review queue
│
▼
delivery queue (retries, backoff, dead-letter)
Engineering
Five decisions and what each was measured to be worth
1. The prompt is deliberately larger than it needs to be
Prompt caching on this model only engages from 4096 tokens of prefix. At 12 worked examples the prefix weighed 3804 tokens — under the line, so the cache silently never engaged and every message paid full price for the same 3800 tokens. Adding a Ukrainian example for each category brought it to 4342 and turned the cache on. The check is a build command, node dist/cli/eval.js --tokens, because "silently not caching" is not something you notice in a log.
2. One-hour cache, not the default five minutes
The stream produces a message every 7–25 minutes. With the default 5-minute cache every single call would have been a cache write at 1.25× the normal price — worse than not caching at all. The one-hour cache costs 2× to write and 0.1× to read, and at this traffic it pays for itself several times over within the hour. The cost per message on the live stand sits near $0.002; without the cache the same prompt would cost about $0.006.
3. The schema does the parsing, not a regex over the model's prose
The response comes back through a JSON schema with enums for category, priority and sentiment, so an invalid category is impossible rather than unlikely. The output is still validated on arrival: if anything fails the check, the message falls back to the keyword baseline and is marked for a human instead of being routed on a guess.
4. Regex kept the order numbers, and that was a measurement, not a preference
Order numbers, amounts, emails and phones were scored both ways on the held-out split. The model reached 71% F1; a handful of regexes reached 97.5%. The model normalises and infers — helpful in prose, wrong when the task is "copy exactly what is written". So the regexes own those four fields and the model owns name, product and deadline, where no regex can help. That combination ships.
5. Confidence alone did not catch a single mistake
The first threshold was 0.6, chosen the way most people choose it — by feel. On 96 messages it flagged nothing at all, because the model never returns a value that low. Its correct answers never went below 0.85, and four of its wrong answers sat at 0.95. A second, free signal fixed it: when the keyword baseline picks a different category, that message goes to a human regardless of how confident the model sounds. Together the two signals catch seven of eight errors.
See the full trade-off curve on the accuracy page →
What is synthetic here, and what is not
Synthetic: the company, the products, the customers and every message in the stream. Vaskra Outdoor does not exist. Messages are assembled from templates with randomised names, order numbers and amounts, so each one is genuinely new text rather than a replayed recording.
Not synthetic: every analysis on this page. The stream goes through the same model, prompt and code path as the message you paste. The accuracy numbers come from a stored run against a labelled set that lives in the repository, and the cost figures are the tokens that were actually billed.
Your text is not stored. Pasted messages are analysed and discarded — only the anonymous cost and latency counters survive. The demo inbox resets to a reference state every six hours.
Guards, because a public AI endpoint is a public wallet
Every path that spends money goes through the same gate: a daily budget in dollars, a daily call ceiling, per-IP limits per minute and per day, a maximum message length, and a concurrency cap. The demo stream carries its own separate budget, so a busy day of visitors cannot starve the live feed and vice versa. When a budget runs out the stand says so plainly rather than failing in a confusing way. IP addresses are salted and hashed, never stored raw.
Stack
Node 22, Fastify, TypeScript, SQLite. Claude Haiku 4.5 with structured outputs and prompt caching. No frontend framework — the page is HTML, one stylesheet and a few modules, which is also why it loads the way it does. The whole thing is one process behind nginx, with the demo reset and the accuracy run as separate commands in the same binary.