Measured, not claimed
A number without a baseline is a decoration.
So here are three predictors on the same 96 messages the model has never seen: the model, a keyword rule engine written the way most teams would write it, and the floor — always guessing the most common category.
Method
How these numbers were produced
128 labelled messages, split 32 / 96
Eight categories, sixteen messages each, half English and half Ukrainian, across email, chat and web form. The 32-message dev split is used for prompt work. The 96-message test split is what you see reported here.
The prompt never sees the test split
The 16 worked examples inside the system prompt come from the dev split only. Nothing in the reported 96 messages appears in the prompt, so this is a held-out measurement rather than a recital.
The baseline is a real attempt
Weighted keyword lists in both languages, regex for order numbers and amounts, and a tie-break order. Written from the category definitions and deliberately not tuned against the test set — a straw man would flatter the model.
You can check it
The full labelled set lives at data/testset.jsonl in the repository, one JSON object per message with its gold category, priority and entities. Every error the model made is listed further down this page.
The same set moves by a point or two
The model samples, so re-running this exact set does not give the exact same score. Three consecutive runs gave 91.7%, 92.7% and 93.8%. This page always shows the latest stored run with its timestamp, not the best one. Read the headline as "about 92%", and treat the gap to the baseline as the real result.
Per category
Where each predictor wins and loses
F1 balances precision and recall, so a category the model over-uses is penalised as well as one it misses. Twelve test messages per category.
The rest of the pipeline
Priority, entities and language
Entity scoring covers the four fields with an objectively correct answer: order number, amount, email and phone. Name, product and deadline are extracted but not scored, because reasonable people disagree on their exact boundaries.
Knowing when to stop
What it costs to catch the mistakes
The model's confidence is well calibrated at the top and useless at the bottom: correct answers never dropped below 0.85, but four of the wrong ones sat at 0.95. Confidence alone is not enough, so a second, free signal is used — disagreement with the keyword baseline. This is the resulting trade-off on the same 96 messages.
Full disclosure
Every message the model got wrong
Including the model's own one-line reasoning. Some of these are genuine judgement calls where a human agent could reasonably file the message either way — that is worth seeing rather than hiding.
Cost and speed