Demonblade Labs · Issue 004
I Built A Product Assistant That Is Not Allowed To Guess
A catalogue assistant for a building materials maker, built from its public pages, that answers a tradie's question in about six milliseconds, shows the exact sentence every answer came from, and has four verdicts it may give. One of them is "not in the published data". It makes no model call at answer time, and the 25 answers it still gets wrong are named here, in public.
If you buy, sell or build product chatbots, this one is for you. It sells nothing and it costs nothing to read. The prototype was built unsolicited from a manufacturer's public website, so the company is not named here: they have not seen it yet, and they should see it before anyone else does. Nothing of theirs was touched, no account was created, no page was scraped behind a login. What you get is the design, the measurements and the misses.
What you will get out of it
- The four words an assistant is allowed to say, and why the fourth is the whole product.
- How an answer carries its own receipt: the sentence, the document and the page.
- The tradie test: 68 real phrasings, with typos and slang, before and after.
- A blind set written by someone who had not opened the code, and the 25 wrong answers it exposed.
- The speed and cost figures, and why "no running cost" is the weakest argument for it.
- Three questions to ask of any chatbot you are shown.
Chapter 0. The number is 25
Across 1,409 questions, 1,186 in the working bank and 223 in a blind set, the assistant gave 25 answers the published data does not support. Every one is listed by its question in the test suite, quarantined so the pass gate stays honest, and none has been edited away. The bank runs at 86.0 percent pass (947 answered, 73 honest no-data, 12 wrong, 154 missing). The blind set runs at 82.1 percent.
The reason the number is 25 and not zero is the point of the issue. An assistant that is never allowed to guess still guesses sometimes, and the only honest way to run one is to count the guesses out loud. A chatbot that reports a single satisfaction score has hidden that count.
Chapter 1. Four verdicts
Every answer the system gives falls into exactly one of four buckets when it is judged.
- ANSWERED. A product, a quantity, a property or a document, with the sentence it came from.
- HONEST. "Not in the published data." The catalogue does not say, so neither does the assistant.
- MISSING. The data holds the answer and the assistant failed to find it. A bug on my side, logged against the question.
- WRONG. The assistant named something the data does not support. Zero tolerance: each one is named, and the gate does not pass while one is hidden.
The second verdict is the product. When a tradie asks "recommend me one" with no job described, the assistant says it will not pick blind and asks the one question that unlocks a choice: inside or outside, what surface. Over 122 scripted conversations and 245 turns, 92 of them a request to recommend, that behaviour is now tested at every stage of a conversation.
A system that is allowed to say PASS about itself will, eventually. A system that is allowed to guess will, immediately.
Chapter 2. Show the source
Each statement in an answer carries a citation, and the citation has a button: Show the source. It opens the document with the exact sentence highlighted, and where the statement was quoted from a PDF, the page number.
| Measurement (1 November 2026) | Value |
|---|---|
| Golden questions with a known right answer | 4,659 |
| Answered correctly | 4,599 (98.71 percent) |
| Distinct citations across those answers | 4,529 |
| Statements quoted from a PDF that carry the page number | 1,639 of 1,639 |
| Property checks across the golden set | 0 violations |
The 98.71 percent did not move across the last three rounds of work, and it was not meant to. The golden set is the floor: any change that drops it is refused before it merges.
Chapter 3. The tradie test
Nobody on a site types the product name the catalogue uses. I wrote 68 phrasings before the matcher changed: 16 typos, 15 slang, 7 old product names, 14 jargon, 16 plain. One matcher now serves the search box, the product finder and the assistant, with spelling corrected against the catalogue's own vocabulary and old names mapped to the current product.
| Phrasing kind | Right product in the top 3, before | After |
|---|---|---|
| Typos (16) | 3 | 15 |
| Slang (15) | 5 | 12 |
| Old product names (7) | 6 | 7 |
| Jargon (14) | 10 | 10 |
| Plain (16) | 12 | 14 |
| All 68 | 36 (52.9 percent) | 58 (85.3 percent) |
Top-1 went from 45.6 to 73.5 percent, top-6 from 54.4 to 91.2 percent. The jargon row did not move, and that is where the next work is: the ten misses exposed four generic words the synonym groups did not know.
Chapter 4. The blind set, and what it caught
A bank you wrote while reading the code tests what you already thought of. So 223 questions were written without opening the assistant's code, 44.8 percent of them deliberately messy: typos, half sentences, the way a question sounds at seven in the morning. First run, before any triage: 80.5 percent. After triage of the tooling, with no change to the assistant: 82.1 percent (183 of 223).
Three of the wrong answers it caught, because they show the shape of the failure.
- "wall is damp" was answered with a marine acoustic damping page. The stem of "damp" collided with "damping". The data has nothing on damp walls, so the right verdict was HONEST.
- "need glue" returned a specific adhesive grout with no job described. A silent guess. The right move was the clarifying question.
- "how do i fill a 30mm low spot in the slab" named a floor leveller whose published depth range does not reach 30 millimetres. The answer was plausible and unsupported, which is the worst kind.
Each is now a named row. The judge also reclassified seven rows as faults in the bank itself, where the expectation was tighter than the data, and corrected five earlier rows where the bank said "not published" and the product tables held the value. The test set is held to the same standard as the system.
Chapter 5. Speed and cost, and the weakest argument
No model is called when a question is answered. The matcher, the rules and the catalogue data ship to the browser and run there, so the assistant works offline on a phone in a basement car park.
| Measurement | Value |
|---|---|
| Answer time, 1,186-question bank, median | about 6 ms |
| Answer time, 95th percentile (box loaded by three other builds) | about 35 ms |
| Golden set, median | 0.28 ms |
| First screen, gzip, against a 90,000 byte budget | 87,221 bytes |
| Model cost at answer time | US$0 |
| Box the whole thing is built on | about US$30 a month |
Here is the honest part. "No running cost" is the weakest reason to want this. A model-run chat at 100,000 sessions a year costs somewhere between US$360 and US$7,200 in model fees on published prices, which is cheap for any company with a catalogue worth a chatbot. The reason to want this design is the receipt on every sentence and the fourth verdict. A model will answer "wall is damp" fluently, every time, and that is the problem.
Chapter 6. The battery
On the final commit of this round, ten test suites ran with every inner exit code read: catalogue stock checks, image checks, browser checks at desktop and phone widths, the matcher and citation units, voice and thumbnail checks, the job comparison, the brain units and the golden set, the action and cart suites with a 3,794-case property test. 5,005 checks, 0 failed, plus the 4,599 golden answers. The rebuild script runs twice and leaves the tree clean. Everything here was measured on 31 October and 1 November 2026.
The limits of all of this
The things I would attack if I were reading this.
- The golden set and the bank were written by the same system that is being judged. The blind set is the only independent number, and it is 223 questions.
- 82.1 percent on the blind set means roughly one question in six is still missed or declined. Most of those are MISSING, where the data has the answer and the search did not reach it.
- Timing was measured in a headless runtime on a loaded box, not on a tradie's phone over a site connection. The first screen budget was measured on a simulated fast 3G link.
- It covers one maker's public catalogue, 512 products. It has not been shown to the maker. Whether it changes a single purchase is UNKNOWN until it has.
- Spelling correction is deliberately conservative, so it does not yet lift the messy sets. That is a chosen trade for zero wrong.
Your three questions
If you are shown a product chatbot this week, three questions cost nothing. Ask it where the sentence is: can it open the document and highlight the line it used? Ask it something the catalogue does not cover, and listen for whether it says so or answers anyway. Then ask the vendor how many questions it was tested on that were written by someone who had not seen the code, and what the wrong count was. If the answer to the last one is a satisfaction score, you have your answer.
What has to be true before an assistant is allowed in front of a customer: it has a verdict for "I do not know", every answer carries its source, it is tested on questions it has never seen, and its wrong answers are counted in public. That is the bar I hold mine to.
If you run a catalogue, a counter or a chatbot, I would genuinely like to hear what your customers actually type. Reply on X with the worst question you have seen a product search get, or tell me where four verdicts are too few. Would it be okay if I asked you for the one question your current search gets wrong?
Receipts
- Every figure above is a test suite result or a byte count from the project, measured on 31 October and 1 November 2026, or stated arithmetic. The model fee range is modelled on published per-token prices at 100,000 sessions a year.
- The catalogue is the maker's public web pages and data sheets. No private data, no login, no account.
Next issue: The button that ignored a fast thumb.
Previous issues: I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.