TITANOS

Demonblade Labs · Issue 004

I Built A Product Assistant That Is Not Allowed To Guess

A catalogue assistant for a building materials maker, built from its public pages, that answers a tradie's question in about six milliseconds, shows the exact sentence every answer came from, and has four verdicts it may give. One of them is "not in the published data". It makes no model call at answer time, and the 25 answers it still gets wrong are named here, in public.

If you buy, sell or build product chatbots, this one is for you. It sells nothing and it costs nothing to read. The prototype was built unsolicited from a manufacturer's public website, so the company is not named here: they have not seen it yet, and they should see it before anyone else does. Nothing of theirs was touched, no account was created, no page was scraped behind a login. What you get is the design, the measurements and the misses.

What you will get out of it


Chapter 0. The number is 25

Across 1,409 questions, 1,186 in the working bank and 223 in a blind set, the assistant gave 25 answers the published data does not support. Every one is listed by its question in the test suite, quarantined so the pass gate stays honest, and none has been edited away. The bank runs at 86.0 percent pass (947 answered, 73 honest no-data, 12 wrong, 154 missing). The blind set runs at 82.1 percent.

The reason the number is 25 and not zero is the point of the issue. An assistant that is never allowed to guess still guesses sometimes, and the only honest way to run one is to count the guesses out loud. A chatbot that reports a single satisfaction score has hidden that count.


Chapter 1. Four verdicts

Every answer the system gives falls into exactly one of four buckets when it is judged.

  1. ANSWERED. A product, a quantity, a property or a document, with the sentence it came from.
  2. HONEST. "Not in the published data." The catalogue does not say, so neither does the assistant.
  3. MISSING. The data holds the answer and the assistant failed to find it. A bug on my side, logged against the question.
  4. WRONG. The assistant named something the data does not support. Zero tolerance: each one is named, and the gate does not pass while one is hidden.

The second verdict is the product. When a tradie asks "recommend me one" with no job described, the assistant says it will not pick blind and asks the one question that unlocks a choice: inside or outside, what surface. Over 122 scripted conversations and 245 turns, 92 of them a request to recommend, that behaviour is now tested at every stage of a conversation.

A system that is allowed to say PASS about itself will, eventually. A system that is allowed to guess will, immediately.


Chapter 2. Show the source

Each statement in an answer carries a citation, and the citation has a button: Show the source. It opens the document with the exact sentence highlighted, and where the statement was quoted from a PDF, the page number.

Measurement (1 November 2026)Value
Golden questions with a known right answer4,659
Answered correctly4,599 (98.71 percent)
Distinct citations across those answers4,529
Statements quoted from a PDF that carry the page number1,639 of 1,639
Property checks across the golden set0 violations

The 98.71 percent did not move across the last three rounds of work, and it was not meant to. The golden set is the floor: any change that drops it is refused before it merges.


Chapter 3. The tradie test

Nobody on a site types the product name the catalogue uses. I wrote 68 phrasings before the matcher changed: 16 typos, 15 slang, 7 old product names, 14 jargon, 16 plain. One matcher now serves the search box, the product finder and the assistant, with spelling corrected against the catalogue's own vocabulary and old names mapped to the current product.

Phrasing kindRight product in the top 3, beforeAfter
Typos (16)315
Slang (15)512
Old product names (7)67
Jargon (14)1010
Plain (16)1214
All 6836 (52.9 percent)58 (85.3 percent)

Top-1 went from 45.6 to 73.5 percent, top-6 from 54.4 to 91.2 percent. The jargon row did not move, and that is where the next work is: the ten misses exposed four generic words the synonym groups did not know.


Chapter 4. The blind set, and what it caught

A bank you wrote while reading the code tests what you already thought of. So 223 questions were written without opening the assistant's code, 44.8 percent of them deliberately messy: typos, half sentences, the way a question sounds at seven in the morning. First run, before any triage: 80.5 percent. After triage of the tooling, with no change to the assistant: 82.1 percent (183 of 223).

Three of the wrong answers it caught, because they show the shape of the failure.

Each is now a named row. The judge also reclassified seven rows as faults in the bank itself, where the expectation was tighter than the data, and corrected five earlier rows where the bank said "not published" and the product tables held the value. The test set is held to the same standard as the system.


Chapter 5. Speed and cost, and the weakest argument

No model is called when a question is answered. The matcher, the rules and the catalogue data ship to the browser and run there, so the assistant works offline on a phone in a basement car park.

MeasurementValue
Answer time, 1,186-question bank, medianabout 6 ms
Answer time, 95th percentile (box loaded by three other builds)about 35 ms
Golden set, median0.28 ms
First screen, gzip, against a 90,000 byte budget87,221 bytes
Model cost at answer timeUS$0
Box the whole thing is built onabout US$30 a month

Here is the honest part. "No running cost" is the weakest reason to want this. A model-run chat at 100,000 sessions a year costs somewhere between US$360 and US$7,200 in model fees on published prices, which is cheap for any company with a catalogue worth a chatbot. The reason to want this design is the receipt on every sentence and the fourth verdict. A model will answer "wall is damp" fluently, every time, and that is the problem.


Chapter 6. The battery

On the final commit of this round, ten test suites ran with every inner exit code read: catalogue stock checks, image checks, browser checks at desktop and phone widths, the matcher and citation units, voice and thumbnail checks, the job comparison, the brain units and the golden set, the action and cart suites with a 3,794-case property test. 5,005 checks, 0 failed, plus the 4,599 golden answers. The rebuild script runs twice and leaves the tree clean. Everything here was measured on 31 October and 1 November 2026.


The limits of all of this

The things I would attack if I were reading this.


Your three questions

If you are shown a product chatbot this week, three questions cost nothing. Ask it where the sentence is: can it open the document and highlight the line it used? Ask it something the catalogue does not cover, and listen for whether it says so or answers anyway. Then ask the vendor how many questions it was tested on that were written by someone who had not seen the code, and what the wrong count was. If the answer to the last one is a satisfaction score, you have your answer.

What has to be true before an assistant is allowed in front of a customer: it has a verdict for "I do not know", every answer carries its source, it is tested on questions it has never seen, and its wrong answers are counted in public. That is the bar I hold mine to.


If you run a catalogue, a counter or a chatbot, I would genuinely like to hear what your customers actually type. Reply on X with the worst question you have seen a product search get, or tell me where four verdicts are too few. Would it be okay if I asked you for the one question your current search gets wrong?

Receipts

Next issue: The button that ignored a fast thumb.

Previous issues: I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.