Demonblade Labs · Issue 006
The Exam The Builder Never Saw
Today's round made the product assistant conversational. On my own probes, honest no answers climbed from 24 to 38 of 46. On a final exam the builder never saw, the same category went from 13 to 11 of 23. A builder's own probe gains are hypotheses.
If you build, buy or sell anything with a chat assistant in it, this one is for you. It sells nothing and it costs nothing to read. This follows Issues 004 and 005: the same unsolicited prototype, built from a building materials maker's public website, so the company is still not named. Nothing of theirs was touched.
What you will get out of it
- Two measurements of the same round that disagree.
- What an independent reviewer found behind green tests, in plain words.
- How the fixes were verified and taken live.
- The two numbers that are still open.
Chapter 0. The number is 71
The final exam has 120 questions. It was written and held back so the builder could not tune to it. The version before the round scored 64. The version after scored 71 of 120. That is 7 more correct answers, and it is real progress.
But the honest no category, the questions where the right answer is a plain no, went from 13 to 11 of 23. It went down, while my own probes for the same category went from 24 to 38 of 46.
| Measure | Before | After |
|---|---|---|
| Own probes, honest no (46) | 24 | 38 |
| Final exam, overall (120) | 64 | 71 |
| Final exam, honest no (23) | 13 | 11 |
The probes I write while fixing a problem are the questions I already know how to answer. They show the fix works on the shape I was looking at. They do not show the assistant got better. A builder's own probe gains are hypotheses.
Chapter 1. Green suites are not a verdict
The suites were green. Then an independent adversary reviewer, who had built none of it, read the change and found 3 blocking problems and 5 serious ones that a buyer would see. The three blocking ones, in plain words:
- A product from earlier in the conversation was carried into the answer to a quantity question, so the answer could be about the wrong product.
- Some product range names were being treated as aliases for individual products, which sends a buyer to the wrong item.
- "Yes, happy to help" was said for things the company does not sell. A cheerful yes to a product the maker does not make is the failure this assistant exists to prevent.
Chapter 2. Fixed, re-run by me, then live
All three were fixed. I then verified the fixes myself instead of taking the builder's word for it.
| Check | Result |
|---|---|
| Golden set | 4,600 |
| Unit review | 49 of 49 |
| Conversation and cart tests in a real browser | green |
| The reviewer's own probes | all fixed |
| Live on both previews, files hash-matched | 4 of 4 on each |
The hash match means what is serving is byte for byte what was tested. Nothing was merged to the main branch, and the previous version is kept for a one step revert.
The limits of all of this
- The final exam is 71 of 120. I am not rounding that into a pass.
- The honest no category is 11 of 23 against 13 before the round. That regression is not closed.
- The earlier size, accessibility and performance misses from Issues 004 and 005 are still on the list.
- It covers one maker's public catalogue, and whether any of it changes a purchase is UNKNOWN until they have seen it.
Your three questions
Do you have an exam the builder has never seen, held where the builder cannot read it? When your own probes go up and the held-back exam does not, which one do you believe? Has someone who did not build it tried to break it before a buyer did?
What has to be true before I call a gain real: it shows up on questions I never saw, an independent reviewer has tried to break it, I re-run the fixes against the real build, and what is live matches what was tested by hash. Today only some of that is met.
If you build assistants, I would genuinely like to hear how you keep a test set honest. Reply on X with the thing that stopped you grading your own homework. Would it be okay if I asked you what the one question is that your assistant must answer with a no?
Receipts
- Every figure above is a test run result or a count from the project, measured on 2 November 2026.
- The catalogue is the maker's public web pages and data sheets. No private data, no login, no account.
Next issue: Twelve blockers behind a green suite.
Previous issues: The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.