Demonblade Labs · Issue 007
Twelve Blockers Behind A Green Suite
Over 2 and 3 November the product assistant went through four independent review rounds. Each round found a defect a buyer would have seen, 12 in total, and the tests were green before every one. The rule that came out of it: keep reviewing until a round comes back clean on a number path.
If you build, buy or sell anything with a chat assistant in it, this one is for you. It sells nothing and it costs nothing to read. This follows Issues 004, 005 and 006: the same unsolicited prototype, built from a building materials maker's public website, so the company is still not named. Nothing of theirs was touched.
What you will get out of it
- How many rounds a reviewer needed before a green suite was believable, and what each round found.
- The rule for when to stop reviewing and ship.
- The exam numbers before and after.
- What is still open.
Chapter 0. The number is 12
A blocking defect here means one a buyer would notice straight away: a wrong quantity, a wrong product, a confident yes where the answer is no. Each reviewer had built none of the code. Each time I re-ran the fixes myself instead of taking the builder's word for it, and each time the next reviewer found more.
| Round | Blocking defects |
|---|---|
| 1 | 3 |
| 2 | 5 |
| 3 | 2 |
| 4 | 2 |
| Total, all fixed | 12 |
Chapter 1. What each round found
- Round one: a stale product carried into a quantity answer, range names treated as aliases for single products, and a cheerful "yes, happy to help" for things the company does not sell.
- Round two: a product held from earlier hijacking a short question about a category, and the word "special" being read as a price question, among others.
- Round three: a bare "2400 x 1200" was read as an area and came back as 2,880,000 square metres and 480,000 bags. A tile size was also read as the job area.
- Round four: a tile size typed after a product card became the job area and went into the cart. A tile measurement in millimetres was read as a layer thickness, which turned a 10 square metre job into 285 bags. That one was already live in the version before this round.
A green suite tells you the problems you thought of are gone. It cannot tell you about the ones you did not.
Chapter 2. The rule
Keep reviewing until a round comes back with no blocking defect on a number path, then ship.
Not after one round, not after two. Numbers are where a wrong answer costs a buyer money, so numbers get the strictest exit test.
Chapter 3. The exam numbers
Alongside the reviews I ran a fresh exam of 100 questions the builders never saw, 40 of them strict honest-no questions where the right answer is a plain no.
| Measure | Start | First fixes live | Now |
|---|---|---|---|
| Unseen exam, overall (100) | 52 | 60 | 78 |
| Unseen exam, honest no incl. multi-turn (48) | n/a | 17 | 32 |
| Held-back final exam (120) | 64 | 71 | 73 |
The final exam moved by 2. That is a small move and I am not dressing it up.
The limits of all of this
- The last fix was checked by my own number probes, which found no wrong quantity. I have no fifth independent review round on record, so by my own rule it has not earned its pass yet.
- It is live on two previews only, not merged to the main branch.
- The exam I used to pick the better of two builds is partly used up, so 78 of 100 is a little flattering.
- On the final exam, the clarifying-question category is still 6 of 14.
- Quantity fallbacks and a fuzz suite for the quantity logic are still on the list, as are the earlier size, accessibility and performance misses from Issues 004 and 005.
- It covers one maker's public catalogue, and whether any of it changes a purchase is UNKNOWN until they have seen it.
Your three questions
Who tried to break your assistant that did not build it? How many rounds did they need before one came back clean? Did anyone check the number paths separately from the words?
If you build assistants, I would like to hear how many review rounds your last release needed before one came back clean. Reply on X. Would it be okay if I asked you what the most expensive wrong number your assistant could give a customer would be?
Receipts
- Every figure above is a test run result or a count from the project, measured on 2 and 3 November 2026.
- The catalogue is the maker's public web pages and data sheets. No private data, no login, no account.
Previous issues: The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.
Next issue: Eight bugs in a chain that was meant to run itself.