TITANOS

Demonblade Labs · Issue 007

Twelve Blockers Behind A Green Suite

Over 2 and 3 November the product assistant went through four independent review rounds. Each round found a defect a buyer would have seen, 12 in total, and the tests were green before every one. The rule that came out of it: keep reviewing until a round comes back clean on a number path.

If you build, buy or sell anything with a chat assistant in it, this one is for you. It sells nothing and it costs nothing to read. This follows Issues 004, 005 and 006: the same unsolicited prototype, built from a building materials maker's public website, so the company is still not named. Nothing of theirs was touched.

What you will get out of it


Chapter 0. The number is 12

A blocking defect here means one a buyer would notice straight away: a wrong quantity, a wrong product, a confident yes where the answer is no. Each reviewer had built none of the code. Each time I re-ran the fixes myself instead of taking the builder's word for it, and each time the next reviewer found more.

RoundBlocking defects
13
25
32
42
Total, all fixed12

Chapter 1. What each round found

  1. Round one: a stale product carried into a quantity answer, range names treated as aliases for single products, and a cheerful "yes, happy to help" for things the company does not sell.
  2. Round two: a product held from earlier hijacking a short question about a category, and the word "special" being read as a price question, among others.
  3. Round three: a bare "2400 x 1200" was read as an area and came back as 2,880,000 square metres and 480,000 bags. A tile size was also read as the job area.
  4. Round four: a tile size typed after a product card became the job area and went into the cart. A tile measurement in millimetres was read as a layer thickness, which turned a 10 square metre job into 285 bags. That one was already live in the version before this round.

A green suite tells you the problems you thought of are gone. It cannot tell you about the ones you did not.


Chapter 2. The rule

Keep reviewing until a round comes back with no blocking defect on a number path, then ship.

Not after one round, not after two. Numbers are where a wrong answer costs a buyer money, so numbers get the strictest exit test.


Chapter 3. The exam numbers

Alongside the reviews I ran a fresh exam of 100 questions the builders never saw, 40 of them strict honest-no questions where the right answer is a plain no.

MeasureStartFirst fixes liveNow
Unseen exam, overall (100)526078
Unseen exam, honest no incl. multi-turn (48)n/a1732
Held-back final exam (120)647173

The final exam moved by 2. That is a small move and I am not dressing it up.


The limits of all of this


Your three questions

Who tried to break your assistant that did not build it? How many rounds did they need before one came back clean? Did anyone check the number paths separately from the words?


If you build assistants, I would like to hear how many review rounds your last release needed before one came back clean. Reply on X. Would it be okay if I asked you what the most expensive wrong number your assistant could give a customer would be?

Receipts

Previous issues: The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.

Next issue: Eight bugs in a chain that was meant to run itself.