TITANOS

Demonblade Labs · Issue 010

Busy is not finished: 561 jobs done, and every final gate held

If you run agents, cron jobs or any pipeline you leave alone, this one is for you. It sells nothing and costs nothing to read. It is about my own build at TitanOS in Brisbane, so nobody else is named.

On 6 and 7 November three build chains ran overnight: my main system, the enterprise build and the website. About 530 jobs. The heartbeat on my phone said 561 DONE and the number kept rising.

All three final acceptance gates were held.

The main re-home gate held at 23:02. The enterprise acceptance held at 18:44, because an approval check let a person approve their own request. The website acceptance held at 20:24, with three of nine modules rejected: an auto-fix that could break mail, a marketplace job funded from an unpaid checkout, and investor figures with no labels.

Agents fixed every one of those blockers by 04:34. The gates still sat held for 6 to 9 hours. The commander only re-armed a held stage when one of its dependencies finished, and the fixes were separate jobs, so nothing ever told the gate that its problem was gone. The counter kept climbing the whole time.

Fix one: a held stage re-arms when the fixes land

The fix shipped at about 05:15. A held stage now re-arms when an .unblock file newer than the hold appears, and the planners touch that file when fixes close the blockers. I proved it by mutation: the old code fails the 2 new tests. Within about an hour both re-runs passed: enterprise ACCEPT 4/4, website ACCEPT 9/9.

Fix two: count the finish line, not the activity

The second lesson came the same morning, from one plain question: how much has actually got done? Of the 561 DONE, about 130 were planning jobs and about 410 were real code jobs. Only 11 of roughly 427 branches had been merged into the product. So now every chain starts with a merge batch, the planners are throttled while more than 30 branches are waiting, and the heartbeat carries a MERGED x/y figure next to the DONE count.

Fix three: a babysitter that lives outside the lane

The third piece is a two layer babysitter on the heartbeat. Layer one is code reflexes for every failure I have already seen: a dead pid, an infrastructure refusal burning a retry, a held gate whose fixes landed, a stale commander, a chain with free slots and nothing in them. Layer two is one capped agent for the unexpected. It has 34 tests, and they are proven by mutation too. A reader on X pointed out that a lane cannot report its own death, so the check lives outside the lane. Silence is the alarm.

If you want to copy one thing: put a finish line number on your dashboard, not only an activity number. Mine is merged branches out of total branches.

What is still open

Plainly: nothing is live. The modules were accepted on stand-in adapters. Next is a real local database, Stripe test mode and an independent security review.

The lesson I am keeping: busy is not finished. A done counter can rise all night while the finish line does not move.

Which number on your dashboard counts activity when it should count finished work?


Previous issues: The ninth bug: my heartbeat went quiet when I needed it most, Eight bugs in a chain that was meant to run itself, Twelve blockers behind a green suite, The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.

Next issue: Zero green in 58: I was tuning the wrong dial.