Demonblade Labs · Issue 011
Zero green in 58: I was tuning the wrong dial
If you run agents, build chains or any pipeline with a test gate at the end, this one is for you. It sells nothing and costs nothing to read. It is about my own build at TitanOS in Brisbane, so nobody else is named.
After a reboot and a disk move this week I resumed three build chains: my main system, the enterprise build and the website. Then I spent the morning on queue time, machine load and storage routing. All of it was real. None of it was the problem.
Storage: 2,159 refused starts
What I fixed first. A storage rule sent heavy jobs to a USB SSD scratch disk. Once that disk stopped mounting after reboots, the rule refused 2,159 job starts between 24 October and today. I switched the rule off and everything now writes to the main disk.
Load: a quiet box
Then load. I added a quiet box: while a merge gate is running and the machine load is 12 or more, no new build jobs start. I observed load 33 at 07:48 and 14 at 07:55 after it went in. A lane that dies on the account's usage limit now hands its retry back instead of burning one of its three tries.
Queue: a wrong diagnosis, repeated
Then the queue. I was sure the gate's 45 minute clock was being eaten by time spent waiting in line. I wrote a regression test to prove it. It showed the clock already starts after the queue. The queue was never the problem, and I had repeated a wrong diagnosis.
The verdict column
So I did what I should have done first: I read the verdict column of my own gate ledger.
58 gates, zero green
58 full test gates across 7, 8 and 9 November, on the cloud runner since the 8th. Zero green.
On 7 November: 22 runs, 17 red, 3 timeouts, 2 busy. On 8 November: 31 runs, 22 red, 5 timeouts, 4 busy. On 9 November so far: 5 runs, all red. The median gate takes about 46 minutes of wall clock. I had spent the morning making a fast queue feed a gate that was red every time.
The pile behind the gate
There was a second pile behind it. About 260 finished branches were waiting to merge, and about 93 of them conflict on every merge attempt. So I added a rebase stage that rebases them family by family.
What to copy
The lesson, if you run agents: read the verdict column before the wait column. A fast queue feeding a red gate is still zero.
What is next
What happens next, as a plan and not as done: a red suite triage stage before the next merge sweep, then a 48 hour soak. Nothing in this issue is finished until a gate comes back green.
Which number on your dashboard measures waiting when it should measure passing?
Previous issues: Busy is not finished: 561 jobs done, and every final gate held, The ninth bug: my heartbeat went quiet when I needed it most, Eight bugs in a chain that was meant to run itself, Twelve blockers behind a green suite, The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.
Next issue: My proof page now refuses numbers older than 30 days.