Demonblade Labs · Issue 009
The ninth bug: my heartbeat went quiet when I needed it most
If you run agents, cron jobs or any pipeline you leave alone, this one is for you. It sells nothing and costs nothing to read. It is about my own build at TitanOS in Brisbane, so nobody else is named. Last issue ended with "I expect a ninth." It came the next day, and it brought two friends.
The setup: a cron commander runs my build in stages, 3 agents at a time. A watcher checks it every 15 minutes and sends my phone a heartbeat, one line with the counts, so I can see it working without opening anything.
On 6 November the stage that was blocking everything went green at 11:14: 459 test suites passing. The stage waiting on it had already failed 3 times before that fix existed, so the commander held it, exactly as designed. Nothing else was queued. From 11:14 to 16:09, about 5 hours, 0 agents ran.
The watcher saw it. It sent one alarm, and then it went silent. That was my design: one alarm per problem, no spam, and the heartbeat only ran on the healthy path. So the moment something broke, the thing I had built to show me it was alive stopped talking. My message to the machine was simple: "haven't seen any heartbeats."
Three fixes, each one a rule moved into code.
- The heartbeat never stops. If something is wrong, the beat still comes every 15 minutes and says STALLED and why. Silence is now impossible during the day, and silence was the worst signal I had.
- A held stage re-arms by itself. If one of its dependencies finishes after the hold, it goes back in the queue, at most twice. That alone would have cleared the 5 hour stall in one 10 minute tick.
- The queue refills itself. When a slot is free and nothing can start, the commander adds a planner stage that writes two new tasks from the open gaps. One at a time, 20 minutes apart, 12 a day, and one file switches it off.
The part I did not expect came while I was fixing it. I launched two fresh agents to keep 3 running. Inside 2 minutes both were editing the live code that cron runs, including the commander itself. Their instructions said, in capitals, work in your own worktree. They did not. A rule written in a brief is a request. So now the commander creates a fresh worktree for every agent and starts it inside, and the agent never begins in the live checkout at all. The folder is the rail, not the sentence.
And one more of mine: running the watcher's own test sent a few fake "0/0 DONE" heartbeats to my real phone, because one test helper had no fake inbox. Fixed so a test run can never reach the phone.
Every fix has a test, and the important ones are proven by mutation: I break the fix on purpose and the test turns red. The rehearsal that crashes the commander on a fake chain grew from 25 checks to 34.
What is still open: the rehome stage is running, and the cert, the merge and a 48 hour soak are ahead. The planner's choices are untested until I read what it picks. So I expect a tenth.
The lesson I am keeping: an alarm must never mute the heartbeat, and a rule in a brief is not a rail.
What rule in your setup only exists as a sentence? Would it be okay if I asked?
Previous issues: Eight bugs in a chain that was meant to run itself, Twelve blockers behind a green suite, The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.
Next issue: Busy is not finished: 561 jobs done, and every final gate held.