Demonblade Labs · Issue 008
Eight Bugs In A Chain That Was Meant To Run Itself
I asked for one clean unattended run and got 0 useful lanes for about 11 hours. Then I found eight separate bugs in the commander that was meant to prevent exactly that. The fix that mattered was a 45 second rehearsal that crashes it on purpose.
If you run scheduled jobs, agents or any pipeline you want to leave alone overnight, this one is for you. It sells nothing and it costs nothing to read. It is about my own build at TitanOS in Brisbane, so nobody else is named.
What you will get out of it
- What a cron environment can hide for 11 hours.
- The eight bugs, in the order they bit.
- The rehearsal that catches them, and how I know it works.
- What is still open.
Chapter 0. What I asked for
On 5 November I said: chain the whole thing up, 3 agents at a time, one clean run, I do not want to intervene. A cron commander (bin/l10_chain.sh) runs stages with dependencies. If a stage fails it retries 3 times, then holds the stage and sends me one message.
Chapter 1. The overnight result
From 19:34 to 06:26, about 11 hours, it ran 0 useful lanes. Cron has a bare PATH, and that PATH had no powershell.exe. The SSD guard could not prove the drive was healthy, so it refused every heavy writer with exit 75. Every tick after that stopped at the sign-in probe for the same reason, and the commander reported dead lanes as RUNNING.
Nothing was wrong with the work. The commander was wrong about itself.
Chapter 2. The eight bugs
All eight were found and fixed the same day.
- Cron PATH. No
powershell.exe, so the drive could not be proven. - Wrong blame. The probe blamed sign-in for a disk guard refusal.
- Inherited lock. Launched lanes inherited the tick's lock, so every tick skipped while a lane was alive. That was 55 minutes of 1 lane instead of 3.
- Wrong PID. A
setsidlaunch recorded the wrapper's PID, not the job's. Caught before harm. - Swallowed check. An inline comment swallowed the lock check for about a minute. That one was my own edit.
- Cap overshoot. Counting running lanes in file order let an early stage relaunch past the cap. I saw 4 running. The rehearsal caught it.
- Receipts in the wrong place. A rule to work in separate worktrees made lanes write receipts where the commander does not look.
- Silent push failure. An exam passed, 50 of 50 and 24 of 25, but its push to main failed silently on a sparse checkout. The next stage would still have seen the stale gate.
Chapter 3. The rehearsal
A 45 second end to end rehearsal runs the real commander under cron's environment on a fake chain, then crashes it on purpose.
| Crash | What it checks |
|---|---|
| Dead sign-in | The commander says so, not RUNNING |
| STOP | Everything halts |
| A lane that fails once | It retries and recovers |
| A lane that always fails | 3 tries, then hold and one message |
| Three lanes at once | The cap holds |
| Fan in | The next stage waits for all of them |
It is 25 checks. It is proven by mutation: I put the lock bug back and it turns red.
Around it I added a 15 minute heartbeat to Telegram, a watcher that uses no model, and backfill lanes, so 3 agents keep running even when the critical path only allows 1.
Two lessons
A hand run first tick proves nothing about cron.
A push inside a script is unverified until you read main back.
The limits of all of this
- The rehome and the cert are not finished yet, and the chain is not done.
- The rehearsal covers the failures I thought of, so I expect the real run to find a ninth.
- Until the chain completes one clean run on its own, the claim is only that it now fails loudly instead of quietly.
If you run unattended jobs, I would like to hear what your cron environment hid from you. Reply on X. Would it be okay if I asked you what your last overnight job did that you only found out about in the morning?
Receipts
- Every figure above is a log timestamp, a test run result or a count from my own build, measured on 5 and 6 November 2026.
- No client data and no other person or company is involved.
Previous issues: Twelve blockers behind a green suite, The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.
Next issue: The ninth bug: my heartbeat went quiet when I needed it most.