TITANOS

Demonblade Labs · Issue 008

Eight Bugs In A Chain That Was Meant To Run Itself

I asked for one clean unattended run and got 0 useful lanes for about 11 hours. Then I found eight separate bugs in the commander that was meant to prevent exactly that. The fix that mattered was a 45 second rehearsal that crashes it on purpose.

If you run scheduled jobs, agents or any pipeline you want to leave alone overnight, this one is for you. It sells nothing and it costs nothing to read. It is about my own build at TitanOS in Brisbane, so nobody else is named.

What you will get out of it


Chapter 0. What I asked for

On 5 November I said: chain the whole thing up, 3 agents at a time, one clean run, I do not want to intervene. A cron commander (bin/l10_chain.sh) runs stages with dependencies. If a stage fails it retries 3 times, then holds the stage and sends me one message.

Chapter 1. The overnight result

From 19:34 to 06:26, about 11 hours, it ran 0 useful lanes. Cron has a bare PATH, and that PATH had no powershell.exe. The SSD guard could not prove the drive was healthy, so it refused every heavy writer with exit 75. Every tick after that stopped at the sign-in probe for the same reason, and the commander reported dead lanes as RUNNING.

Nothing was wrong with the work. The commander was wrong about itself.


Chapter 2. The eight bugs

All eight were found and fixed the same day.

  1. Cron PATH. No powershell.exe, so the drive could not be proven.
  2. Wrong blame. The probe blamed sign-in for a disk guard refusal.
  3. Inherited lock. Launched lanes inherited the tick's lock, so every tick skipped while a lane was alive. That was 55 minutes of 1 lane instead of 3.
  4. Wrong PID. A setsid launch recorded the wrapper's PID, not the job's. Caught before harm.
  5. Swallowed check. An inline comment swallowed the lock check for about a minute. That one was my own edit.
  6. Cap overshoot. Counting running lanes in file order let an early stage relaunch past the cap. I saw 4 running. The rehearsal caught it.
  7. Receipts in the wrong place. A rule to work in separate worktrees made lanes write receipts where the commander does not look.
  8. Silent push failure. An exam passed, 50 of 50 and 24 of 25, but its push to main failed silently on a sparse checkout. The next stage would still have seen the stale gate.

Chapter 3. The rehearsal

A 45 second end to end rehearsal runs the real commander under cron's environment on a fake chain, then crashes it on purpose.

CrashWhat it checks
Dead sign-inThe commander says so, not RUNNING
STOPEverything halts
A lane that fails onceIt retries and recovers
A lane that always fails3 tries, then hold and one message
Three lanes at onceThe cap holds
Fan inThe next stage waits for all of them

It is 25 checks. It is proven by mutation: I put the lock bug back and it turns red.

Around it I added a 15 minute heartbeat to Telegram, a watcher that uses no model, and backfill lanes, so 3 agents keep running even when the critical path only allows 1.


Two lessons

A hand run first tick proves nothing about cron.

A push inside a script is unverified until you read main back.


The limits of all of this


If you run unattended jobs, I would like to hear what your cron environment hid from you. Reply on X. Would it be okay if I asked you what your last overnight job did that you only found out about in the morning?

Receipts

Previous issues: Twelve blockers behind a green suite, The exam the builder never saw, The button that ignored a fast thumb, I built a product assistant that is not allowed to guess, I built a gym that judges its own best strategies as luck, I freed 334 gigabytes and the disk got smaller, Why more AI agents don't automatically mean more output. All issues: Demonblade Labs.

Next issue: The ninth bug: my heartbeat went quiet when I needed it most.