Unattended runs: telling "ran and correctly did nothing" apart from "never ran"

[noagogo]

I run Claude unattended on a schedule — in my case from a local scheduler that shells out to claude -p, but I think anyone driving this from a schedule: trigger in Actions ends up with the same problem, so I'm asking here.

Each of my jobs starts with a gate step: "if this period's entry already exists in the log, exit immediately." A no-op is therefore a normal, frequent, healthy outcome — most runs of a weekly job correctly do nothing.

The problem is that a healthy no-op and a job that never fired look identical from the outside. Both leave me with no new output and either exit 0 or no process at all. I only discover something has been dead when a deliverable someone was waiting on fails to appear.

This has now bitten me three times, always the same shape: a job that had no entry of its own in the schedule config, and instead "rode along" inside another job's instructions. It silently stopped firing when that host job was edited, and nothing anywhere recorded the absence. The most recent one had never fired at all, and I only caught it because I happened to be reading the config for an unrelated reason.

What I have now:

• one schedule ledger as the single source of truth; the runner reads only that, and no job may piggyback on another job's trigger
• the purely date-based part of the gate ("is today even a candidate?") is handled in shell, so I'm not paying for a full model spin-up just to read one row and exit
• every job proves itself through its artifact, not its logs — the weekly one appends a dated heading to a file, the monthly one appends a dated line, and "nothing changed" is written out explicitly instead of left blank
• a separate audit reads those artifacts and asks "is there an entry for this period?"

Where I'm still stuck:

1. A monthly job gives me 12 chances a year to notice it's dead, and each miss costs a full month. Weekly jobs self-correct quickly; monthly ones don't.
2. The auditor is itself a scheduled job, so it can go silent in exactly the same way. Adding a second auditor just moves the problem up a level.
3. Once a periodic job exits 0 my runner writes a stamp and that period is settled — it won't retry, even if the run was a silent failure rather than a genuine no-op. Exit code alone can't tell me which happened.

For people running this unattended over months rather than days: what actually catches "this never ran"?

• Do you push a heartbeat to something external that alerts on silence rather than on errors?
• Do you record an expected-next-run timestamp that a different system audits?
• Do you make every run emit a positive "I ran, and here is what I decided, and here is why I did nothing" record, so absence of that record is itself the alarm?

I'm not looking for a better scheduler — the scheduling part works fine. It's the noticing I haven't solved.

[noagogo]

Following up on my own question — I went and did the reading I should have done before posting, so I'll narrow this down.

The standard answer here is a dead man's switch / heartbeat: the job pings an external service on each run, and the service alerts on silence rather than on errors. Healthchecks.io, Dead Man's Snitch and similar all work this way, and there's a lot of prior discussion of the pattern. That solves two of my three problems outright:

• the host being down, or the scheduler never firing at all → the ping simply doesn't arrive, and something outside my machine notices
• "who watches the watcher" → the watcher is now somebody else's uptime problem, not another job on the same box

So those two are answered, and I should have found that before asking.

What I don't think heartbeats solve, and what I'd still like to hear about:

A healthy no-op pings exactly like a real run does. My jobs gate themselves ("if this period's entry already exists, exit"), so most executions correctly do nothing — and they still exit 0, so they still ping. That means a job which has silently stopped doing its actual work, but is still being invoked and still exiting cleanly, keeps the switch happy indefinitely. Silence-based alerting can't see it, because there is no silence.

The best idea I have so far is to stop treating the ping as a boolean. Instead of "I ran", send "I ran, and here is what I decided, and here is why I did nothing" — then alert not only on silence but on the wrong shape of activity: N consecutive no-ops for a job that should have produced something by now, or a run whose declared outcome doesn't match the artifact that should have appeared. That turns it from liveness monitoring into something closer to an expectation check.

I haven't found prior art for that second part, which is really what I was asking about. If anyone is running something like "assert that this job's output changed this period, not just that the job executed", I'd like to hear how you express the expectation without hand-maintaining a rule per job.

[ryanportfolio]

A no-op doesn't have to ping success. Healthchecks has a /log endpoint that records an event without touching the check's status. Ping the plain URL only on the run that actually produced this period's entry and /log on every other run.

Then set the check's period to how often the job is supposed to produce something (ie 1 month for the monthly job), not how often the scheduler invokes it.

A job that's still being invoked and still exiting 0 but has silently stopped producing then only ever sends /log so the check goes late right around when the deliverable was due.

"how do you express the expectation without hand-maintaining a rule per job?"

The only per-job config is period + grace on the check plus the artifact path and period format in your ledger row.

PERIOD=$(date +%Y-%m)   # format from the ledger row
had_entry=$(grep -c "^## $PERIOD" "$ARTIFACT" || true)

run_job

if ! grep -q "^## $PERIOD" "$ARTIFACT"; then
  curl -fsS -m 10 --retry 3 --data-raw "$PERIOD: unsettled" "$PING_URL/log"
elif [ "$had_entry" -eq 0 ]; then
  curl -fsS -m 10 --retry 3 --data-raw "$PERIOD: produced" "$PING_URL"
  touch "stamps/$JOB-$PERIOD"
else
  curl -fsS -m 10 --retry 3 --data-raw "$PERIOD: no-op" "$PING_URL/log"
fi

The stamp comes from the artifact so a silent failure leaves the period unsettled and the next invocation retries it; invoke the monthly job daily and you get ~30 chances to self-correct instead of 1.

However this could false alarm for any job that can go a full period with nothing to produce...

And /log really doesn't restart the late timer as Check.ping() only updates last_ping on success and fail pings.

[noagogo]

This is exactly the piece I was missing, thank you.

The part that reframed it for me isn't /log itself, it's setting the check's period to how often the job is supposed to produce something rather than how often the scheduler invokes it. I had been treating "monthly job" as one thing, so "a monthly cadence gives me 12 chances a year to notice" felt like a law rather than a consequence of conflating those two numbers. Decoupling them and invoking daily to get ~30 chances to self-correct is the answer to the actual problem, and it doesn't need any new monitoring concept — just a different period on the check.

Deriving the stamp from the artifact instead of from the exit code is the other half. My scheduler currently writes a stamp when the process exits 0, which is precisely why a silent failure settles the period and never retries. Reading the artifact for this period's marker instead means an unproduced period simply stays open, and the retry falls out of the existing invocation loop rather than needing a separate repair path.

On your caveat about jobs that can legitimately go a whole period with nothing to produce: I think that one's avoidable by making "nothing to report" an artifact write rather than a silence. One of my monthly jobs already requires writing "no change" explicitly for each item, precisely because a blank section is ambiguous between "checked, nothing moved" and "never looked". If every job must emit a period marker even when the answer is nothing, then produced and no-op collapse into the same observable and the false alarm goes away. It costs one line of output per period.

Confirmed your point about /log not restarting the late timer — that's what makes the whole thing work rather than just being cosmetic, since otherwise the no-op pings would keep the check green forever.