Signal Graph Automation — Engineering Plan & Runbook

1System map

Four producers feed one consumer chain. Everything left of the chain runs continuously or pre-dawn; the chain itself is the single 05:00Z heartbeat that turns yesterday's records into a rendered, delivered signal graph.

2Workflow inventory

3The daily heartbeat — node-by-node

30 nodes on the deployed chain (surqRbwaJYHx9GL9, versionId 32f93e26, read from origin/main — the branch copy is pre-0b; always name the ref). Two triggers converge on one linear spine; every executable stage is wired to a single halt node. Numbering below is execution order.

Execution topology

• Triggers: Cron 05:00 Reykjavik (0 5 * * *, tz Atlantic/Reykjavik = UTC) and Manual Trigger — both feed resolve · batch date. A manual run is byte-identical to a cron run except for the trigger node.
• Wrapper contract (every Execute Command node): out=$({ … } 2>&1); rc=$?; [ rc != 0 ] && printf … >&2 && exit rc — all output captured; on failure the full text is re-emitted on stderr, because n8n keeps only stderr on the error branch (probe exec 52793).
• Halt topology: 16 stages carry onError: continueErrorOutput → error branch → halt · print + stop, which throws HALT at <leg>\n<refusal text> (first 6000 chars). The n8n execution log + Error Sentinel are the red verdict surface. The only exception is leg 0b: continueRegularOutput — a mint failure flows forward as data, never halts.

Phase A — resolve + preflights (~1 s total)

Phase B — ground the day (~2–3 min)

Phase C — readiness gates (~30 s)

Phase D — content gates (the long middle)

Phase E — publish (~1–2 min)

Phase F — load + deliver (~1 min)

Phase G — verdict / halt

Measured timing profile

Key leg mechanics

• 0 · SYNC — the checkout is a consumer, never a producer: git fetch + reset --hard origin/main. Untracked store imports survive; tracked rewrites are discarded on purpose. (Trap: land_payload_pr.py only resets on merge days — the chain resets every day so no-change days never run on stale code.)
• 0b · MINT — Execute Workflow into 8uhLTBmQDcs2Yti1, waits, continues on error. Window = batch + 2 prior days, so a /Backup copy the mirror missed one morning is minted the next, not never. Runs after the mirror (by cron ordering) and before 1a so the imported repo copy already carries the url.
• 1a · IMPORT — import_store_parcels.py copies store bytes verbatim into the repo checkout; a differing repo copy is overwritten. This is why origin_url must be written on the STORE record, never repo-side.
• qc — check_quotes.py: every quote must be a contiguous, byte-identical span of its source body (whitespace collapse + unicode fold only; single↔double quotes are NOT folded — deliberate). Fails → chain halts, Error Sentinel fires, nothing renders.
• 6a · LOAD — jsonBody = the leg's stdout verbatim (re-serialization broke the digest once: every payload 422'd). 200 = upsert + server read-back; 409 DIGEST MOVED is never auto-replaced from the chain — replace is a human decision through the re-drive tool.

Leg ownership — who gets the red line

4Data contracts that hold this together

5Runbook — the daily read

Timeline (all UTC; Reykjavik = UTC)

Reading the verdict

0  · SYNC      checkout = origin/main <sha>
0b · ORIGIN URLS (mint, non-gating)   selected N · merged N · refused 0
1a · IMPORT    written N · unchanged N · unreviewed ≤2 · refused 0
qc             N quotes across M parcels — PASS
3  · CONVICTION …
4b · LAND      PR #N merged
6a · LOAD      state NEW|UNCHANGED · read_back true
6b · DELIVER   one delivery row

• Healthy baselines: refused 0 everywhere; unreviewed ≤ 2 (shells awaiting S2); 6a state: NEW on first load of a batch, UNCHANGED on idempotent re-runs.
• Checking an execution: n8n_executions list workflowId=surqRbwaJYHx9GL9; on error, get <id> mode=error names the halting leg and quotes the decisive line. Node data is invisible while a run is in flight — status only.

Re-running a batch

• Full chain re-run (after a content fix): execute the chain workflow directly (production mode). Legs before the gate are idempotent — 0b skips has_url, 1a reports unchanged, 4b lands only diffs. resolve computes batch = yesterday, so a re-run on day D+1 still targets batch D.
• Load-only replay (payload changed on origin/main, graph already loaded): re-drive webhook — GET /webhook/signal-graph-redrive-load?batch=YYYY-MM-DD[&replace=1][&dry=1]. replace is the explicit human override for 409 DIGEST MOVED.
• Mint-only: GET /webhook/origin-url-mint?batch=YYYY-MM-DD[&dry=1][&days=N][&all=1]. dry=1 resolves links, writes nothing — the standing readiness control.

6Failure playbooks (each one already exercised)

7Manual tools

8Instance traps (paid for once, written down)

• Execute Command runs ONCE per execution (executeOnce: true): N items in → 1 run. Batch work goes through one base64 argv (pack/unpack) — and if the payload can exceed ~128 KB, not through argv at all (Linux MAX_ARG_STRLEN → spawn E2BIG).
• /Current refuses public links (settings_error/not_authorized) — mint on /Backup, always. This is why the mirror is load-bearing, not a convenience.
• Credential scopes differ per credential: the HTTP OAuth cred is sharing-scoped (no files.content.read); content download/upload goes through the Dropbox-node cred.
• MCP test client times out at 120 s; the execution continues server-side — read the execution log, don't re-fire.
• A url written repo-side only is reverted by the next 1a import — store record or nowhere.
• Verify landings by content (git diff branch origin/main empty), never by ancestry — rebase rewrites SHAs.