Substack email ingestion pipeline

Purpose and current boundary

This is the production path for Substack publication emails delivered to Fastmail. It converts a qualifying email into an OKF source-document parcel, writes that parcel to the parcels bucket, and relies on the L2 projector to make the current revision retrievable by Mímir.

Live n8n workflow: 37ZscHy7SyDYx1CC — viska-fastmail-ingest-transform — active.

The older wig8zrxkyoLLeO9G workflow, Viska Substack Ingest (email/eml -> research_findings), is an Outlook-era research_findings lane. It is not the current Fastmail → OKF parcels → L2 retrieval path described here. Do not use it for recovery or infer current serving behaviour from it.

flowchart LR
  M[Fastmail / Substack mailbox] --> I[Email Trigger IMAP]
  I --> B[Build OKF Parcel]
  B --> A[Assert Contract]
  A --> S[Sign Parcel Upload]
  S --> H[Handle Sign Outcome]
  H --> P[PUT parcel markdown]
  P --> L1[parcels bucket / L1]
  L1 --> X[OKF bucket projector / L2]
  X --> D[(okf_parcels)]
  D --> R[Mímir retrieval: stable rows only]

Intake and scheduling

Email Trigger (IMAP) is event-driven, not cron-driven.

The trigger only searches when Fastmail emits its new-mail callback. Activation and a reconnect reopen the connection; neither backfills mail already sitting in the folder. A non-matching wake-marker may cause a search but is an unreliable diagnostic, not a recovery mechanism. JMAP polling is not part of this pipeline; historical reconciliation belongs to ViskaRes.

Workflow nodes

Parcel construction contract

The transform writes an OKF v0.2 source-document with:

• original, bare email Message-ID in source_meta.origin as the email-lane dedup identity;
• publication/source identity resolved from the known alias registry or the publication URL;
• title, author, published and retrieved timestamps;
• Markdown body, images/video media, completeness/truncation information, and a structured links array;
• source_meta.extraction_contract: okf-parcel-n8n-substack/4 and matching producer stamp process:okf-parcel-n8n-substack/4.

parcel_id and object_key include the extraction-contract revision plus origin URL, title, and body. A corrected extraction contract therefore creates a new, deterministic object rather than colliding with the old result. Re-running the same email under the same contract produces the same parcel identity.

Link and prose policy

The structured links array is intentionally stricter than source prose:

• persist only HTTPS URLs after query allowlisting;
• strip tracking/capability parameters not on the allowlist;
• exclude product chrome such as unsubscribe, profile/app/read-in-app and open.substack.com/chat/ links;
• omit opaque substack.com/redirect/<uuid> targets rather than fetching or resolving them;
• retain the anchor text in the Markdown body with [unresolved Substack citation omitted] when an opaque redirect is omitted.

This preserves evidence that a citation existed without persisting an unverifiable target. Paywall gates become explicit completeness metadata; footer notices are removed without declaring a complete article truncated.

Storage, projection, and serving

1. L1 / parcels bucket stores the immutable Markdown parcel at parcels/substack/<source>-<published>-<hash>.md.
2. L2 / okf_parcels is a re-runnable projection of bucket objects, not an authoring destination. The projector chooses the highest source_meta.extraction_contract for a shared origin. It retires an older row as status='deprecated', preserves its bytes and records superseded_by; it does not delete historical bucket objects.
3. Serving must select status='stable'. Mímir applies that filter to body retrieval, structured parcel retrieval, and visibility probes so retired revisions never reappear in search results.

The production v4 audit measured 98 stable Substack bodies, 9 retained links (7 distinct), zero opaque redirects, non-HTTPS URLs, capability parameters, chrome/chat, suspicious/unrelated URLs, or unclassified URLs. It retained 86 explicit unresolved-citation markers. The deployed retrieval control selected Citrini v4 ...:967b6a58 (47,170 characters, five parts, 61 markers, no truncation).

Backfill and missed-email recovery

Use scripts/recover-substack-email.js only for a confirmed missed publication, listener downtime, or a known absent execution.

1. Read the workflow descriptor and the failed execution first; diagnose the named node rather than guessing.
2. Confirm workflow 37ZscHy7SyDYx1CC is active and the IMAP mailbox and binding above remain unchanged.
3. Run with the injected bindings and explicit mutation approval:

   N8N_ALLOW_MUTATIONS=1 node scripts/recover-substack-email.js

4. The script searches the Substack mailbox with the same two publication headers and fetches raw MIME without marking it read.
5. fetchRawMessages() waits for the IMAP message and every body stream to end before parsing. This prevents partial, non-deterministic MIME bodies.
6. It runs the live Build OKF Parcel transform twice per fetched email and stops if the parcel Markdown differs. It prints the resulting parcel identities, not credentials.
7. It creates a temporary authenticated n8n webhook and temporary header credential, posts each parsed email into Build OKF Parcel, then removes both webhook nodes and the credential in finally.
8. Success evidence is either a synchronous upload response or explicit already_delivered; the latter means the object already exists and is an idempotent outcome. Public n8n execution history can omit temporary-webhook runs.
9. For corrected historical content, confirm the new bucket object and L2 projection before declaring recovery complete. Mímir's stable-only audit is the acceptance control for serving safety and relevance.

Never expose credential values, use a synthetic source-email fixture, invoke an unauthenticated webhook, or use the Fastmail MCP token as a JMAP bearer.

Operational runbook

Normal operation

• Leave the active IMAP trigger running; there is no scheduled pull to fire manually.
• Monitor failed executions with npm run debug -- <executionId> and inspect the named node.
• For any workflow change: npm run sync before editing, npm run validate after, then deploy and activate only the changed workflow.
• When the transformed parcel changes, use a higher extraction-contract revision so L2 can identify it as a correction.

Incident: publication is missing

1. Confirm that it is a publication, not an account notice or other transactional Substack mail.
2. Check whether the listener has an execution. Do not treat reconnect/re-activation as a backfill.
3. Run the recovery procedure above.
4. Verify the bucket object and L2 row are present and stable; request Mímir stable-only retrieval verification when the correction affects persisted corpus content.

Incident: unsafe or irrelevant persisted link

1. Audit stored values read-only; do not request or execute the persisted URL to judge it.
2. Correct the producer's extractor and add a focused red case.
3. Increase source_meta.extraction_contract, deploy, and recover the affected source emails twice to demonstrate deterministic output.
4. Have L2 admit/project the newer contract, retire the old revision, and re-run the Mímir stable-only audit.

Deliverables and acceptance evidence

Known separate concern

Four duplicate live Oilprice origins in the RSS lane predate this work and have no extraction-contract metadata. They are not generated by the Fastmail/Substack pipeline and need an R6 decision on which capture wins. Do not fold them into Substack email recovery.