§1Overview & Client User-Journey
The IBKR Flex ingest feature is the data pipeline that turns Viska's live Interactive Brokers account into the fund state the viska.gg dashboard renders. An n8n workflow on Hostinger pulls daily Flex statements from IBKR, normalizes them to metadata, hands the raw XML to a standalone apply-svc on Railway that re-parses it and is the sole writer of the trading.* schema, which projects up into the public.* views the frontend reads.
This is a thin-transport / authoritative-writer split: n8n moves bytes and holds zero database or storage credentials, while apply-svc owns every write to trading.* behind a single bearer gate. apply_service.py:35-44 The pipeline fires nightly via a 22:30 cron on the live n8n workflow; an on-demand viska.gg pull is DESIGN-APPROVED, NOT BUILT. on-demand-webhook-design.md:6,109-117
End-to-End Client User Journey
THE FLOW — IBKR ACCOUNT → DASHBOARD
- 1. Trigger. Either the nightly 22:30 cron on the live n8n workflow (
ADeTjBZiHOMIXzvr, activeVersionId6aa4c859) fires, OR — once built — a client opens viska.gg and activates an on-demand pull. The on-demand path (2nd Webhook trigger, 202-async, native Header Auth, 5-min cooldown) is NOT BUILT. chronicle.md:35-38; on-demand-webhook-design.md:6,109-117 - 2. IBKR Flex pull. n8n runs a 2-call async handshake against the IBKR Flex Web Service:
SendRequest(pinned hostndcdyn, query id1384817,v=3, hardcoded because$envis blocked on the instance) returns a reference + statement<Url>;GetStatementthen polls that dynamic URL ({{ FlexStatementResponse.Url }}, observed hostgdcdyn) in a bounded loop ($runIndex < 6) untilLast30CalendarDaysresolves to oneFlexQueryResponsecarrying ~22 daily<FlexStatement>elements. flex-pull-...json:85; chronicle-2026-06-22:389 - 3. Normalize (metadata only). The
Normalizenode parses each statement to metadata — report dates, position counts, filename — driving guards and the upload key, never the DB write. flex_to_sections.py:6-11; multiday-contract:50-52 - 4. Usable-statement guard. The
Statement Usable?node (position_count > 0 AND report_date notEmpty, commitf8a0304) gates the path. False → Flag Unmapped Conids → Respond, skipping the write so an empty next-day snapshot (IBKRtoDatefloat) never overwrites real data. git show f8a0304; chronicle.md:17-32 - 5. Sign + upload. n8n calls
apply-svc /sign-upload(static bearer) to mint a server-signed URL, then PUTs the raw XML to the privateflex-inboxbucket. The PUT leg isauthentication: none— the signed URL self-authenticates; the old HS256FLEX_UPLOADER_JWTleg is DEAD. apply_service.py:66-70,326; 20260621-01:24-26 - 6. Apply. n8n POSTs
{bucket, object_path}toapply-svc /apply(same bearer). apply-svc downloads + re-parses the raw XML — the SSOT for report_date and positions — and writestrading.*with the owner connection. flex_to_sections.py:6-11; apply_service.py:146 - 7. trading.* persisted. Idempotent on
file_hash = sha256("flex:"+account+":"+report_date)→trading.statements UNIQUE(file_hash); Flex Phase-1 lands 4 streams (instruments, positions, NAV, FX) per report date. flex_import.py:20-31; flex_to_sections.py:10-12 - 8. Dashboard renders. viska.gg / ViskaFront reads the
public.*projections (positions_current,nav_latest,cash_current,fx_rates…) — nevertrading.v_*directly — currently authenticated-only behind CF Access. useDashboardData.ts:76,108,152
Current State — KPI Strip
Round-Trip GREEN ×1
End-to-end pull→apply succeeded once (exec 35607, reportDate:20260619 → positions:21, fx:5, nav_rows:3). chronicle-2026-06-22:502-503
Multi-Day Fix CODE-CONFIRMED, DEPLOY-UNCONFIRMED
Fix A (build_sections_by_day+emit_days) committed f810998, 75 tests green. Live Railway image SHA unfingerprinted — "22/22 in prod" is unverified. flex_to_sections.py:145; ibkr_import.py:359
Write-Path Auth LIVE
Single static bearer FLEX_APPLY_WEBHOOK_SECRET guards both /sign-upload and /apply (401-on-unauth live-confirmed); /healthz → 200. apply_service.py:66-70; live-probe
On-Demand Pull NOT BUILT
2nd webhook trigger design-approved; blocked on N8N_API_URL/N8N_API_KEY + auth-secret mint. Nightly cron is the only live trigger. on-demand-webhook-design.md:6,109-117
Components at a Glance
| Component | Host | Role (one line) |
|---|---|---|
n8n workflow ADeTjBZiHOMIXzvr | Hostinger | Thin transport: pulls Flex XML, normalizes to metadata, signs+uploads, POSTs apply — zero DB/storage cred, zero trading writes. flex-pull-...json:85; rescope-ops:83-89 |
| apply-svc | Railway (flex-apply-svc-production.up.railway.app) | Sole SSOT writer: re-parses raw XML, writes trading.* with owner conn, single bearer gate; separate service from viska-mimir-agent. apply_service.py:35-44; live-probe /healthz 200 |
| flex-inbox bucket | Supabase Storage | Private raw-XML drop; signed-URL PUT in, service_role read (bypasses RLS) out. 20260621-01:18-19,24-26 |
| trading.* schema | ViskaDB (Supabase) | 11 tables; importer writes 9, Flex path lands 4 streams; idempotent on UNIQUE(file_hash). ibkr_import.py:181-352; flex_to_sections.py:10-12 |
| public.* projections | ViskaDB (Supabase) | FE read contract (positions_current, nav_latest, cash_current, fx_rates …); authenticated-only since 06-19. useDashboardData.ts:76,108,152; 20260619-03 |
| viska.gg / ViskaFront | CF Pages (CF Access) | Dashboard surface; reads public.*, never trading.v_*; CF Access fronts the portal only, not ingest. useDashboardData.ts:76; on-demand-webhook-design.md:136-137 |
| Mímir bot | Railway (viska-mimir-agent) | Separate consumer: reads trading.v_* via trading_bot_ro role — not part of the ingest write-path. 20260610-08:42-47 |
§2Hosting Topology & Dependencies
The IBKR Flex feature is a two-host pipeline: an n8n orchestrator on Hostinger (thin transport, zero DB/storage credential) and an apply-svc on Railway that re-parses the raw XML and is the sole SSOT writer of trading.*. Three external/managed services back them: IBKR Flex Web Service (source), Supabase Postgres + Storage (sink), and Cloudflare Access (dashboard auth only — not in the ingest path). The two Railway services — apply-svc and the Mastra/Mímir agent — are distinct and must never be conflated.
CANONICAL TOPOLOGY (FIXED)
n8n on Hostinger (workflow ADeTjBZiHOMIXzvr, "Viska Macro Daily — IBKR Flex Pull"). apply-svc on Railway (flex-apply-svc-production.up.railway.app), a separate service from the Mastra/Mímir agent (Railway project viska-mimir-agent). flex-pull-ADeTjBZiHOMIXzvr.json:85 live-probe /healthz 200 Never place n8n on Railway; never conflate the two Railway services.
The chain (end to end)
Schedule (22:30 Mon-Fri) ─┐
├→ Init → Flex SendRequest (httpQueryAuth, ndcdyn) → poll loop ($runIndex < 6)
Webhook (POST, headerAuth)┘ → Flex GetStatement (dynamic {{ FlexStatementResponse.Url }}, observed gdcdyn)
→ Parse XML → Normalize (multi-day) → "Statement Usable?" guard
(position_count > 0 AND report_date notEmpty)
→ Sign Upload (POST /sign-upload, bearer) → PUT raw XML (signed URL, auth:none)
→ flex-inbox bucket (flex/<report_date>/<execId>.xml)
→ Apply POST (POST /apply, bearer) → apply-svc RE-PARSES XML → writes trading.*
→ object moved to processed/ on success
n8n holds zero trading writes and zero storage credential; apply-svc is the SSOT parser of the raw XML and the only writer to trading.*. The Upsert Positions node (f694c363…) was removed — replaced by Sign Upload → Upload → Apply POST. flex-multiday-normalize-apply-contract.md:54-56 flex-pull-rescope-ops.md:23-24,31,83-89
NODE IDS ≠ COMMITS
28992ad5… (Normalize), 902d95a5… (Flag Unmapped Conids), 4531f9c5… (GetStatement), f694c363… (Upsert Positions, removed) are n8n node ids, not git SHAs — no such SHAs exist in the repo. Real commits: Fix A f810998, Fix B 2f12e5c, guard f8a0304, dedup b1664ff, HMAC→Bearer 7466abf, signed-URL 3545985/d182bcb. flex-pull-rescope-ops.md:30,32
Dependency table
| Component | Host | Role | Talks to |
|---|---|---|---|
n8n workflow ADeTjBZiHOMIXzvrLIVE active for 22:30 cron (activeVersionId 6aa4c859) |
Hostinger | Orchestrator: dual trigger (Schedule 22:30 Mon-Fri + on-demand Webhook headerAuth), Flex 2-call pull, XML parse, multi-day Normalize, "Statement Usable?" guard, sign-upload, raw-XML PUT, apply trigger. Thin transport — no DB write. | IBKR Flex Web Service (outbound httpQueryAuth); apply-svc /sign-upload + /apply (bearer); Supabase Storage signed-URL (PUT); dashboard backend (inbound webhook caller) |
apply-svc (apply_service.py, Flask + psycopg) |
Railway — flex-apply-svc-production.up.railway.app; /healthz→200 {"ok":true}. Separate Railway service from Mastra/Mímir (viska-mimir-agent). |
Two bearer-auth endpoints sharing one secret: POST /sign-upload (mints short-lived Supabase signed UPLOAD URL, service-role server-side) and POST /apply (re-parses raw XML = SSOT for report_date+positions, writes trading.* in one txn, moves object to processed/ on success). Multi-day loop emit_days emits all statements. |
Supabase Postgres (VISKA_DB_URL); Supabase Storage REST (VISKA_SUPABASE_URL + service-role key); receives inbound from n8n |
| Supabase Postgres | Supabase (project URL via VISKA_SUPABASE_URL) |
trading.* SSOT — statements (UNIQUE(file_hash)), positions_snapshots, nav_snapshots, fx_balances; ops.health_events alert sink. service_role has ALL on trading.*. |
apply-svc (write); FE readers via public.* projections; Mímir bot via trading.v_* (trading_bot_ro) |
| Supabase Storage | Supabase (same project) | Private flex-inbox bucket — landing zone for raw FlexQueryResponse XML. Bucket server-locked (caller-supplied bucket → 400). Object moved to processed/ after apply. service_role reads bypass RLS. |
n8n (PUT via signed URL); apply-svc (sign/read/move via service-role) |
| IBKR Flex Web Service | External (interactivebrokers.com) | Source of holdings. 2-call contract: SendRequest (returns ReferenceCode + a <Url>) → GetStatement (raw FlexQueryResponse XML). Query 1384817, v=3, Last30CalendarDays → ~22 daily <FlexStatement>. |
Receives n8n SendRequest/GetStatement (outbound only — no IBKR webhook/push; the on-demand trigger is Viska-side) |
| Cloudflare | Cloudflare edge | CF Access fronts the dashboard / artifacts Pages portal only. NOT in the IBKR ingest path — the on-demand webhook uses native n8n Header Auth, explicitly chosen over a CF edge shield. | Dashboard auth only PARTIAL (CF Access → Supabase JWT, P2) |
| Mastra / Mímir agent | Railway project viska-mimir-agent |
Mímir Slack research assistant runtime — DISTINCT from apply-svc. No role in the IBKR feature. Listed only to disambiguate the two Railway services. | (out of IBKR scope) |
| Dashboard backend (ViskaFront domain) | UNKNOWN (ViskaFront domain) | Holds the webhook Header-Auth secret (never in browser JS); fires POST <webhook>, gets 202, polls trading-data freshness. Contract only — not built in these records. |
n8n webhook (caller) |
flex-pull-rescope-ops.md:14,23-24,62 flex-multiday-normalize-apply-contract.md:54-56,63-68 apply_service.py:35-44 cio-ratification-ibkr-feed-gaps-A-H.md:63 on-demand-webhook-design.md:136-137
IBKR host nuance — ndcdyn vs gdcdyn
Live workflow hardcodes SendRequest to ndcdyn.interactivebrokers.com/AccountManagement/FlexWebService/SendRequest. During live wiring, SendRequest's response <Url> came back on the gdcdyn host — so both GetStatement URLs were patched to dynamic {{ FlexStatementResponse.Url }} (was hardcoded ndcdyn) to follow whichever host IBKR returns. Net: SendRequest pinned ndcdyn; GetStatement dynamic (observed gdcdyn). flex-pull-ADeTjBZiHOMIXzvr.json:85 chronicle-2026-06-22:389(3)
Credential inventory (bindings only — no values)
| Binding | Where held | Consumed by | Never held by |
|---|---|---|---|
IBKR Flex token — n8n cred Viska: IBKR Flex Token, type httpQueryAuth (token as query param). ≈6h life → rotation reminder. |
n8n (Hostinger) credential store | n8n Flex SendRequest node only |
apply-svc; Supabase; dashboard; workflow body |
FLEX_APPLY_WEBHOOK_SECRET — static bearer; n8n cred Viska: Flex Webhook Auth (6zo7MIzIBcUrjswl, httpHeaderAuth). ONE secret for BOTH /sign-upload and /apply, constant-time compare, 401-before-I/O. |
n8n credential store (header value); apply-svc env FLEX_APPLY_WEBHOOK_SECRET |
n8n (sends Authorization: Bearer); apply-svc (constant-time verify, else 401) |
LLM context / workflow JSON; Supabase |
VISKA_DB_URL — service_role/owner Postgres conn string |
apply-svc env | apply-svc (apply SQL + ops.health_events alert insert) |
n8n (zero DB creds — by design); dashboard |
VISKA_SUPABASE_URL — Supabase project URL (Storage REST base) |
apply-svc env | apply-svc (Storage REST) | n8n; dashboard |
VISKA_SUPABASE_SERVICE_ROLE_KEY — service_role/secret key (Storage sign-upload + read + move). dispatch-060 called it SB_SECRET_KEY → reconciled to this name. |
apply-svc env only | apply-svc (mints signed URLs; reads/moves objects). Key never leaves the service — pure core never receives it (test-asserted) | n8n (PUTs via self-authenticating signed URL — zero storage cred); dashboard |
FLEX_INBOX_BUCKET — optional, default 'flex-inbox' |
apply-svc env (optional) | apply-svc (locks bucket; caller-supplied bucket → 400) | n8n (cannot override bucket) |
VISKA_IBKR_FLEX_QUERY_ID — Flex query id (non-secret; live currently hardcoded 1384817 because instance blocks $env in nodes) |
n8n run context (intended) / hardcoded in node (actual) | n8n SendRequest (q= param) |
apply-svc; Supabase |
N8N_API_URL / N8N_API_KEY — n8n REST/god-mode pair (tooling cred, not a runtime feature cred) |
n8n instance (Hostinger) + agent session env (direnv) | ViskaN8N god-mode (fetch/deploy workflow edits) | Not loaded this session — direnv whitelist excludes _Client-Orgs; blocks live fetch/deploy |
RAILWAY_TOKEN — Railway deploy/CLI token (deploy-side, not a runtime feature cred) |
deploy-side; UNSET in agent session | apply-svc deploy/redeploy (Railway) | n8n |
FLEX_UPLOADER_JWTflex_uploader HS256 JWT |
— | — | DROPPED (task 060) — scoped HS256 upload-JWT leg is structurally dead (ES256 prod 403s against HS256 self-mint); replaced by server-signed upload URLs (3545985/d182bcb). Not minted, not wired. The flex_inbox_uploader_insert RLS policy exists in DDL but is vestigial / unused-but-harmless (signed URLs bypass it; service_role reads bypass RLS). |
flex-pull-rescope-ops.md:8-13,16,24,54,62-65,76 apply_service.py:6-9,35-44,46,66-70,182-186,326-340 on-demand-webhook-design.md:111-113 chronicle-2026-06-22:389
KEY ISOLATION INVARIANT (CONFIRMED)
The service-role key and DB URL live ONLY on apply-svc (Railway). n8n holds exactly two feature creds — the IBKR Flex token (query-auth, outbound to IBKR) and FLEX_APPLY_WEBHOOK_SECRET (header-auth, to apply-svc) — plus the non-secret query id. n8n's storage write is a single-use server-minted signed URL (authentication: none), so n8n never touches Supabase storage/DB secrets. flex-pull-rescope-ops.md:24,62-65 apply_service.py:182-186,340
Deploy-state & open caveats
LIVE-DEPLOY FINGERPRINT — UNCONFIRMED
The multi-day Fix A is committed in code (f810998, 75 tests green; build_sections_by_day + emit_days present). flex_to_sections.py:145 ibkr_import.py:359 But the Railway live image running Fix A is UNCONFIRMED — /healthz 200 + 401-on-unauth /apply are the only live confirmations; the running image SHA is unfingerprinted (no railway.toml in ViskaDB; RAILWAY_TOKEN unset). State as: code-confirmed, live-deploy-unconfirmed. Do not claim "22/22 lands in prod." chronicle-2026-06-22:92-100
- Apply POST body key — handler truth =
object_path(apply_service.py:146, 400s on missing); n8n node spec sendspath(rescope-ops:78). This mismatch is an unreconciled OPEN — reconciliation state UNKNOWN. Contract ={bucket, object_path}; never assertpathworks end-to-end. rescope-ops:78,80 - PUT-leg auth shape — whether the signed-upload PUT needs the returned token as a header vs query-string self-auth is OPEN UNKNOWN. rescope-ops:67-70
- On-demand webhook — 2nd Webhook trigger,
onReceived202-async, native Header Auth, n8n 5-min static-data cooldown — DESIGN-APPROVED, NOT BUILT (blocked onN8N_API_URL/N8N_API_KEY+ secret mint). Exact webhook path string UNKNOWN. on-demand-webhook-design.md:6,109-117 - Live node ids for Sign Upload / Upload / Apply POST UNKNOWN (raw JSON gated). rescope-ops
- Supabase project ref behind
VISKA_SUPABASE_URL— binding-only by policy, not read. Whether Cloudflare also serves Pages/DNS for viska.gg surfaces is UNKNOWN; only its CF Access (dashboard auth) role is confirmed.
DECISION 055-A — DELIVERY POSTURE
055-A chose to reuse the in-repo importer (option A), rejecting a forked write path (B, rule #18). A's DEFAULT delivery = file-drop (host-internal, auth:none) + Hades-owned cron; the bearer-auth HTTP apply-svc on Railway is the rule-#19-gated FALLBACK for the synchronous on-demand viska.gg pull. The Railway HTTP endpoint is not 055's primary design — it is the gated fallback. 055/ViskaDB.done:21-50
§3Source Data & the IBKR Flex Handshake
The trading-intelligence pipeline draws one source: the IBKR Flex Web Service. A single Activity Flex query returns the full daily series (~22 <FlexStatement> in one FlexQueryResponse) via a 2-call async handshake on Hostinger n8n. n8n is thin transport holding zero DB/storage credential; the Railway apply-svc re-parses the raw XML and is the sole SSOT writer of trading.*.
n8n (transport)
Hostinger, workflow ADeTjBZiHOMIXzvr — "Viska Macro Daily — IBKR Flex Pull". Pulls XML, uploads to bucket, fires Apply POST. No trading writes. flex-pull-...json:85; chronicle:389
apply-svc (SSOT writer)
Railway flex-apply-svc-production.up.railway.app — separate service from viska-mimir-agent. Re-parses raw XML → trading.*. /healthz 200 live-probe
IBKR Flex
Query 1384817, v=3, Last30CalendarDays. No webhook push — request→poll only. build-spec:24; chronicle:389
3.1 — The Flex query
| Fact | Value | Source |
|---|---|---|
| Flex query id | 1384817 — hardcoded live ($env blocked on the instance) | build-spec:24; chronicle-2026-06-22:389(2) |
| Flex version | v=3, text format | build-spec:24 |
| Query type / scope | Last30CalendarDays — one <FlexStatement> per trading day (~22 in the 06-15 fixture; fromDate==toDate each) | multiday-contract:17-24; chronicle:56 |
| Query category | "Activity Flex Query" → OpenPositions / EquitySummaryInBase / CashReport / FxPositions | build:33; multiday-contract |
| Account (prod) | U22131377 — parsed from XML at runtime (_account_id()), not hardcoded; test fixtures use U999 | flex_to_sections.py:49-55; ibkr_import.py:21 |
A single pull returns the full daily series in one FlexQueryResponse. The earlier "one pull = one report_date" framing was explicitly retracted. multiday-contract:23-24
3.2 — The two-call async handshake
IBKR Flex is a request → poll pattern (no push). Live-proven end-to-end through Normalize by operator test (exec 35607). SendRequest is pinned to ndcdyn in the live node; the <Url> IBKR returns came back on gdcdyn, so both GetStatement URLs follow {{ FlexStatementResponse.Url }} dynamically. flex-pull-...json:85; chronicle:389(3)
Call 1 — SendRequest (request XML)
GET https://ndcdyn.interactivebrokers.com/Universal/servlet/
FlexStatementService.SendRequest?t=<FLEX_TOKEN>&q=1384817&v=3
→ FlexStatementResponse:
<FlexStatementResponse timestamp="...">
<Status>Success</Status> <!-- Success | Warn | Fail -->
<ReferenceCode>NNNNNNNNNN</ReferenceCode>
<Url>https://gdcdyn.interactivebrokers.com/Universal/servlet/
FlexStatementService.GetStatement</Url>
</FlexStatementResponse>
- n8n node Flex SendRequest, auth =
httpQueryAuthcredIBKR flex— token bound as query paramt, never in body. build:24,38; chronicle:389(1) - Parsed by Parse SendRequest XML → gate SendRequest OK?; on
Fail→ Slack Flex Generation Failed. build:24-29; chronicle:389(5) <ReferenceCode>+<Url>confirmed returned live. chronicle:389(3)
XML FIELD-LEVEL CAVEAT
Only <Status>, <ReferenceCode>, and <Url> are the confirmed parsed set. The timestamp attribute and the exact Warn/Fail casing are structural-only — UNKNOWN at field level (no raw SendRequest envelope committed to the repo; payloads are gated). domain finding §2
Call 2 — GetStatement (poll loop)
GET {{ FlexStatementResponse.Url }}?t=<FLEX_TOKEN>&q=<ReferenceCode>&v=3
(node Flex GetStatement, responseFormat:text → raw XML at $json.data)
Poll loop (bounded, NOT fixed 2-shot):
Statement Ready? ──false──▶ Max Tries? {{$runIndex}} < 6
│true │true │false
▼ ▼ ▼
FlexQueryResponse Wait Retry ↺ Flex Timeout Alert
(data document) Flex GetStatement → Respond
- Bounded poll loop
$runIndex < 6— refactored live from a 2-shot scaffold after aStatus=Warnnot-ready race. chronicle:389(4); build:36 - Not-ready signal: GetStatement returns the same
FlexStatementResponseenvelope with<Status>Warn</Status>(still generating) instead of theFlexQueryResponsedata document — the gate distinguishes these. domain finding §2
Live write-path node ids
| Node | Live id | Role |
|---|---|---|
| Flex GetStatement | 4531f9c5-bbdc-4f40-9639-680ab3f5aa2d | raw XML source (responseFormat:text → $json.data) |
| Normalize Flex Dataset | 28992ad5-b0e7-491f-8f5f-7b001802fe6e | emits report_date, object_path (metadata/guard only) |
| Flag Unmapped Conids | 902d95a5-17b8-4380-b33c-5c5f74db6ce7 | reads Normalize; keep |
| Upsert Positions | f694c363-607a-4c18-8a84-57b5a5d8e444 | REMOVED — dead write path; n8n holds zero trading-write cred |
| Respond | e6daefe1-b458-43aa-aae0-bb45d95b7cf4 | terminal |
NODE IDS ≠ COMMIT SHAS
28992ad5…, 902d95a5…, 4531f9c5…, f694c363… are n8n node ids, not git SHAs. Real commits: Fix B 2f12e5c, guard f8a0304, dedup b1664ff, signed-URL 3545985/d182bcb, Fix A f810998. SendRequest/GetStatement/Wait/Init node ids themselves: UNKNOWN (the fetched node table lists only the 5 write-path nodes). rescope-ops:30,32
3.3 — The FlexQueryResponse data document
Reconstructed verbatim from the parser's .get(...) targets (flex_to_sections.py) and the canonical test fixture (test_flex.py:20-48,123-137). Attribute names are the actual strings the parser reads.
<FlexQueryResponse queryName="t" type="AF"> <!-- test_flex.py:20,127 -->
<FlexStatements count="22"> <!-- multiday-contract:17 -->
<FlexStatement accountId="U22131377" <!-- :51-53 _account_id -->
fromDate="20260514" toDate="20260514" <!-- :134-137 period bounds -->
period="Last30CalendarDays"
whenGenerated="20260615;..." > <!-- re-stamped every pull -->
<AccountInformation accountId="U22131377" currency="USD" /> <!-- :54 fallback acct -->
<!-- ===== OpenPositions → instruments + positions_snapshots ===== -->
<OpenPosition reportDate="20260514" <!-- :38,72 latest-rd selector -->
levelOfDetail="SUMMARY" <!-- :74 SUMMARY-only filter -->
symbol="AA" conid="100" description="Alcoa" <!-- :80-81 -->
isin="US0138721065" listingExchange="NYSE" <!-- :81 isin→Security ID -->
multiplier="1" assetCategory="STK" <!-- :82 -->
currency="USD" position="200" <!-- :83 position→Quantity -->
costBasisPrice="30" costBasisMoney="6000" <!-- :84-85 -->
markPrice="35" positionValue="7000" <!-- :85 markPrice→Close -->
fifoPnlUnrealized="1000" <!-- :86 →Unrealized P/L -->
percentOfNAV="..." fxRateToBase="..." /> <!-- Normalize-only; NOT flex_to_sections -->
<!-- ... one OpenPosition per holding, 25–31 per day ... -->
<!-- ===== EquitySummaryInBase → nav_snapshots ===== -->
<EquitySummaryByReportDateInBase reportDate="20260514" <!-- :93-94 -->
currency="USD" cash="500" stock="1000" <!-- :96-97 -->
total="1500" /> <!-- :98 total→NAV total -->
<!-- ===== FxPositions → fx_balances ===== -->
<FxPosition reportDate="20260514" levelOfDetail="SUMMARY" <!-- :105-108 -->
functionalCurrency="USD" fxCurrency="EUR" <!-- :110 fxCurrency→Description -->
quantity="1000" costBasis="-1100" <!-- :113 -->
value="1100" unrealizedPL="5" /> <!-- :113-114 -->
<!-- ===== Activity-Flex extras: parsed by n8n Normalize only ===== -->
<CashReport> <!-- multiday-contract:106-108 -->
<CashReportCurrency currency="USD"
endingCash="..." endingSettledCash="..." />
</CashReport>
<!-- Trades / Dividends / CashTransaction: CSV path, NOT Flex Phase-1 -->
</FlexStatement>
<!-- ... 21 more FlexStatement, one per trading day ... -->
</FlexStatements>
</FlexQueryResponse>
Parse guards (flex_to_sections.py): _safe_parse (:25-33) rejects any <!DOCTYPE>/<!ENTITY> (XXE / billion-laughs guard, stdlib-only) — a genuine FlexQueryResponse carries neither. levelOfDetail filter (:74, :108): only SUMMARY (or empty) rows pass; LOT-level rows dropped. flex_to_sections.py:25-33,74,108
3.4 — Source stream → trading.* coverage matrix
Two parsers consume the document with different coverage: flex_to_sections.py (the live apply-svc path) lands the Phase-1 snapshot only; ibkr_import.emit() has the full machinery but the trade/dividend/cash/interest sections populate only on the CSV path.
| Source stream (XML) | → section key | → trading.* | Flex path? | Source |
|---|---|---|---|---|
| OpenPosition (instruments) | Financial Instrument Information | trading.instruments | YES | flex_to_sections:67,80-87; ibkr_import:164-187 |
| OpenPosition (positions) | Open Positions | trading.positions_snapshots | YES | flex_to_sections:68,83-88; ibkr_import:235-257 |
| EquitySummaryByReportDateInBase | Net Asset Value | trading.nav_snapshots | YES | flex_to_sections:91-100; ibkr_import:259-281 |
| FxPosition | Forex Balances | trading.fx_balances | YES | flex_to_sections:102-115; ibkr_import:339-352 |
| Trades | Trades | trading.trades | CSV only | ibkr_import:208-233; flex_to_sections:10-12 |
| Dividends + Withholding Tax | Dividends, Withholding Tax | trading.dividends | CSV only | ibkr_import:297-323 |
| Deposits & Withdrawals | Deposits & Withdrawals | trading.cash_flows | CSV only | ibkr_import:283-295 |
| Interest | Interest | trading.fees_interest | CSV only | ibkr_import:325-337 |
| CashReport / CashReportCurrency | — | no trading.* target | Normalize reads; no landing | multiday-contract:106-108 |
| CashTransaction | — | UNKNOWN | — | in no importer file/spec |
SCOPE — FLEX PHASE-1 LANDS 4 STREAMS
The live autonomous Flex path lands 4 streams — instruments, positions, NAV, FX — at the latest reportDate per statement. Trades / Dividends / Interest / Cash were deliberately deferred (they span all daily statements and feed Mímir P&L, not the public.* FE views). The shared emit() already has the machinery; the Flex adapter just doesn't populate those sections yet. percentOfNAV / fxRateToBase / endingCash reach Normalize output but map to no trading.* column — metadata/guard only. flex_to_sections.py:10-12; multiday-contract:75-76
3.5 — From XML to trading.*: the write path
The raw FlexQueryResponse reaches trading.* via the Railway apply-svc, not n8n. n8n Normalize output is metadata/guard/filename only — it never drives the DB write.
n8n (thin transport):
Sign Upload ──POST /sign-upload (bearer)──▶ apply-svc mints signed URL
Upload Flex XML ──PUT raw FlexQueryResponse to signed URL (auth: none)──▶ flex-inbox bucket
Apply POST ──POST /apply (bearer, body {bucket, object_path})──▶ apply-svc
apply-svc (SSOT writer):
fetch object (service_role) → flex_to_sections.build_sections_by_day
→ ibkr_import.emit_days → ONE idempotent BEGIN…COMMIT → trading.*
Apply POST body — the key contract
POST /apply
Authorization: Bearer <FLEX_APPLY_WEBHOOK_SECRET> (constant-time compare, 401-before-I/O)
Content-Type: application/json
{
"bucket": "flex-inbox",
"object_path": "flex/U22131377/20260619/FlexQueryResponse.xml"
}
OPEN — object_path vs path (UNRECONCILED)
Handler truth is object_path — apply_service.py:146 reads payload.get("object_path") and 400s when it is missing. The n8n node spec, however, sends {bucket, path} (rescope-ops:78). Which side was reconciled live is UNKNOWN — flagged OPEN in the spec itself (:80). The end-to-end contract is written as {bucket, object_path} (handler truth); never assert path works end-to-end. The PUT-leg auth shape (token-as-header vs query-string self-auth) is likewise UNKNOWN/OPEN (rescope-ops:67-70). apply_service.py:146; rescope-ops:78,80,67-70
Multiday landing & idempotency
Multiday emit
build_sections_by_day (flex_to_sections:145-162) yields one tuple per distinct OpenPosition reportDate (each single-day, period_start==period_end); emit_days (ibkr_import:359-382) stacks all N days in one transaction with per-day temp tables _stmt_<i>. apply_service.py:94-122
Idempotency — semantic hash
file_hash = sha256("flex:" + account + ":" + reportDate) → trading.statements UNIQUE(file_hash). One row per (account, reportDate); re-pull = no-op. NOT a raw-bytes hash — IBKR re-stamps whenGenerated every pull (the shipped bug, fixed b1664ff). flex_import.py:20-31; test_flex.py:154-177
The empty-statement guard
IF "Statement Usable?" (position_count > 0 AND report_date notEmpty) sits between Normalize and Sign Upload (commit f8a0304, activeVersionId 6aa4c859). False branch → Flag Unmapped Conids → Respond, skipping Sign/Upload/Apply (no empty overwrite). Reason: IBKR toDate floats → empty next-day snapshot (exec 35606, reportDate 2026-06-20, 0 positions). git show f8a0304; chronicle.md:17-32
3.6 — Credential bindings (names only)
| Binding | Type | Where | Source |
|---|---|---|---|
IBKR flex (token t) | n8n httpQueryAuth | SendRequest / GetStatement | chronicle:389(1) |
Viska: Flex Webhook Auth (6zo7MIzIBcUrjswl) = FLEX_APPLY_WEBHOOK_SECRET | n8n httpHeaderAuth | Sign Upload + Apply POST (single shared bearer) | rescope-ops:54,76 |
FLEX_APPLY_WEBHOOK_SECRET, VISKA_DB_URL, VISKA_SUPABASE_URL, VISKA_SUPABASE_SERVICE_ROLE_KEY, FLEX_INBOX_BUCKET (opt) | apply-svc env (Railway) | apply-svc; service-role key never leaves apply-svc | apply_service.py:35-44,182-186 |
FLEX_UPLOADER_JWT / flex_uploader HS256 leg | DEAD/dropped | HS256 self-mint 403s vs ES256 prod; replaced by server-minted signed URLs (task 060, 3545985/d182bcb). flex_inbox_uploader_insert RLS policy is vestigial unused-but-harmless | apply_service.py:6-9,46; 20260621-01:32-38 |
3.7 — UNKNOWN / unconfirmed (this domain)
- CashTransaction — named in the prompt but in no importer file or spec; no section key, no table. Cannot confirm it is consumed. (CashReport/CashReportCurrency is read by n8n Normalize but lands nowhere.) UNKNOWN
- SendRequest/GetStatement not-ready XML attributes beyond
<Status>/<ReferenceCode>/<Url>— raw envelope not committed (gated). UNKNOWN - SendRequest / GetStatement / Wait / Init node ids — fetched node table lists only the 5 write-path nodes. UNKNOWN
- Apply PUT-leg auth shape + the
path-vs-object_pathlive reconciliation — both OPEN. UNKNOWN - Sign Upload / Upload / Apply POST live node ids — raw JSON gated. UNKNOWN
- Live image SHA running Fix A on Railway — unfingerprinted (
RAILWAY_TOKENunset). Fix A is code-confirmed (f810998, 75 tests) but live-deploy-unconfirmed; do not claim "22/22 lands in prod." UNKNOWN
§4The n8n Workflow — Node by Node
Workflow ADeTjBZiHOMIXzvr — "Viska Macro Daily — IBKR Flex Pull" — runs on Hostinger n8n and is the thin transport of the IBKR ingest path. It pulls a multi-day Flex statement from IBKR, guards it, and drops the raw XML to the Railway apply-svc via a signed URL. It holds zero DB/storage credential and does zero trading writes — the apply-svc re-parses the raw XML and is the sole SSOT writer of trading.*. flex-pull-...json:85; multiday-contract:50-52
TOPOLOGY — NEVER CONFLATE
n8n on Hostinger (workflow ADeTjBZiHOMIXzvr). apply-svc on Railway (flex-apply-svc-production.up.railway.app) — a separate service from the Mastra/Mímir agent (viska-mimir-agent Railway project). n8n is never on Railway; the two Railway services are never the same thing. live-probe /healthz 200; CLAUDE.md topology
Live state & evolution
Active LIVE
active:true since operator publish 08:42 2026-06-22, armed for the 22:30 cron. activeVersionId 6aa4c859 (post "Statement Usable?" guard). chronicle.md:35-38,25
Node count
17 = design baseline (build spec); grew to 19+ live during cred-wiring + poll-loop refactor + guard insert. The live graph below is authority. build-spec:20-30; chronicle-2026-06-22:389
Round-trip
GREEN end-to-end once (exec 35607): pull → Sign Upload 200 → Upload PUT 200 → Apply 200. chronicle-2026-06-22:501-503
Live node graph (17-node design → current wiring)
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ TRIGGERS (two entry points, converge on Init Params) │
└─────────────────────────────────────────────────────────────────────────────────────┘
[Daily Schedule (EOD)] scheduleTrigger cron 0 30 22 * * 1-5 (22:30 Mon–Fri)
│
├──────────────────────────────────────┐
[On-Demand Pull] │ webhook, authentication: headerAuth
│ (2nd entry; 202 onReceived async) │ DESIGN-APPROVED, cooldown NOT BUILT
└──────────────────────────────────────┤
▼
[Init Params] Set/Code — seeds run params
│
▼
[Flex SendRequest] httpRequest, auth=httpQueryAuth
q={query 1384817}, v=3, t=text ── cred "IBKR flex"
host PINNED ndcdyn → returns FlexStatementResponse XML
│
▼
[Parse SendRequest XML] xml node
│
▼
┌──── [SendRequest OK?] IF (Status == Success?) ────┐
│ true false │
▼ ▼
[Wait For Report] wait [Flex Generation Failed] Slack
│ │
▼ ▼
┌────────►[Flex GetStatement] httpRequest [Respond]
│ id 4531f9c5-bbdc-4f40-9639-680ab3f5aa2d
│ url = {{ FlexStatementResponse.Url }} (DYNAMIC — observed gdcdyn host)
│ responseFormat: text → $json.data = raw FlexQueryResponse XML
│ │
│ ▼
│ [Parse Statement XML] xml node
│ │
│ ▼
│ ┌─── [Statement Ready?] IF (statement generated?) ───┐
│ │ false true │
│ ▼ ▼
│ [Max Tries?] IF ({{$runIndex}} < 6) [Normalize Flex Dataset]
│ │ true │ false id 28992ad5-b0e7-491f-8f5f-7b001802fe6e
│ ▼ ▼ │ Code (flex-normalize.js body)
│ [Wait Retry] [Flex Timeout Alert] Slack │ multi-day: Σ positions,
│ │ │ │ latest report_date, days[]
└─────┘ ▼ ▼
(loop GetStatement) [Respond] ┌── [Statement Usable?] IF ──┐
│ true false │
│ (pos_count>0 AND │ (empty/next-day
│ report_date notEmpty) │ snapshot — skip)
▼ │
[Sign Upload] httpRequest POST │
{apply-svc}/sign-upload (headerAuth) │
cred "Viska: Flex Webhook Auth" │
body {bucket:flex-inbox, object_path} │
→ {ok,bucket,path,signedUrl,token} │
│ │
▼ │
[Upload Flex XML] httpRequest PUT │
url = {{ Sign Upload.signedUrl }} │
auth: none (signed URL self-auth) │
body = {{ Flex GetStatement.data }} │
(raw 22-statement XML, application/xml) │
│ │
▼ │
[Apply POST] httpRequest POST │
{apply-svc}/apply (headerAuth) │
body {bucket, object_path: ...path} │
→ {ok,reportDate,positions,fx,nav_rows}│
│ │
▼ ▼
[Flag Unmapped Conids] ◄───────────────┘
id 902d95a5-17b8-4380-b33c-5c5f74db6ce7
│
▼
[Respond]
id e6daefe1-b458-43aa-aae0-bb45d95b7cf4
REMOVED: [Upsert Positions] id f694c363-607a-4c18-8a84-57b5a5d8e444 (postgres)
disabled → REMOVED; replaced by Sign Upload → Upload → Apply POST.
n8n now holds ZERO trading-write credential.
Triggers
Daily Schedule (EOD) — scheduleTrigger
Fires the EOD pull on cron 0 30 22 * * 1-5 = 22:30 local, Mon–Fri. Output → Init Params. build-spec:22; webhook-design:36
On-Demand Pull — webhook, authentication: headerAuth
Authenticated POST entry for a dashboard "pull latest" button; converges on the same chain (one guard, one apply path). responseMode: onReceived → 202 immediately, chain runs async server-side to survive Flex's 30s–3min latency. Header-auth secret held by the dashboard backend, never in browser JS (rule #19). The headerAuth trigger node exists; the 5-min static-data cooldown gate is DESIGN-APPROVED, NOT BUILT — blocked on N8N_API_URL/N8N_API_KEY + auth-secret mint. Exact path string UNKNOWN. webhook-design:6,44-51,109-117,135-137
Pull & parse
| Node | Type | Behaviour |
|---|---|---|
| Init Params | Set/Code | Seeds run parameters. Exact field contents UNKNOWN. build-spec:24 |
| Flex SendRequest | httpRequest · httpQueryAuth | Flex Web Service call 1 — requests statement generation. q=1384817, v=3, t=text. Host pinned ndcdyn. Query id hardcoded (non-secret) because $env.VISKA_IBKR_FLEX_QUERY_ID is blocked on the instance. Query 1384817 is configured Last30CalendarDays → one <FlexStatement> per trading day (~22 days). Cred IBKR flex (Flex token, ≈6h life; never in workflow body). build-spec:24; chronicle-2026-06-22:389(1,2); flex-pull-...json:85 |
| Parse SendRequest XML | xml | Parses SendRequest → JSON (Status, ReferenceCode, Url). XML node used to dodge the create_workflow_from_code regex-mangle bug (zero regex in Code nodes). build-spec:18,24 |
| SendRequest OK? | IF | Branches on Status == Success. true → Wait For Report. false → Flex Generation Failed (Slack). build-spec:24-29; chronicle-2026-06-22:389(5) |
| Wait For Report | wait | Initial delay before the first GetStatement poll. Exact duration UNKNOWN. → Flex GetStatement. build-spec:24 |
Flex GetStatement4531f9c5-bbdc-4f40-9639-680ab3f5aa2d | httpRequest · responseFormat text | Flex Web Service call 2 — fetches the generated statement. URL is dynamic {{ FlexStatementResponse.Url }} (the <Url> IBKR returns, observed host gdcdyn) — patched off the original hardcoded ndcdyn. Output: $json.data = unmodified FlexQueryResponse XML (all ~22 statements) — this is what Upload PUTs. Re-entered by the Wait Retry loop. chronicle-2026-06-22:389(3); rescope-ops:29,65-66 |
| Parse Statement XML | xml | Parses GetStatement XML → JSON for the readiness gate + Normalize. → Statement Ready?. build-spec:24 |
Poll / retry loop
Statement Ready?
IF — is the statement generated (vs "not ready" race)? true → Normalize. false → Max Tries?. chronicle-2026-06-22:389(4)
Max Tries?
IF {{$runIndex}} < 6 — bounds the poll loop to 6 attempts. true → Wait Retry. false → Flex Timeout Alert. chronicle-2026-06-22:389(4)
Wait Retry
wait — delays, then loops back to Flex GetStatement to re-poll. Replaced the original 2-shot retry scaffold. Exact duration UNKNOWN. chronicle-2026-06-22:389(4); build-spec:28,36
Normalize & empty-statement guard
NORMALIZE IS METADATA/GUARD ONLY — NOT THE DB WRITE
The Normalize Flex Dataset node (28992ad5-b0e7-491f-8f5f-7b001802fe6e, Code) parses the FlexQueryResponse into a guard/metadata/filename payload. It does not drive the DB write — apply-svc re-parses the raw XML authoritatively. percentOfNAV/fxRateToBase/endingCash are carried as metadata only (no trading.* column). multiday-contract:34-52,75-76
Output shape (Fix B logic, scripts/lib/flex-normalize.js:46-53, 7/7 tests green; iterates all statements, keys positions on conid, preserves NULLs via toN/toS never 0):
{ report_date: <latest day>, // → Sign Upload filename + Respond
position_count: <Σ across all days>, // → guard (passes if ANY day has data)
statement_count, report_dates[],
date_from, date_to,
days[] } // each: {report_date, positions[], nav, cash, fx, position_count}
LIVE NODE BODY IS DEFERRED — LIB ≠ LIVE
Fix B (multi-day Normalize) is committed (2f12e5c, lib flex-normalize.js, 7/7), but the live n8n Normalize node body update is DEFERRED to the A+B-green deploy (Class C, viska-pm-routed; branch feat/flex-multiday-normalize, local only). The live node currently still parses day-0 only (the stmts[0] collapse — harmless to the DB write per Fix-A analysis, since apply-svc re-parses raw XML). The lib body is not diffed against the live node. multiday-contract:78-82; chronicle.md:54-62
Statement Usable? — IF (inserted 2026-06-22)
Empty-statement guard between Normalize and Sign Upload. Condition: position_count > 0 AND report_date notEmpty. true → Sign Upload (apply path). false → Flag Unmapped Conids → Respond, skipping Sign/Upload/Apply (no empty overwrite). Reason: IBKR Flex's floating toDate returns an empty next-day/weekend snapshot that would otherwise PUT empty XML and overwrite good trading.* data (live incident: exec 35606, reportDate 2026-06-20, 0 positions). Commit f8a0304, activeVersionId 6aa4c859. A dedicated Slack "no-data" alert on the false branch was left off to dodge the Slack resource-discriminator quirk. git show f8a0304; chronicle.md:17-32
Write path — signed-URL drop (replaces removed Upsert Positions)
UPSERT POSITIONS — REMOVED
Upsert Positions (f694c363-607a-4c18-8a84-57b5a5d8e444, postgres, single-table (report_date,conid) upsert) was disabled then removed — replaced by Sign Upload → Upload → Apply POST. n8n now holds zero trading-write credential and does zero DB writes. rescope-ops:31,83-89; chronicle-2026-06-22:497
| Node | Method / Auth | Body & behaviour |
|---|---|---|
| Sign Upload | httpRequest POST{apply-svc}/sign-uploadhttpHeaderAuth · cred Viska: Flex Webhook Auth | Asks apply-svc to mint a single-use signed upload URL. Body { bucket:"flex-inbox", object_path: {{ Normalize.object_path }} }; bucket server-locked to flex-inbox (override → 400). Returns { ok, bucket, path, signedUrl, token }. Cred binds FLEX_APPLY_WEBHOOK_SECRET (value-bound by Hades, never in-session). → Upload Flex XML. rescope-ops:50-58; chronicle-2026-06-22:495,500 |
| Upload Flex XML | httpRequest PUT{{ Sign Upload.signedUrl }}auth: none | PUTs the raw unmodified statement XML to the server-minted signed URL — the signed URL self-authenticates, so the service-role key stays server-side. Body {{ Flex GetStatement.data }} = raw 22-statement FlexQueryResponse XML, Content-Type: application/xml. The HS256 FLEX_UPLOADER_JWT leg is dead/dropped (ES256 prod 403s); the flex_inbox_uploader_insert RLS policy is vestigial unused-but-harmless. → Apply POST. rescope-ops:60-66; apply_service.py:6-9,46 |
| Apply POST | httpRequest POST{apply-svc}/applyhttpHeaderAuth · same cred | Triggers apply-svc to re-parse the uploaded XML and write trading.*. Body key contract = object_path (handler truth, apply_service.py:146) — server-returned path = SSOT, avoids Normalize↔server drift. Returns { ok, reportDate, positions, fx, nav_rows }. Idempotent on (account, reportDate) via the file_hash grain — safe retry. → Flag Unmapped Conids. rescope-ops:72-80; apply_service.py:146; multiday-contract:63-67 |
OPEN — n8n SENDS path, HANDLER READS object_path
The n8n Apply POST node spec sends {bucket, path} (rescope-ops:78); the apply-svc handler reads payload.get("object_path") and 400s on a missing object_path (apply_service.py:146). The handler-confirmed contract is {bucket, object_path}. Whether this was reconciled live is UNKNOWN — flagged OPEN in the spec itself. The PUT-leg token-as-header vs query-string self-auth question is likewise UNKNOWN (OPEN). rescope-ops:67-70,78,80; apply_service.py:146
Live execution evidence & the two distinct gaps
Single-day Apply POST (live, exec 35607) returned { ok:true, reportDate:20260619, positions:21, fx:5, nav_rows:3 }. chronicle-2026-06-22:502-503 Normalize had parsed 30 positions for that same day. These are two separate phenomena — never conflate:
30 → 21 single-day gap
Inside apply-svc parse/persist (Normalize saw 30, Apply persisted 21 for one day). Cause inferred, not fully root-caused. A Railway/apply-svc concern, not n8n. ViskaN8N chronicle.md:7-14 [inferred: gap cause]
1-of-22 multi-day landing
The by-design latest-reportDate filter in apply-svc (keeps only max(reportDate)), not a stmts[0]/n8n-flatten bug. Fixed by Fix A (emit_days/build_sections_by_day, ViskaDB) + Fix B (multi-day Normalize, ViskaN8N). flex_to_sections.py:56,81,103,115; multiday-contract:42-61
FIX STATE — CODE-CONFIRMED, LIVE-DEPLOY-UNCONFIRMED
Fix A is committed in code (f810998 2026-06-22 23:10, 75 tests green; build_sections_by_day+emit_days present). Fix B is committed (2f12e5c, 7/7) with the live node body still deferred. The Railway live image running Fix A is UNCONFIRMED — no SHA fingerprint, RAILWAY_TOKEN unset, no railway.toml in ViskaDB. Do NOT claim "22/22 lands in prod." Live confirmations are limited to /healthz 200 and 401-on-unauth /apply. flex_to_sections.py:145; ibkr_import.py:359; chronicle 2026-06-22 23:40
Terminal & alert nodes
| Node | Type | Behaviour |
|---|---|---|
Flag Unmapped Conids902d95a5-17b8-4380-b33c-5c5f74db6ce7 | Code | Reads Normalize output, reports positions whose conid is null/unmapped (theme/sub_group join is a downstream DB concern; workflow writes raw IBKR fields only). On the guard-false path it is reached directly, skipping the apply chain. → Respond. rescope-ops:32; build-spec:43-44 |
Responde6daefe1-b458-43aa-aae0-bb45d95b7cf4 | respondToWebhook | Terminal node — returns the run result. Reached from Flag Unmapped Conids (success/guard-false), Flex Generation Failed, and Flex Timeout Alert. build-spec:26; rescope-ops:33 |
| Flex Generation Failed | Slack | Alert when SendRequest fails (SendRequest OK? = false). → Respond. chronicle-2026-06-22:389(5) |
| Flex Timeout Alert | Slack | Alert when the poll loop exhausts retries (Max Tries? = false, $runIndex reached 6 without a ready statement). → Respond. chronicle-2026-06-22:389(4) |
SLACK-NODE QUIRK
Both alert nodes (and the deferred no-data alert): the MCP addNode does not durably persist the Slack resource/operation discriminator → Resource/Operation + the real channel must be re-selected in the n8n UI before the alerts fire correctly. Live Slack channel id UNKNOWN. chronicle-2026-06-22:390; rescope-ops:105; chronicle.md:29-30
Credentials & node-id discipline
n8n holds exactly 2 feature creds
IBKR Flex token (httpQueryAuth, cred IBKR flex) for SendRequest/GetStatement, and FLEX_APPLY_WEBHOOK_SECRET (httpHeaderAuth, cred Viska: Flex Webhook Auth) for both /sign-upload and /apply. Zero DB/storage credential. rescope-ops:24,54,76
Node ids ≠ git commits
28992ad5… (Normalize), 902d95a5… (Flag Unmapped Conids), 4531f9c5… (GetStatement), f694c363… (removed Upsert) are n8n node ids, not SHAs. Real commits: Fix B 2f12e5c, guard f8a0304, dedup b1664ff, HMAC→Bearer 7466abf, signed-URL 3545985/d182bcb, Fix A f810998. rescope-ops:30,32
Open items
- On-demand webhook build (cooldown gate + 202 async) — design-approved, blocked on
N8N_API_URL/N8N_API_KEYREST creds + auth-secret mint. webhook-design:109-117 - Normalize multi-day live node body — committed in lib, live update deferred to A+B-green deploy (Class C, viska-pm-routed PR). chronicle.md:54-62
- Apply body key —
path(n8n) vsobject_path(handler) reconciliation state UNKNOWN; PUT-token shape UNKNOWN. rescope-ops:67-70,80 - Not in sourced docs UNKNOWN: exact webhook path string; Wait/Wait-Retry durations; live Slack channel id; Init Params field contents; live Sign Upload / Upload / Apply POST node ids (raw JSON gated). raw JSON gated
§5Normalize & the Guards
The n8n side of the Flex pull is thin transport. Its "Normalize Flex Dataset" Code node and the two guards around it never write the database — they shape metadata, gate empty statements, and surface unmapped instruments. apply-svc on Railway re-parses the raw XML and is the sole SSOT writer of trading.*. flex-normalize.js:6-11; multiday-contract:50-52
ROLE BOUNDARY (LOAD-BEARING)
The Normalize node output does NOT drive the DB write. It feeds exactly three sinks: the "Statement Usable?" guard, the Sign Upload filename, and the Respond payload. The uploaded raw XML — not the Normalize JSON — is what apply-svc parses authoritatively. flex-normalize.js:6-11
Node map & identity
Statement Ready?[true]
→ Normalize Flex Dataset (node id 28992ad5-b0e7-491f-8f5f-7b001802fe6e)
→ Statement Usable? (IF guard — true/false branch)
true → Sign Upload → Upload Flex XML → Apply POST → Flag Unmapped Conids → Respond
false→ Flag Unmapped Conids → Respond (skips Sign/Upload/Apply)
rescope-ops:30,32,37; chronicle.md:22-24
NODE IDs ARE NOT GIT SHAs
28992ad5-b0e7-491f-8f5f-7b001802fe6e (Normalize) and 902d95a5-17b8-4380-b33c-5c5f74db6ce7 (Flag Unmapped Conids) are n8n node ids, not commits — no such SHAs exist in the repo. The real commits in this domain are Fix B 2f12e5c and the guard f8a0304. rescope-ops:30,32
1. Normalize Flex Dataset
The Code node embeds a copy of normalizeFlexDataset because n8n Code nodes cannot import repo files; the tested SSOT for that body is scripts/lib/flex-normalize.js. Raw workflow JSON was not read (gated), so the lib is asserted-equal to the live node body, not independently diffed. flex-normalize.js:1-5 LIB=SSOT, LIVE NOT DIFFED
What it reads — four Activity-Flex blocks per <FlexStatement>
| XML block | Path | Emitted group |
|---|---|---|
OpenPositions.OpenPosition | stmt.OpenPositions.OpenPosition | positions[] |
EquitySummaryInBase…ByReportDateInBase | stmt.EquitySummaryInBase…[0] | nav |
CashReport.CashReportCurrency | stmt.CashReport.CashReportCurrency | cash[] |
FxPositions.FxPosition | stmt.FxPositions.FxPosition | fx[] |
build-spec:33; flex-normalize.js:23-38
Every block is wrapped [].concat(… || []) so the single-statement-as-object case (n8n's XML parser collapses 1-element arrays to objects) and an empty/absent block both normalize to an array. flex-normalize.js:24,33,36; multiday-contract:80
Fix B — multi-day awareness (commit 2f12e5c)
Root cause: Flex query 1384817 is Last30CalendarDays and returns one <FlexStatement> per trading day (~22, each fromDate==toDate). The pre-Fix-B node read stmts[0] only, so a 22-day file was judged by day-0 alone. multiday-contract:22-23,43; chronicle.md:56-57
Fix B iterates all statements ([].concat(...) then stmts.map) and adds the multi-day contract to the returned object: flex-normalize.js:19-21,42-54
report_dates[]— every day's ISO date, sorted ascending flex-normalize.js:42-43position_count— sum across all days (days.reduce((s,d)=>s+d.position_count,0)), NOT day-0's count flex-normalize.js:48statement_count—days.lengthflex-normalize.js:49date_from/date_to— first/last of sortedreport_datesflex-normalize.js:51-52report_date(top-level) — the latest day, so the Sign Upload filenameflex-<latest>-<execId>.xmlstays sensible while the uploaded file still holds all days flex-normalize.js:44,47; multiday-contract:10,135
Per-day object: { report_date, positions, nav, cash, fx, position_count: positions.length }; a .filter((d)=>d.report_date) drops statements with no resolvable date before they reach the guard. flex-normalize.js:39-40
Top-level emitted shape: { report_date, position_count, statement_count, report_dates, date_from, date_to, days }. report_date + position_count are the keys downstream nodes already read; the rest are additive — no downstream node breaks. Tests: 7/7 green in tests/ingestion/flex-normalize.test.js. flex-normalize.js:46-54; multiday-contract:79-80,137
FIX B IS COMMITTED — LIVE NODE BODY DEFERRED
Fix B is committed (2f12e5c, 7/7), but the live n8n Normalize node body update is DEFERRED to the A+B-green deploy step; the lib has not been diffed against the live node. This is Class C → viska-pm-routed. Fix B fixes guard/metadata correctness only — the load-bearing multi-day landing fix is Fix A in apply-svc (f810998), out of this domain. multiday-contract:81-82; chronicle.md:54-62
conid-keying & NULL discipline
Each position keys on conid: toN(p.conid) — the join key is conid/isin, never ticker; unmapped (conid=null) positions are flagged back, not dropped. Carried per-position: symbol, isin, quantity, mark_price, position_value, cost_basis_price, cost_basis_money, fifo_pnl_unrealized, percent_of_nav, currency, asset_category, fx_rate_to_base. flex-normalize.js:25-29; build-spec:40-41
Two coercers ensure empty/undefined/null map to null, never 0:
toN = (v) => (v===undefined || v===null || v==='') ? null : Number(v) toS = (v) => (v===undefined || v===null || v==='') ? null : String(v) isoDate: YYYYMMDD => YYYY-MM-DD, or null if falsy
flex-normalize.js:13-15; build-spec:42
percentOfNAV, fxRateToBase, and endingCash have no trading.* column — they ride the Normalize JSON as metadata only and never reach the DB. multiday-contract:75-76
2. Guard — "Statement Usable?" (commit f8a0304)
f8a030499160ec3fc78b6fe506f21ac7982e0531 — "feat(flex): empty-statement guard on IBKR Flex pull (ADeTjBZiHOMIXzvr)", 2026-06-22 10:58:22 +0200. Landed in DRAFT then published — activeVersionId 6aa4c859. PUBLISHED git show f8a0304; chronicle.md:25-28
WHY IT EXISTS
Exec 35606 returned report_date 2026-06-20 with ZERO positions — IBKR Flex toDate floats and can return an empty next-day/weekend snapshot. Unguarded, the 22:30 Mon–Fri schedule could push empty XML to /apply and overwrite a good trading.* snapshot. chronicle.md:17-24
Condition: an IF node between Normalize and Sign Upload — position_count > 0 AND report_date notEmpty.
- True → Sign Upload → Upload → Apply POST (the apply path).
- False → Flag Unmapped Conids → Respond, skipping Sign/Upload/Apply; returns
position_count:0, no empty write.
chronicle.md:22-24
With Fix B's summed position_count, the guard blocks only a wholly-empty file (every day empty); any one day with data passes. "Wrong-date" detection (report_date == last trading day) is explicitly out of scope. Applied via MCP update_workflow (5 atomic ops); not test-exec'd because mutating (fires IBKR + apply + DB) → Class C trading-writer critical-path, viska-pm-routed PR. multiday-contract:159; chronicle.md:25-32
3. Guard — "Flag Unmapped Conids" (node id 902d95a5-…)
Reads Normalize's output and reports back positions where conid resolved to null — the conid-mapping feedback path. The workflow writes raw IBKR fields only; the theme/sub_group overlay (trading.instrument_themes) is a downstream DB concern, untouched here. Unmapped positions are surfaced rather than silently dropped. build-spec:41,43-44
It is the penultimate node before Respond on both paths. Across the rescope its inbound connection moved from the removed Upsert Positions → Flag Unmapped Conids to Apply POST → Flag Unmapped Conids; on the guard's false branch it is reached via Normalize → Flag Unmapped Conids → Respond. Marked "keep" through the Upsert Positions removal — Normalize is kept precisely because it drives object_path + Flag Unmapped. rescope-ops:83-89,93; chronicle.md:23-24
UPSERT POSITIONS — REMOVED, NOT DISABLED
The legacy Upsert Positions node (f694c363…) was removed, not merely disabled. n8n holds zero trading-write credential and does zero DB writes — the write path is Sign Upload → Upload → Apply POST, with apply-svc as sole SSOT writer. rescope-ops:31,83-89
Open / unknown in this domain
- Live n8n node body vs
flex-normalize.js— not diffed (raw JSON gated; live update deferred). Lib is the tested SSOT, asserted-equal not verified-equal. multiday-contract:81-82 - The downstream 30→21 single-day count gap lives inside apply-svc parse/persist, not in Normalize — Normalize parsed 30 for that day; the loss is on the persist side. Cause INFERRED, not root-caused. Distinct from the multi-day 1-of-22 landing. ViskaN8N chronicle.md:7-14
- Sign Upload / Upload / Apply POST live node ids — UNKNOWN (raw JSON gated). —
§6Write Path — Sign-Upload, Apply, Real Payloads
The IBKR Flex mutating write path that lands raw FlexQueryResponse XML into Supabase trading.*. Three n8n nodes drive two Railway apply-svc endpoints plus one direct Supabase Storage PUT. n8n is thin transport holding zero DB/storage credential; apply-svc re-parses the raw XML and is the sole SSOT writer of trading.*.
INFRA GROUND TRUTH
n8n runs on Hostinger (workflow ADeTjBZiHOMIXzvr "Viska Macro Daily — IBKR Flex Pull") flex-pull-ADeTjBZiHOMIXzvr.json:85. apply-svc runs on Railway at flex-apply-svc-production.up.railway.app — a separate service from the Mastra/Mímir agent (viska-mimir-agent). Never conflate the two Railway services. /healthz → 200 {"ok":true} live-probe; apply_service.py:357-359
6.0 Topology
Flex GetStatement (raw 22-stmt FlexQueryResponse XML, Hostinger n8n)
→ Parse Statement XML → Normalize Flex Dataset (emits report_date + object_path; metadata/guard/filename ONLY, never the DB write)
→ Statement Usable? guard (position_count > 0 AND report_date notEmpty)
→ [1] Sign Upload POST {apply-svc}/sign-upload bearer → {ok, bucket, path, signedUrl, token}
→ [2] Upload Flex XML PUT signedUrl auth=none raw XML → object in flex-inbox bucket
→ [3] Apply POST POST {apply-svc}/apply bearer → {ok, account, reportDate, report_dates, statements, positions, fx, nav_rows}
→ Flag Unmapped Conids → Respond
The raw 22-statement XML reaches apply-svc on the wire; apply-svc re-parses it as SSOT for report_date + positions — the n8n Normalize date only names the storage object. on-demand-webhook-design.md:70-76; flex-multiday-normalize-apply-contract.md:50-52
LIVE NODE IDS — PARTIAL
Only Normalize Flex Dataset = 28992ad5-b0e7-491f-8f5f-7b001802fe6e is in readable markdown flex-pull-rescope-ops.md:26-30. These are n8n node ids, NOT git SHAs (no such commits exist in the repo) flex-pull-rescope-ops.md:30,32. Sign Upload / Upload Flex XML / Apply POST live node ids: UNKNOWN (raw workflow JSON gated). Real commits are distinct: Fix B 2f12e5c, guard f8a0304, dedup b1664ff, HMAC→Bearer 7466abf, signed-URL 3545985/d182bcb, Fix A f810998.
6.1 Node [1] — Sign Upload
n8n-nodes-base.httpRequest, POST {apply-svc}/sign-upload. Auth = httpHeaderAuth credential Viska: Flex Webhook Auth (6zo7MIzIBcUrjswl) binding env var FLEX_APPLY_WEBHOOK_SECRET — the single shared bearer for BOTH endpoints, constant-time compared. flex-pull-rescope-ops.md:50-58; apply_service.py:36-37,66-70,326 Handler handle_sign_upload apply_service.py:177-219
Request body (object_path from the Normalize node):
POST /sign-upload
Authorization: Bearer <FLEX_APPLY_WEBHOOK_SECRET>
Content-Type: application/json
{
"bucket": "flex-inbox",
"object_path": "flex/2026-06-12/01HXXXXEXECID.xml"
}
Server-side input hardening:
bucketis never caller-controlled — a present non-defaultbucketis refused400 "bucket override not allowed"apply_service.py:85-91,206-208- Body is OPTIONAL; a missing
object_path⇒ generatedincoming/<uuid>.xmlapply_service.py:171-174,209 object_pathmust match_OBJECT_PATH_RE = [A-Za-z0-9_\-/]{1,200}\.xml, no.., not absolute; else400 "object_path must be a safe .xml object"apply_service.py:75-82,210-211- Mint is server-side:
POST {VISKA_SUPABASE_URL}/storage/v1/object/upload/sign/<bucket>/<path>with the service-role key, which stays on the server (injected closure; pure core never receives it; test-asserted) apply_service.py:182-186,250-271,340
Response 200 apply_service.py:218-219,265-270:
{
"ok": true,
"bucket": "flex-inbox",
"path": "flex/2026-06-12/01HXXXXEXECID.xml",
"signedUrl": "https://<viska-supabase-host>/storage/v1/object/upload/sign/flex-inbox/flex/2026-06-12/01HXXXXEXECID.xml?token=<jwt>",
"token": "<jwt>"
}
Error codes: 401 bad/missing bearer · 400 malformed body / non-xml object_path / bucket override · 502 storage mint failure. apply_service.py:191-192,215-216
6.2 Node [2] — Upload Flex XML
n8n-nodes-base.httpRequest, PUT. flex-pull-rescope-ops.md:60-70
| Field | Value |
|---|---|
| method | PUT |
| url | ={{ $('Sign Upload').item.json.signedUrl }} (server-minted, single-use) |
| authentication | none — the signed URL self-authenticates; the service-role key never leaves apply-svc flex-pull-rescope-ops.md:63 |
| body | ={{ $('Flex GetStatement').item.json.data }} — the raw, unmodified FlexQueryResponse XML (all 22 daily statements), Content-Type: application/xml flex-pull-rescope-ops.md:65-66 |
This PUTs the bytes straight to Supabase Storage flex-inbox — a private bucket. No apply-svc round-trip on this leg. 20260621-01-flex-inbox-bucket.sql:21-24
OPEN — PUT AUTH SHAPE
Whether the PUT needs the returned token as an Authorization: Bearer / x-upsert header, or the query-string token self-authenticates, is unresolved in the spec ("confirm the apply-svc's exact shape before wiring. Do NOT guess."). Status of that confirmation: UNKNOWN. flex-pull-rescope-ops.md:67-70
6.3 Node [3] — Apply POST
n8n-nodes-base.httpRequest, POST {apply-svc}/apply (same Railway host). Same httpHeaderAuth credential = FLEX_APPLY_WEBHOOK_SECRET. retryOnFail + onError continue — apply is idempotent so retry is safe. flex-pull-rescope-ops.md:72-80 Handler handle_apply apply_service.py:125-168
UNRECONCILED OPEN — APPLY BODY KEY (path vs object_path)
The n8n node sends {bucket, path} flex-pull-rescope-ops.md:78, but the apply-svc handler reads payload.get("object_path"), NOT path apply_service.py:146. On the current code, a body keyed path yields 400 "missing object_path" apply_service.py:147-148. The spec flags this exact mismatch as an unresolved OPEN flex-pull-rescope-ops.md:80. The handler-confirmed contract is {bucket, object_path}. Whether the live node was rewired to object_path, or apply-svc changed to accept path, is UNKNOWN — never assert path works end-to-end.
Request body (handler-truth contract):
POST /apply
Authorization: Bearer <FLEX_APPLY_WEBHOOK_SECRET>
Content-Type: application/json
{
"bucket": "flex-inbox",
"object_path": "flex/2026-06-12/01HXXXXEXECID.xml"
}
Response 200 — spreads the _build_and_emit summary apply_service.py:117-119,168:
{
"ok": true,
"account": "U22131377",
"reportDate": "20260612",
"report_dates": ["2026-05-14", "2026-05-15", "…", "2026-06-12"],
"statements": 22,
"positions": 660,
"fx": 110,
"nav_rows": 22
}
Field semantics: reportDate = latest report_date (backward-compat); report_dates = per-day period_end list (ascending); statements = day count; positions/fx/nav_rows = summed across all days. apply_service.py:117-119
LIVE SINGLE-DAY FIRE (exec 35607, reportDate 20260619)
A real single-day Apply POST returned ok:true with positions:21, fx:5, nav_rows:3 ViskaN8N/chronicle.md:9-12; chronicle-2026-06-22:502-503. Note Normalize parsed 30 positions for that day — the 30→21 single-day count gap lives inside apply-svc parse/persist CAUSE INFERRED ViskaN8N chronicle.md:7-14. This is distinct from the multi-day 1-of-22 landing issue (§6.7) — never conflate the two.
Error codes: 401 bad/missing bearer · 400 malformed JSON / missing-or-unsafe object_path / bucket override · 500 fetch/parse/apply failure (no commit + durable alert). apply_service.py:130-159
6.4 The Write Transaction — handle_apply
One pure core with injected I/O (fetch_object / apply_sql / move_object / alert) so it is unit-testable with no flask/psycopg/network. apply_service.py:30-33
- 1. Bearer check → 401 before any I/O apply_service.py:133-134
- 2. Body parse + bucket/path hardening — same guards as sign-upload apply_service.py:136-150
- 3. Fetch XML from
flex-inboxwith service-role key:GET {VISKA_SUPABASE_URL}/storage/v1/object/<bucket>/<path>apply_service.py:227-235 - 4.
_build_and_emit→build_sections_by_day(path)yields one tuple per distinct OpenPosition reportDate (ascending); per dayfile_hash = flex_file_hash(account, rd);emit_days(days,"flex_api",timezone)→ ONEBEGIN…COMMITtransaction apply_service.py:94-122,110; flex_to_sections.py:145-162; ibkr_import.py:359-382 - 5.
apply_sql(sql)— one autocommit connection; the emittedBEGIN…COMMITis the txn apply_service.py:274-281 - 6. On success:
move_object(bucket, object_path, "processed/"+object_path)viaPOST .../storage/v1/object/move— best-effort; a move failure does NOT un-commit apply_service.py:161-166,238-247 - 7. On any exception:
alert(...)→ insertops.health_events, return 500 — no commit; object left in inbox for retry apply_service.py:156-159,284-299
SQL shape (emit_days): single BEGIN; → SET LOCAL timezone → per day emit(..., stmt_tmp="_stmt_<i>", wrap=False) → COMMIT;. Per-day CREATE TEMP TABLE _stmt_<i> ON COMMIT DROP captures that day's statement_id via the RETURNING id upsert, avoiding single-statement collapse. ibkr_import.py:190-197,359-382 Every write is ON CONFLICT: instruments ON CONFLICT (conid) DO UPDATE ibkr_import.py:184-187; statements ON CONFLICT (file_hash) DO UPDATE … RETURNING id ibkr_import.py:196; trades/positions/dividends/cash_flows/fees_interest ON CONFLICT DO NOTHING ibkr_import.py:199-206; nav_snapshots ON CONFLICT (statement_id) DO NOTHING ibkr_import.py:281; fx_balances ON CONFLICT (statement_id, currency) DO NOTHING ibkr_import.py:352. Re-apply = no-op.
6.5 Idempotency Grain — Semantic file_hash
# flex_import.py:20-31
def flex_file_hash(account, rd):
return hashlib.sha256(("flex:%s:%s" % (account, rd)).encode()).hexdigest()
- Dedup key =
sha256("flex:{account}:{report_date}")— semantic, NOT raw XML bytes. IBKR stampswhenGeneratedfresh every pull, so two pulls of the same reportDate are byte-different yet the same snapshot; a bytes-hash would mint a duplicate every re-pull (the shipped bug, fixedb1664ff). flex_import.py:23-29 - This hash lands in
trading.statements.file_hashcarryingCONSTRAINT statements_file_hash_key UNIQUE (file_hash)⇒ exactly one statement row per (account, reportDate);statement_idis the per-day key for all child rows. 20260610-01-trading_schema_core.sql:23,29 source='flex_api'admitted bystatements_source_check CHECK (source = ANY (ARRAY['csv','flex_api'])). 20260610-01-trading_schema_core.sql:30-31- CSV path divergence:
ibkr_import.main()keeps a file-bytessha256— there a re-import IS the identical file, so bytes are correct identity. ibkr_import.py:403-404; flex_import.py:28-29
6.6 Error / Alert Path → ops.health_events
On 500, _real_alert writes stderr + inserts; meta carries {"bucket":…, "object_path":…}; alerting is best-effort (wrapped to never raise). apply_service.py:157-158,284-299
INSERT INTO ops.health_events (source,severity,error_class,message,meta)
VALUES ('flex_apply', %s, %s, %s, %s)
source='flex_apply' is admitted by health_events_source_check CHECK (source IN ('layer1','watchdog','flex_apply')) — the base CHECK in 20260611-01 allowed only ('layer1','watchdog'); the flex-inbox migration extends it. 20260621-01-flex-inbox-bucket.sql:48-50; 20260611-01-ops_schema_core.sql:14 The table is retry-safe: the object stays in flex-inbox (not moved to processed/) on failure, so a re-fire re-attempts cleanly.
6.7 The Two Distinct Gaps — Never Conflate
30→21 single-day count gap
Inside apply-svc parse/persist — Normalize saw 30 positions, Apply persisted 21 for one day (exec 35607). Cause INFERRED, not fully root-caused. ViskaN8N chronicle.md:7-14
1-of-22 multi-day landing
The by-design latest-reportDate filter (flex_to_sections.py:56,81,103,115) kept only max(reportDate). NOT a stmts[0]/flatten bug. Fixed by Fix A (emit_days, ViskaDB) + Fix B (multi-day Normalize, ViskaN8N). flex-multiday-normalize-apply-contract.md:42-52
| Fix | State | Evidence |
|---|---|---|
Fix A — build_sections_by_day + emit_days | CODE-COMMITTED f810998, 75 tests green | flex_to_sections.py:145; ibkr_import.py:359 |
| Fix B — multi-day Normalize lib | CODE-COMMITTED 2f12e5c, flex-normalize.js, 7/7 tests | ViskaN8N chronicle.md:54; flex-normalize.js:46-54 |
| Fix B — live Normalize node body update | DEFERRED to A+B-green deploy (lib≠live, not diffed) | flex-multiday-normalize-apply-contract.md:81-82 |
| Railway live image running Fix A | DEPLOY-UNCONFIRMED — no SHA fingerprint; RAILWAY_TOKEN unset; no railway.toml in ViskaDB | chronicle 2026-06-22 23:40 |
| Live 22-day end-to-end landing in prod | UNKNOWN — no confirming chronicle line | flex-multiday-normalize-apply-contract.md:139-143 |
DEPLOY NUANCE — STATE AS CODE-CONFIRMED, LIVE-DEPLOY-UNCONFIRMED
The multiday fix is merged in code (f810998); the live image running it is unconfirmed. The only live confirmations are /healthz 200 and 401-on-unauth /apply; the running image SHA is unfingerprinted. Never claim "22/22 lands in prod."
6.8 Empty-Statement Guard — "Statement Usable?"
An IF node between Normalize and Sign Upload: position_count > 0 AND report_date notEmpty (commit f8a0304, activeVersionId 6aa4c859). False branch → Flag Unmapped Conids → Respond, skipping Sign/Upload/Apply (no empty overwrite). git show f8a0304; ViskaN8N chronicle.md:17-32 Reason: IBKR toDate floats → empty next-day snapshot (exec 35606, reportDate 2026-06-20, 0 positions). The Upsert Positions node (f694c363…) was removed — replaced by this Sign Upload → Upload → Apply path; n8n holds zero trading-write credential. flex-pull-rescope-ops.md:31,83-89
6.9 Credential Bindings (env var NAMES only — values via Hades, never read)
| Env var | Purpose |
|---|---|
FLEX_APPLY_WEBHOOK_SECRET | static shared bearer for BOTH /sign-upload and /apply |
VISKA_DB_URL | service_role/owner Postgres conn (apply SQL + health_events insert) |
VISKA_SUPABASE_URL | Supabase project URL (Storage REST base) |
VISKA_SUPABASE_SERVICE_ROLE_KEY | service_role key — Storage sign / read / move; one key, server-side only |
FLEX_INBOX_BUCKET | optional; default flex-inbox |
apply_service.py:35-44,62,326-330 n8n side: ONE standing credential Viska: Flex Webhook Auth (6zo7MIzIBcUrjswl, httpHeaderAuth) binding FLEX_APPLY_WEBHOOK_SECRET. n8n carries exactly 2 feature creds (IBKR Flex token + this webhook secret) and zero standing upload token. flex-pull-rescope-ops.md:24,54,76,97
FLEX_UPLOADER_JWT — DEAD/DROPPED
The 057 FLEX_UPLOADER_JWT (HS256 self-mint) is structurally dead — it 403s against Viska's ES256 prod; replaced by server-minted signed URLs (task 060, 3545985/d182bcb) apply_service.py:6-9,46,182-185. The flex_inbox_uploader_insert RLS policy exists in DDL but is vestigial / unused-but-harmless — signed URLs bypass it and service_role reads bypass RLS. Never describe it as the live upload auth. 20260621-01:32-38; apply_service.py:14-15
6.10 Security Posture (rule #19 — mutating + network-reachable ⇒ authenticate)
- Both endpoints require
Authorization: Bearer <secret>, constant-time compared, 401 before any I/O (live-confirmed). Bearer (not body-HMAC) because this n8n blocks$envin nodes and Code nodes can't read credentials, making HMAC-over-body unwireable. apply_service.py:18-23,66-70,133,194 bucketnever caller-controlled;object_pathanchored to a traversal-free.xmlcharset, url-quoted before forming any Storage URL. apply_service.py:25-28,73-82,230,259- Upload PUT auth =
none— the server-minted signed URL self-authenticates; service-role key never leaves apply-svc. flex-pull-rescope-ops.md:63; apply_service.py:182-186 flex-inboxis a private bucket; only the apply-svc service-role key reads it (service_role bypasses RLS); no anon/authenticated SELECT policy. 20260621-01-flex-inbox-bucket.sql:21-24- XML parse refuses any DOCTYPE/ENTITY before parsing (XXE + billion-laughs guard, stdlib-only, no
defusedxml). flex_to_sections.py:16-33 - CF Access fronts the dashboard / artifacts Pages portal only — it is NOT in the IBKR ingest path; apply-svc's static bearer is the sole caller-auth. grep null; on-demand-webhook-design.md:136-137
DECISION 055-A — THE RAILWAY HTTP ENDPOINT IS THE GATED FALLBACK, NOT THE PRIMARY DESIGN
055-A chose to reuse the in-repo importer (A), rejecting a forked write path (B, rule #18). A's DEFAULT delivery = file-drop (raw XML, host-internal, auth:none) + Hades-owned cron. The bearer-auth HTTP apply-svc on Railway is the rule-#19-gated FALLBACK, taken only because the operator refined to a synchronous on-demand viska.gg HTTP pull. Never present the Railway HTTP endpoint as 055's primary choice. 055/ViskaDB.done:21-50
6.11 Confirmed-State & Open Items (honest)
| Item | State |
|---|---|
apply-svc on Railway, /healthz 200, 401-on-unauth /apply | LIVE-CONFIRMED |
Real single-day fire (exec 35607) landed positions:21/fx:5/nav_rows:3, ok:true | CONFIRMED |
Workflow live + active, 22:30 cron (activeVersionId 6aa4c859); round-trip GREEN once | CONFIRMED |
Fix A in code (build_sections_by_day + emit_days present) | CODE-CONFIRMED |
| Railway live image running Fix A | UNKNOWN (no SHA fingerprint) |
| Live 22-day end-to-end landing fired-and-verified in prod | UNKNOWN |
Apply body key — path (n8n node) vs object_path (handler) | OPEN — handler truth = object_path |
| Upload PUT auth shape (header token vs query-string self-auth) | OPEN |
| Sign Upload / Upload / Apply live node ids | UNKNOWN (raw workflow JSON gated) |
§7Supabase Landing — the trading.* Schema
apply-svc re-parses the raw Flex XML and is the sole SSOT writer of trading.*; n8n holds zero DB/storage credential and does zero trading writes (the Upsert Positions node was removed). apply_service.py:182-186; rescope-ops:31,83-89 The schema is created CREATE SCHEMA IF NOT EXISTS trading at 20260610-01-trading_schema_core.sql:9, commented "Viska macro slhf. book of record (IBKR). Append-only import; statement_id lineage on every event row. Bot/API reads go through trading.v_* views." 20260610-01:11-12
11 tables · importer writes 9
trading.* is 11 tables. The importer writes 9 directly; instrument_themes is operator-seeded taxonomy; candles is the n8n nightly EOD job, NOT the IBKR/Flex importer. 20260610-01/02; ibkr_import.py:181-352
ID pattern
All 11 tables use surrogate PK id bigint GENERATED BY DEFAULT AS IDENTITY — EXCEPT instruments, whose PK is the natural key conid. Every event table carries statement_id bigint NOT NULL FK → trading.statements(id) for import lineage. 20260610-01:37-49; 20260610-02
Append-only
Importer uses ON CONFLICT DO NOTHING against the per-table dedup indexes below. Re-import is a no-op. ibkr_import.py:199
Migration provenance
20260610-01-trading_schema_core.sql—statements,instruments,instrument_themes20260610-0120260610-02-trading_book.sql—trades,positions_snapshots,nav_snapshots,cash_flows,dividends,fees_interest,fx_balances,candles20260610-0220260610-06-trades-nullconid-dedup.sql—tradesNULL-conid partial dedup index 20260610-06:14-1620260610-07-fees-interest-dedup.sql—fees_interestbusiness-key dedup +positions_snapshotsNULL-conid partial dedup 20260610-07:18-2520260619-01-flex-symbol-theme-close.sql— DATA backfill intoinstrument_themes(RIOl/AMRC), no DDL 20260619-0120260621-01-flex-inbox-bucket.sql—storage.buckets/storage.objects(flex-inbox), NOT atrading.*table 20260621-01
CITATION CORRECTIONS
20260619-02-* does not exist (the 20260619 sequence is 01, 03, 04, 06). candles lives in 20260610-02:175-193, NOT in 20260617-02-market-data-tables.sql — confirmed by grep. grep; 20260610-02:175-193
Table matrix
| table | holds | PK | dedup-key (UNIQUE) | citation |
|---|---|---|---|---|
| statements LINEAGE ANCHOR | one row per imported statement file — account, period, base_currency, file_hash re-import key, source∈{csv,flex_api}, meta jsonb |
id (statements_pkey) |
UNIQUE(file_hash) statements_file_hash_key — semantic sha256("flex:"+account+":"+report_date), one row per (account, reportDate); re-pull = no-op |
20260610-01:17-32; flex_import.py:20-31 |
| instruments | IBKR security master — one row per conid; symbol, isin, description, listing_exch, currency, multiplier, asset_category |
conid (instruments_pkey, natural key — no surrogate id) |
PK on conid is the dedup; idx instruments_symbol_idx(symbol) |
20260610-01:37-50 |
| instrument_themes OPERATOR-SEEDED | operator-editable taxonomy (Gulli's IS themes as DATA, not code); symbol→theme/sub_theme, conid backfilled, valid_from/valid_to validity range. Grain: one ACTIVE theme per symbol |
id (instrument_themes_pkey) |
partial UNIQUE INDEX instrument_themes_active_symbol_ux(symbol) WHERE valid_to IS NULL — exactly one active theme/symbol |
20260610-01:58-78 |
| positions_snapshots | per statement × open position; qty, cost_price, cost_basis, close_price, value, unrealized_pl, as_of |
id (positions_snapshots_pkey) |
UNIQUE(statement_id, conid) positions_snapshots_ux + partial positions_snapshots_nullconid_ux(statement_id, symbol) WHERE conid IS NULL |
20260610-02:48-68; 20260610-07:23-25 |
| nav_snapshots | per-statement NAV decomposition; cash, stock, interest_accruals, dividend_accruals, total, twr_pct, as_of |
id (nav_snapshots_pkey) |
UNIQUE(statement_id) nav_snapshots_statement_ux — one NAV row per statement |
20260610-02:75-90 |
| fx_balances | per statement × currency cash/FX position; qty, cost_basis_usd, value_usd, unrealized_pl_usd | id (fx_balances_pkey) |
UNIQUE(statement_id, currency) fx_balances_ux |
20260610-02:156-169 |
| trades FLEX: DEFERRED | one row per execution; conid, symbol, executed_at, side∈{BUY,SELL}, signed qty, t_price, proceeds, commission, basis, realized_pl, mtm_pl, IBKR codes (O/C/P) |
id (trades_pkey) |
trades_dedup_ux(conid, executed_at, qty, t_price, proceeds) (no IBKR exec id) + partial trades_dedup_nullconid_ux(symbol, executed_at, qty, t_price, proceeds) WHERE conid IS NULL |
20260610-02:11-43; 20260610-06:14-16 |
| cash_flows FLEX: DEFERRED | one row per deposit/withdrawal; settle_date, currency, description, amount | id (cash_flows_pkey) |
cash_flows_dedup_ux(statement_id, settle_date, currency, amount, md5(COALESCE(description,''))) |
20260610-02:95-109 |
| dividends FLEX: DEFERRED | one row per dividend/withholding line; pay_date, symbol (parsed from description), gross, withholding, currency | id (dividends_pkey) |
dividends_dedup_ux(statement_id, pay_date, COALESCE(symbol,''), COALESCE(gross,0), COALESCE(withholding,0)) |
20260610-02:114-130 |
| fees_interest FLEX: DEFERRED | fees + broker interest rows; posted_date, type∈{fee,interest,broker_interest,other}, currency, description, amount |
id (fees_interest_pkey) |
fees_interest_dedup_ux(statement_id, posted_date, type, currency, amount, md5(COALESCE(description,''))) — added in -07 (PK-only originally → re-import double-counted interest) |
20260610-02:135-149; 20260610-07:18-20 |
| candles N8N NIGHTLY | EOD OHLC market data (n8n nightly job, NOT the IBKR importer); conid, symbol, d date, o/h/l/c, volume, source default 'alpaca' |
id (candles_pkey) |
UNIQUE(symbol, d, source) candles_ux |
20260610-02:175-191 |
FLEX PHASE-1 LANDS 4 STREAMS, NOT 9
The Flex path lands only 4 streams — instruments, positions_snapshots, nav_snapshots, fx_balances — at the statement's report date. trades, dividends, cash_flows, fees_interest are CSV-only / deferred on the Flex path. flex_to_sections.py:10-12 source='flex_api' is admitted by the statements.source CHECK. 20260610-01:30-31 The single-day Apply POST (exec 35607, reportDate:20260619) returned positions:21, fx:5, nav_rows:3. chronicle-2026-06-22:502-503
Importer write-set
The IBKR/Flex importer (importer/ibkr_import.py) writes 9 tables directly: statements (lineage anchor, written first), instruments ibkr_import.py:181, nav_snapshots :273, and via the insert_rows(table,...) helper (all ON CONFLICT DO NOTHING) :199: trades :229, positions_snapshots :251, cash_flows :295, dividends :321, fees_interest :337, fx_balances :350.
instrument_themes— NOT importer-written; operator-seeded (20260610-05seed +20260619-01backfill); importer backfillsconidonly. 20260610-01:54-57candles— NOT importer-written; n8n nightly EOD job. 20260610-02:172
flex-inbox Storage bucket (the raw-XML drop)
Not a trading.* table — the durable raw-XML drop apply-svc reads. Source: migrations/20260621-01-flex-inbox-bucket.sql (Task 057a).
| object | holds | key / dedup | citation |
|---|---|---|---|
storage.buckets row flex-inbox |
PRIVATE bucket (public=false), the durable raw-XML corpus for Flex statements; service_role reads bypass RLS |
PK id='flex-inbox'; ON CONFLICT (id) DO UPDATE SET public=false (enforces private even if pre-existing) |
20260621-01:24-26 |
storage.objects RLS |
RLS asserted ON (ALTER TABLE storage.objects ENABLE ROW LEVEL SECURITY) |
— | 20260621-01:29 |
FLEX_UPLOADER_JWT / flex_inbox_uploader_insert — DEAD & VESTIGIAL
The migration defines INSERT policy flex_inbox_uploader_insert 20260621-01:32-38: FOR INSERT TO authenticated WITH CHECK (bucket_id = 'flex-inbox' AND (auth.jwt() ->> 'app_role') = 'flex_uploader'), with env binding FLEX_UPLOADER_JWT 20260621-01:56. This HS256 self-mint leg is structurally DEAD — it 403s against Viska's ES256 prod and was replaced by server-minted signed URLs (task 060, 3545985/d182bcb). apply_service.py:6-9,46 The RLS policy exists in DDL but is unused-but-harmless: signed URLs self-authenticate the PUT and service_role reads bypass RLS entirely. Do NOT describe FLEX_UPLOADER_JWT as the live upload auth — the live caller-auth is the static bearer FLEX_APPLY_WEBHOOK_SECRET on /sign-upload and /apply. apply_service.py:66-70,326
Side effect (same migration, not a trading.* table): the ops.health_events source CHECK is extended to admit 'flex_apply' so a failed import lands a durable status-page-visible event — CHECK (source IN ('layer1','watchdog','flex_apply')). 20260621-01:48-50
RLS / read posture
All trading.* tables are anon-REVOKED (20260610-04-trading_rls.sql); the trading schema carries no anon surface. 20260619-01:17-20 Bot/API reads route through trading.v_* views (20260610-03-trading_views.sql) under the trading_bot_ro role — consumed by the Mímir bot. 20260610-08:42-47 The frontend (viska.gg / ViskaFront) reads public.* projections (positions_current, nav_latest, cash_current, holdings_profiles, allocation_history, market_quotes, fx_rates), never trading.v_* directly — the data-streams bible is STALE on this point. useDashboardData.ts:76,108,152; 05-data-streams.md:9-13 All public.* are currently authenticated-only (anon revoked 06-19). 20260619-03/04/06
APPLY-STATE CAVEAT
All trading.* / public.* migrations are FORWARD (authored, not prod-applied) except 20260619-03 (PR #20, Hades rc=0). The prod-applied state of the 06-17/06-19 projection + anon-revoke chain is UNKNOWN from the repo. The RLS migration bodies (-04) were snippet-tier, not deep-read; this matrix is deep-read from the DDL. migration headers
§8Consumption — Views & Read Contract
The read surface is two tiers with a deliberate trust boundary: the internal book-of-record layer trading.v_* (consumed by the Mímir Slack bot via a dedicated trading_bot_ro role) and the portal-projection layer public.* (consumed by the viska.gg dashboard). The frontend never reads trading.v_* directly — it reads public.* projections, and the data-streams bible that still names trading.v_* as FE targets is stale.
THE STALE BIBLE — FE READS public.*, NOT trading.v_*
The FE data-streams bible ViskaFront/docs/bible/05-data-streams.md:9-13 lists trading.v_* as the FE read targets. That doc is STALE (dated 06-11, predates the 053 public.* projection layer of 06-17). The implemented contract is public.* — confirmed in .from("positions_current"), .from("nav_latest"), .from("cash_current") ViskaFront/src/hooks/useDashboardData.ts:76,108,152 and .from("market_quotes"), .from("fx_rates") useMarketData.ts:35,68. Any claim that the dashboard SELECTs trading.v_* is wrong.
The two consumers
Mímir Slack bot → trading.v_*
Reads the 4 book-of-record views + the round_trips matview via a separate Hades-deposited trading_bot_ro role 20260610-08:42-47. Contract: "the bot SELECTs views, never raw tables, never embeds book-of-record numbers into the vector corpus" 20260610-03:1-4. The role is not anon, not service_role 20260610-04:7-10.
viska.gg dashboard (ViskaFront) → public.*
Reads the 7-stream public.* contract (task 053) through one Supabase client (VITE_SUPABASE_URL / VITE_SUPABASE_ANON_KEY) carrying an authenticated session JWT behind a magic-link RequireAuth wall src/lib/supabase.ts:4-7 src/App.tsx:2,28. Missing env/empty → static fallback snapshot + DataSourceBanner src/lib/fallback-data.ts; 05-data-streams.md:36-37.
CRITICAL AUTH-POSTURE CAVEAT — THE ANON ERA WAS REVERSED
The public.* views were created anon-readable on 2026-06-17 (053; GRANT SELECT … TO anon, authenticated) but anon was REVOKED across the entire fund + market + greining surface on 2026-06-19 under the Heph "no anon-readable Viska tables" directive. The current posture of every public.* fund/market view is authenticated-only — the "anon" rows in the table below reflect this end state; the 06-17 anon grant is historical.
- Fund views → authenticated 20260619-03-revoke-anon-fund-views.sql:26-35
- allocation_history + base → authenticated 20260619-03:39-51
- market_quotes / fx_rates (+ raw bases) → authenticated 20260619-06-greining-market-anon-to-authenticated.sql:52-77
- 74-object internal anon sweep 20260619-04-revoke-anon-scoped.sql:29-80
Read-contract — every view, exposure, consumer, posture
| View | Exposes | Consumer | Auth-posture (current) | Citation |
|---|---|---|---|---|
trading.v_positions_current |
latest-statement positions: conid, symbol, currency, qty, cost_price/basis, close_price, value, unrealized_pl, theme, sub_theme, description, listing_exch | Mímir bot (trading_bot_ro); base for public.positions_current |
authenticated + trading_bot_ro; anon REVOKED. security_invoker=true |
20260610-03-trading_views.sql:72-98; 20260610-04:57-61; 20260610-08:42-47 |
trading.v_nav_latest |
most-recent NAV snapshot: statement_id, as_of, cash, stock, interest/dividend_accruals, total, twr_pct, account, period_end, file_hash (LIMIT 1) | Mímir bot; base for public.nav_latest |
authenticated + trading_bot_ro; anon REVOKED. security_invoker=true |
20260610-03:144-161; 20260610-08:42-47 |
trading.v_pnl_monthly |
per month×currency: realized_pl, buy_volume, sell_volume, trade_count | Mímir bot (FE greining "stale" target) | authenticated + trading_bot_ro; anon REVOKED |
20260610-03:103-113; 20260610-08:42-47 |
trading.v_round_trips |
round-trip lots: conid, symbol, trip_seq, entry/exit_at, hold_days, qty_opened/closed, entry/exit_price, realized_pl, commission, return_pct, is_closed, theme, sub_theme | Mímir bot | authenticated + trading_bot_ro; anon REVOKED. Reads trading.round_trips matview |
20260610-03:118-139; 20260610-08:38-39 |
trading.round_trips (matview) |
derived closed/open lot sequences per conid (REFRESH after each import) | backs v_round_trips only |
authenticated, service_role, trading_bot_ro; anon REVOKED. Matviews carry no RLS — grant-only |
20260610-03:14-66; 20260610-04:54-55; 20260610-08:38-39 |
public.positions_current |
ticker, name, shares, cost_basis, market_value, theme, sub_group, march_mtd (NULL flag), ytd_mtm (unrealized_pl proxy). Order: market_value desc | viska.gg useDashboardData.ts:76 | authenticated-only (anon revoked 06-19). SECURITY DEFINER — crosses trading anon-revoke boundary | 20260617-01-public-fund-anon-views.sql:27-43; 20260619-03:26,32 |
public.nav_latest |
nav_ibkr, nav_total (==nav_ibkr flag), twr_pct, march_mtd (NULL flag), as_of | viska.gg useDashboardData.ts:108 | authenticated-only. SECURITY DEFINER over trading.v_nav_latest |
20260617-01:47-58; 20260619-03:27,33 |
public.cash_current |
currency, local_amount, usd_amount (IBKR trading.fx_balances latest stmt; off-IBKR ISK bankabók NOT included — flag) |
viska.gg useDashboardData.ts:152 | authenticated-only. SECURITY DEFINER | 20260617-01:65-80; 20260619-03:28,34 |
public.holdings_profiles |
ticker, description, moat (curated copy; EMPTY until operator/research authors — flag, not fabricated) | viska.gg useDashboardData.ts:186 (fallback) | authenticated-only. SECURITY DEFINER over public.viska_tickers (registry itself anon-revoked 20260609-04) |
20260617-03-profiles-allocation.sql:18-25; 20260619-03:29,35 |
public.allocation_history |
week, gull, kopar, uran, platina, al, orkuinnvidir, orkuframleidsla, ai, annad, total (computed). Order: week asc | viska.gg useDashboardData.ts:198 | authenticated-only. security_invoker=true → caller RLS on base allocation_snapshots gates rows (anon dropped from policy 06-19) |
20260617-03:56-66; 20260619-03:39-51 |
public.allocation_snapshots (base table) |
weekly theme-allocation buckets (numeric); written by producers/allocation_snapshot.py via service_role |
producer write target; read via allocation_history |
authenticated-only (anon grant + anon RLS policy both removed); service_role write |
20260617-03:30-52; 20260619-03:43-51; 20260619-04:86-93 |
public.market_quotes |
source, ticker, price, currency, captured_at (DISTINCT ON (source,ticker) latest). FE: captured_at desc, limit 50 | viska.gg useMarketData.ts:35,98 | authenticated-only (was anon 06-17, revoked 06-19). security_invoker=true over market_quotes_raw |
20260617-02-market-data-tables.sql:32-41; 20260619-06:71-75 |
public.fx_rates |
base, quote, rate, source, captured_at (DISTINCT ON (base,quote) latest). FE: captured_at desc, limit 20 | viska.gg useMarketData.ts:68 | authenticated-only (revoked 06-19). security_invoker=true over fx_rates_raw |
20260617-02:64-73; 20260619-06:71-75 |
public.market_quotes_raw / public.fx_rates_raw (base tables) |
ingest targets (ViskaN8N market-data lane task 054); producers/alpaca_quotes.py / fx_rates_flex.py write via service_role |
producer write; read via the two views above | authenticated-only (anon RLS policy swapped → authenticated_select, anon grant revoked 06-19); service_role write preserved |
20260617-02:11-29,44-61; 20260619-06:59-68 |
Consumer contracts — who and how
Mímir bot least-privilege
SELECT-only on the 4 v_* views + the round_trips matview, plus the 6 view-touched base tables (trades, positions_snapshots, nav_snapshots, statements, instruments, instrument_themes) with a role-targeted RLS policy trading_bot_ro_read — required because security_invoker=true views evaluate base-table privilege against the invoking role 20260610-08:1-16,52-68. candles, cash_flows, dividends, fees_interest, fx_balances are deliberately NOT granted → bot structurally cannot read them ("never raw tables" enforced by least-privilege).
Producers (write side)
Recurring writers feed the consumption layer: alpaca_quotes.py→market_quotes_raw, fx_rates_flex.py→fx_rates_raw, allocation_snapshot.py→allocation_snapshots, all via service_role. ViskaDB emits SQL only (--emit-sql), never runs --write with real creds (rule #04) producers/README.md:14-19,54-59. allocation_snapshot.BUCKET_MAP mirrors trading.instrument_themes as taxonomy SSOT producers/README.md:34-38.
CF ACCESS IS NOT IN THE READ PATH
Cloudflare Access fronts the dashboard / artifacts Pages portal only — it is the gate the magic-link session sits behind for the browser. It is not in the IBKR ingest path. FE read-auth is the authenticated Supabase session JWT + RLS; ingest-write-auth is the apply-svc static bearer (§ write-path). Do not place CF in the data read or ingest chain on-demand-webhook-design.md:136-137.
APPLY-STATE CAVEAT — FORWARD vs PROD
Every migration header states "FORWARD migration (NOT applied to prod here — Hades-summon apply)". Only the WS-1 trading layer (20260610-*) is confirmed reached prod via Hades (20260610-08 fixes a Hades prod-apply finding, so -04 was applied). The prod-applied state of the 20260617-* / 20260619-* public.* projection + anon-revoke chain is UNKNOWN from the repo (the 06-19 sessions cite tasks/hades-apply-revoke-anon/Hades.done, not read). The lone confirmed-applied projection-era migration is 20260619-03 (PR #20, Hades rc=0). Do not claim the authenticated-only posture is live in prod beyond 20260619-03.
Data-quality flags carried in the views
public.positions_current.march_mtd= NULL — MTD P&L not in flex snapshot, needs period calc 20260617-01:37public.nav_latest.nav_total == nav_ibkruntil off-IBKR ISK bankabók cash source exists;march_mtdNULL 20260617-01:51,53public.cash_currentomits the off-IBKR ISK bankabók row — no source table yet 20260617-01:64,80trading.v_nav_latest.twr_pctmay be NULL → FE shows "n/a" 05-data-streams.md:9public.holdings_profiles.description/moatEMPTY until operator/research authors — flag, not fabricated 20260617-03:25
READ-CONTRACT IN ONE LINE
FE → public.* (7 projections, authenticated-session only, anon revoked 06-19). Mímir bot → trading.v_* via trading_bot_ro. Nobody reads trading.v_* from the browser; nobody serves public.* to anon anymore. The bible naming trading.v_* as FE targets is the one stale artifact to ignore.
§9Security & Privacy
How the IBKR/Flex trading pipeline is kept safe — no unauthenticated mutation (rule #19), least-privilege roles, credential isolation — and private — RLS lockdown, the anon-revoke arc, a private storage bucket. All DDL below is the authored intended posture: every migration header except 20260619-03 (PR #20, Hades-applied rc=0) self-labels FORWARD — not applied to prod. See the apply-state caveat at the end.
SINGLE GATE, TWO MUTATING ENDPOINTS
apply-svc (Railway flex-apply-svc-production.up.railway.app) exposes two public, mutating endpoints — /apply and /sign-upload — each guarded by one static shared bearer FLEX_APPLY_WEBHOOK_SECRET, constant-time compared, checked before any I/O. The 401-on-unauth /apply live probe is the one confirmation the gate is active in prod. apply_service.py:66-70,133-134,194-195
9.1 Write-path authentication (rule #19)
The rule-#19 predicate UNSAFE = MUTATING ∧ PUBLIC ∧ UNAUTHENTICATED is not satisfied: the endpoints are mutating and public, but the bearer removes the unauthenticated leg. apply_service.py:66-70
- Constant-time compare, both endpoints:
_valid_bearerreturnshmac.compare_digest("Bearer " + secret, auth_header)— rejects an empty header and a bare token without theBearerprefix. apply_service.py:66-70 - Bearer, not body-HMAC (commit
7466abf): this n8n instance blocks$envin nodes and Code nodes can't read credentials, so HMAC-over-body is unwireable.X-Viska-Signature→Authorization: Bearer; n8n binds it as anhttpHeaderAuthcredential so the value never enters node/LLM context. Native header-auth from vault — not path obscurity. apply_service.py:17-23 - Single shared bearer for both
/sign-uploadand/apply. apply_service.py:4,36,326 - Upload PUT auth =
authentication: none— the server-minted signed upload URL self-authenticates; n8n carries no upload token. The HS256FLEX_UPLOADER_JWTleg is dead/dropped (see 9.3). apply_service.py:6-9,46 - Replay = no-op given
(account, reportDate)semantic idempotency (see 9.6). apply_service.py:23
OPEN — APPLY POST BODY KEY (path vs object_path)
Handler truth: /apply reads payload.get("object_path") and 400s if absent — contract is {bucket, object_path}. apply_service.py:146 The n8n node spec sends {bucket, path}. rescope-ops:78 This mismatch is flagged OPEN in the spec itself rescope-ops:80 — UNKNOWN whether reconciled live. The PUT-leg token shape (header vs query-string self-auth) is likewise UNKNOWN. rescope-ops:67-70 Never assert the path form works end-to-end.
9.2 Task-060 hardening — bucket scope-lock & path-traversal
Folded into commit d182bcb (2 MEDIUM security-review findings); hardens both /apply and /sign-upload since they share the file and bearer. apply_service.py:75-91
| Control | Mechanism | Effect |
|---|---|---|
| Path-traversal anchor | _safe_object_path: str AND ".." not in path AND not path.startswith("/") AND _OBJECT_PATH_RE.fullmatch(path) where _OBJECT_PATH_RE = [A-Za-z0-9_\-/]{1,200}\.xml apply_service.py:75-82 | fullmatch rejects ? # space and any . except the .xml suffix → kills query-param injection, absolute paths, ../ before the Storage URL forms |
| Bucket scope-lock → 400 | _resolve_bucket: bucket is never caller-controlled; a present non-default bucket returns (None, "bucket override not allowed") apply_service.py:85-91 | A shared-bearer holder cannot reach other project buckets |
| Defense-in-depth url-quote | urllib.parse.quote(path, safe="/") on fetch/move/sign apply_service.py:230,259 | Redundant escaping even though path is already anchored |
Tested: test_bucket_override_rejected_400 (asserts applied_sql is None — foreign bucket never fetched) test_apply_service.py:192-196, test_path_traversal_rejected_400 test_apply_service.py:199-205, test_unsafe_paths_rejected over ("../x.xml","/x.xml","x.txt","a..b/x.xml","x.xml?q=1","a b.xml","x.xml#f","",None,123) test_apply_service.py:215-217; auth: test_missing_bearer_401 / test_bad_bearer_401 / test_raw_token_without_bearer_prefix_401 test_apply_service.py:71-87. 75 importer tests green after the multiday fix f810998.
9.3 Private storage — flex-inbox bucket
The raw-XML drop is a private Supabase Storage bucket. 20260621-01:24-26
- Private by construction:
INSERT INTO storage.buckets … ('flex-inbox','flex-inbox', false) ON CONFLICT DO UPDATE SET public = false— forces private even if it pre-existed;storage.objectsRLS asserted. 20260621-01:24-29 - No SELECT/UPDATE/DELETE policy for anon or authenticated → apply-svc reads/moves with service_role, which BYPASSES RLS; reads happen server-side only. 20260621-01:18-19,40-42
VESTIGIAL — flex_inbox_uploader_insert RLS POLICY
The migration defines flex_inbox_uploader_insert — FOR INSERT TO authenticated, WITH CHECK (bucket_id='flex-inbox' AND auth.jwt()->>'app_role'='flex_uploader'). 20260621-01:32-38 This policy exists in DDL but is unused-but-harmless: task-060 replaced the upload leg with server-minted signed URLs (commits 3545985/d182bcb) after the 057 HS256 FLEX_UPLOADER_JWT leg 403'd — Viska prod signs JWTs with ES256, so an HS256 self-mint is rejected. apply_service.py:6-9,14-15 Net: n8n holds zero standing upload token; signed URLs bypass this policy and service_role reads bypass RLS. Describe FLEX_UPLOADER_JWT as dropped/dead, never as live upload auth.
9.4 Credential isolation — who holds what
Bindings only (env-var names; values Hades-provisioned). Consistent with rule #04 — credential VALUES are the one thing Viska escalates out, to Hades. apply_service.py:35-46
| Component | Holds | Does NOT hold |
|---|---|---|
| n8n (Hostinger, ViskaN8N) | IBKR Flex token (httpQueryAuth) + webhook bearer (httpHeaderAuth) — value never in node/LLM ctx rescope-ops:24,54,76 | NO service-role key, NO DB URL, NO standing storage/upload token (060 retired FLEX_UPLOADER_JWT) apply_service.py:46 |
| apply-svc (Railway) | FLEX_APPLY_WEBHOOK_SECRET, VISKA_DB_URL (service_role PG conn), VISKA_SUPABASE_URL, VISKA_SUPABASE_SERVICE_ROLE_KEY, FLEX_INBOX_BUCKET (opt) apply_service.py:35-44 | — |
service-role key never leaves apply-svc: the injected sign_upload closure closes over the key; the pure handle_sign_upload core never receives it, so it cannot appear in any response (test-asserted). apply_service.py:182-186,340
9.5 RLS posture — trading.* and mirror schemas
The pattern across all four data schemas: service_role full · authenticated SELECT-only · anon NOTHING. 20260610-04:13-14
trading.* (11 tables)
Schema USAGE granted to authenticated, service_role; REVOKE ALL … FROM anon — anon can't traverse the schema. RLS on statements, instruments, instrument_themes, trades, positions_snapshots, nav_snapshots, cash_flows, dividends, fees_interest, fx_balances, candles. Policies: service_role_full_access (FOR ALL) · authenticated_read_only (FOR SELECT, no INSERT/UPDATE/DELETE). Future objects inherit authenticated-SELECT / service_role-ALL. 20260610-04:13-26,29-43,64-65
ops.* (3 tables)
health_events, heartbeats, component_status; anon revoked. Dedicated ops_daemon_rw role — daemons connect via pooler with no JWT claims, so JWT-claim policies can't fire; explicit role-targeted policy gives INSERT/UPDATE/SELECT (no DELETE). 20260611-02:1-7,18-19
signals.* (Council)
universe, market_candles, market_fundamentals, market_calendar, signal_runs, backtest_runs, takes; anon revoked. Runtime viska_council_rt (views + INSERT on takes only) and a nightly engine role are separate Hades-deposited roles. Conviction-score RLS is the same family. 20260615-02:9-14,16-18
Curated views + matview
round_trips matview + v_positions_current / v_pnl_monthly / v_round_trips / v_nav_latest → SELECT to authenticated+service_role, REVOKE from anon. 20260610-04:54-61
9.6 Least-privilege read role — trading_bot_ro (Mímir)
The 4 trading.v_* views are security_invoker = true, so granting the Mímir bot SELECT on views only let it read nothing (invoker semantics evaluate base-table privilege against the calling role). 20260610-08:1-8
- Resolution = OPTION (a): keep the heph-reviewed invoker posture; grant base-table SELECT on exactly the 6 view-touched tables (
trades, positions_snapshots, nav_snapshots, statements, instruments, instrument_themes) +round_tripsmatview + the 4 views, plus a role-targeted RLS read policytrading_bot_ro_read. 20260610-08:8-16,29-47,52-68 - Deliberately NOT granted:
candles, cash_flows, dividends, fees_interest, fx_balances— no curated view exposes them, so the bot cannot read them. "Never raw tables" is enforced by convention (the role's contract is thev_*views). 20260610-08:14-16 - Role is Hades-provisioned vault-side, not created in the migration. 20260610-08:18-22,42-47
9.7 Anon-surface posture — the anon-revoke arc
Trigger: Heph directive 2026-06-19 "no anon-readable Viska tables" + a live rule-#04 breach (public.ssot_credentials was anon-readable). Hades enumerated 84 of 131 public.* objects anon=t at the start. 20260619-04-revoke-anon-scoped.AUDIT.md:1-6
| Migration | What it did | Apply state |
|---|---|---|
20260617-01-public-fund-anon-views | The original anon projection later reversed — SECURITY DEFINER views positions_current / nav_latest / cash_current over trading.v_*, anon SELECT (anon never touches trading.* base tables, only curated columns) 20260617-01:14-21,83-85 | FORWARD |
20260619-03-revoke-anon-fund-views | Fund feed → authenticated: REVOKE anon on positions_current, nav_latest, cash_current, holdings_profiles; same for allocation_history invoker-view + base allocation_snapshots (drops anon from base-table RLS policy) 20260619-03:26-29,32-35,39-51 | APPLIED PR #20, rc=0 |
20260619-04-revoke-anon-scoped | 74 internal/sensitive objects REVOKE'd, ssot_credentials first (rule-#04 breach, TIER 1). TIER-1 also: client_book (PII), lake_*, mimir_*, viska_agent_memory*. TIER-2: ssot infra, embeddings/RAG/docs, reports, market raw. 20260619-04:33,111 | FORWARD |
20260619-06-greining-market-anon-to-authenticated | 12 KEEP FE surfaces: 8 → authenticated (theme_intelligence, broker_theme_calls, off_radar, theme_corroboration, market_quotes(_raw), fx_rates(_raw)), 4 → revoked internal (theme_score_audit, blind_spot_clusters, gics_sectors, sector_sentiment_scores). Writer policies (n8n_report_rw, service_role_write) untouched. 20260619-06:21-77,79-100 | FORWARD |
Why grant-revoke is sufficient: a GRANT SELECT TO anon is required for anon to read even under a permissive anon RLS policy, so revoking the grant blocks anon in all cases — the GRANT-vs-RLS-effective split is an exposure ledger, not a correctness gate. 20260619-04:8-12 Legacy thread 20260609-04-revoke-legacy-anon-grants revokes leftover full-DML anon/authenticated grants on public.viska_chat_users (defense-in-depth; RLS already denied them). 20260609-04:8-14
END-STATE ANON POSTURE
Trading / fund / greining / market data is auth-gated (authenticated session reads, server-authoritative); credentials / PII / agent-memory / internal substrate is fully revoked from anon. Pipelines (n8n / mimir / research) write via service_role, so the anon revoke affects no writer. 20260619-04.AUDIT.md:94-98
9.8 Read contract & auth-domain whitelist
- FE reads
public.*projections (positions_current, nav_latest, cash_current, holdings_profiles, allocation_history, market_quotes, fx_rates) — nevertrading.v_*directly (the data-streams bible is stale). Allpublic.*are currently authenticated-only (anon revoked 06-19). useDashboardData.ts:76,108,152 - Mímir bot reads
trading.v_*viatrading_bot_ro. 20260610-08:42-47 - Email-domain whitelist gates who can hold an authenticated session — no native GoTrue "allowed domain" knob, so a DB-layer trigger on
auth.users: predicateis_viska_allowed_email=@viskasjodir.isORadmin@boas.dev;enforce_viska_email_whitelistBEFORE INSERT AND BEFORE UPDATE OF email (blocks out-of-domain creation AND re-pointing);security definer,search_path=''. All viska.gg auth = Supabase magic-link, NO Cloudflare Access. 20260615-05:5-11,25-36,46-47,61-69
9.9 CF Access — NOT in the ingest path
SINGLE GATE ON THE INGEST PATH IS THE BEARER
CF Access fronts the dashboard / artifacts Cloudflare Pages portal only — a separate surface. Grep of viska-artifacts/ + ViskaFront/ for flex-apply-svc / FLEX_APPLY_WEBHOOK_SECRET returns nothing: the dashboard never calls apply-svc directly. The CF-Access references that exist (viska-artifacts/_assets/auth.mjs, functions/api/_supabase-auth.mjs) belong to the portal, not the trading path. apply-svc's only caller-auth is the static bearer — CF Access / Railway private networking would be a stronger shield but are not present. on-demand-webhook-design.md:136-137 The defense-in-depth note "deploy with Railway private networking / only-from-n8n if available" is design intent, not confirmed live. apply_service.py:24
The planned on-demand webhook (2nd trigger on the same workflow, onReceived 202-async, n8n-native Header Auth chosen over a CF edge shield, 5-min cooldown) is DESIGN-APPROVED, NOT BUILT — blocked on N8N_API_URL/N8N_API_KEY + an auth-secret mint; the headerAuth trigger node exists, the cooldown gate does not. The auth secret lives in the dashboard backend, never in browser JS. on-demand-webhook-design.md:6,47-49,109-117
9.10 Decision 055-A — gated fallback, not primary
Decision 055-A chose reuse the in-repo importer (A), rejecting a forked write path (B, rule #18 vaporware). A's DEFAULT delivery = file-drop (raw XML, host-internal, auth:none) + a Hades-owned cron (credential-free stdlib emitter, owner-conn apply). The bearer-auth HTTP apply-svc on Railway is the rule-#19-gated FALLBACK, taken only because the operator refined to a synchronous on-demand viska.gg HTTP pull. Never present the Railway HTTP endpoint as 055's primary design. 055/ViskaDB.done:21-50
9.11 Idempotency — semantic dedup as a safety property
Idempotency is a security control (replay-safe writes), not only a correctness one: file_hash = sha256("flex:"+account+":"+report_date) → trading.statements UNIQUE(file_hash) → one row per (account, reportDate); a re-pull is a no-op. This is a semantic hash, NOT a raw-bytes hash (the raw-bytes form was the shipped bug, fixed b1664ff). apply-svc re-parses the raw XML and is the sole SSOT writer of trading.*; n8n holds zero DB/storage credential and does zero trading writes (the Upsert Positions node was removed). flex_import.py:20-31; 20260610-01:23,29
APPLY-STATE CAVEAT (LOAD-BEARING)
Every migration header except 20260619-03 (PR #20, Hades rc=0) self-labels FORWARD — not applied to prod — Hades-summon apply: the RLS/grant DDL above is the authored intended posture, applied to prod only on a Hades summon. The prod-applied state of the 06-17/06-19 projection + anon-revoke chain is UNKNOWN from the repo. apply-svc deploy LIVE-confirm was BLOCKED at last record (chronicle 2026-06-22 23:40): RAILWAY_TOKEN unset, no railway.toml in ViskaDB — service is up (/healthz 200, 401 on unauth /apply) but the running image SHA is unfingerprinted. The 401-on-unauth observation is the one live confirmation the bearer gate is active. Do not claim the multiday fix runs in prod, and do not claim any public.* table is "anon-revoked in prod" beyond 20260619-03. on-demand-webhook-design.md / chronicle:2026-06-22 23:40
§10The Real Decision Arcs (not blind)
Eight load-bearing decisions in the IBKR-Flex → trading.* pipeline, each with what was chosen, what was rejected, the driving incident, and a file/SHA citation. The first one — 055-A — carries the named trap: writers who skim it report the Railway HTTP apply-svc as the design. It is not. It is the rule-#19 FALLBACK; the default was a host-internal file-drop + Hades-owned cron.
| # | Decision | Chose | Rejected | Why (incident / evidence) | Citation |
|---|---|---|---|---|---|
| 1 | Flex→trading.* write architecture |
A — reuse the in-repo importer (flex_to_sections + ibkr_import.emit) |
B — mint viska_trading_writer + fork a 2nd n8n write path |
Write path already built/proven (Board #8 #2 Done); B = redundant cred + divergent write path + new mutating surface for zero new capability (rule #18). | 055/ViskaDB.done:12-19 |
| 1b | A's delivery model — the nuance | DEFAULT = file-drop + Hades-owned cron (credential-free stdlib emitter, owner-conn apply) | A public HTTP POST→trading.* endpoint as the default |
A public mutating endpoint = MUTATING∧PUBLIC∧UNAUTH (rule #19); whatever hosts it holds service_role anyway. The bearer-auth HTTP apply-svc is the gated FALLBACK, taken only for the synchronous on-demand viska.gg pull. |
055/ViskaDB.done:21-50 |
| 2 | apply-svc caller auth | Static Bearer (Authorization: Bearer, constant-time eq) |
HMAC over body (X-Viska-Signature) |
n8n blocks $env in nodes and Code nodes can't read creds → HMAC-over-body dead on arrival. Bearer binds via native httpHeaderAuth; value never in node/LLM ctx. Still rule-#19-ok (vault header-auth, not obscurity). |
ViskaDB chronicle.md · 7466abf |
| 3 | Raw-XML upload auth to private bucket | Server-minted signed upload URL (/sign-upload, service-role mints short-lived PUT) |
HS256 FLEX_UPLOADER_JWT scoped app_role=flex_uploader |
Viska Supabase signs ES256 in prod → the HS256 uploader JWT 403s live. Service-role key never leaves apply-svc (pure core never receives it; test-asserted). | ViskaDB chronicle.md · 3545985 / d182bcb |
| 4 | trading.statements dedup key |
Semantic (account, reportDate) → file_hash = sha256("flex:"+account+":"+reportDate) |
Raw-XML-bytes hash (the shipped bug); whenGenerated; (account,fromDate,toDate) |
IBKR stamps whenGenerated/ordering fresh each pull → same reportDate never byte-collides → every re-pull mints new file_hash → duplicate snapshot. Reuses UNIQUE(file_hash), zero migration. |
055/ViskaDB.done:57-84 · b1664ff |
| 5 | Multi-day landing (22 days collapsed to 1) | Fix A — apply-svc emit_days (drop the latest-rd filter) + Fix B — multi-day Normalize |
Keeping the by-design latest-reportDate filter; a (report_date,conid) migration; chasing a phantom stmts[0] bug |
Query 1384817 = Last30CalendarDays → 22 daily <FlexStatement>; apply-svc deliberately dropped non-latest days (flex_to_sections.py:56,81,103,115). Schema already per-day idempotent → ZERO DDL. |
flex-multiday…contract.md:40-72 · f810998 |
| 6 | Empty-statement guard | IF "Statement Usable?" (position_count>0 AND report_date notEmpty) before Sign Upload; False branch skips Sign/Upload/Apply |
No guard (status quo) | IBKR Flex toDate floats: exec 35606 returned reportDate 2026-06-20, ZERO positions. Unguarded, the 22:30 cron could push empty XML to /apply and overwrite a good snapshot. |
ViskaN8N chronicle.md · f8a0304 |
| 7 | On-demand pull trigger | 2nd Webhook trigger on the same workflow; 202 async; native Header Auth; 5-min static-data cooldown | Clone workflow; synchronous wait; secret query param; CF edge shield; Supabase timestamp row | DRY (one guard, one /apply). Flex 30s–3min latency → async 202. Browser-exposed bearer ≡ unauth → secret stays in dashboard backend (rule #19). Cooldown blocks button-mashing. |
on-demand-webhook-design.md:44-58,127-137 |
| 8 | CIO ratification of feed gaps | A–P decision packet (recommended defaults + alternatives per item), awaiting CIO sign-off | Wiring any feed/spend/posture pre-ratification (nothing wired) | Downstream streams authored + verified-viable but parked behind CIO-only calls (feed mechanism, vendor spend, ISK-NAV, public-data posture). A & D T-deep w/ source ledgers; B/C/E/F ground-truthed vs the live 06-18 statement. | cio-ratification-ibkr-feed-gaps-A-H.md:29-70 |
Per-decision detail
1 + 1B — REUSE IMPORTER (A), AND THE FILE-DROP DEFAULT vs HTTP FALLBACK
The 055 verdict reads literally: "A (reuse the importer) — but not as a public endpoint." 055/ViskaDB.done:12 Rule #18 (no-vaporware): flex_import.py already wires flex_to_sections.build_sections(xml) → ibkr_import.emit() into idempotent SQL for the full migration-053 shape — Board #8 #2 Done. 055/ViskaDB.done:13-18 B rejected for three concrete costs: a redundant viska_trading_writer cred, a forked second write path ViskaN8N would maintain against every emit() fix, and a new mutating surface — "three costs for zero capability A doesn't already have." 055/ViskaDB.done:18-19
The load-bearing nuance: A's default delivery was explicitly NOT HTTP. The chosen default is the importer's own model — "a credential-free stdlib SQL emitter; a Hades-owned scheduled job (owner conn) runs the parse→emit→apply. No public endpoint, no n8n trading cred, no second write path." 055/ViskaDB.done:26-28 The contract row specifies transport = "file drop (raw XML) — n8n writes, importer job reads. NOT a public POST" with "auth: none needed: drop is host-internal." 055/ViskaDB.done:40-46
The bearer-auth Railway HTTP apply-svc is the rule-#19-gated FALLBACK, taken only because the operator refined to a synchronous on-demand viska.gg pull needing an HTTP writer. The done file foresaw exactly this: "If viska-pm/ViskaN8N insist on pull-driven HTTP instead of a drop: gate it per rule #19 — CF Tunnel + Access or HMAC header-verify Worker; service_role stays in vault on the job host, never in n8n." 055/ViskaDB.done:48-50 Writers must never present the Railway HTTP endpoint as 055's primary design.
2 — HMAC → STATIC BEARER (the n8n $env wall)
Task 057 shipped HMAC X-Viska-Signature or 401. ViskaN8N's wiring hit a hard wall: HMAC-over-body needs the secret in a Code node, but this n8n instance blocks $env in nodes and Code nodes can't read creds → HMAC dead on arrival. The 057 contract had always allowed "HMAC OR bearer," so the swap was X-Viska-Signature → Authorization: Bearer <secret> with constant-time eq + bare-token reject; n8n binds via native httpHeaderAuth so the value never enters node/LLM context. Env var name FLEX_APPLY_WEBHOOK_SECRET unchanged; the same single bearer authenticates BOTH /sign-upload and /apply, 401-before-I/O (live-confirmed). ViskaDB chronicle.md · 7466abf · apply_service.py:66-70,326
3 — flex_uploader HS256 JWT → SERVER-MINTED SIGNED URL
Task 057 originally granted n8n a narrow upload path via a scoped HS256 JWT (app_role=flex_uploader). It 403'd live: Viska's Supabase signs ES256 in prod, so an HS256-self-minted uploader JWT is rejected. Task 060 replaced it — a bearer-authed /sign-upload route where a valid bearer makes apply-svc's service-role mint a short-lived Supabase signed PUT URL for the private flex-inbox bucket; n8n PUTs raw XML to it under authentication: none (the signed URL self-authenticates). The service-role key never leaves apply-svc. Downstream consequence: the flex_inbox_uploader_insert RLS policy from migration 20260621-01:32-56 still exists in DDL but is vestigial — unused-but-harmless (signed URLs and service-role reads both bypass it); writers must describe FLEX_UPLOADER_JWT as dead/dropped, never as live upload auth. ViskaDB chronicle.md · 3545985 / d182bcb · apply_service.py:6-9,46,182-186
4 — DEDUP KEY: raw-bytes-hash BUG → semantic (account, reportDate)
The shipped bug: flex_import.py:31-32 hashed raw XML bytes — correct for CSV (re-import = identical bytes), wrong for Flex. IBKR stamps whenGenerated/ordering fresh on every pull, so the same reportDate never byte-collides ⇒ every re-pull mints a NEW file_hash ⇒ NEW statements.id ⇒ duplicate snapshot. Fix: file_hash = sha256("flex:"+account+":"+reportDate), reusing trading.statements UNIQUE(file_hash) for zero schema migration. The chain closes because children are per-statement (positions_snapshots_ux(statement_id,conid), nav_snapshots_statement_ux, fx_balances(statement_id,currency)): same-day re-pull → same synthetic hash → ON CONFLICT (file_hash) DO UPDATE … RETURNING id returns the existing id → children no-op. Rejected: whenGenerated (drifts intraday → would dup), raw bytes (the bug itself), (account,fromDate,toDate) (multi-column where one suffices). Shipped b1664ff, 4 FlexDedupKeyTest cases (TDD). 055/ViskaDB.done:57-84 · flex_import.py:20-31 · b1664ff
5 — MULTI-DAY: latest-reportDate filter → Fix A (emit_days) + Fix B (multi-day Normalize)
Symptom: a 30-day pull landed only 1 report_date. The investigation corrected the dispatch's framing. Verify-only ground truth: running the real parser on a fresh 22-statement pull emitted "1 distinct report_date (latest only), 21 dropped — CONFIRMED. Root cause CORRECTS the dispatch: NOT stmts[0] / n8n flattening — it's a by-design latest-reportDate filter (flex_to_sections.py:56,81,103,115)." A Phase-2 implementer chasing stmts[0] in Python finds nothing. flex-multiday-normalize-apply-contract.md:40-72
Fix A — load-bearing (ViskaDB/apply-svc)
Drop the latest-rd filter, emit every day. build_sections_by_day + ibkr_import.emit_days (one BEGIN..COMMIT, per-day _stmt_<i> temp tables, ON CONFLICT throughout). Smoke: "22/22 distinct report_dates land, idempotent." CODE-CONFIRMED 75 tests. ZERO DDL — idempotency already in schema via the statement_id grain. f810998 2026-06-22 23:10 · flex_to_sections.py:145 · ibkr_import.py:359
Fix B — guard/metadata only (ViskaN8N)
Multi-day-aware Normalize so a 22-day file isn't judged by day-0. Logic extracted to scripts/lib/flex-normalize.js (TDD 7/7). Does NOT drive the DB write — it feeds the guard, the Sign Upload filename, and Respond. position_count = sum across all days; report_date = latest day. The live Normalize node body update is DEFERRED. 2f12e5c · flex-normalize.js:46-54
Sequence: "Fix A is the gate. Fix B without Fix A still lands one day… B alone does nothing for the landing." contract.md:153 Live-deploy state: the Railway image running Fix A is UNCONFIRMED — /healthz 200 but no image-SHA fingerprint, RAILWAY_TOKEN unset, no railway.toml in ViskaDB. State as code-confirmed, live-deploy-unconfirmed; never claim "22/22 lands in prod." chronicle 2026-06-22 23:40
6 — EMPTY-STATEMENT GUARD (floating Flex toDate)
While debugging the 30-vs-21 gap, exec 35606 (2 min before 35607, same query 1384817) returned report_date 2026-06-20 with ZERO positions — IBKR Flex toDate floats and can return an empty next-day/weekend snapshot. Unguarded, the 22:30 cron could push empty XML to /apply and overwrite a good trading.* snapshot. Fix: an IF "Statement Usable?" node between Normalize and Sign Upload (position_count>0 AND report_date notEmpty); True → Sign Upload (apply path), False → Flag Unmapped Conids → Respond (skips Sign/Upload/Apply, returns position_count:0, no empty write). Shipped commit f8a0304, published as activeVersionId 6aa4c859. Not test-exec'd (mutating — fires IBKR+apply+DB). ViskaN8N chronicle.md:17-32 · git show f8a0304
7 — ON-DEMAND WEBHOOK (202-async + 5-min cooldown) — DESIGN-APPROVED, NOT BUILT
Driver: a dashboard button to pull latest IBKR data on demand when the client traded intraday. Choices: a 2nd Webhook trigger on the same workflow ADeTjBZiHOMIXzvr (DRY — one guard, one apply path) over a clone workflow; onReceived 202 async over synchronous wait (Flex 30s–3min latency would time out a held conn); native Header Auth over a secret query param ("path obscurity ≠ auth") or a CF edge shield ("no provider signature here") — the secret lives in the dashboard backend, never browser JS, since "a browser-exposed bearer is equivalent to unauthenticated"; n8n static-data 5-min cooldown over a Supabase timestamp row (no new DB grant), blocking double-clicks while allowing a genuine intra-session re-pull. DESIGN-APPROVED, NOT BUILT — blocked on N8N_API_URL/N8N_API_KEY (excluded by direnv whitelist) + apply-svc idempotency answer + auth-secret mint. The exact webhook path string is UNKNOWN (TBD at build). on-demand-webhook-design.md:44-58,109-117,127-137
8 — CIO RATIFICATION A–P GATES (nothing wired pre-sign-off)
A single decision surface parks 8 feed gaps + the public-data posture behind CIO-only calls; nothing wired pre-ratification. Key rows:
- A 🔴 ISK→
nav_total: interim =nav_ibkr_usd + isk_balance/flex_isk_usd_rate(in-statement IBKR Flex ISK→USD rate); medium-term Enable Banking PSD2. Rejects waiting on fully-unattended automation (90–180d SCA re-consent). - D 🔴
market_quotesvendor: primary Twelve Data, fallback Finnhub. Rejects IEX (dead 2024), Polygon (US-only), yfinance (Yahoo ToS — unfit for a client product). - E/F 🟢 no external feed needed (daily Flex FX + derivable allocation).
- P public-data posture — the cross-cutting gate: the merged FE contract exposes fund data via anon Postgres roles, colliding with Hephaistos's "no anon-readable Viska tables." P1 (ratify anon-read) / P2 (auth + revoke anon) / P3 (hybrid); ViskaDB recommends P3, every public stream's live-flip held until ruled.
Confidence tiering is explicit: A & D T-deep with source ledgers; B/C/E/F ground-truthed against the live 06-18 statement, "not inferred." CIO sign-off status: AWAITING as of 06-19. cio-ratification-ibkr-feed-gaps-A-H.md:29-70,106-117
§11Source & Confidence Ledger
This report is a reconciliation + QA pass over nine independently-cited domain findings, not fresh code research. The ledger below is the audit-grade output of that pass: every load-bearing fact carries a status badge and its source. Ten cross-domain contradictions were resolved into the canonical fact sheet; the residual UNKNOWN rows are the honest edge of what the repo can prove.
CONFIRMED
Backed by a primary source — code line, migration header, git object, or a live probe (HTTP status / round-trip). The fact is asserted without caveat.
INFERRED
The artifact is observed but the cause / interpretation is reasoned, not directly cited. State with the inference flagged.
UNKNOWN
No source in-repo resolves it. Flagged OPEN — never asserted in either direction. Most are live-deploy or raw-JSON-gated facts.
Confirmed / Inferred / Unknown ledger
| # | Load-bearing fact | Status | Source |
|---|---|---|---|
| 1 | n8n runs on Hostinger (workflow ADeTjBZiHOMIXzvr, "Viska Macro Daily — IBKR Flex Pull") | CONFIRMED | flex-pull-…json:85 via spec/chronicle |
| 2 | apply-svc on Railway flex-apply-svc-production.up.railway.app, separate from viska-mimir-agent | CONFIRMED | live-probe /healthz 200 |
| 3 | apply-svc /healthz → 200 {"ok":true} | CONFIRMED | live-probe |
| 4 | apply-svc running image SHA = the multiday fix | UNKNOWN | RAILWAY_TOKEN unset; no fingerprint |
| 5 | Fix A (build_sections_by_day+emit_days) committed f810998 2026-06-22 23:10, 75 tests green | CONFIRMED | flex_to_sections.py:145; ibkr_import.py:359 |
| 6 | Fix A live 22-day landing fired-and-verified in prod | UNKNOWN | no prod-apply evidence |
| 7 | Bearer gate active in prod (401 on unauth /apply) | CONFIRMED | live-probe |
| 8 | Flex query id 1384817, v=3, Last30CalendarDays → ~22 daily <FlexStatement> | CONFIRMED | build-spec:24; chronicle:389 |
| 9 | query id hardcoded live ($env blocked on instance) | CONFIRMED | chronicle-2026-06-22:389(2) |
| 10 | SendRequest pinned ndcdyn; GetStatement dynamic {{…Url}} (observed gdcdyn) | CONFIRMED | flex-pull-…json:85; chronicle:389(3) |
| 11 | 2-call async handshake + bounded poll loop ($runIndex < 6) | CONFIRMED | chronicle:389(4) |
| 12 | Idempotency = semantic sha256("flex:"+account+":"+reportDate) → UNIQUE(file_hash) | CONFIRMED | flex_import.py:20-31; 20260610-01:23,29 |
| 13 | apply-svc re-parses raw XML = SSOT for report_date+positions; n8n Normalize is metadata/guard only | CONFIRMED | flex_to_sections.py:6-11; multiday-contract:50-52 |
| 14 | Live single-day Apply POST returned positions:21, fx:5, nav_rows:3, reportDate:20260619 (exec 35607) | CONFIRMED | chronicle-2026-06-22:502-503 |
| 15 | Normalize parsed 30 positions for that day → 30-vs-21 single-day gap inside apply-svc | CONFIRMED INFERRED | ViskaN8N chronicle.md:7-14 (artifact); gap cause inferred |
| 16 | Apply body key contract = object_path (handler); n8n node sends path (OPEN) | CONFIRMED | apply_service.py:146 / rescope-ops:78,80 |
| 17 | PUT-leg auth shape (token as header vs query-string self-auth) | UNKNOWN | OPEN — rescope-ops:67-70 |
| 18 | Statement Usable? guard (position_count>0 AND report_date notEmpty), commit f8a0304, activeVersionId 6aa4c859 | CONFIRMED | git show f8a0304; chronicle.md:17-32 |
| 19 | Fix B multi-day Normalize committed 2f12e5c, lib flex-normalize.js, 7/7 tests | CONFIRMED | chronicle.md:54; flex-normalize.js:46-54 |
| 20 | Live Normalize node body update DEFERRED to A+B-green deploy (lib≠live not diffed) | CONFIRMED | multiday-contract:81-82; chronicle.md:54-62 |
| 21 | 28992ad5…/902d95a5… are n8n node ids, NOT git SHAs | CONFIRMED | rescope-ops:30,32 |
| 22 | Upsert Positions (f694c363…) removed; n8n holds zero trading-write cred | CONFIRMED | rescope-ops:31,83-89 |
| 23 | n8n holds exactly 2 feature creds: IBKR Flex token (httpQueryAuth) + FLEX_APPLY_WEBHOOK_SECRET (httpHeaderAuth) | CONFIRMED | rescope-ops:24,54,76 |
| 24 | apply-svc env: FLEX_APPLY_WEBHOOK_SECRET, VISKA_DB_URL, VISKA_SUPABASE_URL, VISKA_SUPABASE_SERVICE_ROLE_KEY, FLEX_INBOX_BUCKET(opt) | CONFIRMED | apply_service.py:35-44 |
| 25 | Single shared bearer for BOTH /sign-upload and /apply, constant-time compare | CONFIRMED | apply_service.py:66-70,326 |
| 26 | service-role key never leaves apply-svc (injected closure; pure core never receives it; test-asserted) | CONFIRMED | apply_service.py:182-186,340 |
| 27 | FLEX_UPLOADER_JWT / flex_uploader HS256 leg DEAD (ES256 prod 403s); replaced by signed URLs (060) | CONFIRMED | apply_service.py:6-9,46 |
| 28 | flex_inbox_uploader_insert RLS policy exists in DDL but unused-but-harmless | CONFIRMED | 20260621-01:32-38; apply_service.py:14-15 |
| 29 | flex-inbox private bucket; service_role reads bypass RLS | CONFIRMED | 20260621-01:24-26,18-19 |
| 30 | trading.* = 11 tables; importer writes 9 directly; instrument_themes operator-seeded; candles n8n nightly | CONFIRMED | 20260610-01/02; ibkr_import.py:181-352 |
| 31 | Flex path lands 4 streams (instruments, positions, NAV, FX) at latest reportDate; trades/dividends/interest/cash deferred | CONFIRMED | flex_to_sections.py:10-12 |
| 32 | source='flex_api' admitted by CHECK; ops.health_events source extended to flex_apply | CONFIRMED | 20260610-01:30-31; 20260621-01:48-50 |
| 33 | FE reads public.* projections, NOT trading.v_*; data-streams bible STALE | CONFIRMED | useDashboardData.ts:76,108,152; 05-data-streams.md:9-13 |
| 34 | All public.* fund/market views currently authenticated-only (anon revoked 06-19) | CONFIRMED | 20260619-03/04/06 |
| 35 | trading.v_* consumed by Mímir bot via trading_bot_ro role | CONFIRMED | 20260610-08:42-47 |
| 36 | Most migrations are FORWARD (not applied to prod) except 20260619-03 (PR #20, Hades rc=0) | CONFIRMED UNKNOWN | migration headers (confirmed); prod-apply state of 06-17/06-19 chain unknown |
| 37 | CF Access fronts the dashboard/artifacts portal only, NOT the IBKR ingest path; apply-svc bearer is sole gate | CONFIRMED | grep null; on-demand-webhook-design.md:136-137 |
| 38 | On-demand webhook (2nd trigger, 202-async, native Header Auth, 5-min cooldown) DESIGN-APPROVED, NOT BUILT (blocked on N8N_API_URL/N8N_API_KEY + secret mint) | CONFIRMED | on-demand-webhook-design.md:6,109-117 |
| 39 | Exact webhook path string | UNKNOWN | not in any spec/file |
| 40 | Sign Upload / Upload / Apply POST live node ids | UNKNOWN | raw JSON gated |
| 41 | Decision 055-A default = file-drop + Hades cron; HTTP apply-svc = rule-#19 fallback | CONFIRMED | 055/ViskaDB.done:21-50 |
| 42 | CashTransaction consumed anywhere | UNKNOWN | in no importer file/spec |
| 43 | Workflow live + active for 22:30 cron (activeVersionId 6aa4c859); round-trip GREEN once (exec 35607) | CONFIRMED | chronicle.md:35-38; chronicle-2026-06-22:501-503 |
| 44 | percentOfNAV/fxRateToBase/endingCash → no trading.* column (Normalize metadata only) | CONFIRMED | multiday-contract:75-76 |
Contradictions resolved (10)
The reconciliation found ten cross-domain conflicts. Each was resolved to the handler/code/commit truth and folded into the canonical fact sheet; section writers follow the resolution, not the raw domain finding.
C1 · Apply body key
n8n sends {bucket, path}; handler reads object_path and 400s without it. Resolution: contract = {bucket, object_path} (handler truth); the path mismatch is an unreconciled OPEN. apply_service.py:146 / rescope-ops:78,80
C2 · multiday filter — pending or merged?
Schema/transform read it as pending Fix A; writepath found emit_days already in code. Resolution: Fix A is MERGED IN CODE (f810998); the "1-of-22" framing applies only to the pre-f810998 live image. flex_to_sections.py:145; ibkr_import.py:359
C3 · 30 vs 21 positions
Not a contradiction — two distinct phenomena: (a) 30→21 single-day gap inside apply-svc; (b) 1-of-22 multi-day landing (Fix A). Never conflate. ViskaN8N chronicle.md:7-14
C4 · FLEX_UPLOADER_JWT live or dead
Schema documents the RLS policy as active; all others confirm the HS256 leg 403s against ES256 prod. Resolution: JWT leg DEAD, RLS policy vestigial unused-but-harmless; signed URLs are the live auth. apply_service.py:6-9,46
C5 · apply-svc live deploy
/healthz 200 + 401-on-unauth are the ONLY live confirmations; image SHA unfingerprinted (RAILWAY_TOKEN unset). Resolution: code-confirmed, live-deploy-unconfirmed — never claim "22/22 lands in prod." chronicle 2026-06-22 23:40
C6 · ndcdyn vs gdcdyn
Resolution: SendRequest pinned ndcdyn in the live node; GetStatement follows {{ FlexStatementResponse.Url }} dynamically (observed gdcdyn). Source's gdcdyn SendRequest is illustrative. flex-pull-…json:85; chronicle:389(3)
C7 · node id vs commit SHA
28992ad5…/902d95a5… are n8n node ids, not git SHAs. Real Fix-B = 2f12e5c; guard = f8a0304. rescope-ops:30,32
C8 · Decision 055-A drift (named trap)
055-A default = file-drop (auth:none) + Hades cron; the bearer-auth Railway HTTP apply-svc is the rule-#19-gated FALLBACK. Never present the HTTP endpoint as 055's primary choice. 055/ViskaDB.done:21-50
C9 · 17 vs 19 nodes
17 = design baseline; grew to 19+ live after cred-wiring, poll-loop refactor, and the Statement Usable? guard insert. Not a contradiction — cite the live graph. chronicle.md (live graph)
C10 · Upsert Positions present or removed?
Disabled then REMOVED, replaced by Sign Upload → Upload → Apply POST. n8n holds zero trading-write credential. rescope-ops:31,83-89
Residual unknowns — the honest edge
NEVER ASSERT THESE IN EITHER DIRECTION
- Live image SHA running the multiday fix — no fingerprint (
RAILWAY_TOKENunset, norailway.tomlin ViskaDB). UNKNOWN - Prod 22/22 landing fired-and-verified — code-confirmed only. UNKNOWN
- PUT-leg auth shape (header vs query-string self-auth) — OPEN at rescope-ops:67-70. UNKNOWN
- Apply
pathvsobject_pathreconciliation state — flagged OPEN in the spec itself. UNKNOWN - Prod-apply state of the 06-17 / 06-19 projection + anon-revoke migration chain (only
20260619-03confirmed applied). UNKNOWN - Exact on-demand webhook path string and the Sign/Upload/Apply live node ids — raw JSON gated. UNKNOWN
CashTransactionconsumption — appears in no importer file or spec. UNKNOWN
METHODOLOGY
T-deep where the fact is load-bearing: every confirmed row traces to a primary source (code line, migration header, git object, or a live HTTP probe). The two single-source INFERRED tags (row 15 gap cause, row 36 prod-apply state) are explicitly caveated rather than promoted to confirmed. The canonical fact sheet is authority over any individual domain finding on conflict.