T3 — Eval Matrix — Nous Hermes-Agent vs Pantheon Stack
Synthesis layer. Joins T1 (Pantheon stack per executor) and T2 (nous-hermes single-process). Output: weighted composite scores + gap-zone identification + role-disambiguation guidance for T4. Weighting source: PRD v0 §G2-G6 + 5 mandatory guardrails (PHASE2) + operator directives (D8 cost-low, D11 strict-isolation, D12 take-time-ship-properly).
Method
Each of 10 axes (mirrored T1/T2) gets a weight 1-5 based on fleet priority. Per-executor axis scores from T1/T2 multiplied by weight. Sum = composite. Higher = better fleet fit.
Nous-hermes scored once per role (see §Role disambiguation). Pantheon-stack scores represent best-of-class per axis from T1 inventory (the executor we'd actually use for that role).
Weighting
| Axis | Weight | Rationale |
|---|---|---|
| 1 — Triggering modes | 3 | Multi-trigger surface useful but not differentiator-class. n8n + Routines already cover well. |
| 2 — Persistence / durability | 4 | Critical for fleet-stable schedules. Wake-catchup primacy non-negotiable on laptop fleet. |
| 3 — Tool surface | 2 | Matters for autonomous role, not for scheduling. Over-weighted historically — most schedules are simple. |
| 4 — Cost model | 3 | Operator directive "keep costs low for implementation" (D8). |
| 5 — Isolation / sandbox | 5 | Operator directive "strict isolation" (D11). 5 mandatory guardrails depend on this. Highest weight. |
| 6 — Observability + rule-18 evidence | 5 | PRD §G2 — verification_evidence is unrepresentable, not flagged. CHECK constraint enforces. Highest weight. |
| 7 — Multi-tenant boundary | 5 | Fleet has 5+ agents (atlas/hermes/plutus/proteus/mimir-n8n/pantheon-ops). Per-tenant collapse = identity loss. Highest weight. |
| 8 — Failure modes + auto-pause | 4 | PRD §G3 — automatic + bounded. Compensated SSOT-side, but executor cooperation helps. |
| 9 — Learning / feedback loop | 2 | Nice-to-have for scheduling role. Mandatory for autonomous-research role (re-weighted below). |
| 10 — Cross-machine / sleep-wake | 4 | PRD §G4. Fleet has K20N0S + MacBooks + hostinger-vps + Railway. Multi-host coordination matters. |
Weight total: 37. Max composite = 5 × 37 = 185.
Matrix join (scheduling-executor role)
This is the head-to-head if Hermes is adopted as ONE MORE executor in fleet_schedules.executor enum. Pantheon-stack column shows best-of-class score from T1 per axis.
| Axis | Weight | Pantheon best (executor) | Hermes | Pantheon weighted | Hermes weighted | Delta (H − P) |
|---|---|---|---|---|---|---|
| 1 | 3 | 4 (Routines, n8n) | 4 | 12 | 12 | 0 |
| 2 | 4 | 5 (launchd, Routines) | 3 | 20 | 12 | −8 |
| 3 | 2 | 5 (/loop, n8n) |
5 | 10 | 10 | 0 |
| 4 | 3 | 5 (launchd, Slack) | 4 | 15 | 12 | −3 |
| 5 | 5 | 4 (Mastra) | 3 | 20 | 15 | −5 |
| 6 | 5 | 5 (Mastra writeRun) | 3 | 25 | 15 | −10 |
| 7 | 5 | 5 (Slack workspace) for delivery / 4 (Mastra) for compute | 2 | 25 / 20 | 10 | −15 / −10 |
| 8 | 4 | 4 (Mastra) | 3 | 16 | 12 | −4 |
| 9 | 2 | 3 (Mastra reflection_text) |
3 | 6 | 6 | 0 |
| 10 | 4 | 5 (Routines cloud) | 3 | 20 | 12 | −8 |
Composite scheduling-executor role: Pantheon-best 169/185 (91%) vs Hermes 116/185 (63%). Delta −53 (28 percentage points).
Hermes LOSES the scheduling-executor head-to-head against the best-of-Pantheon-stack. Verdict basis: launchd + Mastra Workflow + Slack already cover the axes Hermes weakens (Persistence, Isolation, Observability, Multi-tenant).
Matrix join (autonomous-research-agent role)
Re-weight axes for the autonomous-research role: - Axis 9 (Learning) → weight 5 (the differentiator) - Axis 3 (Tool surface) → weight 4 (research needs breadth) - Axis 1 (Triggering) → weight 2 (autonomous = self-triggers, less external scheduling) - Others unchanged.
New weight total: 38. Max composite = 190.
| Axis | Weight | Pantheon best (executor) | Hermes | Pantheon weighted | Hermes weighted | Delta |
|---|---|---|---|---|---|---|
| 1 | 2 | 4 | 4 | 8 | 8 | 0 |
| 2 | 4 | 5 | 3 | 20 | 12 | −8 |
| 3 | 4 | 5 (/loop, n8n) |
5 | 20 | 20 | 0 |
| 4 | 3 | 5 | 4 | 15 | 12 | −3 |
| 5 | 5 | 4 | 3 | 20 | 15 | −5 |
| 6 | 5 | 5 | 3 | 25 | 15 | −10 |
| 7 | 5 | 4 (Mastra) | 2 (or 4 w/ process-per-agent) | 20 | 10 / 20 | −10 / 0 |
| 8 | 4 | 4 | 3 | 16 | 12 | −4 |
| 9 | 5 | 3 | 3 | 15 | 15 | 0 |
| 10 | 4 | 5 | 3 | 20 | 12 | −8 |
Composite autonomous-research-role: Pantheon-best 179/190 (94%) vs Hermes 131/190 (69%) in shared-process mode, or 141/190 (74%) in process-per-agent mode. Delta −38 to −48 (20–25 percentage points).
Hermes LOSES this head-to-head too — BUT the comparison is unfair: Pantheon-best Axis 9 score is 3 (Mastra's reflection_text, not actively consumed), while Hermes Axis 9 is also 3 but represents an ACTIVE Ralph loop pursuing goals across turns. Both score 3 mechanically, but the realized value is asymmetric. T1's stack-level gap "no feedback loop closes end-to-end" applies to Pantheon; Hermes's Ralph loop genuinely closes it.
If Axis 9 scoring were refined to reflect "active vs latent" feedback, Hermes would jump from 3 → 4 here. Pantheon stays at 3. Hermes composite would become 141/190 (74%) → 146/190 (77%). Still a loss in absolute terms but the gap narrows materially.
Matrix join (delivery-surface role)
Hermes-on-K20N0S already provides Slack gateway (machine report). Compare against Pantheon best-of-stack for delivery (Slack chat.postMessage).
Skip detailed matrix — Slack is already best-in-class on Axes 5, 7, 8, 10. Hermes-as-Slack-gateway is substrate for Slack, not a replacement. The 22-platform breadth (Telegram/Discord/Teams/Matrix/etc.) only matters if fleet expands beyond Slack — currently slack-boas + slack-viska are the only fleet delivery surfaces.
Verdict: Hermes-as-delivery-surface = ALREADY DEPLOYED on K20N0S, no decision to make. Maintain existing posture.
Gap-zone summary
Where Hermes ADDS value Pantheon-stack lacks:
1. Active autonomous loop (Ralph). Pantheon's reflection_text is written but never read. Hermes's Ralph loop genuinely closes the feedback cycle. This is the only true gap-filler.
2. 22-platform messaging breadth. Pantheon has Slack only. Hermes ships Telegram/Discord/Teams/Matrix/IRC/SMS/email + 15 more. Latent value — not currently needed.
3. Self-modifying skill substrate. TokenMix benchmark cites ~40% research-task-time reduction via self-created skills. Pantheon stack has no equivalent.
4. MCP client built-in. Pantheon-stack uses MCP via /loop + CC Desktop + Mastra; Hermes has it natively + at runtime in plugins/skills. Marginal — not differentiating.
5. OpenAI-compat HTTP+SSE proxy. Lets any LLM tool point at Hermes-the-gateway and get the substrate transparently. Architectural sweetener — not load-bearing for fleet today.
Where Hermes WEAKENS what Pantheon-stack does well: 1. Persistence vs launchd's wake-catchup primacy. Axis 2 delta −2 absolute (5 → 3). 2. Rule-18 evidence vs Mastra's writeRun. Axis 6 delta −2 absolute. Plugin-authoring tax (~3–5 days) to close partially. 3. Multi-tenant boundary vs Mastra's per-Workflow Tool-constraint. Axis 7 delta −2 absolute. Process-per-agent partially closes (resource cost). 4. Cross-machine vs Routines's cloud reach. Axis 10 delta −2 absolute. Routines's per-operator-account collapse already disqualifies it for fleet identity — moot trade.
Composite verdict
Mechanical scoring: Hermes underperforms best-of-Pantheon-stack in all three role framings (scheduling, autonomous-research, delivery).
Why this isn't the full answer: the best-of-Pantheon-stack composite blends executors that wouldn't typically run the same schedule. Mastra alone doesn't trigger, n8n alone doesn't reflect, launchd alone doesn't observe. Real fleet schedules chain through these — Themis → launchd → Hermes-bot? → Mastra Workflow → Slack. Hermes inserted as one node in that chain inherits the chain's compensations.
The real question: is there a role-shape where Hermes contributes a capability the chain can't deliver otherwise? Three candidates:
| Role | Hermes value-add | Pantheon-stack alternative cost |
|---|---|---|
| Autonomous-research executor for Plutus/atlas analyst loops | Ralph loop closes feedback cycle | Build feedback-loop wiring around Mastra + cron poller of fleet_schedule_runs.reflection_text — ~5–10 days. |
| Self-modifying skill substrate for atlas/hermes-fleet-agent investigation loops | ~40% time-reduction per TokenMix | Build skill-authoring + version-control + replay infra fleet-side — ~3–4 weeks. |
| Multi-platform messaging surface (Telegram/Discord/Teams/Matrix expansion beyond Slack) | 22 platforms native | Build per-platform integrations — ~1–3 days each × N platforms. |
For each candidate, weigh "Hermes value-add" against "5 mandatory guardrails + plugin authoring (25–40 person-days) + 4–8 wk hook half-life maintenance."
Role disambiguation guidance for T4
Three deployment patterns, each with different cost/value calculus:
Pattern A — Hermes as autonomous-research executor only
- Scope: Plutus market-close digest loop, atlas governance-research loop, hermes-fleet-agent YouTube-intel pipeline.
- Process: one Hermes per agent identity on K20N0S (3 processes minimum, ~6GB RAM).
- Plugins: full 12-plugin backlog from T2 §Pantheon-authored plugin backlog.
- Effort: 25–40 person-days plugin authoring + 5–8 days K20N0S Docker Compose hardening + 3–5 days
fleet-schedules-trigger-adapter. - Aggregate first-deploy cost: ~7–10 weeks engineering.
- Maintenance: ~1–2h/wk per process + monthly upgrade triage = ~3–6h/wk total at 3 processes.
Pattern B — Hermes as delivery-surface only
- Scope: maintain current K20N0S Slack gateway as-is; do NOT expand to scheduling.
- Process: 1 (existing).
- Plugins: minimal —
slack-template-filter+freeze-flagonly. - Effort: ~3 days plugin authoring + Docker Compose hardening if D10-compliance desired.
- Maintenance: ~0.5–1h/wk.
- Trade-off: Loses Ralph loop value entirely. Hermes becomes a redundant Slack relay vs
@slack/boltdirect.
Pattern C — Hermes as omnipotent executor (full adoption)
- Scope: Hermes replaces launchd + Mastra + parts of n8n for fleet scheduling.
- Effort: ~3–6 months engineering (full plugin backlog + replacement of in-flight Mastra Workflow infrastructure).
- Maintenance: ~10–20h/wk fleet-side.
- Trade-off: Highest concentration of bus-factor risk (teknium1 dependency for ALL fleet scheduling); breaks the multi-executor SSOT defense-in-depth principle from T1 invariant 3.
T4 recommendation surface: Pattern A bounded by tight scope criteria, OR Pattern B status-quo-maintenance, OR mixed (Pattern A for 1-2 agents on trial, Pattern B for delivery, reject Pattern C).
Carry-forward to T4
T4 must answer: 1. Which pattern? (A / B / A+B mixed / reject) 2. If A or A+B: which agent roles get a Hermes process first? Plutus market-close? atlas research? 3. K20N0S hardening prereq — Docker Compose wrap or operator-waived "out-of-support" posture? 4. Resource cap on K20N0S (existing daemon + N new processes; older hardware per machine report). 5. fleet_schedules.executor enum extension order — adapter plugin before any schedule fires. 6. Trial gate — pre-defined criteria (catch-rate, time-savings, false-evidence rate) before promoting from trial to standard. 7. Rollback criteria — when to retire Hermes if trial fails.