claudemesh/.artifacts/specs/2026-04-15-broker-ha-statelessness-audit.md

Broker HA readiness — statelessness audit

Single-instance broker is the biggest GA blocker. Moving to 2+ replicas behind a load balancer requires first understanding which state the broker holds in-process that breaks if split across nodes.

Current in-process state (apps/broker/src/index.ts)

Symbol	Line	Per-node?	Survives HA?	Notes
connections	147	yes (WS state)	✅ naturally per-node	WS connections are pinned to a node by L7 routing. Each node holds only its own connections. OK as long as the LB uses sticky sessions or cross-node fan-out.
connectionsPerMesh	148	yes	🟡 per-node count, not global	Used for capacity cap. Global cap requires Redis.
tgTokenRateLimit	151	yes	🟡 per-node	Telegram bot rate limiting; tolerable as per-node.
urlWatches	173	yes	🔴 stuck on one node	If peer disconnects from node A and reconnects on B, the watch stays orphaned on A. Needs DB/Redis, or "pin to owning node". Acceptable risk if watches are per-session ephemeral.
streamSubscriptions	259	yes	🔴 multi-node broken	Sub on A, publish on B → message never reaches A's subscribers. Needs Redis pub/sub for HA.
meshClocks	270	yes	🔴 multi-node broken	Simulated clocks must be single-authority. Solve by pinning one node as clock leader (simple leader election) or by moving clock state to DB.
mcpRegistry	327	yes	🔴 multi-node broken	MCP server catalog cached in memory. If deployed on A but called on B, B doesn't know it exists. Must be DB-backed (partly is already — see mesh_service table). Audit the cache/DB sync path.
mcpCallResolvers	338	yes	✅ per-call ephemeral	In-flight callback resolvers; WS sticks to owning node so this is fine.
scheduledMessages	359	yes	🔴 multi-node broken	Scheduled delivery timers live in-process. Restart loses them. Persistence exists (scheduled_message table) + recovery on startup, but two nodes could both fire the same timer. Needs a leader lock or per-schedule pg_advisory_lock on fire.
sendRateLimit	index.ts:494	yes	🟡 per-node	Each node enforces its own quota; a client spread across nodes could 2x the limit. Tolerable if sticky sessions hold.
hookRateLimit	index.ts:482	yes	🟡 per-node	Same as sendRateLimit.
lastHash (audit.ts:22)	—	yes	🔴 broken on write	Two nodes writing audit rows concurrently will BOTH read the same last hash, BOTH compute a new hash, and both INSERT — the chain forks. Needs SELECT FOR UPDATE or a single audit writer.

Conclusion

Current broker is NOT HA-safe. Five symbols break under multi-instance: urlWatches, streamSubscriptions, meshClocks, mcpRegistry cache, scheduledMessages, lastHash. None are unsolvable, but none are trivial.

Rollout plan for HA

Phase 0 (now) — sticky sessions

Deploy a single broker behind Traefik with loadBalancer.sticky.cookie enabled. WS upgrade inherits the cookie, so reconnects land on the same node. Gives us 1 node of safe HA headroom (i.e., one deploy rollover without user-visible disconnection) without any code changes.

Phase 1 — Active/passive

Two replicas. Traefik routes all traffic to primary; secondary is warm. Primary fails → secondary takes over, all WS connections reset. No code change needed; clients auto-reconnect.

Phase 2 — Active/active for stateless routes

HTTP-only routes (/cli/*, /download, /hook) can round-robin across any number of replicas today. WS routes stay sticky per mesh via Traefik sticky.cookie. Already behind Postgres → each replica reads the same mesh/member/invite rows.

Phase 3 — Full active/active

Migrate the 6 problematic in-memory symbols:

• streamSubscriptions → Redis pub/sub
• meshClocks → leader-elect via Postgres advisory lock on mesh_id
• scheduledMessages → single-writer pattern: whichever replica holds pg_advisory_xact_lock(schedule_id) fires
• urlWatches → DB-backed + each replica owns watches where presence.node_id = this_node
• mcpRegistry → rely on mesh_service table, drop the in-memory cache
• lastHash → wrap audit.ts writes in a transaction that SELECT hash FROM audit_log ... ORDER BY id DESC FOR UPDATE, making concurrent inserts serialize.

Phase 4 — Multi-region

SPOF at Frankfurt (OVH). Move to a managed Postgres with read replicas, one broker cluster per region, global DNS geo-routing. Out of scope for v1.0.0.

Immediate ship: local docker-compose for 2-replica smoke test

packaging/docker-compose.ha-local.yml (TODO) spins up:

• 2x broker (same DATABASE_URL)
• 1x postgres
• 1x traefik with sticky cookie
• 1x locust / synthetic client

Tests:

1. Send to peer connected on node A → delivered.
2. Subscribe on A, publish on B → expect failure (documents the gap).
3. Kill node A → client reconnects to B within Xs.
4. Audit chain verify after concurrent writes from both nodes → expect a fork (documents the gap).

Decision

Ship v1.0.0 on sticky-session single-writer (Phase 0 + Phase 1 warm standby). That closes the "what happens on deploy" story. Phase 3 full HA is v1.1.0 work.