Root causes of the 2026-09-08 mesh blackout on the broker side (spec
.artifacts/specs/2026-09-08-daemon-restart-mesh-blackout.md, Addenda 5-6):
- ws.on("close") checked conn.ws !== ws BEFORE an await and mutated the
lease AFTER it; and under Bun (prod runtime) the ws shim emits the replaced
socket's close event synchronously inside close(). Either way the close of
the DEAD daemon's socket landed on the NEW daemon's reattached lease: grace
started, 90 s eviction timer armed, healthy session evicted while its
socket stayed open — a ghost that peer list and sends could not see.
Fix: decision taken synchronously (src/lease.ts decideCloseAction), lease
swapped onto the new socket BEFORE the old one is closed, dedup drops the
entry before closing, grace timer re-checks leaseState before evicting.
- evictPresenceFully never closed a still-open socket → zombie. Now closes
it (4004 presence_evicted) so the client re-hellos; a 30 s zombie sweeper
closes any OPEN socket with no presence (4005 no_presence).
- Session-id dedup scoped by role: the launch wrapper's control-plane hello
and the daemon's session_hello share the Claude session UUID and used to
kick each other on every relaunch.
- mesh.presence had 2,491,950 rows and no index beyond the PK; every hot
query full-scanned it, pushing hello latency to ~5 s under a reconnect
storm (which tripped the daemon's 5 s ack timeout). Migration 0030 adds
partial indexes on the online rows; hourly retention sweep deletes rows
disconnected > 30 d (batched 20k). Hello handlers log latency_ms.
- sweepOrphanMessages rewritten with the query builder (the raw Date param
threw on every hourly run).
Tests: tests/lease.test.ts (pure decision helpers incl. the incident
sequence), tests/presence-retention.test.ts (retention + orphan sweep
against a real DB). Verified locally: migrations 0000-0030 apply through
the runtime runner on a fresh pgvector DB; SIGKILL/respawn drills against
this broker show reattach with 0 evictions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
0bf7ee9c84