0bf7ee9c84
Root causes of the 2026-09-08 mesh blackout on the broker side (spec
.artifacts/specs/2026-09-08-daemon-restart-mesh-blackout.md, Addenda 5-6):
- ws.on("close") checked conn.ws !== ws BEFORE an await and mutated the
lease AFTER it; and under Bun (prod runtime) the ws shim emits the replaced
socket's close event synchronously inside close(). Either way the close of
the DEAD daemon's socket landed on the NEW daemon's reattached lease: grace
started, 90 s eviction timer armed, healthy session evicted while its
socket stayed open — a ghost that peer list and sends could not see.
Fix: decision taken synchronously (src/lease.ts decideCloseAction), lease
swapped onto the new socket BEFORE the old one is closed, dedup drops the
entry before closing, grace timer re-checks leaseState before evicting.
- evictPresenceFully never closed a still-open socket → zombie. Now closes
it (4004 presence_evicted) so the client re-hellos; a 30 s zombie sweeper
closes any OPEN socket with no presence (4005 no_presence).
- Session-id dedup scoped by role: the launch wrapper's control-plane hello
and the daemon's session_hello share the Claude session UUID and used to
kick each other on every relaunch.
- mesh.presence had 2,491,950 rows and no index beyond the PK; every hot
query full-scanned it, pushing hello latency to ~5 s under a reconnect
storm (which tripped the daemon's 5 s ack timeout). Migration 0030 adds
partial indexes on the online rows; hourly retention sweep deletes rows
disconnected > 30 d (batched 20k). Hello handlers log latency_ms.
- sweepOrphanMessages rewritten with the query builder (the raw Date param
threw on every hourly run).
Tests: tests/lease.test.ts (pure decision helpers incl. the incident
sequence), tests/presence-retention.test.ts (retention + orphan sweep
against a real DB). Verified locally: migrations 0000-0030 apply through
the runtime runner on a fresh pgvector DB; SIGKILL/respawn drills against
this broker show reattach with 0 evictions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
39 lines
1.8 KiB
SQL
39 lines
1.8 KiB
SQL
-- 2026-09-08 daemon-restart mesh blackout, RC-B (spec:
|
|
-- .artifacts/specs/2026-09-08-daemon-restart-mesh-blackout.md, Addendum 5).
|
|
--
|
|
-- mesh.presence had 2.49 M rows and no index beyond the PK. Every hot
|
|
-- query filters on `disconnected_at IS NULL` (13 rows online) and was
|
|
-- full-scanning the table:
|
|
-- * sweepStalePresences — every 30 s: WHERE disconnected_at IS NULL AND last_ping_at < cutoff
|
|
-- * listPeersInMesh — JOIN member … WHERE disconnected_at IS NULL
|
|
-- * handleHookSetStatus — WHERE cwd = ? AND disconnected_at IS NULL
|
|
-- * heartbeat / restorePresence / disconnectPresence — by PK (fine)
|
|
-- Under a reconnect storm those scans pushed hello latency to ~5 s.
|
|
--
|
|
-- Partial indexes keep only the online rows (tiny) so each of those
|
|
-- queries is an index scan regardless of table size. The plain index on
|
|
-- disconnected_at serves the new retention sweep (delete rows disconnected
|
|
-- > 30 d ago in batches).
|
|
--
|
|
-- Plain CREATE INDEX (not CONCURRENTLY): the migration runner wraps each
|
|
-- file in a transaction, and CONCURRENTLY cannot run inside one. On the
|
|
-- 2.5 M-row prod table these three builds take a few seconds each and
|
|
-- block writes to presence for that window; heartbeats retry on the next
|
|
-- pong, so this is acceptable for a one-time build.
|
|
|
|
CREATE INDEX IF NOT EXISTS presence_online_member_idx
|
|
ON mesh.presence (member_id)
|
|
WHERE disconnected_at IS NULL;
|
|
--> statement-breakpoint
|
|
CREATE INDEX IF NOT EXISTS presence_online_last_ping_idx
|
|
ON mesh.presence (last_ping_at)
|
|
WHERE disconnected_at IS NULL;
|
|
--> statement-breakpoint
|
|
CREATE INDEX IF NOT EXISTS presence_online_cwd_idx
|
|
ON mesh.presence (cwd)
|
|
WHERE disconnected_at IS NULL;
|
|
--> statement-breakpoint
|
|
CREATE INDEX IF NOT EXISTS presence_disconnected_at_idx
|
|
ON mesh.presence (disconnected_at)
|
|
WHERE disconnected_at IS NOT NULL;
|