Files
claudemesh/.artifacts/specs/2026-09-08-daemon-restart-mesh-blackout.md
Alejandro Gutiérrez c138e0bceb fix: reattachLease ordering contract + flushed daemon shutdown (20:40Z recurrence = RC-C)
The 20:40Z clean-restart recurrence (spec Addendum 7) is the same RC-A/RC-C
mechanism, with prod evidence: 4 reattached presences (was:"online") hit
"lease grace started" 2.5 s later and were evicted at 20:42:05 — exactly
their disconnected_at. last_ping_at froze because the ping loop skips
offline leases; the freeze is a symptom, not a separate ping bug.

- broker: the swap-before-close ordering now lives in lease.ts#reattachLease
  and is unit-tested with a close callback that fires synchronously (Bun
  emulation): the handler sees ignore_replaced; a mis-behaving handler that
  arms a grace timer still leaves the lease online with no timer.
- cli: WsLifecycle.close() waits (<=1 s) for the socket's close event and
  shutdown() closes all broker clients in parallel, so a clean restart's
  close frames reach the broker before process.exit. Prod showed the old
  daemon's 17 sockets were only reaped by the 75 s stale-pong watchdog,
  which is why the next daemon reattached onto "online" leases (the RC-C
  path) instead of "offline" ones. Test: close() resolves only after the
  fake broker observed the close.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 23:10:27 +01:00

31 KiB
Raw Permalink Blame History

2026-09-08 — Daemon restart → mesh blackout (peer list empty, sessions "invisible")

Status: diagnosed on a live incident, fix pending. Owner: claudemesh repo session. Reporter: flexicar-orchestrator session (evidence gathered read-only from ~/.claudemesh/daemon/daemon.log, live CLI probes, and source on main @ 1.37.0).

Symptom (user-visible)

After the local daemon restarted (launchd com.claudemesh.daemon KeepAlive), claudemesh peer list returned 0 peers on EVERY mesh (flexicar, minery, legalrag, nedas-mesh…) for >40 min, although all 8 Claude sessions were alive. Sends by session pubkey reported queued — no connected peer matches this key. The whole orchestration went dark. User verdict: "in case of daemon restart, sessions should be reconnected — this must not happen."

Timeline (UTC; local = UTC+1)

  • 2026-09-05 20:56:59Z — daemon pid 85205 started (ran ~3 days).
  • 2026-09-08 ~19:27–19:28Z — pid 85205 died without logging a single line (no exit/uncaught/SIGTERM in daemon.log, no crash report in ~/Library/Logs/DiagnosticReports, no memorystatus kill in log show). launchd respawned: pid 93782 at 19:28:07 (died within 6 s — crash-loop once), pid 94504 at 19:28:13 (stable, still running).
  • 19:28:07 and 19:28:13 — sessions_rehydrated revived=8 persisted=8 → rehydration WORKS: registry restored from sessions.json, keypairs re-derived, onRegister created a SessionBrokerClient per session.
  • 19:28:12–19:28:20 — daemon-level mesh WS: broker_ws_hello_sent → broker_hello_ack_timeout (all 9 meshes, 5 s) → broker_ws_closed 1005 → broker_ws_reconnect_scheduled → reconnect → broker_ws_hello_acked at 19:28:20.
  • 19:28:19+ — every per-session WS: session_broker_ws_open_attempt/open_ok/hello_acked for ALL session pubkeys (02b5b7d5, 0fdf7159, 612fa2bc, 9f0e1c09, aee93b3b, e051e316, f34b4b9f, …). Some flapped (aee93b3b ×4 attempts, 2 closed).
  • 20:09 / 20:18Z — drain_max_attempts err=broker_not_open for 2 outbox rows (a session WS was NOT open at drain time despite hello_acked earlier → flapping).
  • 20:09:01 / 20:10:08Z — two sessions relaunched by the user → session_registered via IPC → hello_acked. Still 0 peers.
  • Broker (ic.claudemesh.com): /health ok, uptime 5.34 M s (~62 d, NOT redeployed), db up.

What IS working after the restart

  • Registry rehydration (8/8), keypair re-derivation, attestation re-mint.
  • Per-session broker WS: open + hello_acked for every session (so the broker's in-memory conn map has them, lease online).
  • Inbound pushes: claudemesh inbox still lists prior messages; /v1/events SSE reconnects by itself (mcp/server.ts).

What is BROKEN

  1. list_peers → empty on all meshes. Broker list_peers handler = listPeersInMesh(meshId) = DB query presence WHERE disconnectedAt IS NULL. Empty ⇒ the DB presence rows are marked disconnected while the in-memory lease is online → split-brain between in-memory conn map and the presence table. Hypothesis (p≈0.8): the old daemon died without close frames → broker WS closes → lease grace (90 s) → new daemon reattached within grace (session_hello reattach, restorePresence(pid) fire-and-forget void) → later the DB-level stale-presence sweeper (keyed on lastPingAt) flips the rows back to disconnectedAt = now because pongs on the reattached WS don't refresh lastPingAt for the reattached presence id — or restorePresence lost a race with the sweeper. Either way restorePresence is a one-shot patch on a row the sweeper keeps re-flipping. Needs confirmation from broker logs (ws hello reattach (lease), session_hello reattach, sweeper output) and SELECT id,session_pubkey,disconnected_at,last_ping_at FROM presence WHERE … on the broker DB (OVH via Coolify; SSH needs Tailscale re-auth right now).
  2. daemon.log = 1.0 GB (disk at 95 %). drain.ts:186 logs drain_session_ws_not_ready on EVERY drain tick for EVERY outbox row whose session WS isn't open — 117 797 lines in the last 20 MB, mostly for stale session pubkeys from previous days (4e00fc666559 ×70 676, cb26f393232e ×15 708 …) that will never reconnect. Rows are backed off but never expired; the log grows unbounded (launchd StandardOutPath → no rotation). This is the most plausible cause of the silent death of pid 85205 (I/O or disk pressure) — unproven.
  3. Session WS flapping after rehydration (multiple open/close cycles per session, broker_not_open at drain time). Likely the same-pubkey ghost handling on the broker (session_replaced close of the older socket) racing the reconnect helper. Needs broker logs.
  4. Recovery UX: nothing tells the user the mesh is in this state; peer list just says "no peers". A relaunch of a session does NOT fix it (proven twice today).

Work order (spec-first, then code, then tests, then release)

  1. Reproduce locally: broker + daemon in a test harness: kill daemon with SIGKILL → launchd-style respawn → assert peer list non-empty within 10 s for rehydrated sessions; assert presence rows disconnectedAt IS NULL after reattach AND after one stale-sweep cycle.
  2. Broker fix: make the reattach path authoritative for the DB row: restorePresence awaited + sweeper must not flip a row whose in-memory lease is online (or the pong handler must bump lastPingAt for reattached presences). Add a self-heal: on list_peers, reconcile DB rows against the in-memory connections map (online lease ⇒ include). Log at warn when DB/in-memory disagree.
  3. Daemon fix (drain.ts): rate-limit drain_session_ws_not_ready (log once per row per N minutes), expire outbox rows for session pubkeys with no registry entry after max(attempts)/TTL, and rotate daemon.log (size cap + keep N) — launchd writes stdout directly, so rotate in-process or via newsyslog/logrotate-style config installed by daemon install-service.
  4. Daemon crash forensics: install an uncaughtException/unhandledRejection + SIGTERM logger with flush, and a daemon_exit line, so the next silent death has a cause.
  5. Operator signal: claudemesh doctor (or daemon status) must detect "sessions registered + WS hello_acked but list_peers empty" and say so, with the one-command fix.
  6. Release as 1.37.1 (cli) + broker deploy; verify on this Mac: claudemesh peer list --mesh flexicar shows the 8 sessions without relaunching any of them.

Evidence pointers

  • ~/.claudemesh/daemon/daemon.log (1 GB — grep, don't cat): sessions_rehydrated, broker_hello_ack_timeout, session_broker_ws_*, drain_max_attempts, drain_session_ws_not_ready.
  • ~/.claudemesh/daemon/sessions.json (persisted registry).
  • Broker source: apps/broker/src/index.ts L1925-1960 (member reattach), L2225-2262 (session reattach), apps/broker/src/broker.ts L431-445 restorePresence, L448+ listPeersInMesh, stale sweeper (staleTimer).
  • Daemon source: apps/cli/src/daemon/run.ts L212-300 (onRegister), L312-384 (rehydrate), apps/cli/src/daemon/drain.ts L182-200, apps/cli/src/daemon/session-registry.ts (persist/reaper).

Addendum — outbox state at 20:35Z (claudemesh daemon outbox list)

  • pending rows with attempts=120 038 … 132 380, last_error="session_ws_not_open" → the drain loop has retried the same 4 rows ~130 k times (≈ once per second for days) for session pubkeys that no longer exist. This IS the 1 GB log. Rows must expire.
  • dead rows max_attempts: no connected peer for target (not online…) at attempts=25 → the broker rejected delivery because its DB says the target is offline, while the target's WS is hello_acked → confirms the presence split-brain (item 1). Consequence: after a daemon restart, DMs to live sessions are refused, not just discovery.
  • A fresh send now returns recipientOnline:false, status:queued for a session whose WS hello_acked minutes earlier.

Addendum 2 — broker-side evidence (OVH, container mcn8m74tbxfxbplmyb40b2ia-…, image = commit a9b735a; DB schema mesh.*)

Broker log 19:28:05–19:31Z (counts): 30 ws close — lease grace started · 27 ws evict full · 22 lease grace expired — evicting · 12 session_hello reattach · 9 hello dedup · 5 ws session_hello · 27 ws hello. Two daemons (pid 93782 lived 6 s, then 94504) both rehydrated 8 sessions and raced the broker within the same 90 s grace window while the broker was still processing the dead daemon's socket closes. Mesh-level hellos of 94504 hit broker_hello_ack_timeout (5 s) for all 9 meshes → broker_ws_closed 1005 → reconnect → acked at 19:28:20 (broker was busy with the storm).

Final broker DB state for flexicar (mesh.presence, disconnected_at IS NULL): online = aee93b3b (intranet 2ad05da8), 8494e7e4 (cliente-front relaunched), 612fa2bc (crm-api), 0fdf7159 (intranet f8b797d1), + control-plane daemon-94504. GHOSTS (evicted on the broker, socket still open on the daemon, no session_broker_ws_closed in daemon.log): 9f0e1c09 (orchestrator), 02b5b7d5 (crm-front), e051e316 (cliente-back), f34b4b9f (old cliente-front). Their presences (sUNqkMO9, ePekV8WH, BsheM0ey, Qxauubu7) got ws close — lease grace started at 19:28:17.9–18.3 (1 s AFTER session_hello reattach by 94504 at 19:28:17.0–17.2) and were lease grace expired — evicting at 19:29:43–50. evictPresenceFully does NOT ws.close() the socket it evicts → the daemon keeps a half-open WS forever, never reconnects → session unreachable and invisible, list_peers/DMs from that session go into a void.

Why peer list showed 0 on EVERY mesh from the orchestrator: the CLI inside a launched session is session-scoped (token env) → list_peers travels over the ghost session WS → no answer/empty. The member-scope check is in the next addendum.

hello dedup design flaw: the launch wrapper's member-level hello carries session_id = <claude session UUID>; the broker's session-id dedup then KICKS the live session_hello presence with the same UUID (hello dedup old_presence=a5eDg0aj), the daemon's session_hello then kicks the wrapper's presence (session_hello dedup) — 2 evictions + 2 DB disconnects + a forced WS close per relaunch (seen at 20:10:04–20:10:44 for intranet: a5eDg0aj → FYL5tnMl → MgXl7se3 → CBj9nWzr → JVuLGue1). Control-plane and session presences must not dedup each other by session_id.

Also seen: hourly orphan msg sweep throws DrizzleQueryError … TypeError: The "string" argument must be of type string … Received an instance of Date — the sweep is broken (Date passed where a string is expected).

Fix list (broker): (1) evictPresenceFully must ws.close(1000,"evicted") so the client reconnects; (2) a server-side liveness probe: if a socket is open but its presence is gone, close it; (3) hello dedup must be scoped by role (control-plane vs session); (4) mesh hello_ack must not time out under load (ack before heavy work, or raise/adapt the client timeout); (5) fix the orphan-sweep Date bug. Fix list (daemon): (6) SessionBrokerClient: detect a dead-but-open WS (missing server pings / no hello_ack refresh) and reconnect; (7) after rehydration, verify presence with the broker (sessions/me or a whoami ping) and re-hello if absent; (8) never run two daemons within the same grace window (launchd ThrottleInterval / startup lock) — the 6 s crash-loop doubled the storm.

Addendum 3 — ROOT CAUSE of "peer list empty on every mesh" (daemon side, confirmed in code + log)

apps/cli/src/daemon/broker.ts connect():

this.lifecycle = await connectWsWithBackoff({ ..., onStatusChange: (s) => { this._status = s; ... } });

On 2026-09-08 19:28:17.889Z the first hello_ack timed out (5 s; broker busy with the storm) → connectWsWithBackoff rejected → this.lifecycle was never assigned (broker connect failed for flexicar: Error: hello_ack_timeout, ×9 meshes). But the helper's internal reconnect loop kept running on the same object: broker_status reconnecting → connecting → open at 19:28:20.187Z and onStatusChange set _status = "open". Net state: _status === "open" AND lifecycle === undefined → every method gated on this.lifecycle short-circuits:

  • listPeers(): if (this._status !== "open" || !this.lifecycle) return []; → /v1/peers answers {"peers":[]} in 30 ms (measured; a real broker round trip is ≥200 ms) → peer list = 0 on all 9 meshes, forever, without any warning.
  • isOpen() reads this.lifecycle?.ws → false → send() no-ops → outbox drain logs drain_max_attempts err=broker_not_open for member-keyed rows.
  • listSkills, getState, etc. → same. The control-plane presence (daemon-94504) IS online on the broker (DB row XqS1ExHG, disconnected_at NULL), so the WS is real — the daemon just lost its handle to it.

Fix: connectWsWithBackoff must hand back the lifecycle handle synchronously (before the first ack) and expose readiness separately (await lifecycle.ready), so a first-ack timeout cannot orphan a live reconnecting socket; connect() must not reject while the helper is still retrying; add an invariant check (_status === "open" && !lifecycle → log error + self-heal by re-running connect). Regression test: simulate hello_ack timeout on first attempt + ack on the second → listPeers() must return the broker's list.

Two daemons in 6 s: launchd respawned pid 93782 which died within 6 s (cause unknown — no daemon_shutdown, no crash report; the clean shutdown() handler only logs on SIGINT/SIGTERM) → doubled hellos/reattaches/dedups in the same 90 s grace window. Needs daemon_exit/uncaughtException logging and a launchd ThrottleInterval.

Addendum 4 (CORRECTED 20:55Z) — after a daemon restart, delivery INTO Claude Code survives via the hook fallback; the live SSE push path does NOT

Wrong claim retracted: "no session receives pushes". Verified end-to-end after the clean restart: DM req:ping-3 (20:41:11Z) → crm-api woke up with the injected prompt [claudemesh] peer message arrived while idle - surface new inbox message(s), read its inbox and replied ack:pong3 (20:41:39Z, in the orchestrator's inbox). So messages reach sessions after a restart — through the local hook layer (~/.claude/hooks/claudemesh-wake-watcher.sh for idle sessions, claudemesh-inbox-drain.sh on Stop/UserPromptSubmit, claudemesh hook idle|working), which polls the daemon via CLI/IPC on each tick and therefore survives daemon restarts. Latency = the watcher's poll interval, not immediate.

What is dead after the restart: the MCP plugin's SSE subscription (/v1/events) — the path that injects <channel> blocks mid-turn into a working session. Evidence: lsof shows none of the 10 live claudemesh mcp processes connected to the daemon (neither UDS daemon.sock nor TCP 127.0.0.1:47823), and mcp-*.log has 0 sse_event_received after 20:40Z. Cause: subscribeEvents (mcp/server.ts) reconnects only on res 'end' / req 'error'; res 'error'/'close'/'aborted' (what a killed or process.exit-ed daemon produces) only logs to stderr — no reconnect, no heartbeat. Claude Code does not respawn the stdio server. Consequence: a working session no longer sees peer messages until its next Stop/prompt hook fires, and now-priority interrupts are lost until then.

Fix (unchanged): reconnect on every termination event + SSE heartbeat/idle-timeout + daemon shutdown() ends SSE responses before exit + replay unread on resubscribe + log reconnects to mcp-<pid>.log. Regression test: kill -9 daemon under an active SSE client → resubscribed within 5 s of the new daemon accepting → a DM sent afterwards is channel_emitted (not only hook-drained).

Addendum 5 (claudemesh repo session, 21:50Z) — verification of Addenda 1-4 + two more root causes + final design

Verified against code + logs (broker a9b735a = HEAD, daemon 1.37.0): Addendum 1's hypothesis (stale-presence sweeper flipping rows / pongs not refreshing lastPingAt) is NOT what happened — the pong handler bumps lastPingAt for the reattached presence id correctly. Addendum 2 (evict without closing the socket → ghost) and Addendum 3 (lost lifecycle handle) are confirmed and are the two visible mechanisms. Addendum 2's line "list_peers travels over the ghost session WS" is wrong (the session WS carries no RPCs) — Addendum 3 is the correct explanation for the empty list. Two more root causes found:

RC-A (broker) — why a HEALTHY reattached lease was evicted: async gap in ws.on("close")

index.ts close handler: if (conn.ws !== ws) return; → await savePeerState(...) → if (conn.leaseState === "online") enter grace else evict. The conn.ws === ws check happens BEFORE the await, the lease mutation AFTER. Under DB latency the handler of the dead daemon's socket resumes after the new daemon has already reattached (conn.ws swapped, leaseState = "online") and wrongly (a) starts grace on a lease whose new socket is alive, or (b) if a previous late handler already set offline, evicts immediately. Log proof: ws close — lease grace started for ePekV8/sUNqkMO9/BsheM0ey at 19:28:17.9–18.4, i.e. 0.85–1.2 s after session_hello reattach for the same pids, with no session_broker_ws_closed in daemon.log for those sockets; and 5 ws evict full at 19:28:16.4–16.9 with no lease grace expired (the "already offline" branch). Combined with Addendum 2 (no ws.close() on evict) the new socket becomes a zombie: OPEN on both ends, pings/pongs flow (the ws lib auto-pongs), no presence.

RC-B (broker, degradation) — why the broker was slow enough for the race + the 5 s hello timeouts

mesh.presence has 2,491,950 rows (2,488,046 with disconnected_at older than 30 d) and no index except the PK. sweepStalePresences (every 30 s), listPeersInMesh, handleHookSetStatus and the join in the fan-out paths full-scan 2.5 M rows; the postgres.js pool is the default 10. A restart storm (17 closes + 17 hellos + 9 timed-out re-hellos in ~10 s, each doing 4-8 queries) queued behind the scans → hello acks took 4–5 s (hello_sent 13.25 → hello_acked 17.08–18.32), which is what tripped the daemon's 5 s ack timeout and stretched the close handlers' awaits into the race window. Recurrence: guaranteed to worsen with N (≈ +1 row per session connect, never deleted). Fix = indexes + retention, not a bigger timeout.

Final design (what ships in 1.37.1 / broker deploy)

Broker

  1. Close handler: lease transition is synchronous (no await before it); savePeerState runs after, fire-and-forget; close code+reason logged. Eviction timer re-checks leaseState === "offline" before evicting (defensive).
  2. evictPresenceFully(pid, reason): if the socket is still OPEN → ws.close(4004, "presence_evicted") so the client reconnects and re-hellos. Reason logged.
  3. Zombie sweeper (in the 30 s ping loop): any wss.clients socket that is OPEN, older than 60 s and not referenced by a connections entry → close(4005, "no_presence").
  4. Dedup scoped by role: member hello dedup only kicks control-plane presences; session_hello dedup only kicks session presences (fixes the mutual-kick chain of Addendum 2).
  5. sweepOrphanMessages rewritten with the query builder (fixes the Date-param DrizzleQueryError; the hourly sweep has been failing).
  6. Presence retention: hourly + at startup, batched DELETE FROM mesh.presence WHERE disconnected_at < now() - 30 d (20k/batch). Migration 0030_presence_indexes.sql: partial indexes on (member_id) WHERE disconnected_at IS NULL, (last_ping_at) WHERE disconnected_at IS NULL, plus (disconnected_at) for the retention delete. Hello handlers log latency_ms. Daemon / CLI
  7. ws-lifecycle.ts: createWsLifecycle() returns the handle synchronously; ready resolves on the first ack; a hello-ack timeout only forces a backoff reconnect (never rejects/orphans). close() is always effective. Ack timeout default 5 s → 15 s. Both clients store the handle before awaiting ready. Invariant status==="open" ⇒ lifecycle holds by construction.
  8. drain.ts: drain_session_ws_not_ready logged on attempt 1 and every 60th; not-ready path counts against the attempt cap; rows whose session pubkey has no registered session for >10 min → dead session_gone; any pending row older than 24 h → dead expired.
  9. Rotating log sink (daemon/log-sink.ts): when daemon up --foreground runs without a TTY (launchd/systemd), stdout/stderr are redirected to daemon.log with a 20 MB cap × 5 files and a total-budget purge; the service unit's StandardOut/Err become daemon.launchd.log (pre-init crashes only). Existing oversized log is rotated on first start (incident excerpts preserved in .artifacts/test-runs/).
  10. Forensics: uncaughtException / unhandledRejection / exit / SIGHUP / SIGTERM all write a daemon_exit/daemon_uncaught line synchronously before exit; plist gets ThrottleInterval 10.
  11. MCP SSE (mcp/server.ts subscribeEvents): reconnect on res close/aborted/error and req close, idle watchdog (daemon sends : keepalive every 15 s → no bytes for 45 s ⇒ destroy + reconnect), reconnects logged to mcp-<pid>.log; daemon shutdown() ends open SSE responses before exit.
  12. Operator signal: GET /v1/diagnostics (per-mesh member WS status + per-session WS status + log size); claudemesh daemon status prints it; claudemesh doctor adds: daemon version = CLI version, every member WS open, every registered session's WS open, every registered session present in the broker's peer list (the ghost detector), daemon.log size + stale service unit. Each with the one-command fix (claudemesh daemon restart / claudemesh daemon install-service). peer list prints a warning when the daemon's member WS for that mesh is not open instead of a silent empty list. Tests: vitest for the lifecycle (first ack times out, second succeeds → same handle, listPeers returns the broker's list; close() before ack stops retries), drain expiry/rate-limit, log rotation, broker pure helpers (close-action decision, role-scoped dedup, zombie selection, orphan-sweep query shape). Release/deploy: CLI 1.37.1 (npm + cli-v1.37.1 tag) and broker via git push gitea-vps main → Coolify. Broker deploy first (the new close codes are harmless to old clients), then CLI upgrade + claudemesh daemon restart here, then the SIGKILL drill: kill -9 <daemon pid> → launchd respawn → peer list --mesh flexicar shows every live session within 10 s without relaunching any.

Addendum 6 (claudemesh repo session, 22:15Z) — local reproduction results + RC-C (the actual eviction trigger) + two-daemon hazard

Reproduction harness (all local, nothing touched in prod): new broker (bun src/index.ts) on :7955 against a throwaway pgvector Postgres (docker cmh-test-pg, migrations 0000–0030 applied by the runtime runner); isolated CLI 1.37.1 daemon (CLAUDEMESH_CONFIG_DIR/DAEMON_DIR/SESSIONS_DIR=/tmp/cmh-drill/*, --foreground --no-tcp, no TTY → rotating sink active); two fake sessions registered over IPC with live sleep pids and real session keypairs + parent attestations.

Drill 1 — SIGKILL + respawn (the incident shape): sessions visible in peer list at t+1 s after respawn; broker: ws close 1006 → enter_grace ×3, then session_hello reattach was:offline ×2; daemon: sessions_rehydrated revived=2. No eviction. PASS.

Drill 2 exposed RC-C. A minimal 20-line repro (two session_hello clients, same pubkey) against the local broker shows the old socket's close event logged BEFORE the session_hello reattach line, with action: enter_grace. Verified with an isolated script: under Bun (the production runtime) the ws shim emits the server-side close event SYNCHRONOUSLY inside ws.close(); Node emits it on a later tick. Consequence in the reattach path (oldConn.ws.close(1000,"session_replaced") → oldConn.ws = ws): the close handler runs while conn.ws is still the old socket → sees an online lease → starts grace + arms a 90 s eviction timer → the swap then sets the lease online again but the timer stays armed (the reattach's clearTimeout ran before the close) → 90 s later the healthy, reattached lease is evicted, socket left open → ghost. In the pre-fix handler the same sequence ran with an extra await savePeerState in between, which is why prod logged lease grace started 0.85–1.2 s after reattach (DB latency) and lease grace expired — evicting ~90 s later for exactly the reattached pids (19:29:47–50). This is the primary trigger of the blackout; RC-A (async gap) is the second way the same handler could misfire. Fix: swap conn.ws/lease state BEFORE closing the replaced socket, clear timers AFTER; dedup paths dropConnection() before close(); the grace timer re-checks leaseState before evicting (this guard alone already prevented the eviction in drill 2 — logged lease grace timer fired on online lease — skipped). Pure-logic tests in apps/broker/tests/lease.test.ts.

Drill 2 also exposed a two-daemon hazard (CLI): services/daemon/lifecycle.ts probeDaemon() treated ANY probe failure (incl. a 2.5 s timeout against a busy-but-live daemon) as "stale" and cleanupStaleFiles() unlinked the live daemon's daemon.sock + daemon.pid. The daemon kept running unreachable; the next CLI call auto-spawned a second daemon; both rehydrated the same sessions and kicked each other's sockets with session_replaced every second (20 hellos/s on the broker), forever. Fixes: (a) a probe failure is "stale" only on ECONNREFUSED/ENOENT or a dead pid — otherwise "busy" (files intact, no spawn); (b) the daemon probes the socket for another live daemon before taking the lock, and re-asserts its pid file every 30 s (daemon_pidfile_restored / daemon_pidfile_conflict); (c) the WS lifecycle treats 1000 session_replaced as terminal (whoever replaced us owns the identity; no reconnect fight), surfaced as replaced: true in /v1/diagnostics.

Drill 3 (final build, local broker restarted on the reordered code): (a) two session_hello clients with the same pubkey → the replaced socket's close is now logged code 1000 session_replaced action: ignore_replaced, the reattach reads was: online, no grace timer armed; (b) with daemon.pid deleted by hand, a second daemon up is refused: daemon already running (pid 36531, answering on …/daemon.sock) + daemon_exit code 1 forensics line; (c) SIGKILL + respawn → both sessions visible at t+1 s; (d) the pid-file guard rewrote a deleted pid file within 30 s (daemon_pidfile_restored); (e) broker evictions over the whole drill: 0. Note for future drills: the broker integration suite's cleanupAllTestMeshes() deletes every mesh owned by the test user — including a drill mesh created with setupTestMesh — so run the suite and the drill against different databases (the "1008 unauthorized" seen mid-drill was that, not a product bug).

Addendum 5 — recurrence after the CLEAN restart (22:05Z): reattached presences die, fresh ones survive

Clean claudemesh daemon down at 20:40:31Z → new daemon 72349, sessions_rehydrated 8/8, all 8 session WS hello_acked at 20:40:33Z, peer list = 7 at 20:41Z. By 22:05Z peer list = 4 and mesh.presence online for flexicar = exactly the 4 presences whose connected_at = 20:40:33 (fresh session_hello rows: 9f0e1c09, 02b5b7d5, e051e316, 59d1cfb1) + control-plane. The 3 that vanished (612fa2bc, 0fdf7159, aee93b3b) were the ones the new daemon reattached to pre-existing rows (connected_at 19:28:17 / 20:10:43, last_ping_at frozen at 20:40:33). Neither daemon.log (new file) nor the broker log of the last 20 min shows a close/evict/reattach line for them — they were evicted silently between 20:45Z and 21:45Z, and the daemon still believes their WS is open (no session_broker_ws_closed). Reattach is the broken path: after a reattach the presence stops being pinged (last_ping_at frozen) → the DB stale-presence sweeper flips it → list_peers drops it, and later eviction/close never reaches the daemon. Fresh hellos are healthy. Test to add: reattach within grace → assert last_ping_at keeps advancing and the row stays online for ≥10 min.

Addendum 7 (claudemesh repo session, 22:20Z) — the 20:40Z recurrence IS RC-A/RC-C; broker log + DB evidence; one more mitigation

The "Addendum 5 — recurrence after the CLEAN restart" reading ("reattach stops pinging, evicted silently, no close/evict lines") is corrected by the broker log and mesh.presence rows for the same window:

UTC broker log (mcn8m74…) presences
20:40:33.086–.381 session_hello reattach … was:"online" ×4 OsMWiJPO 612f, JVuLGue1 aee9, fB5eIuRx 0fdf, JOk0n5TA a8d5 (4 reattached, not 3)
20:40:35.699–.810 ws close — lease grace started ×4 (same pids) the DEAD daemon's socket close handler, resumed after await savePeerState (2.5 s of DB latency), lands on the reattached lease = RC-A/RC-C
20:42:05.699–.893 lease grace expired — evicting + ws evict full ×4 disconnected_at = 20:42:05.700/.738/.800/.832 — exact match
20:41:59.9 ws stale terminate ×9 (last_pong_ago_ms 89.9 s) for daemon-94504's control-plane sockets the "clean" shutdown never flushed its close frames

last_ping_at froze at 20:40:33 (= restorePresence at reattach) because the ping loop skips leases in leaseState === "offline" — the lease had been wrongly moved offline at 20:40:35; the freeze is a symptom, not a separate ping bug. The 4 fresh rows kept pinging (last_ping_at 22:07:01 at query time) because nothing touched their lease. So the recurrence is the same mechanism already fixed (Addendum 6 fix list); no reattach-specific ping path exists or is needed.

Why the reattach was was:"online" after a CLEAN restart (new finding): shutdown() awaited b.close() per client sequentially and then process.exit(0); the WS close frames were still unflushed when the process died, so the broker saw no close, kept all 17 leases "online" until its 75 s stale-pong watchdog terminated them (20:41:59), and the new daemon's hellos 1 s after start reattached onto "online" leases → the RC-C path (close of the replaced socket) instead of the safe was:"offline" path. Mitigation shipped in 1.37.1: WsLifecycle.close() now waits (≤1 s) for the socket's close event, and shutdown() closes all clients in parallel before exiting → the broker receives the closes immediately, starts grace, and every restart reattaches onto an offline lease — the safe path even against a pre-fix broker.

Tests added for the suggested scenario: apps/broker/tests/lease.test.ts "reattachLease" — the ordering contract extracted into lease.ts#reattachLease is exercised with a close callback that fires synchronously (Bun emulation): the handler sees ignore_replaced, and even a mis-behaving handler that arms a grace timer leaves the lease online with no timer. apps/cli/tests/unit/ws-lifecycle.test.ts — close() resolves only after the fake broker observed the close frame. A live assertion ("last_ping_at advances ≥10 min after a reattach") belongs to the post-deploy verification on this Mac (SIGKILL + clean-restart drills, then SELECT last_ping_at for the reattached rows after 10 min).