The 20:40Z clean-restart recurrence (spec Addendum 7) is the same RC-A/RC-C mechanism, with prod evidence: 4 reattached presences (was:"online") hit "lease grace started" 2.5 s later and were evicted at 20:42:05 — exactly their disconnected_at. last_ping_at froze because the ping loop skips offline leases; the freeze is a symptom, not a separate ping bug. - broker: the swap-before-close ordering now lives in lease.ts#reattachLease and is unit-tested with a close callback that fires synchronously (Bun emulation): the handler sees ignore_replaced; a mis-behaving handler that arms a grace timer still leaves the lease online with no timer. - cli: WsLifecycle.close() waits (<=1 s) for the socket's close event and shutdown() closes all broker clients in parallel, so a clean restart's close frames reach the broker before process.exit. Prod showed the old daemon's 17 sockets were only reaped by the 75 s stale-pong watchdog, which is why the next daemon reattached onto "online" leases (the RC-C path) instead of "offline" ones. Test: close() resolves only after the fake broker observed the close. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
31 KiB
2026-09-08 — Daemon restart → mesh blackout (peer list empty, sessions "invisible")
Status: diagnosed on a live incident, fix pending. Owner: claudemesh repo session.
Reporter: flexicar-orchestrator session (evidence gathered read-only from ~/.claudemesh/daemon/daemon.log, live CLI probes, and source on main @ 1.37.0).
Symptom (user-visible)
After the local daemon restarted (launchd com.claudemesh.daemon KeepAlive), claudemesh peer list returned 0 peers on EVERY mesh (flexicar, minery, legalrag, nedas-mesh…) for >40 min, although all 8 Claude sessions were alive. Sends by session pubkey reported queued — no connected peer matches this key. The whole orchestration went dark. User verdict: "in case of daemon restart, sessions should be reconnected — this must not happen."
Timeline (UTC; local = UTC+1)
- 2026-09-05 20:56:59Z — daemon pid 85205 started (ran ~3 days).
- 2026-09-08 ~19:27–19:28Z — pid 85205 died without logging a single line (no exit/uncaught/SIGTERM in daemon.log, no crash report in
~/Library/Logs/DiagnosticReports, no memorystatus kill inlog show). launchd respawned: pid 93782 at 19:28:07 (died within 6 s — crash-loop once), pid 94504 at 19:28:13 (stable, still running). - 19:28:07 and 19:28:13 —
sessions_rehydrated revived=8 persisted=8→ rehydration WORKS: registry restored fromsessions.json, keypairs re-derived,onRegistercreated aSessionBrokerClientper session. - 19:28:12–19:28:20 — daemon-level mesh WS:
broker_ws_hello_sent→broker_hello_ack_timeout(all 9 meshes, 5 s) →broker_ws_closed 1005→broker_ws_reconnect_scheduled→ reconnect →broker_ws_hello_ackedat 19:28:20. - 19:28:19+ — every per-session WS:
session_broker_ws_open_attempt/open_ok/hello_ackedfor ALL session pubkeys (02b5b7d5, 0fdf7159, 612fa2bc, 9f0e1c09, aee93b3b, e051e316, f34b4b9f, …). Some flapped (aee93b3b ×4 attempts, 2 closed). - 20:09 / 20:18Z —
drain_max_attempts err=broker_not_openfor 2 outbox rows (a session WS was NOT open at drain time despite hello_acked earlier → flapping). - 20:09:01 / 20:10:08Z — two sessions relaunched by the user →
session_registeredvia IPC → hello_acked. Still 0 peers. - Broker (
ic.claudemesh.com):/healthok, uptime 5.34 M s (~62 d, NOT redeployed), db up.
What IS working after the restart
- Registry rehydration (8/8), keypair re-derivation, attestation re-mint.
- Per-session broker WS: open + hello_acked for every session (so the broker's in-memory conn map has them, lease
online). - Inbound pushes:
claudemesh inboxstill lists prior messages;/v1/eventsSSE reconnects by itself (mcp/server.ts).
What is BROKEN
list_peers→ empty on all meshes. Brokerlist_peershandler =listPeersInMesh(meshId)= DB querypresence WHERE disconnectedAt IS NULL. Empty ⇒ the DB presence rows are marked disconnected while the in-memory lease isonline→ split-brain between in-memory conn map and the presence table. Hypothesis (p≈0.8): the old daemon died without close frames → broker WS closes → lease grace (90 s) → new daemon reattached within grace (session_hello reattach,restorePresence(pid)fire-and-forgetvoid) → later the DB-level stale-presence sweeper (keyed onlastPingAt) flips the rows back todisconnectedAt = nowbecause pongs on the reattached WS don't refreshlastPingAtfor the reattached presence id — orrestorePresencelost a race with the sweeper. Either wayrestorePresenceis a one-shot patch on a row the sweeper keeps re-flipping. Needs confirmation from broker logs (ws hello reattach (lease),session_hello reattach, sweeper output) andSELECT id,session_pubkey,disconnected_at,last_ping_at FROM presence WHERE …on the broker DB (OVH via Coolify; SSH needs Tailscale re-auth right now).- daemon.log = 1.0 GB (disk at 95 %).
drain.ts:186logsdrain_session_ws_not_readyon EVERY drain tick for EVERY outbox row whose session WS isn't open — 117 797 lines in the last 20 MB, mostly for stale session pubkeys from previous days (4e00fc666559 ×70 676, cb26f393232e ×15 708 …) that will never reconnect. Rows are backed off but never expired; the log grows unbounded (launchdStandardOutPath→ no rotation). This is the most plausible cause of the silent death of pid 85205 (I/O or disk pressure) — unproven. - Session WS flapping after rehydration (multiple open/close cycles per session,
broker_not_openat drain time). Likely the same-pubkey ghost handling on the broker (session_replacedclose of the older socket) racing the reconnect helper. Needs broker logs. - Recovery UX: nothing tells the user the mesh is in this state;
peer listjust says "no peers". A relaunch of a session does NOT fix it (proven twice today).
Work order (spec-first, then code, then tests, then release)
- Reproduce locally: broker + daemon in a test harness: kill daemon with SIGKILL → launchd-style respawn → assert
peer listnon-empty within 10 s for rehydrated sessions; assert presence rowsdisconnectedAt IS NULLafter reattach AND after one stale-sweep cycle. - Broker fix: make the reattach path authoritative for the DB row:
restorePresenceawaited + sweeper must not flip a row whose in-memory lease isonline(or the pong handler must bumplastPingAtfor reattached presences). Add a self-heal: onlist_peers, reconcile DB rows against the in-memoryconnectionsmap (online lease ⇒ include). Log atwarnwhen DB/in-memory disagree. - Daemon fix (drain.ts): rate-limit
drain_session_ws_not_ready(log once per row per N minutes), expire outbox rows for session pubkeys with no registry entry aftermax(attempts)/TTL, and rotate daemon.log (size cap + keep N) — launchd writes stdout directly, so rotate in-process or vianewsyslog/logrotate-style config installed bydaemon install-service. - Daemon crash forensics: install an
uncaughtException/unhandledRejection+SIGTERMlogger with flush, and adaemon_exitline, so the next silent death has a cause. - Operator signal:
claudemesh doctor(ordaemon status) must detect "sessions registered + WS hello_acked but list_peers empty" and say so, with the one-command fix. - Release as 1.37.1 (cli) + broker deploy; verify on this Mac:
claudemesh peer list --mesh flexicarshows the 8 sessions without relaunching any of them.
Evidence pointers
~/.claudemesh/daemon/daemon.log(1 GB — grep, don't cat):sessions_rehydrated,broker_hello_ack_timeout,session_broker_ws_*,drain_max_attempts,drain_session_ws_not_ready.~/.claudemesh/daemon/sessions.json(persisted registry).- Broker source:
apps/broker/src/index.tsL1925-1960 (member reattach), L2225-2262 (session reattach),apps/broker/src/broker.tsL431-445restorePresence, L448+listPeersInMesh, stale sweeper (staleTimer). - Daemon source:
apps/cli/src/daemon/run.tsL212-300 (onRegister), L312-384 (rehydrate),apps/cli/src/daemon/drain.tsL182-200,apps/cli/src/daemon/session-registry.ts(persist/reaper).
Addendum — outbox state at 20:35Z (claudemesh daemon outbox list)
pendingrows with attempts=120 038 … 132 380,last_error="session_ws_not_open"→ the drain loop has retried the same 4 rows ~130 k times (≈ once per second for days) for session pubkeys that no longer exist. This IS the 1 GB log. Rows must expire.deadrowsmax_attempts: no connected peer for target (not online…)at attempts=25 → the broker rejected delivery because its DB says the target is offline, while the target's WS is hello_acked → confirms the presence split-brain (item 1). Consequence: after a daemon restart, DMs to live sessions are refused, not just discovery.- A fresh
sendnow returnsrecipientOnline:false, status:queuedfor a session whose WS hello_acked minutes earlier.
Addendum 2 — broker-side evidence (OVH, container mcn8m74tbxfxbplmyb40b2ia-…, image = commit a9b735a; DB schema mesh.*)
Broker log 19:28:05–19:31Z (counts): 30 ws close — lease grace started · 27 ws evict full · 22 lease grace expired — evicting · 12 session_hello reattach · 9 hello dedup · 5 ws session_hello · 27 ws hello. Two daemons (pid 93782 lived 6 s, then 94504) both rehydrated 8 sessions and raced the broker within the same 90 s grace window while the broker was still processing the dead daemon's socket closes. Mesh-level hellos of 94504 hit broker_hello_ack_timeout (5 s) for all 9 meshes → broker_ws_closed 1005 → reconnect → acked at 19:28:20 (broker was busy with the storm).
Final broker DB state for flexicar (mesh.presence, disconnected_at IS NULL): online = aee93b3b (intranet 2ad05da8), 8494e7e4 (cliente-front relaunched), 612fa2bc (crm-api), 0fdf7159 (intranet f8b797d1), + control-plane daemon-94504. GHOSTS (evicted on the broker, socket still open on the daemon, no session_broker_ws_closed in daemon.log): 9f0e1c09 (orchestrator), 02b5b7d5 (crm-front), e051e316 (cliente-back), f34b4b9f (old cliente-front). Their presences (sUNqkMO9, ePekV8WH, BsheM0ey, Qxauubu7) got ws close — lease grace started at 19:28:17.9–18.3 (1 s AFTER session_hello reattach by 94504 at 19:28:17.0–17.2) and were lease grace expired — evicting at 19:29:43–50. evictPresenceFully does NOT ws.close() the socket it evicts → the daemon keeps a half-open WS forever, never reconnects → session unreachable and invisible, list_peers/DMs from that session go into a void.
Why peer list showed 0 on EVERY mesh from the orchestrator: the CLI inside a launched session is session-scoped (token env) → list_peers travels over the ghost session WS → no answer/empty. The member-scope check is in the next addendum.
hello dedup design flaw: the launch wrapper's member-level hello carries session_id = <claude session UUID>; the broker's session-id dedup then KICKS the live session_hello presence with the same UUID (hello dedup old_presence=a5eDg0aj), the daemon's session_hello then kicks the wrapper's presence (session_hello dedup) — 2 evictions + 2 DB disconnects + a forced WS close per relaunch (seen at 20:10:04–20:10:44 for intranet: a5eDg0aj → FYL5tnMl → MgXl7se3 → CBj9nWzr → JVuLGue1). Control-plane and session presences must not dedup each other by session_id.
Also seen: hourly orphan msg sweep throws DrizzleQueryError … TypeError: The "string" argument must be of type string … Received an instance of Date — the sweep is broken (Date passed where a string is expected).
Fix list (broker): (1) evictPresenceFully must ws.close(1000,"evicted") so the client reconnects; (2) a server-side liveness probe: if a socket is open but its presence is gone, close it; (3) hello dedup must be scoped by role (control-plane vs session); (4) mesh hello_ack must not time out under load (ack before heavy work, or raise/adapt the client timeout); (5) fix the orphan-sweep Date bug. Fix list (daemon): (6) SessionBrokerClient: detect a dead-but-open WS (missing server pings / no hello_ack refresh) and reconnect; (7) after rehydration, verify presence with the broker (sessions/me or a whoami ping) and re-hello if absent; (8) never run two daemons within the same grace window (launchd ThrottleInterval / startup lock) — the 6 s crash-loop doubled the storm.
Addendum 3 — ROOT CAUSE of "peer list empty on every mesh" (daemon side, confirmed in code + log)
apps/cli/src/daemon/broker.ts connect():
this.lifecycle = await connectWsWithBackoff({ ..., onStatusChange: (s) => { this._status = s; ... } });
On 2026-09-08 19:28:17.889Z the first hello_ack timed out (5 s; broker busy with the storm) → connectWsWithBackoff rejected → this.lifecycle was never assigned (broker connect failed for flexicar: Error: hello_ack_timeout, ×9 meshes). But the helper's internal reconnect loop kept running on the same object: broker_status reconnecting → connecting → open at 19:28:20.187Z and onStatusChange set _status = "open". Net state: _status === "open" AND lifecycle === undefined → every method gated on this.lifecycle short-circuits:
listPeers():if (this._status !== "open" || !this.lifecycle) return [];→/v1/peersanswers{"peers":[]}in 30 ms (measured; a real broker round trip is ≥200 ms) →peer list= 0 on all 9 meshes, forever, without any warning.isOpen()readsthis.lifecycle?.ws→ false →send()no-ops → outbox drain logsdrain_max_attempts err=broker_not_openfor member-keyed rows.listSkills,getState, etc. → same. The control-plane presence (daemon-94504) IS online on the broker (DB rowXqS1ExHG, disconnected_at NULL), so the WS is real — the daemon just lost its handle to it.
Fix: connectWsWithBackoff must hand back the lifecycle handle synchronously (before the first ack) and expose readiness separately (await lifecycle.ready), so a first-ack timeout cannot orphan a live reconnecting socket; connect() must not reject while the helper is still retrying; add an invariant check (_status === "open" && !lifecycle → log error + self-heal by re-running connect). Regression test: simulate hello_ack timeout on first attempt + ack on the second → listPeers() must return the broker's list.
Two daemons in 6 s: launchd respawned pid 93782 which died within 6 s (cause unknown — no daemon_shutdown, no crash report; the clean shutdown() handler only logs on SIGINT/SIGTERM) → doubled hellos/reattaches/dedups in the same 90 s grace window. Needs daemon_exit/uncaughtException logging and a launchd ThrottleInterval.
Addendum 4 (CORRECTED 20:55Z) — after a daemon restart, delivery INTO Claude Code survives via the hook fallback; the live SSE push path does NOT
Wrong claim retracted: "no session receives pushes". Verified end-to-end after the clean restart: DM req:ping-3 (20:41:11Z) → crm-api woke up with the injected prompt [claudemesh] peer message arrived while idle - surface new inbox message(s), read its inbox and replied ack:pong3 (20:41:39Z, in the orchestrator's inbox). So messages reach sessions after a restart — through the local hook layer (~/.claude/hooks/claudemesh-wake-watcher.sh for idle sessions, claudemesh-inbox-drain.sh on Stop/UserPromptSubmit, claudemesh hook idle|working), which polls the daemon via CLI/IPC on each tick and therefore survives daemon restarts. Latency = the watcher's poll interval, not immediate.
What is dead after the restart: the MCP plugin's SSE subscription (/v1/events) — the path that injects <channel> blocks mid-turn into a working session. Evidence: lsof shows none of the 10 live claudemesh mcp processes connected to the daemon (neither UDS daemon.sock nor TCP 127.0.0.1:47823), and mcp-*.log has 0 sse_event_received after 20:40Z. Cause: subscribeEvents (mcp/server.ts) reconnects only on res 'end' / req 'error'; res 'error'/'close'/'aborted' (what a killed or process.exit-ed daemon produces) only logs to stderr — no reconnect, no heartbeat. Claude Code does not respawn the stdio server. Consequence: a working session no longer sees peer messages until its next Stop/prompt hook fires, and now-priority interrupts are lost until then.
Fix (unchanged): reconnect on every termination event + SSE heartbeat/idle-timeout + daemon shutdown() ends SSE responses before exit + replay unread on resubscribe + log reconnects to mcp-<pid>.log. Regression test: kill -9 daemon under an active SSE client → resubscribed within 5 s of the new daemon accepting → a DM sent afterwards is channel_emitted (not only hook-drained).
Addendum 5 (claudemesh repo session, 21:50Z) — verification of Addenda 1-4 + two more root causes + final design
Verified against code + logs (broker a9b735a = HEAD, daemon 1.37.0): Addendum 1's hypothesis (stale-presence sweeper flipping rows / pongs not refreshing lastPingAt) is NOT what happened — the pong handler bumps lastPingAt for the reattached presence id correctly. Addendum 2 (evict without closing the socket → ghost) and Addendum 3 (lost lifecycle handle) are confirmed and are the two visible mechanisms. Addendum 2's line "list_peers travels over the ghost session WS" is wrong (the session WS carries no RPCs) — Addendum 3 is the correct explanation for the empty list. Two more root causes found:
RC-A (broker) — why a HEALTHY reattached lease was evicted: async gap in ws.on("close")
index.ts close handler: if (conn.ws !== ws) return; → await savePeerState(...) → if (conn.leaseState === "online") enter grace else evict. The conn.ws === ws check happens BEFORE the await, the lease mutation AFTER. Under DB latency the handler of the dead daemon's socket resumes after the new daemon has already reattached (conn.ws swapped, leaseState = "online") and wrongly (a) starts grace on a lease whose new socket is alive, or (b) if a previous late handler already set offline, evicts immediately. Log proof: ws close — lease grace started for ePekV8/sUNqkMO9/BsheM0ey at 19:28:17.9–18.4, i.e. 0.85–1.2 s after session_hello reattach for the same pids, with no session_broker_ws_closed in daemon.log for those sockets; and 5 ws evict full at 19:28:16.4–16.9 with no lease grace expired (the "already offline" branch). Combined with Addendum 2 (no ws.close() on evict) the new socket becomes a zombie: OPEN on both ends, pings/pongs flow (the ws lib auto-pongs), no presence.
RC-B (broker, degradation) — why the broker was slow enough for the race + the 5 s hello timeouts
mesh.presence has 2,491,950 rows (2,488,046 with disconnected_at older than 30 d) and no index except the PK. sweepStalePresences (every 30 s), listPeersInMesh, handleHookSetStatus and the join in the fan-out paths full-scan 2.5 M rows; the postgres.js pool is the default 10. A restart storm (17 closes + 17 hellos + 9 timed-out re-hellos in ~10 s, each doing 4-8 queries) queued behind the scans → hello acks took 4–5 s (hello_sent 13.25 → hello_acked 17.08–18.32), which is what tripped the daemon's 5 s ack timeout and stretched the close handlers' awaits into the race window. Recurrence: guaranteed to worsen with N (≈ +1 row per session connect, never deleted). Fix = indexes + retention, not a bigger timeout.
Final design (what ships in 1.37.1 / broker deploy)
Broker
- Close handler: lease transition is synchronous (no await before it);
savePeerStateruns after, fire-and-forget; close code+reason logged. Eviction timer re-checksleaseState === "offline"before evicting (defensive). evictPresenceFully(pid, reason): if the socket is still OPEN →ws.close(4004, "presence_evicted")so the client reconnects and re-hellos. Reason logged.- Zombie sweeper (in the 30 s ping loop): any
wss.clientssocket that is OPEN, older than 60 s and not referenced by aconnectionsentry →close(4005, "no_presence"). - Dedup scoped by role: member
hellodedup only kickscontrol-planepresences;session_hellodedup only kickssessionpresences (fixes the mutual-kick chain of Addendum 2). sweepOrphanMessagesrewritten with the query builder (fixes theDate-param DrizzleQueryError; the hourly sweep has been failing).- Presence retention: hourly + at startup, batched
DELETE FROM mesh.presence WHERE disconnected_at < now() - 30 d(20k/batch). Migration0030_presence_indexes.sql: partial indexes on(member_id) WHERE disconnected_at IS NULL,(last_ping_at) WHERE disconnected_at IS NULL, plus(disconnected_at)for the retention delete. Hello handlers loglatency_ms. Daemon / CLI ws-lifecycle.ts:createWsLifecycle()returns the handle synchronously;readyresolves on the first ack; a hello-ack timeout only forces a backoff reconnect (never rejects/orphans).close()is always effective. Ack timeout default 5 s → 15 s. Both clients store the handle before awaitingready. Invariantstatus==="open" ⇒ lifecycleholds by construction.drain.ts:drain_session_ws_not_readylogged on attempt 1 and every 60th; not-ready path counts against the attempt cap; rows whose session pubkey has no registered session for >10 min →dead session_gone; any pending row older than 24 h →dead expired.- Rotating log sink (
daemon/log-sink.ts): whendaemon up --foregroundruns without a TTY (launchd/systemd), stdout/stderr are redirected todaemon.logwith a 20 MB cap × 5 files and a total-budget purge; the service unit's StandardOut/Err becomedaemon.launchd.log(pre-init crashes only). Existing oversized log is rotated on first start (incident excerpts preserved in.artifacts/test-runs/). - Forensics:
uncaughtException/unhandledRejection/exit/SIGHUP/SIGTERMall write adaemon_exit/daemon_uncaughtline synchronously before exit; plist getsThrottleInterval 10. - MCP SSE (
mcp/server.ts subscribeEvents): reconnect onres close/aborted/errorandreq close, idle watchdog (daemon sends: keepaliveevery 15 s → no bytes for 45 s ⇒ destroy + reconnect), reconnects logged tomcp-<pid>.log; daemonshutdown()ends open SSE responses before exit. - Operator signal:
GET /v1/diagnostics(per-mesh member WS status + per-session WS status + log size);claudemesh daemon statusprints it;claudemesh doctoradds: daemon version = CLI version, every member WS open, every registered session's WS open, every registered session present in the broker's peer list (the ghost detector), daemon.log size + stale service unit. Each with the one-command fix (claudemesh daemon restart/claudemesh daemon install-service).peer listprints a warning when the daemon's member WS for that mesh is not open instead of a silent empty list. Tests: vitest for the lifecycle (first ack times out, second succeeds → same handle,listPeersreturns the broker's list;close()before ack stops retries), drain expiry/rate-limit, log rotation, broker pure helpers (close-action decision, role-scoped dedup, zombie selection, orphan-sweep query shape). Release/deploy: CLI 1.37.1 (npm +cli-v1.37.1tag) and broker viagit push gitea-vps main→ Coolify. Broker deploy first (the new close codes are harmless to old clients), then CLI upgrade +claudemesh daemon restarthere, then the SIGKILL drill:kill -9 <daemon pid>→ launchd respawn →peer list --mesh flexicarshows every live session within 10 s without relaunching any.
Addendum 6 (claudemesh repo session, 22:15Z) — local reproduction results + RC-C (the actual eviction trigger) + two-daemon hazard
Reproduction harness (all local, nothing touched in prod): new broker (bun src/index.ts) on :7955 against a throwaway pgvector Postgres (docker cmh-test-pg, migrations 0000–0030 applied by the runtime runner); isolated CLI 1.37.1 daemon (CLAUDEMESH_CONFIG_DIR/DAEMON_DIR/SESSIONS_DIR=/tmp/cmh-drill/*, --foreground --no-tcp, no TTY → rotating sink active); two fake sessions registered over IPC with live sleep pids and real session keypairs + parent attestations.
Drill 1 — SIGKILL + respawn (the incident shape): sessions visible in peer list at t+1 s after respawn; broker: ws close 1006 → enter_grace ×3, then session_hello reattach was:offline ×2; daemon: sessions_rehydrated revived=2. No eviction. PASS.
Drill 2 exposed RC-C. A minimal 20-line repro (two session_hello clients, same pubkey) against the local broker shows the old socket's close event logged BEFORE the session_hello reattach line, with action: enter_grace. Verified with an isolated script: under Bun (the production runtime) the ws shim emits the server-side close event SYNCHRONOUSLY inside ws.close(); Node emits it on a later tick. Consequence in the reattach path (oldConn.ws.close(1000,"session_replaced") → oldConn.ws = ws): the close handler runs while conn.ws is still the old socket → sees an online lease → starts grace + arms a 90 s eviction timer → the swap then sets the lease online again but the timer stays armed (the reattach's clearTimeout ran before the close) → 90 s later the healthy, reattached lease is evicted, socket left open → ghost. In the pre-fix handler the same sequence ran with an extra await savePeerState in between, which is why prod logged lease grace started 0.85–1.2 s after reattach (DB latency) and lease grace expired — evicting ~90 s later for exactly the reattached pids (19:29:47–50). This is the primary trigger of the blackout; RC-A (async gap) is the second way the same handler could misfire. Fix: swap conn.ws/lease state BEFORE closing the replaced socket, clear timers AFTER; dedup paths dropConnection() before close(); the grace timer re-checks leaseState before evicting (this guard alone already prevented the eviction in drill 2 — logged lease grace timer fired on online lease — skipped). Pure-logic tests in apps/broker/tests/lease.test.ts.
Drill 2 also exposed a two-daemon hazard (CLI): services/daemon/lifecycle.ts probeDaemon() treated ANY probe failure (incl. a 2.5 s timeout against a busy-but-live daemon) as "stale" and cleanupStaleFiles() unlinked the live daemon's daemon.sock + daemon.pid. The daemon kept running unreachable; the next CLI call auto-spawned a second daemon; both rehydrated the same sessions and kicked each other's sockets with session_replaced every second (20 hellos/s on the broker), forever. Fixes: (a) a probe failure is "stale" only on ECONNREFUSED/ENOENT or a dead pid — otherwise "busy" (files intact, no spawn); (b) the daemon probes the socket for another live daemon before taking the lock, and re-asserts its pid file every 30 s (daemon_pidfile_restored / daemon_pidfile_conflict); (c) the WS lifecycle treats 1000 session_replaced as terminal (whoever replaced us owns the identity; no reconnect fight), surfaced as replaced: true in /v1/diagnostics.
Drill 3 (final build, local broker restarted on the reordered code): (a) two session_hello clients with the same pubkey → the replaced socket's close is now logged code 1000 session_replaced action: ignore_replaced, the reattach reads was: online, no grace timer armed; (b) with daemon.pid deleted by hand, a second daemon up is refused: daemon already running (pid 36531, answering on …/daemon.sock) + daemon_exit code 1 forensics line; (c) SIGKILL + respawn → both sessions visible at t+1 s; (d) the pid-file guard rewrote a deleted pid file within 30 s (daemon_pidfile_restored); (e) broker evictions over the whole drill: 0. Note for future drills: the broker integration suite's cleanupAllTestMeshes() deletes every mesh owned by the test user — including a drill mesh created with setupTestMesh — so run the suite and the drill against different databases (the "1008 unauthorized" seen mid-drill was that, not a product bug).
Addendum 5 — recurrence after the CLEAN restart (22:05Z): reattached presences die, fresh ones survive
Clean claudemesh daemon down at 20:40:31Z → new daemon 72349, sessions_rehydrated 8/8, all 8 session WS hello_acked at 20:40:33Z, peer list = 7 at 20:41Z. By 22:05Z peer list = 4 and mesh.presence online for flexicar = exactly the 4 presences whose connected_at = 20:40:33 (fresh session_hello rows: 9f0e1c09, 02b5b7d5, e051e316, 59d1cfb1) + control-plane. The 3 that vanished (612fa2bc, 0fdf7159, aee93b3b) were the ones the new daemon reattached to pre-existing rows (connected_at 19:28:17 / 20:10:43, last_ping_at frozen at 20:40:33). Neither daemon.log (new file) nor the broker log of the last 20 min shows a close/evict/reattach line for them — they were evicted silently between 20:45Z and 21:45Z, and the daemon still believes their WS is open (no session_broker_ws_closed). Reattach is the broken path: after a reattach the presence stops being pinged (last_ping_at frozen) → the DB stale-presence sweeper flips it → list_peers drops it, and later eviction/close never reaches the daemon. Fresh hellos are healthy. Test to add: reattach within grace → assert last_ping_at keeps advancing and the row stays online for ≥10 min.
Addendum 7 (claudemesh repo session, 22:20Z) — the 20:40Z recurrence IS RC-A/RC-C; broker log + DB evidence; one more mitigation
The "Addendum 5 — recurrence after the CLEAN restart" reading ("reattach stops pinging, evicted silently, no close/evict lines") is corrected by the broker log and mesh.presence rows for the same window:
| UTC | broker log (mcn8m74…) |
presences |
|---|---|---|
| 20:40:33.086–.381 | session_hello reattach … was:"online" ×4 |
OsMWiJPO 612f, JVuLGue1 aee9, fB5eIuRx 0fdf, JOk0n5TA a8d5 (4 reattached, not 3) |
| 20:40:35.699–.810 | ws close — lease grace started ×4 (same pids) |
the DEAD daemon's socket close handler, resumed after await savePeerState (2.5 s of DB latency), lands on the reattached lease = RC-A/RC-C |
| 20:42:05.699–.893 | lease grace expired — evicting + ws evict full ×4 |
disconnected_at = 20:42:05.700/.738/.800/.832 — exact match |
| 20:41:59.9 | ws stale terminate ×9 (last_pong_ago_ms 89.9 s) for daemon-94504's control-plane sockets |
the "clean" shutdown never flushed its close frames |
last_ping_at froze at 20:40:33 (= restorePresence at reattach) because the ping loop skips leases in leaseState === "offline" — the lease had been wrongly moved offline at 20:40:35; the freeze is a symptom, not a separate ping bug. The 4 fresh rows kept pinging (last_ping_at 22:07:01 at query time) because nothing touched their lease. So the recurrence is the same mechanism already fixed (Addendum 6 fix list); no reattach-specific ping path exists or is needed.
Why the reattach was was:"online" after a CLEAN restart (new finding): shutdown() awaited b.close() per client sequentially and then process.exit(0); the WS close frames were still unflushed when the process died, so the broker saw no close, kept all 17 leases "online" until its 75 s stale-pong watchdog terminated them (20:41:59), and the new daemon's hellos 1 s after start reattached onto "online" leases → the RC-C path (close of the replaced socket) instead of the safe was:"offline" path. Mitigation shipped in 1.37.1: WsLifecycle.close() now waits (≤1 s) for the socket's close event, and shutdown() closes all clients in parallel before exiting → the broker receives the closes immediately, starts grace, and every restart reattaches onto an offline lease — the safe path even against a pre-fix broker.
Tests added for the suggested scenario: apps/broker/tests/lease.test.ts "reattachLease" — the ordering contract extracted into lease.ts#reattachLease is exercised with a close callback that fires synchronously (Bun emulation): the handler sees ignore_replaced, and even a mis-behaving handler that arms a grace timer leaves the lease online with no timer. apps/cli/tests/unit/ws-lifecycle.test.ts — close() resolves only after the fake broker observed the close frame. A live assertion ("last_ping_at advances ≥10 min after a reattach") belongs to the post-deploy verification on this Mac (SIGKILL + clean-restart drills, then SELECT last_ping_at for the reattached rows after 10 min).