Alejandro Gutiérrez dd351a7846 fix(cli): daemon restart no longer blacks out the mesh (1.37.1) ...

Daemon side of the 2026-09-08 incident (spec Addenda 3, 5, 6):

- ws-lifecycle: createWsLifecycle() returns the handle synchronously with
  ready exposed separately. A first hello-ack timeout used to REJECT the
  connect promise while the helper kept reconnecting on its own, so
  DaemonBrokerClient.lifecycle stayed null forever and every RPC gated on
  it (listPeers, send, ...) short-circuited: peer list answered [] on all 9
  meshes for as long as the daemon lived. Ack timeout 5 s → 15 s (a timeout
  is now just a reconnect). 1000 session_replaced is terminal for a
  lifecycle (two daemons no longer fight over the same sessions).
- drain: 24 h outbox TTL (expired), session_gone after 10 min, attempt cap on
  the not-ready path, drain_session_ws_not_ready logged on attempt 1 and
  every 60th (was every tick: 4 rows × ~130 k attempts = a 1 GB log).
- Rotating log sink (20 MB × 5, total budget) behind an explicit --log-sink
  flag that the launchd/systemd unit and the detached spawn pass; units
  point StandardOut/Err at daemon.launchd.log and pin ThrottleInterval 10.
- Exit forensics: daemon_exit / daemon_uncaught on every exit path; SIGHUP
  is a clean shutdown.
- Singleton hardening: the CLI stale-probe no longer unlinks a live
  daemon's pid/socket on a slow probe (only ECONNREFUSED/ENOENT or a dead
  pid is stale); the daemon probes the socket for another live daemon
  before taking the lock and re-asserts its pid file every 30 s.
- MCP SSE: resubscribe on every termination event with a 45 s idle
  watchdog; the daemon ends open SSE streams before exiting.
- Operator signal: GET /v1/diagnostics; daemon status shows per-mesh and
  per-session WS state; new daemon restart; doctor adds version match,
  member/session WS open, ghost detector (every registered session must
  appear in the broker peer list), log size + stale unit; peer list warns
  when the daemon is not connected to the broker for a mesh.

Tests: ws-lifecycle (timeout → same handle, close mid-connect,
session_replaced terminal), daemon-broker-lost-handle (Addendum 3
regression), drain-expiry, log-sink, sse-resubscribe (socket destroyed
without end), daemon-probe-classify. Local drills against the fixed broker:
SIGKILL + respawn → sessions visible at t+1 s, second daemon refused,
pid file self-healed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-08 22:22:58 +01:00

4.1 KiB

Raw Permalink Blame History

View Raw