fix(runtime-state): govern redis connections

docs/operations/redis-runtime-runbook.md

@@ -0,0 +1,67 @@
+# Runtime Redis Operations Runbook
+
+This runbook covers Aether runtime Redis connection pressure incidents. It is
+not a substitute for fixing application-level connection churn.
+
+## Normal Expectations
+
+- Each `RuntimeState` Redis backend initializes a fixed set of long-lived
+  connection lanes: fast, stream, blocking stream, and admin.
+- `connected_clients` should stay near a small fixed number per app instance,
+  plus health checks and ad hoc admin clients.
+- `total_connections_received` should not grow linearly with request volume.
+- Large TIME_WAIT spikes between app and Redis indicate a regression or a
+  separate process repeatedly opening Redis connections.
+
+## Emergency Mitigation
+
+1. Disable the retry source first, such as expired Codex/OAuth keys causing a
+   retry storm.
+2. Restart the app to stop continued connection creation:
+
+   ```sh
+   docker compose restart app
+   ```
+
+3. On a Linux host, temporarily widen the ephemeral port range and enable safe
+   TIME_WAIT reuse:
+
+   ```sh
+   sudo sysctl -w net.ipv4.ip_local_port_range="10000 65535"
+   sudo sysctl -w net.ipv4.tcp_tw_reuse=1
+   ```
+
+4. Do not enable `tcp_tw_recycle`; it is obsolete and unsafe with NAT.
+
+Docker Desktop on macOS runs containers inside a Linux VM. Host-level macOS
+`sysctl` changes do not necessarily affect the VM network namespace.
+
+## Checks
+
+Use Redis `INFO clients` and `INFO stats` to inspect:
+
+- `connected_clients`
+- `total_connections_received`
+
+Use OS socket tooling on the Redis host or container namespace to inspect
+TIME_WAIT counts. Persistent growth after the runtime Redis refactor means a
+different code path or process is still opening short-lived Redis connections.
+
+## File Descriptor Limits
+
+Aether's compose files intentionally do not set container `ulimits.nofile`.
+Redis connection churn must be fixed in application code, not hidden by larger
+file descriptor limits.
+
+For high-concurrency production hosts, set file descriptor policy at the
+runtime or service-manager layer instead:
+
+- Docker daemon default ulimit, for example `default-ulimits` in
+  `/etc/docker/daemon.json`.
+- systemd service limits such as `LimitNOFILE=` for Docker or the process
+  supervisor.
+- Managed container platform resource settings, when Docker daemon settings are
+  not available.
+
+Keep Redis `maxclients` below the effective Redis process `nofile` limit with
+room for persistence files, replicas, and admin connections.