Aether/docs/operations/redis-runtime-runbook.md

Runtime Redis Operations Runbook

This runbook covers Aether runtime Redis connection pressure incidents. It is not a substitute for fixing application-level connection churn.

Normal Expectations

• Each RuntimeState Redis backend initializes a fixed set of long-lived connection lanes: fast, stream, blocking stream, and admin.
• connected_clients should stay near a small fixed number per app instance, plus health checks and ad hoc admin clients.
• total_connections_received should not grow linearly with request volume.
• Large TIME_WAIT spikes between app and Redis indicate a regression or a separate process repeatedly opening Redis connections.

Emergency Mitigation

1. Disable the retry source first, such as expired Codex/OAuth keys causing a retry storm.

2. Restart the app to stop continued connection creation:

   docker compose restart app
   

3. On a Linux host, temporarily widen the ephemeral port range and enable safe TIME_WAIT reuse:

   sudo sysctl -w net.ipv4.ip_local_port_range="10000 65535"
   sudo sysctl -w net.ipv4.tcp_tw_reuse=1
   

4. Do not enable tcp_tw_recycle; it is obsolete and unsafe with NAT.

Docker Desktop on macOS runs containers inside a Linux VM. Host-level macOS sysctl changes do not necessarily affect the VM network namespace.

Checks

Use Redis INFO clients and INFO stats to inspect:

• connected_clients
• total_connections_received

Use OS socket tooling on the Redis host or container namespace to inspect TIME_WAIT counts. Persistent growth after the runtime Redis refactor means a different code path or process is still opening short-lived Redis connections.

File Descriptor Limits

Aether's compose files intentionally do not set container ulimits.nofile. Redis connection churn must be fixed in application code, not hidden by larger file descriptor limits.

For high-concurrency production hosts, set file descriptor policy at the runtime or service-manager layer instead:

• Docker daemon default ulimit, for example default-ulimits in /etc/docker/daemon.json.
• systemd service limits such as LimitNOFILE= for Docker or the process supervisor.
• Managed container platform resource settings, when Docker daemon settings are not available.

Keep Redis maxclients below the effective Redis process nofile limit with room for persistence files, replicas, and admin connections.