# Runtime Redis Operations Runbook

This runbook covers Aether runtime Redis connection pressure incidents. It is
not a substitute for fixing application-level connection churn.

## Normal Expectations

- Each `RuntimeState` Redis backend initializes a fixed set of long-lived
  connection lanes: fast, stream, blocking stream, and admin.
- `connected_clients` should stay near a small fixed number per app instance,
  plus health checks and ad hoc admin clients.
- `total_connections_received` should not grow linearly with request volume.
- Large TIME_WAIT spikes between app and Redis indicate a regression or a
  separate process repeatedly opening Redis connections.

## Emergency Mitigation

1. Disable the retry source first, such as expired Codex/OAuth keys causing a
   retry storm.
2. Restart the app to stop continued connection creation:

   ```sh
   docker compose restart app
   ```

3. On a Linux host, temporarily widen the ephemeral port range and enable safe
   TIME_WAIT reuse:

   ```sh
   sudo sysctl -w net.ipv4.ip_local_port_range="10000 65535"
   sudo sysctl -w net.ipv4.tcp_tw_reuse=1
   ```

4. Do not enable `tcp_tw_recycle`; it is obsolete and unsafe with NAT.

Docker Desktop on macOS runs containers inside a Linux VM. Host-level macOS
`sysctl` changes do not necessarily affect the VM network namespace.

## Checks

Use Redis `INFO clients` and `INFO stats` to inspect:

- `connected_clients`
- `total_connections_received`

Use OS socket tooling on the Redis host or container namespace to inspect
TIME_WAIT counts. Persistent growth after the runtime Redis refactor means a
different code path or process is still opening short-lived Redis connections.

## File Descriptor Limits

Aether's compose files intentionally do not set container `ulimits.nofile`.
Redis connection churn must be fixed in application code, not hidden by larger
file descriptor limits.

For high-concurrency production hosts, set file descriptor policy at the
runtime or service-manager layer instead:

- Docker daemon default ulimit, for example `default-ulimits` in
  `/etc/docker/daemon.json`.
- systemd service limits such as `LimitNOFILE=` for Docker or the process
  supervisor.
- Managed container platform resource settings, when Docker daemon settings are
  not available.

Keep Redis `maxclients` below the effective Redis process `nofile` limit with
room for persistence files, replicas, and admin connections.