mirror of
https://github.com/fawney19/Aether.git
synced 2026-09-11 05:30:19 +08:00
68 lines
2.4 KiB
Markdown
68 lines
2.4 KiB
Markdown
# Runtime Redis Operations Runbook
|
|
|
|
This runbook covers Aether runtime Redis connection pressure incidents. It is
|
|
not a substitute for fixing application-level connection churn.
|
|
|
|
## Normal Expectations
|
|
|
|
- Each `RuntimeState` Redis backend initializes a fixed set of long-lived
|
|
connection lanes: fast, stream, blocking stream, and admin.
|
|
- `connected_clients` should stay near a small fixed number per app instance,
|
|
plus health checks and ad hoc admin clients.
|
|
- `total_connections_received` should not grow linearly with request volume.
|
|
- Large TIME_WAIT spikes between app and Redis indicate a regression or a
|
|
separate process repeatedly opening Redis connections.
|
|
|
|
## Emergency Mitigation
|
|
|
|
1. Disable the retry source first, such as expired Codex/OAuth keys causing a
|
|
retry storm.
|
|
2. Restart the app to stop continued connection creation:
|
|
|
|
```sh
|
|
docker compose restart app
|
|
```
|
|
|
|
3. On a Linux host, temporarily widen the ephemeral port range and enable safe
|
|
TIME_WAIT reuse:
|
|
|
|
```sh
|
|
sudo sysctl -w net.ipv4.ip_local_port_range="10000 65535"
|
|
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
|
|
```
|
|
|
|
4. Do not enable `tcp_tw_recycle`; it is obsolete and unsafe with NAT.
|
|
|
|
Docker Desktop on macOS runs containers inside a Linux VM. Host-level macOS
|
|
`sysctl` changes do not necessarily affect the VM network namespace.
|
|
|
|
## Checks
|
|
|
|
Use Redis `INFO clients` and `INFO stats` to inspect:
|
|
|
|
- `connected_clients`
|
|
- `total_connections_received`
|
|
|
|
Use OS socket tooling on the Redis host or container namespace to inspect
|
|
TIME_WAIT counts. Persistent growth after the runtime Redis refactor means a
|
|
different code path or process is still opening short-lived Redis connections.
|
|
|
|
## File Descriptor Limits
|
|
|
|
Aether's compose files intentionally do not set container `ulimits.nofile`.
|
|
Redis connection churn must be fixed in application code, not hidden by larger
|
|
file descriptor limits.
|
|
|
|
For high-concurrency production hosts, set file descriptor policy at the
|
|
runtime or service-manager layer instead:
|
|
|
|
- Docker daemon default ulimit, for example `default-ulimits` in
|
|
`/etc/docker/daemon.json`.
|
|
- systemd service limits such as `LimitNOFILE=` for Docker or the process
|
|
supervisor.
|
|
- Managed container platform resource settings, when Docker daemon settings are
|
|
not available.
|
|
|
|
Keep Redis `maxclients` below the effective Redis process `nofile` limit with
|
|
room for persistence files, replicas, and admin connections.
|