chore: disable Redis persistence by default, document triage and policy

This commit is contained in:
elky
2026-07-01 14:15:16 +08:00
parent 7c32be46ca
commit a0f7074e59
2 changed files with 63 additions and 4 deletions
+1 -4
View File
@@ -37,9 +37,7 @@ services:
redis:
image: redis:7-alpine
container_name: aether-redis
command: redis-server --appendonly yes --appendfsync everysec --save 60 1000 --requirepass ${REDIS_PASSWORD} --maxclients ${REDIS_MAXCLIENTS:-10000}
volumes:
- redis_data:/data
command: redis-server --dir /tmp --appendonly no --save "" --requirepass ${REDIS_PASSWORD} --maxclients ${REDIS_MAXCLIENTS:-10000}
ports:
- "127.0.0.1:${REDIS_PORT:-6379}:6379"
healthcheck:
@@ -107,4 +105,3 @@ services:
volumes:
postgres_data:
mysql_data:
redis_data:
+62
View File
@@ -3,6 +3,68 @@
This runbook covers Aether runtime Redis connection pressure incidents. It is
not a substitute for fixing application-level connection churn.
## Persistence Policy
The bundled `docker-compose.yml` treats Redis as a low-latency runtime
coordination layer by default: locks, cache affinity, semaphores, and runtime
streams. Postgres remains the source of truth. The default Redis persistence
policy is passed directly to `redis-server` in `docker-compose.yml`:
```sh
--dir /tmp --appendonly no --save ""
```
This avoids request-path latency spikes from AOF fsync and background snapshot
forks. The trade-off is that Redis runtime state can be lost if the Redis
container or host crashes before workers have flushed queued records to the
database. The default `dir /tmp` also prevents old files in the mounted
data directory from being loaded as stale runtime state; the persistence
disable itself is `--appendonly no` and `--save ""`.
Only deployments that intentionally want Redis runtime streams to survive a
crash should restore persistence in the Redis command:
```sh
--dir /data --appendonly yes --appendfsync everysec --save 60 1000
```
Expect higher tail latency when Redis persistence shares disks with Postgres or
application logs.
## Latency Triage
Redis `INFO commandstats` reports `latency_percentiles_usec_*` values in
microseconds. For example `p99=2007` means about 2 ms, not 2 seconds.
Use these checks before attributing app stalls to Redis:
```sh
redis-cli -p 6379 -a "$REDIS_PASSWORD" LATENCY DOCTOR
redis-cli -p 6379 -a "$REDIS_PASSWORD" LATENCY LATEST
redis-cli -p 6379 -a "$REDIS_PASSWORD" SLOWLOG GET 20
redis-cli -p 6379 -a "$REDIS_PASSWORD" INFO persistence
redis-cli -p 6379 -a "$REDIS_PASSWORD" INFO commandstats
redis-cli -p 6379 -a "$REDIS_PASSWORD" INFO clients
```
For immediate mitigation on an existing container that is running with AOF
enabled:
```sh
redis-cli -p 6379 -a "$REDIS_PASSWORD" CONFIG SET appendfsync no
redis-cli -p 6379 -a "$REDIS_PASSWORD" CONFIG SET appendonly no
redis-cli -p 6379 -a "$REDIS_PASSWORD" CONFIG SET save ""
```
Active defrag can help when `mem_fragmentation_ratio` is high, but it is not an
AOF fsync fix. Enable it only after confirming the Redis build supports it:
```sh
redis-cli -p 6379 -a "$REDIS_PASSWORD" CONFIG SET activedefrag yes
redis-cli -p 6379 -a "$REDIS_PASSWORD" CONFIG SET active-defrag-ignore-bytes 50mb
redis-cli -p 6379 -a "$REDIS_PASSWORD" CONFIG SET active-defrag-threshold-lower 10
```
## Normal Expectations
- Each `RuntimeState` Redis backend initializes a fixed set of long-lived