feat: add postgres to single-node migration

2026-09-01 17:00:21 +08:00 · 2026-05-16 15:22:42 +08:00
parent 75b7319465
commit 4b12ec8913
13 changed files with 5386 additions and 68 deletions
--- a/docs/operations/pg-to-single-node-migration.md
+++ b/docs/operations/pg-to-single-node-migration.md
@@ -0,0 +1,246 @@
+# Postgres to Aether Single Node Migration
+
+Chinese version: [pg-to-single-node-migration.zh-CN.md](pg-to-single-node-migration.zh-CN.md)
+
+This runbook migrates an existing Docker Compose Postgres deployment to Aether
+single-node. In this repository, **single-node** means the default SQLite installer mode:
+`install.sh --mode single-node`, a system service backed by SQLite. The Docker Compose
+single-node template is `docker-compose.single-node.yml`, exposed through `--mode compose-single-node`.
+
+The migration script is:
+
+```bash
+scripts/migrate-pg-to-single-node.sh
+```
+
+If the target should stay on Docker Compose instead of becoming a system
+service, use the image-based Compose migration script:
+
+```bash
+scripts/migrate-pg-compose-to-single-node.sh
+```
+
+Both migration scripts pull/install the target single-node version before
+downtime, stop only the source `app`, copy Postgres records directly into a
+temporary SQLite DB without writing a JSONL file, replace the target
+`aether.db`, and start single-node.
+
+You can also use the installer as the unified entrypoint and let `--mode`
+select the migration target:
+
+```bash
+# In interactive mode, first choose the target deployment mode:
+#   1) Docker Compose standard deployment (Postgres + Redis)
+#   2) Docker Compose single-node deployment (SQLite)
+#   3) System service single-node deployment (SQLite)
+# After choosing 2 or 3, choose the data initialization mode:
+#   1) Fresh initialization (do not migrate existing data)
+#   2) Migrate from an existing Docker Compose PG database
+install.sh
+
+# Migrate into a new single-node Docker Compose directory.
+install.sh \
+  --mode compose-single-node \
+  --migrate-from-compose /root/Aether/docker-compose.yml \
+  --compose-dir /opt/aether-single \
+  --replace-existing
+
+# Migrate into the system service + SQLite layout.
+sudo install.sh \
+  --mode single-node \
+  --migrate-from-compose /root/Aether/docker-compose.yml \
+  --replace-existing
+```
+
+Interactive mode first asks for the target deployment shape. If the target is
+`compose-single-node` or `single-node`, the installer then asks for the data
+initialization mode: fresh initialization, or migration from an existing Docker
+Compose PG database. If you choose migration, it tries to detect the source PG
+Compose file from `docker compose ls`, then verifies that the Compose config
+contains the default `app` and `postgres` services. If exactly one match is
+found, it is used as the default prompt value. If detection is ambiguous or
+fails, the installer stops; rerun it with `--migrate-from-compose` to specify
+the source compose path.
+
+The installer only normalizes the entrypoint: `compose-single-node` delegates to
+`scripts/migrate-pg-compose-to-single-node.sh`, while `single-node` delegates to
+`scripts/migrate-pg-to-single-node.sh`.
+
+## What It Does
+
+The script keeps the production cutover window short:
+
+1. Reads the source Compose `.env`.
+2. Builds a single-node env file that preserves `JWT_SECRET_KEY`, `ENCRYPTION_KEY` or
+   `AETHER_GATEWAY_DATA_ENCRYPTION_KEY`, admin settings, port, and app config.
+3. Installs the single-node release with `install.sh --mode single-node --skip-start`.
+4. Preflights SQLite migrations with the installed single-node binary.
+5. Pulls the target single-node image, confirms its `copy` command supports the
+   current migration domains, and verifies that its Docker image ID matches the
+   currently running source `app` image ID.
+6. Checks for non-empty Postgres tables not covered by the migration domains.
+7. Applies the compressed body and HTTP body detail policy. The default is full,
+   and you can opt into an omit mode for large artifacts.
+8. Checks that the work directory and target SQLite directory have enough free
+   disk space for the temporary and final SQLite files.
+9. Stops only the source `app` service, leaving Postgres and Redis running.
+10. Copies source Postgres records directly into a temporary SQLite database
+   without generating JSONL files.
+11. Replaces the target SQLite DB, including SQLite `-wal`/`-shm` sidecar files
+    when present, and starts the single-node service.
+
+The image check compares Docker image IDs, not just tag strings. If both source
+and target say `latest` but resolve to different image IDs, migration stops.
+Upgrade the source PG Compose `app` to the target single-node version first,
+verify it is healthy, then run the migration. The scripts also check that the
+target image supports `stats`, `auxiliary`, and the request-body omit flag; using
+a new script with an old image stops before cutover to avoid missing data.
+
+## Production Cutover
+
+Before production cutover, take a normal server backup or snapshot. Then run:
+
+```bash
+sudo scripts/migrate-pg-to-single-node.sh \
+  --source-compose /root/Aether/docker-compose.yml \
+  --replace-existing
+```
+
+For Docker Compose single-node cutover instead of a system service:
+
+```bash
+scripts/migrate-pg-compose-to-single-node.sh \
+  --source-compose /root/Aether/docker-compose.yml \
+  --replace-existing
+```
+
+The source Postgres compose directory and target single-node compose directory
+can be different. For example:
+
+```bash
+install.sh \
+  --mode compose-single-node \
+  --migrate-from-compose /root/Aether/docker-compose.yml \
+  --compose-dir /opt/aether-single \
+  --replace-existing
+```
+
+Equivalently, call the lower-level script and pass each target path explicitly:
+
+```bash
+scripts/migrate-pg-compose-to-single-node.sh \
+  --source-compose /root/Aether/docker-compose.yml \
+  --target-compose /opt/aether-single/docker-compose.single-node.yml \
+  --target-env /opt/aether-single/.env.single-node \
+  --target-db /opt/aether-single/data/aether.db \
+  --replace-existing
+```
+
+During cutover, the script stops and removes only the source `app` container to
+free the fixed `aether-app` container name. Postgres, Redis, and their volumes
+remain in place for rollback.
+
+Defaults:
+
+| Setting | Default |
+| --- | --- |
+| Source Compose | `docker-compose.yml` |
+| Single Node install root | `/opt/aether` |
+| Single Node config dir | `/etc/aether` |
+| Target SQLite DB | `/opt/aether/data/aether.db` |
+| Source app service | `app` |
+| Source Postgres service | `postgres` |
+| Single Node service | `aether-gateway` |
+
+The script writes migration artifacts under `./data/pg-to-single-node-<timestamp>` next
+to the source Compose file unless `--work-dir` is provided.
+
+## Rollback
+
+The script leaves the original Postgres and Redis volumes in place. If cutover
+finishes but you need to roll back:
+
+```bash
+sudo systemctl stop aether-gateway
+cd /root/Aether
+docker compose -f docker-compose.yml up -d app
+```
+
+For the Compose single-node script, rollback is the same idea: start the app
+again from the original Postgres compose file.
+
+If the migration fails before cutover completes, the script attempts to restart
+the source `app` service automatically. Pass `--keep-source-stopped-on-error` if
+you want to inspect the stopped source deployment manually instead.
+
+## Data Coverage Guard
+
+The current migration covers these persistent domains: users, API keys,
+providers, provider keys, endpoints, models, global models, auth modules, OAuth
+links, user groups, proxy nodes, system configs, wallets, usage, and billing
+data.
+
+Before stopping the app, and again after the source app has stopped, the script
+checks the source Postgres database for non-empty tables outside that migration
+coverage. It does not run source Postgres migrations or backfills during the
+cutover. It ignores lifecycle metadata tables such as `_sqlx_migrations` and
+`schema_backfills`. Any other non-empty uncovered table blocks the migration.
+
+## Request Body Detail Policy
+
+The production migration migrates all migratable data by default. The only
+optional exclusion is request body detail data.
+
+When you choose to skip request bodies, the migration does not copy
+`usage_body_blobs`, `usage_http_audits`, or legacy `usage` request body columns
+such as `request_body`, `provider_request_body`, `response_body`,
+`client_response_body`, and `*_body_compressed`.
+
+Interactive installation lets you choose:
+
+```text
+1) Full migration: migrate all migratable data, including request body details
+2) Skip request bodies: migrate all other data; skip only request body large fields and HTTP body detail tables; source PG is unchanged
+```
+
+For non-interactive full runs:
+
+```bash
+scripts/migrate-pg-to-single-node.sh \
+  --request-body-mode full
+```
+
+For non-interactive omit runs:
+
+```bash
+scripts/migrate-pg-to-single-node.sh \
+  --request-body-mode omit
+```
+
+`omit` only skips writing those large artifacts and detail tables into the
+target SQLite database. It does not delete or clear the source Postgres data.
+
+If a table is intentionally excluded, allow it explicitly:
+
+```bash
+scripts/migrate-pg-to-single-node.sh \
+  --allow-non-exported-table legacy_custom_table
+```
+
+Use that only after confirming the table is not required in the single-node target.
+
+## Notes
+
+- Single Node requires root or sudo because it writes `/opt/aether`, `/etc/aether`, and
+  the system service definition.
+- The script does not decrypt or re-encrypt provider keys. It preserves the
+  original encryption key and moves encrypted data as-is.
+- Existing target SQLite databases, including `-wal`/`-shm` sidecars, are not
+  replaced unless `--replace-existing` is provided.
+- Disk space checks use `pg_database_size(current_database()) * 2 + 1 GiB` as the
+  conservative estimate for one SQLite copy. If the work directory and target DB
+  directory are on the same filesystem, the script requires enough space for both
+  the temporary and final SQLite files. With `--request-body-mode omit`, the
+  estimate subtracts `usage_body_blobs` and `usage_http_audits` relation sizes.
+- For non-standard source Compose files, set `--app-service` and
+  `--postgres-service` to match the service names.
--- a/docs/operations/pg-to-single-node-migration.zh-CN.md
+++ b/docs/operations/pg-to-single-node-migration.zh-CN.md
@@ -0,0 +1,230 @@
+# Postgres 到 Aether Single Node 迁移
+
+英文版：[pg-to-single-node-migration.md](pg-to-single-node-migration.md)
+
+本文档用于把现有 Docker Compose Postgres 部署迁移到 Aether
+single-node。当前版本里，**single-node** 指默认 SQLite 安装模式：
+`install.sh --mode single-node`，也就是系统服务加 SQLite。Docker Compose
+单机模板是 `docker-compose.single-node.yml`，安装脚本入口是
+`--mode compose-single-node`。
+
+迁移脚本：
+
+```bash
+scripts/migrate-pg-to-single-node.sh
+```
+
+如果目标形态仍然要保持 Docker Compose，而不是系统服务，使用镜像版迁移脚本：
+
+```bash
+scripts/migrate-pg-compose-to-single-node.sh
+```
+
+两种迁移脚本都会先拉取/安装目标 single-node 版本，再停止源 `app`，把 Postgres
+记录直接写入临时 SQLite DB，不落 JSONL 中间文件；复制成功后替换目标
+`aether.db`，最后启动 single-node。
+
+也可以直接用安装脚本作为统一入口，由 `--mode` 选择迁移目标：
+
+```bash
+# 交互式执行时，先选择目标部署模式：
+#   1) Docker Compose 标准部署（Postgres + Redis）
+#   2) Docker Compose 单节点部署（SQLite）
+#   3) 系统服务单节点部署（SQLite）
+# 选择 2 或 3 后，再选择数据初始化方式：
+#   1) 全新初始化（不迁移现有数据）
+#   2) 从现有 Docker Compose PG 数据库迁移
+install.sh
+
+# 迁移到新的 single-node Docker Compose 目录
+install.sh \
+  --mode compose-single-node \
+  --migrate-from-compose /root/Aether/docker-compose.yml \
+  --compose-dir /opt/aether-single \
+  --replace-existing
+
+# 迁移到系统服务 + SQLite
+sudo install.sh \
+  --mode single-node \
+  --migrate-from-compose /root/Aether/docker-compose.yml \
+  --replace-existing
+```
+
+交互模式会先选择目标部署形态。如果目标是 `compose-single-node` 或
+`single-node`，安装脚本会再询问数据初始化方式：全新初始化，或从现有 Docker
+Compose PG 数据库迁移。选择迁移后，脚本会通过 `docker compose ls` 自动探测源
+PG Compose 文件，并确认该 Compose 配置里存在默认的 `app` 和 `postgres` 服务；
+如果能唯一识别，会作为默认值带入提示。探测不到或存在多个候选时会直接中止；
+此时请用 `--migrate-from-compose` 显式指定源 compose 路径。
+
+安装脚本只是统一参数入口：`compose-single-node` 会委托给
+`scripts/migrate-pg-compose-to-single-node.sh`，`single-node` 会委托给
+`scripts/migrate-pg-to-single-node.sh`。
+
+## 迁移内容
+
+脚本会尽量缩短生产停机窗口：
+
+1. 读取源 Compose 目录下的 `.env`。
+2. 生成 single-node 环境文件，保留 `JWT_SECRET_KEY`、`ENCRYPTION_KEY` 或
+   `AETHER_GATEWAY_DATA_ENCRYPTION_KEY`、管理员配置、端口和应用配置。
+3. 执行 `install.sh --mode single-node --skip-start`，提前安装 single-node
+   release，但不启动服务。
+4. 使用已安装的 single-node 二进制预检 SQLite schema migration。
+5. 拉取目标 single-node 镜像，确认其 `copy` 命令支持当前迁移域，并检查源
+   `app` 当前运行镜像 ID 与目标镜像 ID 一致。
+6. 检查 Postgres 里是否存在当前迁移域没有覆盖、但又非空的表。
+7. 检查请求体明细迁移策略；默认全部迁移，也可以选择只跳过请求体明细。
+8. 检查 work-dir 和目标 SQLite 目录是否有足够空间容纳临时库和正式库。
+9. 只停止源 Compose 的 `app` 服务，保留 Postgres 和 Redis 运行，方便回滚。
+10. 从源 Postgres 直接复制记录到临时 SQLite 数据库，不生成 JSONL 中间文件。
+11. 复制完成后替换目标 SQLite DB，包括 SQLite `-wal`、`-shm` 边车文件，
+    然后启动 single-node 系统服务。
+
+镜像一致性检查比较的是 Docker 镜像 ID，不只是 tag 字符串。即使源和目标都写着
+`latest`，只要实际镜像 ID 不同，迁移也会中止。请先把源 PG Compose 的 `app`
+升级到目标 single-node 相同版本，确认运行正常后再迁移。迁移脚本也会检查目标镜像
+是否支持 `stats`、`auxiliary` 和请求体跳过开关；如果只是换了脚本但镜像还是旧版本，
+脚本会直接中止，避免漏迁。
+
+## 生产切换
+
+切换前先做一次常规服务器备份或快照。确认后执行：
+
+```bash
+sudo scripts/migrate-pg-to-single-node.sh \
+  --source-compose /root/Aether/docker-compose.yml \
+  --replace-existing
+```
+
+如果要迁移到 Docker Compose single-node，而不是系统服务：
+
+```bash
+scripts/migrate-pg-compose-to-single-node.sh \
+  --source-compose /root/Aether/docker-compose.yml \
+  --replace-existing
+```
+
+源 Postgres Compose 目录和目标 single-node Compose 目录可以不一样。例如：
+
+```bash
+install.sh \
+  --mode compose-single-node \
+  --migrate-from-compose /root/Aether/docker-compose.yml \
+  --compose-dir /opt/aether-single \
+  --replace-existing
+```
+
+等价地，也可以直接调底层脚本并显式传入每个目标路径：
+
+```bash
+scripts/migrate-pg-compose-to-single-node.sh \
+  --source-compose /root/Aether/docker-compose.yml \
+  --target-compose /opt/aether-single/docker-compose.single-node.yml \
+  --target-env /opt/aether-single/.env.single-node \
+  --target-db /opt/aether-single/data/aether.db \
+  --replace-existing
+```
+
+切换时脚本只会停止并移除源 `app` 容器，用来释放固定的 `aether-app`
+容器名；Postgres、Redis 和它们的 volume 都会保留，方便回滚。
+
+默认路径和服务名：
+
+| 配置项 | 默认值 |
+| --- | --- |
+| 源 Compose 文件 | `docker-compose.yml` |
+| single-node 安装目录 | `/opt/aether` |
+| single-node 配置目录 | `/etc/aether` |
+| 目标 SQLite DB | `/opt/aether/data/aether.db` |
+| 源 app 服务 | `app` |
+| 源 Postgres 服务 | `postgres` |
+| single-node 服务 | `aether-gateway` |
+
+除非显式传入 `--work-dir`，脚本会把迁移产物写到源 Compose 文件旁边的
+`./data/pg-to-single-node-<timestamp>`。
+
+## 回滚
+
+脚本会保留原 Postgres 和 Redis volume。迁移已经完成但需要回滚时：
+
+```bash
+sudo systemctl stop aether-gateway
+cd /root/Aether
+docker compose -f docker-compose.yml up -d app
+```
+
+对于 Compose single-node 脚本，回滚思路相同：重新用原 Postgres compose 文件
+拉起 `app`。
+
+如果迁移在切换完成前失败，脚本默认会尝试自动拉起源 `app` 服务。需要失败后
+保持源应用停止以便人工排查时，增加：
+
+```bash
+--keep-source-stopped-on-error
+```
+
+## 数据覆盖保护
+
+当前迁移覆盖的持久化域包括：用户、API Key、供应商、供应商 Key、
+端点、模型、全局模型、认证模块、OAuth 关联、用户组、代理节点、系统配置、
+钱包、用量和计费数据。
+
+脚本会在停机前，以及源 `app` 停止之后，再检查一次源 Postgres：如果发现
+当前迁移域没有覆盖的非空表，会直接中止迁移，避免漏迁。切换期间不会对源
+Postgres 执行 migrations 或 backfills。生命周期元数据表 `_sqlx_migrations`
+和 `schema_backfills` 会被忽略。
+
+## 请求体明细策略
+
+single-node SQLite 生产迁移默认迁移所有可迁移数据，唯一可选的跳过项是请求体明细。
+
+选择“不迁移请求体”时，不会迁移 `usage_body_blobs`、`usage_http_audits`，也不会迁移 `usage`
+表里的 `request_body` / `provider_request_body` / `response_body` /
+`client_response_body` / `*_body_compressed` 等请求体大字段。
+
+交互安装时可以选择：
+
+```text
+1) 全部迁移：迁移所有可迁移数据，包括请求体明细
+2) 不迁移请求体：迁移其他所有数据；仅跳过请求体大字段和 HTTP 请求体明细，源 PG 不清除
+```
+
+非交互执行时，全部迁移可以显式指定：
+
+```bash
+scripts/migrate-pg-to-single-node.sh \
+  --request-body-mode full
+```
+
+不迁移请求体可以显式指定：
+
+```bash
+scripts/migrate-pg-to-single-node.sh \
+  --request-body-mode omit
+```
+
+`omit` 只是不把这些大字段和明细表写进目标 SQLite，不会删除或清空源 Postgres。
+
+如果某个表确认不需要迁移，可以显式允许：
+
+```bash
+scripts/migrate-pg-to-single-node.sh \
+  --allow-non-exported-table legacy_custom_table
+```
+
+只有在确认该表对 single-node 目标库不重要时才这样做。
+
+## 注意事项
+
+- single-node 安装需要 root 或 sudo 权限，因为会写入 `/opt/aether`、
+  `/etc/aether` 和系统服务定义。
+- 脚本不会解密或重新加密供应商密钥；它会沿用源环境的加密密钥，并原样迁移已加密数据。
+- 已存在的目标 SQLite DB，包括 `-wal`、`-shm` 边车文件，只有在传入
+  `--replace-existing` 时才会被替换。
+- 空间检查会用 `pg_database_size(current_database()) * 2 + 1 GiB` 作为单份
+  SQLite 的保守估算。如果 work-dir 和目标 DB 目录在同一个文件系统，会要求同时
+  容纳临时 SQLite 和正式 SQLite。选择 `--request-body-mode omit` 时，
+  估算会扣除 `usage_body_blobs` 和 `usage_http_audits` 的表空间。
+- 非标准 Compose 服务名需要通过 `--app-service` 和 `--postgres-service`
+  明确指定。