i/Aether
1
0
Fork 0
mirror of https://github.com/fawney19/Aether.git synced 2026-10-07 01:47:47 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat: revamp analytics dashboards and harden database migrations

Browse Source 
Add dashboard and overview analytics, health monitoring, provider expense tracking, and announcement updates across the gateway and frontend.

Keep schema migrations free of historical backfills while preserving automatic backfill execution. Bound migration deadlines, run schema preparation before Compose replacement, and anonymize deleted dashboard users.

Include the current documentation cleanup and regression coverage.
This commit is contained in:
elky
2026-10-01 11:48:17 +08:00
parent 60b89cc840
commit 066ea87d72
327 changed files with 31728 additions and 20645 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/aether-data/runtime/schema/README.md
+69
View File
@@ -109,3 +109,72 @@ remains byte-for-byte stable when composed:
The Rust migration tests compose these manifests too, so fragment drift is
caught during `cargo test -p aether-data split_baseline_sources_match_executable_migrations`.
## Statistics Migrations
The statistics release retains its applied migration history and includes
incremental upgrades for databases that ran the earlier overview and dashboard
definitions. Concurrent index operations remain separate because PostgreSQL
cannot run them inside a transaction.
| Version | Change |
|---|---|
| `20260911000000` | Overview facts, attribution, aggregate tables, and transaction-owned dirty-event queue. Attribution indexes are created while the new table is empty. |
| `20260917000000` | Original account-attribution migration, retained byte-for-byte for databases that already applied it. |
| `20260917000100` | Upgrade the original attribution trigger to the dirty-event queue before later concurrent index builds. |
| `20260918000000` | Create the replacement settlement covering index concurrently. |
| `20260918000100` | Drop the previous settlement covering index concurrently, after its replacement succeeds. |
| `20260919000000` | Dashboard aggregates, activation boundary, and retention support. |
| `20260920000000` | Create the credited-payment lookup index concurrently. |
| `20260920120000` | Provider expense records. |
| `20260921010000` | Add retention support to existing dashboard schemas; safe when the initial dashboard migration already includes it. |
| `20260921020000` | Add the attribution-owner lookup index concurrently on existing databases. |
| `20260921020100` | Create the usage metadata actor index concurrently. |
| `20261001000000` | Remove deleted-user attribution from dashboard activity on future user deletion; schema-only upgrade without rewriting historical rows. |
Do not remove an applied migration after folding its changes into an earlier
schema definition. Existing databases retain its version in `_sqlx_migrations`
and do not rerun earlier versions when their SQL changes. Preserve that history
and provide incremental migrations for any remaining schema differences.
These migrations do not backfill historical requests. Dashboard totals start at
the stored activation boundary. Background maintenance compacts dashboard minute
details older than 35 days in bounded batches, preserving cumulative totals and
the narrow activity counts; it does not delete source usage. JSONL backups include
the dashboard snapshot and its integrity manifest so retained totals can survive
restoration after source usage has expired.
Schema migrations and historical backfills remain separate phases. Normal `auto`
startup and `db prepare` still apply pending scripts from `backfills/postgres`
after schema migration; `verify-only` still requires both phases to be current.
The statistics schema migrations above do not embed a historical data rebuild.
The new dashboard's activation boundary is not moved by legacy backfills, so
they do not restore pre-activation dashboard totals.
Deleted users are excluded from dashboard active-user reads even when an older
version left orphan activity rows. The anonymization upgrade installs rules for
future deletions without cleaning old rows during migration; those old activity
rows age out through the existing 35-day retention task.
The overview worker can still rebuild a historical hour/day when normal writes
change facts in that bucket. That work runs after startup with bounded batches
and query deadlines; it is not a full historical rebuild during migration.
The migration runner defaults to a 1-second lock wait, a 10-second deadline per
transactional migration, and a 15-minute deadline per concurrent index migration.
Timeouts are configurable through `AETHER_POSTGRES_MIGRATION_LOCK_TIMEOUT_MS`,
`AETHER_POSTGRES_MIGRATION_TIMEOUT_MS`, and
`AETHER_POSTGRES_MIGRATION_CONCURRENT_TIMEOUT_MS`; none accepts zero. An independent
control connection attempts to terminate the migration session on failure or
cancellation. An interrupted concurrent index build can leave an invalid index;
the runner removes that index before retrying its migration.
For Compose deployments, `update.sh` applies schema migrations with the new image
before replacing the running app. A migration failure stops the update; already
committed migrations remain applied. Its `local-build` mode delegates to
`deploy.sh` and does not use this separate migration step. Allow for brief table
locks and I/O pressure from concurrent index scans during the upgrade. Keeping
historical backfills out of schema migrations does not make index creation
constant-time: concurrent indexes still scan existing rows and can take minutes
on a large database. The existing app stays running during the Compose migration
preflight.