mirror of
https://github.com/fawney19/Aether.git
synced 2026-09-01 17:00:21 +08:00
Add multi-database data layer
Introduce aether-data-schema and driver-specific schema generation for Postgres, MySQL, and SQLite. Split data backends, lifecycle, repositories, and gateway runtime integration across database drivers. Verified with cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace.
This commit is contained in:
107
docs/architecture/data-schema-inventory.md
Normal file
107
docs/architecture/data-schema-inventory.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# Aether Data Schema Inventory
|
||||
|
||||
This inventory is the maintenance map for the three SQL drivers. The executable
|
||||
`sqlx` migrations remain under `crates/aether-data/migrations/{postgres,mysql,sqlite}`.
|
||||
Do not split already-shipped migration files without also deciding how to handle
|
||||
existing `_sqlx_migrations` rows.
|
||||
|
||||
The maintainable schema source is under `crates/aether-data/schema`. Manifests
|
||||
there compose back into the executable SQL files and are checked by tests, so
|
||||
the split source is not just documentation.
|
||||
|
||||
The schema directory separates human-maintained sources from generated/runtime
|
||||
outputs:
|
||||
|
||||
| Layer | Purpose |
|
||||
|---|---|
|
||||
| `schema/logical/*.toml` | Human-maintained long-term table-structure source. |
|
||||
| `schema/drivers/{postgres,mysql,sqlite}/**` | Human-maintained executable-SQL fragments while generation is being promoted incrementally. |
|
||||
| `schema/bootstrap/postgres/**` | Human-maintained source fragments for the Postgres empty-database bootstrap snapshot. |
|
||||
| `schema/generated/**` | Machine-written SQL from logical schema; checked in for audit and drift detection only. |
|
||||
| `migrations/**` | Runtime SQL artifacts composed from manifests. |
|
||||
|
||||
Generated SQL is not hand-maintained and runtime code does not load it. It
|
||||
exists to prove that the logical schema can emit driver SQL and to provide the
|
||||
candidate replacement for handwritten fragments.
|
||||
|
||||
## Logical Type Map
|
||||
|
||||
| Logical type | Postgres | MySQL | SQLite | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `id` | `varchar/text` | `varchar` | `text` | Repository DTOs treat ids as strings. |
|
||||
| `bool` | `boolean` | `tinyint(1)/boolean` | `integer` | Repositories normalize to Rust `bool`. |
|
||||
| `time_unix` | `bigint` or legacy `timestamptz` | `bigint` | `integer` | New cross-driver paths prefer unix seconds/ms. |
|
||||
| `json` | `json/jsonb` | `text/json-compatible` | `text` | Application parses through `serde_json::Value`. |
|
||||
| `decimal_money` | `numeric` or `double precision` legacy | `double` | `real` | Wallet precision should be reviewed before new money tables. |
|
||||
| `blob` | `bytea` | `longblob/blob` | `blob` | Used for compressed body payloads. |
|
||||
| `enum` | `enum` or `varchar` legacy | `varchar` | `text` | Repository contracts own allowed values. |
|
||||
|
||||
## Baseline Source Plan
|
||||
|
||||
The current executable SQL files are intentionally kept stable for runtime
|
||||
compatibility. Their maintainable sources are:
|
||||
|
||||
| Driver | Executable SQL | Source manifest |
|
||||
|---|---|---|
|
||||
| Postgres baseline | `migrations/postgres/20260403000000_baseline.sql` | `schema/drivers/postgres/baseline/manifest.txt` |
|
||||
| Postgres empty-database snapshot | `aether-data` build output (`OUT_DIR/empty_database_snapshot.sql`) | `schema/bootstrap/postgres/manifest.txt` |
|
||||
| MySQL baseline | `migrations/mysql/20260403000000_baseline.sql` | `schema/drivers/mysql/baseline/manifest.txt` |
|
||||
| SQLite baseline | `migrations/sqlite/20260403000000_baseline.sql` | `schema/drivers/sqlite/baseline/manifest.txt` |
|
||||
|
||||
All driver manifests are kept as a small set of numbered SQL fragments. Postgres
|
||||
uses execution-phase fragments (`001_types_and_tables.sql`,
|
||||
`002_defaults.sql`, `003_constraints.sql`, `004_indexes.sql`,
|
||||
`005_foreign_keys.sql`, `006_footer.sql`) so pg_dump ordering remains stable
|
||||
when composed. MySQL and SQLite use similarly numbered domain fragments. After
|
||||
editing fragments, run:
|
||||
|
||||
```bash
|
||||
bash crates/aether-data/schema/compose_schema.sh compose
|
||||
bash crates/aether-data/schema/compose_schema.sh check
|
||||
```
|
||||
|
||||
Use `schema/logical/*.toml` for new table structure first; handwritten driver
|
||||
fragments remain for executable migration compatibility and generator gaps.
|
||||
|
||||
`crates/aether-data/schema/logical/*.toml` is the single-maintenance source for
|
||||
table structure. `aether-data-schema` renders it to
|
||||
`schema/generated/{postgres,mysql,sqlite}/baseline`, and
|
||||
`compose_schema.sh check` verifies that the generated SQL is current and that
|
||||
required executable SQL tables are represented in logical schema. The generated
|
||||
directory carries its own machine-generated README and per-file `Do not edit`
|
||||
headers; changes there should come only from `compose_schema.sh generate`.
|
||||
|
||||
## Table Inventory
|
||||
|
||||
| Area | Tables | Owner | Generation target |
|
||||
|---|---|---|---|
|
||||
| Identity/auth | `users`, `api_keys`, `management_tokens`, `user_preferences`, `user_sessions`, `user_oauth_links` | `repository/users`, `repository/auth`, `repository/management_tokens`, auth modules | Good first candidate for schema manifest/query helper generation. |
|
||||
| Provider catalog | `providers`, `provider_api_keys`, `provider_endpoints`, `models`, `global_models`, `api_key_provider_mappings`, `provider_usage_tracking` | `repository/provider_catalog`, `repository/global_models`, scheduler read paths | Keep complex selection SQL handwritten; generate basic CRUD only. |
|
||||
| Auth config | `auth_modules`, `oauth_providers`, `ldap_configs` | `repository/auth_modules`, `repository/oauth_providers`, `repository/users` | Good candidate for generated CRUD. |
|
||||
| Proxy nodes | `proxy_nodes`, `proxy_node_events` | `repository/proxy_nodes` | Good candidate for generated CRUD plus handwritten heartbeat update. |
|
||||
| Wallet/billing | `wallets`, `wallet_transactions`, `wallet_daily_usage_ledgers`, `payment_orders`, `payment_callbacks`, `refund_requests`, `redeem_code_batches`, `redeem_codes`, `billing_rules`, `dimension_collectors` | `repository/wallet`, `repository/billing`, `repository/settlement` | Keep settlement/ledger math explicit; generate table definitions and simple reads. |
|
||||
| Usage/audit | `usage`, `usage_body_blobs`, `usage_http_audits`, `usage_routing_snapshots`, `usage_settlement_snapshots`, `request_candidates`, `audit_logs` | `repository/usage`, `repository/candidates`, `repository/audit` | Keep core write/audit queries handwritten. |
|
||||
| Runtime tasks | `video_tasks`, `gemini_file_mappings`, `announcements`, `announcement_reads` | `repository/video_tasks`, `repository/gemini_file_mappings`, `repository/announcements` | Good candidate for generated CRUD except polling claim logic. |
|
||||
| Stats | `stats_*`, `schema_backfills` | backend aggregation modules | Keep aggregation SQL per-driver; generate table/index definitions only. |
|
||||
| System | `system_configs` | `repository/system` through backend dispatch | Good candidate for generated CRUD. |
|
||||
|
||||
## Logical Schema Coverage
|
||||
|
||||
Logical schema currently covers the clean baseline table set plus portable
|
||||
MySQL/SQLite table-creation migrations. Postgres-only historical follow-up
|
||||
migrations remain driver-specific until their schema is normalized or promoted
|
||||
as explicit generated/override fragments.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
1. Keep driver-specific SQL inside driver-specific migration/repository files.
|
||||
2. Use logical type names in docs and future schema manifests, not raw database
|
||||
type names.
|
||||
3. Keep `jsonb` only in Postgres migrations/repositories/tests.
|
||||
4. Prefer generated helpers for simple CRUD first; do not rewrite complex usage,
|
||||
billing, stats, or candidate-selection queries until contract tests cover the
|
||||
behavior.
|
||||
5. When adding a new table, update this inventory and add it to the export domain
|
||||
plan if it must move across databases.
|
||||
6. If a baseline fragment changes, run `compose_schema.sh compose` before tests
|
||||
so the executable SQL artifact is regenerated from the source manifest.
|
||||
Reference in New Issue
Block a user