Aether/crates/aether-data/runtime

aether-data

aether-data is the runtime data-access crate. It owns concrete SQL/database drivers, concrete repository implementations, migration/backfill/export workflows, and the composition layer that wires those pieces into the rest of the application.

It does not own the cross-crate DTO contracts. Shared repository records and errors that are consumed by scheduler, billing, admin, usage runtime, and video task crates live in the sibling ../contracts crate (aether-data-contracts).

Directory Map

Path	Responsibility
src/database.rs	Logical SQL driver selection and shared pool configuration.
src/config.rs	Data-layer config for SQL drivers and repository wiring.
src/maintenance.rs	Maintenance DTOs and aggregation summaries used by backend dispatch and runtime maintenance entrypoints.
src/driver/{postgres,mysql,sqlite}.rs	Thin compatibility facades for the selected adapter crates. Low-level pools, transactions, and leases live in aether-data-postgres, aether-data-mysql, and aether-data-sqlite.
src/repository	Compatibility modules that re-export contracts and selected SQL adapters, plus concrete in-memory implementations. Driver-specific request-path SQL lives in the adapter crates.
src/backend	Composition root. Builds concrete driver backends and exposes app-facing read/write/worker/lock/lease handles.
src/backend/{maintenance,stats,wallet,system}.rs	Backend-owned maintenance, aggregation, wallet ledger, and system config workflows that are not normal request-path repositories.
src/lifecycle/migrate.rs and src/lifecycle/migrate/*	Compatibility entry points and Postgres snapshot bootstrap adapter.
src/lifecycle/backfill.rs	Backfill entry points and backfill discovery.
src/lifecycle/export.rs	Cross-database export/import workflows.
../adapters/{postgres,mysql,sqlite}/migrations	Executable sqlx migrations owned and embedded by each adapter crate.
schema	Schema maintenance workspace for logical definitions, driver fragments, and generated output.
schema/logical	Human-maintained logical table definitions used by aether-data-schema.
schema/drivers/{postgres,mysql,sqlite}	Human-maintained driver fragments that compose back into executable SQL while generation is being promoted.
schema/bootstrap/postgres	Human-maintained source fragments for the Postgres bootstrap snapshot. build.rs composes them into the runtime embedded artifact during crate builds.
schema/generated	Machine-written SQL generated from logical schema for audit and drift detection only.
schema/overrides	Rare driver-specific SQL escape hatch. Keep README-only until a real override is needed.
backfills/{postgres,mysql,sqlite}	Executable backfill SQL grouped by driver.

Layering Rules

The crate is easiest to read as five layers:

1. Contracts: DTOs, input structs, repository traits, and DataLayerError. Prefer aether-data-contracts for anything that another crate needs to compile against.
2. Driver primitives: aether-data-postgres, aether-data-mysql, and aether-data-sqlite connect to infrastructure and expose pools, runners, and executable migrations; the matching src/driver/*.rs files only preserve the existing import paths.
3. Repository implementations: aether-data-{postgres,mysql,sqlite} translate contract types to driver-specific SQL. src/repository/<domain> keeps the compatibility import path and owns the in-memory implementation where one exists.
4. Backend composition: backend chooses one SQL driver from config and wires repository implementations into app-facing handles. Backend-owned runtime maintenance workflows live in focused backend modules rather than in the driver pool files.
5. Maintenance workflows: lifecycle/migrate, lifecycle/backfill, lifecycle/export, and schema manage database lifecycle outside normal request handling.

Do not add domain queries to low-level pool modules. Do not add driver selection logic inside individual repository implementations. Keep cross-crate contracts out of aether-data unless they are implementation-only.

Repository Layout

Most domain repositories use this shape:

src/repository/<domain>/
  mod.rs      # exports trait/type names and concrete implementations
  types.rs    # implementation-local DTOs when they are not already in contracts
  postgres.rs # Postgres implementation
  mysql.rs    # MySQL implementation
  sqlite.rs   # SQLite implementation
  memory.rs   # tests/dev in-memory implementation

Use explicit driver filenames for repository implementations. Do not introduce new generic sql.rs modules for driver-specific code.

SQL Driver Policy

The project supports three SQL drivers at the repository/backend boundary: Postgres, MySQL, and SQLite. That does not mean every raw SQL file is shared. The portable contract is the Rust shape and behavior; the physical SQL stays driver-specific where syntax, indexes, JSON support, timestamps, locking, or upsert semantics differ.

Use logical types in design docs and reviews:

Logical type	Postgres	MySQL	SQLite
json	json or jsonb	json or text JSON	text JSON
bool	boolean	boolean / tinyint(1)	integer
time_unix	bigint or legacy timestamp	bigint	integer
money_decimal	numeric / legacy double	double	real

jsonb is acceptable only in Postgres SQL. MySQL and SQLite migrations must not contain jsonb; this is guarded by migration tests. Prefer serde_json::Value or typed Rust structs at the repository boundary so callers do not depend on the physical storage type.

Feature Matrix

aether-data enables only PostgreSQL by default so local checks do not compile all SQLx drivers:

cargo check -p aether-data
cargo check -p aether-data --no-default-features --features mysql
cargo check -p aether-data --no-default-features --features sqlite
cargo check -p aether-data --no-default-features --features all-drivers

Services selecting MySQL or SQLite in Cargo.toml must also disable the default Postgres feature:

aether-data = { workspace = true, default-features = false, features = ["mysql"] }

The gateway explicitly enables all-drivers for deployment compatibility. New services should select only the driver they deploy with. A configured driver that is not enabled in the build returns an explicit configuration error rather than silently constructing an empty backend.

Schema Maintenance

Executable migrations live under each adapter crate's migrations/ directory. Moving their physical files does not change SQLx migration versions or deployed database history because SQLx records migration version and checksum, not the workspace source path.

The large baseline SQL files are maintained through schema fragments. New table-structure work should start in schema/logical/*.toml and be generated into driver-specific SQL before it is composed into executable migrations:

bash crates/aether-data/runtime/schema/compose_schema.sh generate
bash crates/aether-data/runtime/schema/compose_schema.sh compose
bash crates/aether-data/runtime/schema/compose_schema.sh check

schema/generated/** is machine-written by aether-data-schema; it is checked in to make generator drift reviewable, not because runtime reads it. Edit schema/logical/*.toml instead. Use schema/overrides/** only as an exception bucket for driver-specific SQL that cannot live cleanly in logical schema or the normal driver fragments.

compose_schema.sh check also verifies that required baseline/portable table-creation SQL is represented in schema/logical. This is the guardrail that keeps table structure from drifting back into three manually maintained definitions.

For executable fragments that have not been promoted to generated output yet, edit fragments under schema/drivers/{postgres,mysql,sqlite} directly, run compose, then run check. Do not edit baseline executable SQL and fragments independently.

When adding a table:

1. Add or update schema/logical/*.toml first for the table structure.
2. Run schema/compose_schema.sh generate and inspect the generated driver SQL.
3. Add or update the executable driver-specific migration/fragments only for deployment compatibility or generator gaps.
4. Add or update repository contracts in aether-data-contracts if other crates need the new shape.
5. Add driver repository implementations only for the drivers that are actually supported for that domain.
6. Wire new repositories through src/backend/read.rs or src/backend/write.rs only after the implementation exists for each selected driver.
7. Update docs/architecture/data-layer.md when ownership, import, or logical type policy changes.

Known Cleanup Targets

These are intentionally staged to keep the multi-database refactor reviewable:

1. Retire the compatibility re-export paths once downstream crates use aether-data-contracts for contracts and aether-data only as the runtime composition facade.
2. Group repository domains once file-level names are stable. Likely groups: identity, auth config, provider catalog, runtime tasks, wallet/billing, usage, stats, and proxy nodes.
3. Continue shrinking Postgres stats SQL modules where useful by moving shared SQL fragments and row-mapping helpers behind focused backend/stats/* modules.
4. Split very large usage, wallet, and stats implementations into focused query/mapping/read/write/test modules before considering any additional crates. New crate boundaries should follow stable ownership, not file size alone.