# Usage Counter Root Fix This worktree implements the first production cut of the root fix for usage-related lock contention and hot-row pressure. ## Problem The current Postgres write path mixes: - request facts (`usage`, audit snapshots, settlement state) - shared counters (`api_keys`, `provider_api_keys`, `global_models`) - provider quota window JSON updates - wallet settlement writes That means a single request can hold a transaction while touching multiple shared rows, and high-frequency traffic can serialize on the same `api_key_id` / `provider_api_key_id` / wallet rows. Relevant code paths: - `crates/aether-data/src/repository/usage/postgres/mod.rs` - `crates/aether-data/src/repository/settlement/postgres.rs` - `crates/aether-usage-runtime/src/runtime.rs` ## Goal Remove shared-hot-row updates from the request path without losing correctness. The request path must become: 1. write immutable request facts 2. write durable delta records 3. commit All shared counters must be derived later by a worker. ## Target Architecture ### 1. Facts first Keep `usage` as the source of truth for per-request facts: - request identity - status transitions - billing state - token / cost / latency payloads - audit snapshots ### 2. Durable counter outbox Add a new append-only delta table, for example: `usage_counter_deltas` Each row should represent one logical contribution: - `kind` (`api_key`, `provider_api_key`, `model`, `provider_monthly`, `proxy_node`, `management_token`, `api_key_last_used`, `window`) - `target_id` - delta fields - request id / revision - `processed_at` This table is the bridge between request facts and derived counters. ### 3. Flush worker Add a background worker that: 1. reads unprocessed deltas with `FOR UPDATE SKIP LOCKED` 2. aggregates them in memory by `(kind, target_id)` 3. applies one UPDATE per target 4. marks the source delta rows processed in the same transaction This keeps memory useful as a buffer, but never as the only source of truth. ### 4. Read models Move high-read counters to separate tables or compact materialized read models: - `api_key_usage_counters` - `provider_api_key_usage_counters` - `model_usage_counters` - `provider_monthly_usage_counters` - `provider_api_key_window_usage_counters` Keep the original business tables (`api_keys`, `provider_api_keys`, `global_models`) for configuration and compatibility only. ## Implementation Boundary This branch should land the fix in phases, but the direction must not change: 1. Postgres gets the durable outbox and counter flush path first, because the reported production lock wait is on Postgres row locks. 2. MySQL and SQLite keep their current usage write behavior until their smaller deployment paths are moved to the same contract. 3. Request-path Postgres writes may still write `usage`, audit blobs, routing snapshots, and settlement pricing snapshots. They must not directly update shared aggregate rows. 4. Compatibility mirror columns may be updated by the flush worker only, never by the request transaction. 5. Dashboard/statistics read paths should be migrated after the write pressure is removed, otherwise we risk mixing a large read refactor with the lock fix. The first executable migration creates the outbox. The first code patch makes `upsert_usage_record` enqueue deltas in the same transaction as the usage fact write, then a background worker batches those deltas with `FOR UPDATE SKIP LOCKED`. Dedicated counter read-model tables remain a follow-up after the request-path lock pressure is removed. ## Locking Model ### Keep - advisory lock per `request_id` for idempotent request transitions - wallet row lock only inside settlement, where correctness depends on it ### Remove from request path - direct `UPDATE api_keys` - direct `UPDATE provider_api_keys` - direct `UPDATE global_models` - direct `UPDATE providers.monthly_used_usd` - direct `FOR UPDATE` on provider quota JSON for request-level usage windows ## Memory Cache Rules Allowed: - short TTL snapshot cache for read-only admin UI data - short TTL + single-flight cache for provider/model candidate selection rows - worker-side delta aggregation buffer - per-key last-used-at max tracking before flush Not allowed: - using memory as the only accounting source - using memory as the only settlement source - depending on a clipped Redis stream as the only record of usage ## Rollout Plan 1. Stop counting `pending` / `streaming` as shared counter contributions. 2. Introduce the delta outbox and worker. 3. Redirect request path to facts + outbox only. 4. Migrate reads to the new counter tables. 5. Decommission synchronous hot-row updates. 6. Move provider quota windows out of request transactions. ## Implemented In This Branch - `usage_counter_deltas` durable outbox for api key, provider api key, model, provider monthly, proxy node, management token, and api key last-used counters. - Postgres usage upsert writes request facts plus outbox rows, not shared counter rows. - Postgres settlement enqueues provider monthly usage deltas instead of updating `providers.monthly_used_usd` in the request transaction. - Gateway request-adjacent proxy node, management token, and api key last-used writes enqueue durable deltas and fall back to direct writes only when the usage writer is unavailable. - Gateway provider/model candidate selection reads use a 5 second in-memory TTL cache with per-key single-flight. Provider/routing catalog writes invalidate this cache alongside provider transport and scheduler affinity caches. - Gateway maintenance worker flushes deltas in batches and aggregates in memory inside the worker before applying compatibility counter updates. - Daily quota lookup index on `(user_entitlement_id, usage_date)` removes the avoidable aggregate scan in quota settlement checks. - Hot counter columns are widened to `bigint` where old bootstrap schemas still used `integer`, preventing long-running counter overflow. ## Integration Pressure Tests The hotspot benchmarks start a managed local Postgres instance, run the migrations, seed a single hot target, then monitor `pg_stat_activity` while the load is running. Use a separate target directory when the main worktree target lock is not writable. Usage write path, one hot `api_key` / `provider_api_key` / `global_model`: ```sh CARGO_TARGET_DIR=/tmp/aether-rootfix-target \ cargo run -p aether-testkit --bin usage_counter_hotspot_baseline -- \ --requests 5000 \ --concurrency 200 \ --flush-interval-ms 50 \ --monitor-interval-ms 20 \ --output /tmp/usage_counter_hotspot_after_5000.json ``` Settlement path, one hot provider monthly counter: ```sh CARGO_TARGET_DIR=/tmp/aether-rootfix-target \ cargo run -p aether-testkit --bin usage_settlement_hotspot_baseline -- \ --requests 5000 \ --concurrency 200 \ --flush-interval-ms 50 \ --monitor-interval-ms 20 \ --output /tmp/usage_settlement_hotspot_after_5000.json ``` Auxiliary hot counters, one hot proxy node / management token / api key last-used target: ```sh CARGO_TARGET_DIR=/tmp/aether-rootfix-target \ cargo run -p aether-testkit --bin usage_aux_counter_hotspot_baseline -- \ --requests 5000 \ --concurrency 200 \ --flush-interval-ms 50 \ --monitor-interval-ms 20 \ --output /tmp/usage_aux_counter_hotspot_after_5000.json ``` Latest local run on this worktree: - usage hotspot: 5000 requests, 200 concurrency, p95 173 ms, 0 failures, 15000 outbox rows processed, 0 pending rows, 0 `api_keys` / `provider_api_keys` / `global_models` update waiters. - settlement hotspot: 5000 requests, 200 concurrency, p95 39 ms, 0 failures, 5000 provider monthly deltas processed, `providers.monthly_used_usd = 5.0`, 0 provider update waiters. Run the auxiliary counter hotspot after changing outbox schemas or gateway fallback routing; it should drain all pending outbox rows and report zero request-path waiters for `proxy_nodes`, `management_tokens`, and `api_keys`. ## Runtime Observability The same outbox health signals used by the pressure tools are exposed through admin runtime endpoints: - `GET /api/admin/system/stats` - `GET /api/admin/monitoring/system-status` - `GET /api/admin/stats/performance/providers` These responses include `usage_counter`: - `status`: `idle`, `catching_up`, or `backlogged` - `outbox_pending_rows` - `outbox_processed_rows` - `oldest_pending_created_at_unix_secs` - `oldest_pending_age_secs` - `latest_processed_at_unix_secs` - `pending_by_kind` Operational alerting should page when `status = backlogged`, when pending rows continue growing across several flush intervals, or when the oldest pending age stays above one minute. A transient non-zero backlog is acceptable during catch up bursts. ## Remaining Correctness Locks Wallet debit settlement and daily quota consumption still use database locks because they protect money/quota correctness, not derived counters. Removing those locks safely requires a separate wallet debit ledger/reservation worker: 1. request settlement writes an immutable debit intent keyed by `request_id` 2. a per-wallet worker claims intents with `FOR UPDATE SKIP LOCKED` 3. the worker applies balance changes and writes final settlement snapshots 4. request-facing APIs read `pending/settled/insufficient_quota` from the settlement snapshot Do not replace this with memory-only balance caches. A cache may accelerate read-side availability estimates, but the durable ledger must remain the source of truth. ## Acceptance Criteria - request transactions no longer update shared counter rows - counter updates become batchable and replayable - lock wait time on `api_keys` / `provider_api_keys` drops sharply under concurrency - wallet settlement remains correct and isolated - dashboard/statistics reads stay fast from dedicated read models ## Open Decisions - table names for counter read models - whether provider window counters live in Postgres only or are dual-written into Redis for display latency - flush cadence and batch size defaults - whether to keep compatibility snapshot columns as low-frequency mirrors