Aether/docs/architecture/usage-counter-rootfix.md

Usage Counter Root Fix

This worktree implements the first production cut of the root fix for usage-related lock contention and hot-row pressure.

Problem

The current Postgres write path mixes:

• request facts (usage, audit snapshots, settlement state)
• shared counters (api_keys, provider_api_keys, global_models)
• provider quota window JSON updates
• wallet settlement writes

That means a single request can hold a transaction while touching multiple shared rows, and high-frequency traffic can serialize on the same api_key_id / provider_api_key_id / wallet rows.

Relevant code paths:

• crates/aether-data/src/repository/usage/postgres/mod.rs
• crates/aether-data/src/repository/settlement/postgres.rs
• crates/aether-usage-runtime/src/runtime.rs

Goal

Remove shared-hot-row updates from the request path without losing correctness.

The request path must become:

1. write immutable request facts
2. write durable delta records
3. commit

All shared counters must be derived later by a worker.

Target Architecture

1. Facts first

Keep usage as the source of truth for per-request facts:

• request identity
• status transitions
• billing state
• token / cost / latency payloads
• audit snapshots

2. Durable counter outbox

Add a new append-only delta table, for example:

usage_counter_deltas

Each row should represent one logical contribution:

• kind (api_key, provider_api_key, model, provider_monthly, proxy_node, management_token, api_key_last_used, window)
• target_id
• delta fields
• request id / revision
• processed_at

This table is the bridge between request facts and derived counters.

3. Flush worker

Add a background worker that:

1. reads unprocessed deltas with FOR UPDATE SKIP LOCKED
2. aggregates them in memory by (kind, target_id)
3. applies one UPDATE per target
4. marks the source delta rows processed in the same transaction

This keeps memory useful as a buffer, but never as the only source of truth.

4. Read models

Move high-read counters to separate tables or compact materialized read models:

• api_key_usage_counters
• provider_api_key_usage_counters
• model_usage_counters
• provider_monthly_usage_counters
• provider_api_key_window_usage_counters

Keep the original business tables (api_keys, provider_api_keys, global_models) for configuration and compatibility only.

Implementation Boundary

This branch should land the fix in phases, but the direction must not change:

1. Postgres gets the durable outbox and counter flush path first, because the reported production lock wait is on Postgres row locks.
2. MySQL and SQLite keep their current usage write behavior until their smaller deployment paths are moved to the same contract.
3. Request-path Postgres writes may still write usage, audit blobs, routing snapshots, and settlement pricing snapshots. They must not directly update shared aggregate rows.
4. Compatibility mirror columns may be updated by the flush worker only, never by the request transaction.
5. Dashboard/statistics read paths should be migrated after the write pressure is removed, otherwise we risk mixing a large read refactor with the lock fix.

The first executable migration creates the outbox. The first code patch makes upsert_usage_record enqueue deltas in the same transaction as the usage fact write, then a background worker batches those deltas with FOR UPDATE SKIP LOCKED. Dedicated counter read-model tables remain a follow-up after the request-path lock pressure is removed.

Locking Model

Keep

• advisory lock per request_id for idempotent request transitions
• wallet row lock only inside settlement, where correctness depends on it

Remove from request path

• direct UPDATE api_keys
• direct UPDATE provider_api_keys
• direct UPDATE global_models
• direct UPDATE providers.monthly_used_usd
• direct FOR UPDATE on provider quota JSON for request-level usage windows

Memory Cache Rules

Allowed:

• short TTL snapshot cache for read-only admin UI data
• short TTL + single-flight cache for provider/model candidate selection rows
• worker-side delta aggregation buffer
• per-key last-used-at max tracking before flush

Not allowed:

• using memory as the only accounting source
• using memory as the only settlement source
• depending on a clipped Redis stream as the only record of usage

Rollout Plan

1. Stop counting pending / streaming as shared counter contributions.
2. Introduce the delta outbox and worker.
3. Redirect request path to facts + outbox only.
4. Migrate reads to the new counter tables.
5. Decommission synchronous hot-row updates.
6. Move provider quota windows out of request transactions.

Implemented In This Branch

• usage_counter_deltas durable outbox for api key, provider api key, model, provider monthly, proxy node, management token, and api key last-used counters.
• Postgres usage upsert writes request facts plus outbox rows, not shared counter rows.
• Postgres settlement enqueues provider monthly usage deltas instead of updating providers.monthly_used_usd in the request transaction.
• Gateway request-adjacent proxy node, management token, and api key last-used writes enqueue durable deltas and fall back to direct writes only when the usage writer is unavailable.
• Gateway provider/model candidate selection reads use a 5 second in-memory TTL cache with per-key single-flight. Provider/routing catalog writes invalidate this cache alongside provider transport and scheduler affinity caches.
• Gateway maintenance worker flushes deltas in batches and aggregates in memory inside the worker before applying compatibility counter updates.
• Daily quota lookup index on (user_entitlement_id, usage_date) removes the avoidable aggregate scan in quota settlement checks.
• Hot counter columns are widened to bigint where old bootstrap schemas still used integer, preventing long-running counter overflow.

Integration Pressure Tests

The hotspot benchmarks start a managed local Postgres instance, run the migrations, seed a single hot target, then monitor pg_stat_activity while the load is running. Use a separate target directory when the main worktree target lock is not writable.

Usage write path, one hot api_key / provider_api_key / global_model:

CARGO_TARGET_DIR=/tmp/aether-rootfix-target \
cargo run -p aether-testkit --bin usage_counter_hotspot_baseline -- \
  --requests 5000 \
  --concurrency 200 \
  --flush-interval-ms 50 \
  --monitor-interval-ms 20 \
  --output /tmp/usage_counter_hotspot_after_5000.json

Settlement path, one hot provider monthly counter:

CARGO_TARGET_DIR=/tmp/aether-rootfix-target \
cargo run -p aether-testkit --bin usage_settlement_hotspot_baseline -- \
  --requests 5000 \
  --concurrency 200 \
  --flush-interval-ms 50 \
  --monitor-interval-ms 20 \
  --output /tmp/usage_settlement_hotspot_after_5000.json

Auxiliary hot counters, one hot proxy node / management token / api key last-used target:

CARGO_TARGET_DIR=/tmp/aether-rootfix-target \
cargo run -p aether-testkit --bin usage_aux_counter_hotspot_baseline -- \
  --requests 5000 \
  --concurrency 200 \
  --flush-interval-ms 50 \
  --monitor-interval-ms 20 \
  --output /tmp/usage_aux_counter_hotspot_after_5000.json

Latest local run on this worktree:

• usage hotspot: 5000 requests, 200 concurrency, p95 173 ms, 0 failures, 15000 outbox rows processed, 0 pending rows, 0 api_keys / provider_api_keys / global_models update waiters.
• settlement hotspot: 5000 requests, 200 concurrency, p95 39 ms, 0 failures, 5000 provider monthly deltas processed, providers.monthly_used_usd = 5.0, 0 provider update waiters.

Run the auxiliary counter hotspot after changing outbox schemas or gateway fallback routing; it should drain all pending outbox rows and report zero request-path waiters for proxy_nodes, management_tokens, and api_keys.

Runtime Observability

The same outbox health signals used by the pressure tools are exposed through admin runtime endpoints:

• GET /api/admin/system/stats
• GET /api/admin/monitoring/system-status
• GET /api/admin/stats/performance/providers

These responses include usage_counter:

• status: idle, catching_up, or backlogged
• outbox_pending_rows
• outbox_processed_rows
• oldest_pending_created_at_unix_secs
• oldest_pending_age_secs
• latest_processed_at_unix_secs
• pending_by_kind

Operational alerting should page when status = backlogged, when pending rows continue growing across several flush intervals, or when the oldest pending age stays above one minute. A transient non-zero backlog is acceptable during catch up bursts.

Remaining Correctness Locks

Wallet debit settlement and daily quota consumption still use database locks because they protect money/quota correctness, not derived counters. Removing those locks safely requires a separate wallet debit ledger/reservation worker:

1. request settlement writes an immutable debit intent keyed by request_id
2. a per-wallet worker claims intents with FOR UPDATE SKIP LOCKED
3. the worker applies balance changes and writes final settlement snapshots
4. request-facing APIs read pending/settled/insufficient_quota from the settlement snapshot

Do not replace this with memory-only balance caches. A cache may accelerate read-side availability estimates, but the durable ledger must remain the source of truth.

Acceptance Criteria

• request transactions no longer update shared counter rows
• counter updates become batchable and replayable
• lock wait time on api_keys / provider_api_keys drops sharply under concurrency
• wallet settlement remains correct and isolated
• dashboard/statistics reads stay fast from dedicated read models

Open Decisions

• table names for counter read models
• whether provider window counters live in Postgres only or are dual-written into Redis for display latency
• flush cadence and batch size defaults
• whether to keep compatibility snapshot columns as low-frequency mirrors