`parse_codex_websocket_usage_limit_error` backfills `primary_reset_at` and
`primary_reset_after_seconds` from the error body whenever the embedded headers
did not supply them. Now that a named per-model limit no longer claims the
unprefixed window headers, that absence is exactly what a Spark 429 produces —
and `resets_at` on such an error is the Spark window's reset, so the backfill
put model-scoped timing back on the account's own quota.
Gate the backfill on the same ownership rule the window parsing uses.
Reported by Cursor Bugbot on the fork PR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`parse_codex_usage_headers` reads the unprefixed `x-codex-primary/secondary-*`
headers as the account's own quota. They are not: they carry whichever limit
governed the request, and `x-codex-active-limit` names it — `premium` for the
plan's own limit, or a metered feature such as `codex_bengalfox` for a named
per-model limit. On a request billed against a named limit the unprefixed
headers repeat that limit's windows verbatim.
So a single request to a model with its own limit writes that model's windows
into the account slots. The paid-window swap then makes it worse: a named
limit's secondary window is active, unlike the plan's disabled one, so the swap
promotes the model's weekly window into the account's weekly slot — the slot
the UI labels and the scheduler reads through `quota_usage_ratio`.
It also sticks. Both weekly windows share `window_minutes`, so
`codex_quota_same_window_identity` treats them as one window, and
`codex_quota_merge_same_window` drops an observation whose deadline is earlier
than the stored one. The two weeks start at different instants, so every later
account observation looks like a stale sample of a window that already rolled
over and is discarded until the model window's own deadline passes.
Observed on a `pro` key running both model families: one `gpt-5.3-codex-spark`
request replaced the account weekly window with the Spark weekly one, and the
~3000 plan-limit responses over the next 100 minutes were all discarded. The
account's real weekly usage never landed, and its reset time was reported nine
hours late.
The header set describes itself — every named limit announces
`x-codex-<feature>-limit-name` and carries its windows under the same prefix —
so parse the named families directly and only claim the unprefixed windows for
the account when no announced limit owns them. Responses without
`x-codex-active-limit` keep the previous behaviour.
This also stops the Spark windows from going stale: they were only ever written
by the `wham/usage` admin probe even though every response carries them.
Refs #746
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The response.completed fallback rebuilt every call item with responsesCallInput(), which returns '{}' for a function_call lacking arguments. Since '{}' is truthy, ensureToolCall overwrote arguments already collected from streamed delta events. Guard the completed branch with responsesCallHasInput (matching the output_item.done branch) so empty/default inputs no longer clobber streamed args, and align its dedupe key with the streaming phase to avoid duplicate tool-call rendering when an item has no id. Drop the now-dead '工具调用' fallbacks since responsesCallName never returns empty.
The stale-pending cleanup task previously hardcoded status_code=504 and a
generic timeout message for every usage row it finalized. When a request
had already been observed as failing — e.g. upstream Connection reset by
peer, watchdog 504, or an authenticated 4xx — the cleanup overwrote that
context with a misleading "服务器超时" outcome and 504 status, hiding the
real cause from the dashboards and customer.
Pull the most recent failed/cancelled candidate per stale request_id and,
if present, finalize the usage row with the candidate's status_code
(defaulting to 502 when none was recorded) and error_message. Requests
that have no terminal candidate (truly stuck pending/streaming) keep the
existing 504 + timeout-message behavior, since they really are timeouts
from the cleanup's perspective. Applied to all three SQL backends with
parameterized UPDATE statements.
The Postgres failed-candidate lookup orders by
COALESCE(finished_at, started_at, created_at) DESC, matching the MySQL
and SQLite ORDER BY clauses so the three backends pick the same
"most recent terminal candidate" under every NULL combination of timing
columns.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When the endpoint forces upstream_stream_policy=force_non_stream while
the client streams, the local stream candidate watchdog still preferred
timeouts.first_byte_ms — a non-stream upstream produces no early first
byte, so the watchdog fired before the HTTP request_timeout and aborted
otherwise-healthy attempts at ~300s.
Read upstream_is_stream from report_context and invert the priority:
non-stream upstreams use total_ms first, falling back to first_byte_ms
and then the default; streaming upstreams keep the previous order.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "upstream_is_stream" JSON key flows from the AI execution report
context producer (aether-ai-serving::report_context) through several
consumers — usage runtime metadata copy/move, gateway watchdog, sync
execution decision, observability handlers, and the per-driver usage
repositories. Each site spelled the key as a bare string literal, so a
producer-side rename would silently degrade every consumer to its
fallback (typically assuming streaming) with no compile-time signal.
Introduce a single pub const UPSTREAM_IS_STREAM_KEY in
aether-ai-formats (the lowest crate every consumer already depends on),
re-export from the crate root, and route producer + all map-style
consumers through it. The change is purely a string-literal → constant
swap; behaviour is identical.
Sites left as literals (intentional):
- `json!({"upstream_is_stream": ...})` macro keys, which must be string
literals at the macro layer; these are also API-response payload
field names (an external contract that should not silently track
internal report-context renames).
- SQL column accessors (`try_get::<...>("upstream_is_stream")`), which
refer to the database schema column, not the JSON key.
- Test fixtures and assertions, which validate the on-the-wire contract
and should keep verifying the actual string.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
