Compare commits

..
1662 Commits
Author SHA1 Message Date
elky 535ee098c3 fix(auth): reject unsigned admin identity headers 2026-08-18 11:12:17 +08:00
ZheFox b45df89ce4 Merge pull request #730 from zhefox/fix/responses-websocket-current
feat(gateway): add OpenAI Responses WebSocket mode
2026-08-17 21:14:50 +08:00
ZheFox c8118edf36 fix(ws): harden Responses connection lifecycle
Revalidate control policy per turn, isolate downstream credentials, and make planner/turn ownership cancellation-safe.

Preserve opaque protocol events, align configurable timeout semantics, and extend end-to-end security and settlement coverage.
2026-08-17 18:50:29 +08:00
AAEE86 4a0775c4ea style: apply cargo fmt across gateway and aether-ai crates 2026-08-17 14:53:53 +08:00
AAEE86 6fc02dad3e fix(ws): restore redacted PII in provider frames before client delivery
Responses WebSocket 只实现了脱敏的一半:请求侧 mask 之后,provider 事件帧在推给
客户端之前没有还原,于是 session 映射内的占位符以 <AETHER:EMAIL:...> 的形式直接
透给客户端。这里补齐响应侧,语义与 HTTP 路径对齐。

- 还原点是 relay loop 的最后一跳(send_client_message 之前、capture_client_frame
  之前),对应 HTTP 的 restore_sync_response_body / StreamingResponseRestorer 所在
  位置。审计与终态观测继续消费脱敏态事件,只有发往客户端的那一份拷贝被还原。
- 复用 privacy::restore_json_strings(改为 pub(crate))与
  RedactionSession::restore_text,不复制任何还原逻辑:只还原本 session mask 过的
  映射,未映射的占位符原样保留;type / model / id 等协议字段不可能命中 sentinel,
  因此不受影响。批量 {"chunks":[...]} 帧一并递归还原。
- session 生命周期:mask 仍然是 per-turn(slot 依旧每轮新建),但 session 改由连接
  持有,按有界 FIFO 留最近 8 轮。理由是 WS 的会话历史留在上游,continuation 只发
  增量输入,per-turn 释放会漏还原后续响应里回显的更早轮次占位符;HTTP 不会漏,是
  因为它每次重发整段历史、重新 mask 会派生出同一个 sentinel。被挤出窗口的轮次退回
  「占位符原样透传」,不会错误还原成别的值。
- 未命中还原时不改写字节;连接上没有任何 mask session 时(未启用脱敏)连事件 clone
  都不做。

测试:redaction.rs 新增 8 条单测(还原命中/批量帧/未映射占位符原样/未命中不改写/
无 session 不介入/空 session 不留存/审计侧入参不被改写/跨轮还原/窗口有界);
responses_websocket_e2e 新增一条用例,mock 上游回显收到的 input,断言上游只看到
占位符而客户端拿到真实邮箱。
2026-08-17 14:53:46 +08:00
AAEE86 dbf2809bd6 fix(ws): settle the previous attempt before transparent retry replanning
评审第 2 条。配额透明重试原来的顺序是「detach 旧 attempt → 规划并绑定新
attempt → 把旧 attempt 的结算排进队列」。规划因此读到的是旧 attempt 还没投射的
health / adaptive / pool 状态,而且旧 attempt 仍占着自己的 pool key lease——替代
key 的挑选看到的是一把仍被占用的 key,最坏情况下判成「无可用供应商」而放弃一次
本可以成功的重试。

普通的新 turn 早就挡住了这件事:client.rs 在处理 response.create 前调用
await_pending_turn_finalization,注释写的正是「不要让新 turn 基于陈旧的 health /
adaptive / pool 状态规划」。透明重试是同一个问题的另一条入口,漏了这一步。

现在顺序是:detach → 释放准入 → 结算旧 attempt 并等它落地 → 规划/绑定新 attempt。

新增 lifecycle::settle_turn_finalization:与 queue_turn_finalization 的区别只在于
「等」。后者把 handle 挂在连接上让 relay loop 继续跑,用在结算之后不再读取共享
状态的出口;前者用在必须先看到结算结果才能继续的路径上。

顺序用类型固定,而不是靠注释:settle_turn_finalization 返回
PreviousAttemptSettled,retry_active_turn_after_quota_exhaustion 要求这个参数。
凭证只能由 lifecycle 颁发(结算完成,或明确「没有 attempt 要结算」),所以把顺序
写反连编译都过不了。

重试失败路径随之变化:旧 attempt 已经结算,不再 resume 回去。logical turn 仍停在
Replanning,后续分支的 end() / finalize_active_turn 只清 logical turn、不交出
attempt,因此不存在重复结算。结算 outcome 取值不变(两条路径用的都是
terminal_outcome.unwrap_or_else(upstream_closed),而这条分支里 terminal_outcome
必为 Some——usage_limit_error 成立意味着有一个已解析的 error 终态帧)。

代价(都落在「重试失败」这一侧,且只影响已终态 attempt 的报告注解,不影响计费):
- 那条最终转发给客户端的 429 事件不再进旧 attempt 的 client capture;
  provider 侧 capture 早在 observe_upstream_frame 里就记下了。
- 如果转发 429 给客户端也失败,record_client_delivery_aborted 落在一个已经结算的
  attempt 上,成为 no-op。

测试:
- lifecycle:await_turn_finalization_handle 必须「等到落地」而不是「排进队列」
  (C6 依赖的性质);结算完成后规划才读状态的顺序型断言(计数器替身);结算任务
  panic 也必须放行调用方,不能卡死 relay loop。
- turn_state:Replanning 状态下 end() 不再交出第二个 attempt(无重复结算)。
- e2e 新增 provider_quota_exhaustion_transparently_retries_onto_another_key:
  mock 上游首轮只回 Codex 的 429 usage_limit_reached,网关换到第二把 key 重放同一个
  response.create;断言客户端看不到 429、上游被连两次、两次用的不是同一把 key、两个
  attempt 各留一条终态行(429 的那条 + 计费的那条)。已验证它在改动前后都通过——
  它覆盖的是整条路径可用,顺序由上面的单测确定性覆盖。
  夹具随之参数化出 ProviderFixture::CodexKeyPair:透明重试只有 Codex adapter 会
  开启,而 codex 候选要求 auth_type = oauth,所以这个夹具用未过期的 oauth 凭证。
2026-08-17 14:53:40 +08:00
AAEE86 1d3051cb89 refactor(ws): structured terminal observation without SSE text round-trips
评审第 5 条:Responses WebSocket 收到的本来就是结构化协议事件,但为了复用面向
SSE 的 push_line,观测路径要先把每个事件序列化成 data: {json}\n\n,解析器再
decode 回 Value——一次纯粹的往返。这个「伪 SSE」形状是随手拼的,一旦拼装函数
以后被加上换行或分块逻辑,观测结果就会和真实事件悄悄分叉。

aether-ai-formats:
- OpenAIResponsesProviderState::push_line 机械拆成 decode + push_event,
  push_line 现在只做解码。协议状态机一行未动,diff 里除函数签名外只有
  &value → value(value 从拥有改成借用,持有结构化事件的传输不必为了调用它
  先克隆一份)。
- StreamingStandardTerminalObserver::push_event 走 TerminalStreamParser::Standard,
  service tier 的记录方式与 push_line 完全相同。openai:image 的终态状态机按 SSE
  行做增量解析、没有结构化入口,返回 AiSurfaceFinalizeError 让调用方
  disable_with_error 标记 parser_error,而不是静默丢事件、把摘要留成「未观察到
  终态」。ProviderStreamParser 的其余三个格式同样返回 Err:机械拆分随时可做,
  但不建无调用方的接口。

WS 侧:
- 新增 responses/observation.rs 的 ResponsesStructuredTerminalObserver,直接消费
  frame.protocol_events() 借出的事件。包一层的意义是让「不再拼 SSE」成为类型层面
  的事实——这个类型没有任何接受字节的方法,改回 push_line 不可能悄悄发生。
  finish() 里的 Ok(None) / Err → disable_with_error 兜底也一并收进来。
- body capture 不动,仍然是 SSE 形状(data: 开头、\n\n 结尾):
  aether_usage_runtime::report 用 line.strip_prefix("data:") 解析被捕获的 body
  判定 StreamCapturedTerminalState,而它是 stream_report_represents_failure 的一个
  OR 项,换成结构化 JSON 会让终态判定恒为 Missing。capture_sse_event /
  capture_client_frame / websocket_event_as_sse_line 全部保留,原因写在模块文档
  注释里。这一层只换观测,不换捕获。

差分测试(8 个,aether-ai-formats):同一组事件序列分别走 push_line 与
push_event,断言 ExecutionStreamTerminalSummary 完全相等——批量 delta 序列、
completed 带 usage、合法 incomplete、error、response.failed、未知事件、
service tier、缺终态;外加 openai:image 拒绝结构化入口。两条入口不可能有
过滤差异:任何 Value 序列化出来都不会命中 decode_json_data_line 的 empty /
":" / "event:" / [DONE] 四个过滤条件。

turn.rs 里三个既有的 WS 观测测试改走结构化入口;SSE 形状的断言留在 capture 一侧。
验收:crates/aether-usage 零 diff。
2026-08-17 14:53:33 +08:00
AAEE86 59e27524da refactor(gateway): extract transport-neutral execution attempt lifecycle
评审第 4 条:responses/turn.rs 实际复制了一整套 HTTP execution lifecycle——
usage 写入、candidate 状态流转、health/adaptive 效果投射、pool key lease 释放、
body capture、账单失败判定,与 HTTP 的顺序和超时语义只能靠人工对齐。

新增 execution_runtime/attempt_lifecycle.rs,把一次 provider attempt 的记账收成
transport 中立的三段:

  ExecutionAttemptLifecycle::begin        pending usage 行 + Pending candidate
  ExecutionAttemptLifecycle::mark_started usage stream_started + Streaming candidate(幂等)
  ExecutionAttemptLifecycle::settle       终态四段,顺序不可重排:
                                            1 usage terminal(detachable,不可丢)
                                            2 candidate terminal
                                            3 provider 效果 + 超时兜底释放 lease
                                            4 execution report(作废账单不提交)

顺序、5s 分段超时常量、detachable 语义、「每个效果分支都释放 lease」「作废账单
一律不提交 report」这些不变量全部保持原样。

一并上移的辅助设施:
- AttemptStageGuard 取代 await_websocket_lifecycle_stage /
  await_detachable_lifecycle_stage,把「等多久」参数化:WS 用 Bounded(5s),
  HTTP 接线时用 Unbounded 即保持它现在的语义。
- AttemptBodyCapture 取代 append_capture / encode_stream_capture,把
  「缓冲 + 截断标志」两个字段收成一个类型(WS 侧四个字段变两个)。捕获内容
  仍然是 SSE 形状:usage runtime 按 data: 行解析被捕获的 body 来判定
  StreamCapturedTerminalState,换成结构化 JSON 会让终态判定恒为 Missing。
- C2/C3 的结算表本来就不含任何 WS 类型,随之上移。效果表分支与注释逐字未改,
  仅按新位置改名为 AttemptProviderEffect / classify_attempt_provider_effect。
  responses/settlement.rs 只保留 WS 专属的一件事:把 relay loop 的结算信号
  ResponsesWebSocketTurnOutcome 翻译成两个正交事实。

ResponsesProviderAttempt 现在只持有 WS 专有状态:lifecycle 句柄、deadline、
终态观测器、两侧 capture、准入、provider/delivery 事实。plan / trace_id /
report_kind / report_context / candidate 起始时间戳都归 lifecycle。

HTTP 侧不接线:execution_runtime/stream/execution.rs 的
DirectPassthroughFinalizerCore(38 字段)与 failover / oauth 重试 / prefetch 深度
纠缠,无法在「行为等价 + 单 commit 可验证」的前提下改动。逐调用点映射表写在
模块文档注释里作为后续 PR 的接线依据。验收:git diff 对
execution_runtime/stream/ 与 crates/aether-usage 均为零 diff。

新增 6 个测试:效果段超时后仍走兜底 lease 释放、Unbounded 会一直等、detachable
写入在调用方停止等待后仍跑完、settle 四段顺序(计数器替身)、body capture 的
SSE 形状与编码状态(并显式记录默认上限是 usize::MAX,截断分支不可达)、
candidate error_type 映射。
2026-08-17 14:53:25 +08:00
AAEE86 247e7105a2 fix(ws): bill a provider-reached terminal even when client delivery fails
评审第 5 条后半:provider 终态已经到达、只是 gateway 写客户端 socket 失败时,
relay loop 用 client_disconnected() 覆盖了结算信号,于是一条供应商已经完成推理
并消耗了 token 的响应被记成 void billing、candidate 记 Cancelled、不投射供应商
效果、也不提交 execution report。上游成本凭空消失。

结算表只改一行:作废账单的条件从
    provider.cancelled_by_provider() || delivery.is_aborted()
收紧为
    provider.cancelled_by_provider() || (delivery.is_aborted() && !provider.is_terminal())

于是 Terminal{cancelled=false} + delivery Aborted 与 delivery Complete 落在同一侧:
Billed、candidate Success 或 Failed、投射供应商效果、提交 execution report。
状态码随之变成纯 provider 事实(不再把 200 改写成 499);作废分支的 provider
状态码本身就是 499,取值不变。

依据:供应商已经完成推理并消耗 token,客户端还能用 previous_response_id 续取
这条响应。供应商没给出终态时(客户端先走了)仍然作废,这一侧未改。

配套改动:
- connection.rs 写客户端失败处改为 record_client_delivery_aborted(reason) +
  settle_signal_for_client_delivery_failure(terminal_outcome):provider 终态已到达
  就用那条终态作结算信号,不再无条件覆盖。投递失败原因也不再谎称
  「客户端在终态前断开」。
- 投递结果记在 attempt 上而非 logical turn 上:结算按 attempt 进行,且配额透明
  重试时各 attempt 的投递结果彼此独立。
- report_context 新增 websocket_client_delivery="aborted" 与
  websocket_client_delivery_reason,只增字段不改既有字段,便于事后区分
  「客户端拿到了」和「客户端没拿到但已计费」。
- candidate error_type 新增 client_delivery_failed(原先这个场景写的是
  websocket_cancelled)。它排在供应商侧分类之前:这条记录之所以特别正是因为
  内容没送到客户端,供应商侧判定仍由 candidate_status 与 error_message 保留。
- finish_summary 改用作废判定而非「投递失败」判定:provider 终态已到达时摘要
  必须保留真实的 finish_reason 与 usage,否则计费记录会被写坏。

e2e 期望值变化:client_disconnect_mid_turn_still_settles_the_usage_row 改名为
client_disconnect_before_any_provider_output_settles_a_void_row,并补上
「不计费 + status=cancelled + status_code=499」的断言。原用例的 mock 行为是
StallAfterCreated(只发 response.created 就静默),provider 从未给出终态,所以
它走的是未改动的作废一侧;原来的文档注释说「must still be billed」与实际语义
不符,一并纠正。真正被修正的那一行无法在 e2e 里确定性触发——它取决于 relay
loop 的 select! 先观察到上游终态帧还是先观察到已关闭的客户端 socket,是构造性
竞态——因此由 relay 级单测确定性覆盖,e2e 里以注释指向这两个单测。

新增 7 个测试:结算表修正行(并与「投递成功」逐字段对照,只有 candidate 错误
分类不同)、无终态时仍作废、供应商声明取消即使送达也不计费、结算信号选择、
已记录的投递失败不被结算信号覆盖、relay 级「终态到达 + 客户端已关闭 ⇒ Billed /
Success / ProviderSuccess / 已提交 report 且 usage 完整保留」及其镜像、
report_context 只增不改。
2026-08-17 14:53:12 +08:00
AAEE86 dc3743aecf refactor(ws): 拆分 LogicalTurn 与 ProviderAttempt,结算改表驱动
评审第 5 条:一个 ResponsesWebSocketTurn 同时代表 logical turn 和 provider
attempt,finalize() 又用 outcome.cancelled() 一个布尔驱动 billing、candidate
状态和供应商效果,于是 provider 终态已经到达、只是最后一跳写客户端失败时,
供应商事实会被 Cancelled 覆盖掉。

- ResponsesWebSocketTurn → ResponsesProviderAttempt,
  ActiveResponsesWebSocketTurn → ActiveProviderAttempt:类型名字明确它只代表
  一次上游执行,logical turn 由 C1 落地的 LogicalTurn 承担。
- 新增 settlement.rs:AttemptProviderOutcome × AttemptClientDelivery 两个正交
  事实,classify_attempt_settlement 一张表推出 status_code / billing /
  candidate 状态 / candidate 错误分类 / 供应商效果 / 是否提交 execution report。
- attempt 观察到 provider 终态即记录 provider_outcome。结算信号
  ResponsesWebSocketTurnOutcome 只回答「为什么现在结算」:ProviderTerminal 与
  Failure 对 provider 是权威的,Cancelled 只描述客户端/连接层面的停止,不再
  覆盖已观察到的 provider 事实。
- candidate 状态与 candidate 错误分类分开输出:现状存在
  「missing_terminal=true 而记账层判 Success」的组合(report kind 不要求观察到
  终态事件时),会写出 status=Success + error_type=stream_missing_terminal_event,
  这个组合必须原样保留。

classify_responses_websocket_turn_effect 的判定表原样搬入 settlement.rs,分支
和顺序均未改动,两个既有不变量测试随之迁移。

行为等价。结算表当前口径与拆分前完全一致:客户端投递失败仍与「供应商声明取消」
落在同一侧(作废账单、candidate 记 Cancelled、只释放 lease、不提交 execution
report),即使 provider 终态已经到达——这一行由
settlement_table_row_client_delivery_failure_currently_voids_a_reached_terminal
锁住现状,修正它是下一步独立的行为修正。

新增 15 个测试:outcome → 双事实映射表逐行(含 stream_timeout 只在 504 失败一族
成立、provider 终态即使 504 也不投射流式超时)、结算表逐行、投递失败时
forced_error 必须为 None、已观察终态不被 Cancelled 覆盖、以及跨整张表的
「每个分支都释放 pool key lease」「作废账单一律不提交 report」不变量。
2026-08-17 14:53:05 +08:00
AAEE86 1c5ee5228c refactor(ws): 用 ResponsesTurnState 收敛连接 turn 状态
评审第 2 条:BoundResponsesConnection 用 response_in_flight、active_turn、
active_response_create 三个可独立变化的字段编码同一件事,8 种组合里只有 3 种
合法,非法组合只能靠调用点的 if 和「记得同时改另外两个字段」来避免。

三字段合并为一个 ResponsesTurnState:

  Idle                                  没有进行中的 logical turn
  Responding { logical, attempt }        logical 与 attempt 必须同时存在
  Replanning { logical }                 attempt 已取走去结算/重绑,logical 仍在

Replanning 不是新概念:配额透明重试期间现状就处于这个状态,只是靠
Option::take 意外得到。转换只能走 begin / detach_attempt / resume / end,
response_in_flight 与「是否接受新 response.create」都由变体推导。

由此消除的运行时不变量(原来全靠调用点自觉):
- 有 attempt 必有 logical turn
- response_in_flight 与 attempt 同生共死(原来 client 写失败后
  active_turn=None 而 response_in_flight 仍为 true)
- logical turn 结束时必须清 attempt:原来 `active_response_create = None`
  在 connection.rs 里手写 13 处,漏一处就残留;现在只有 end() 一个出口
- 上游绑定返回的连接不再自带 response_in_flight=true 的半成品状态

同时删除 update_response_in_flight:Started 帧把已经是 true 的字段再设一次,
Close 帧因为没有解析出的 frame 而根本不触发,是纯冗余写;它在 Idle 态收到
Started 帧时还会把 response_in_flight 置真,从而永久阻塞后续 response.create。

行为等价。ActiveResponsesWebSocketRequest 改名 LogicalTurn 并随状态机移入
新的 turn_state.rs;状态机对 attempt 类型泛型化,测试用轻量替身驱动同一套
转换逻辑,无需 AppState 或真实 socket。
2026-08-17 14:52:57 +08:00
AAEE86 9d80281b53 fix(ws): route WS planning and continuation through PII redaction 2026-08-17 14:52:49 +08:00
AAEE86 3b036299d4 fix(ws): enforce absolute upstream handshake and initial-message deadlines 2026-08-17 14:52:39 +08:00
AAEE86 f70ae68273 fix(ws): treat max_output_tokens incomplete as legitimate terminal 2026-08-17 14:52:33 +08:00
AAEE86 1353d76e07 feat(frontend): Responses WebSocket 配置与用量展示
provider 表单支持开启 Responses WebSocket;用量列表、状态与详情
区分 WebSocket 请求。
2026-08-17 14:52:25 +08:00
AAEE86 621a528083 test(ws): Responses WebSocket 端到端套件接入 CI
补齐 aether-integration-tests 的 responses_websocket_e2e 集成测试,
并把 CI 的 scenario 任务从 --bins 改为 --bins --tests,否则该套件
不会被执行。
2026-08-17 14:51:24 +08:00
AAEE86 a498875591 feat(gateway): Responses WebSocket 连通性探针
新增 aether-codex-ws-probe 与 aether-openai-responses-ws-probe 两个
二进制,用于在不暴露凭据的前提下验证上游 WebSocket 端点可用性:凭据
只从环境变量读取,不写入日志。公共流程放在
bin/support/responses_ws_probe.rs,各 profile 只负责自己的鉴权与
请求头要求。
2026-08-17 14:51:18 +08:00
AAEE86 71b54070e8 feat(gateway): Codex/OpenAI Responses WebSocket 代理模式
在 /v1/responses 上支持 WebSocket 升级,把客户端帧中继到上游 Codex /
OpenAI Responses WebSocket 端点,同时保持既有的路由、鉴权、配额与用量
语义:

- 路由与准入:control/route/ai.rs 识别 WebSocket 升级请求;
  websocket/ingress.rs 复用 API Key 鉴权、IP 规则与并发许可,并引入
  独立的 WebSocket 连接许可
- 中继:websocket/responses/* 按 connection / session / turn 分层,
  帧解析归一化、socket 写入有界、continuation 保持调度亲和性
- 配额:orchestration/codex_quota_breaker.rs 在账号配额耗尽时熔断并
  自动恢复,不再直接断开客户端连接
- 用量:每个 turn 的终态用量落库,request_metadata 记录
  websocket_mode / websocket_transport,管理端与 usage 视图暴露
  is_websocket
- 管理端:provider 可配置 Responses WebSocket 开关
2026-08-17 14:50:33 +08:00
ZheFox 9a0d346ff3 Merge pull request #728 from zhefox/main
fix(gateway): stop candidate persistence retry storms
2026-08-17 14:28:11 +08:00
ZheFox 32944538e9 fix(gateway): stop candidate persistence retry storms 2026-08-17 13:49:12 +08:00
ZheFox 0b17026eab Merge pull request #726 from zhefox/main
fix(codex): restore upstream model discovery
2026-08-15 20:16:12 +08:00
ZheFox b13d9b9b40 fix(codex): restore upstream model discovery 2026-08-15 19:36:28 +08:00
ZheFox b7fca851b8 Merge pull request #724 from zhefox/main
fix(codex): serve versioned dynamic model catalogs
2026-08-14 19:08:53 +08:00
zhefox 810c3dfe2b fix(codex): serve versioned dynamic model catalogs 2026-08-14 18:41:44 +08:00
elky a1d64e5239 fix(routing): preserve allowlist edits and save state 2026-08-14 11:43:24 +08:00
elky fb33ea57b0 Merge PR #715: decouple routing model overrides 2026-08-14 11:10:21 +08:00
elky 5b0c763086 fix(codex): fence concurrent quota updates 2026-08-14 09:28:07 +08:00
elky f3a12c1008 fix(ai): preserve Codex image edit validation 2026-08-13 11:31:17 +08:00
elky ca35e09eaa Merge pull request #718 from zjm54321/fix/custom-image-edit-json 2026-08-13 11:00:15 +08:00
elky 8cf381b0c3 feat(codex): add OAuth fingerprint convergence 2026-08-13 09:57:17 +08:00
elky 654c4f6978 fix(auth): preserve turnstile while typing email 2026-08-12 16:56:18 +08:00
elky edb8362adc fix(provider): omit default model test temperature 2026-08-12 16:56:18 +08:00
elky 29fa4aed19 perf(gateway): raise default server pool floor 2026-08-12 16:56:18 +08:00
zjm54321 41e93858e1 refactor(ai): simplify image edit serialization 2026-08-11 00:36:33 +08:00
zjm54321 8d918d0459 fix(ai): serialize image edits with images array 2026-08-11 00:30:00 +08:00
ZheFox 3a759fae89 Merge pull request #712 from zhefox/fix/claude-code-conversion-api-key-toggle
fix: restore Claude Code conversion and user API key toggles
2026-08-05 14:53:03 +08:00
zhefox 985ff3c36a test(gateway): align claude_code endpoint reconciliation 2026-08-05 14:34:20 +08:00
zhefox 908d4f2603 style(provider): apply rustfmt to claude_code tests 2026-08-05 13:58:46 +08:00
zhefox 4d67569873 fix(gateway): support claude_code cross-format Claude messages 2026-08-05 13:57:20 +08:00
ZheFox 1aab31a148 Merge pull request #710 from zhefox/main
fix: align Responses compatibility, routing, and model permissions
2026-08-03 19:31:04 +08:00
zhefox aedff9a704 fix(provider): validate mapped model reasoning effort 2026-08-03 19:22:51 +08:00
zhefox 669f4bddc5 fix: align Responses routing and model permissions 2026-08-03 18:48:01 +08:00
zbs 1a4eede34d fix(routing): decouple model overrides from allowed scope 2026-08-03 07:52:49 +08:00
elky 0318808db9 fix(providers): allow transfer limits on creation 2026-07-31 13:35:08 +08:00
elky 06f5d3c8c0 fix(gateway): complete worker registration cleanup 2026-07-31 11:32:07 +08:00
elky 082407fa51 Merge PR #697: prevent duplicate worker registrations 2026-07-31 11:11:14 +08:00
fawney19 6688ee26db Merge pull request #702 from MMEXA/fix/reconcile-auth-channel-mismatch-formats
fix(gateway): 修复批量更新 API 格式时的认证通道状态冲突
2026-07-31 10:28:37 +08:00
elky beb003b7ad feat(models): add external catalog proxy selection 2026-07-31 09:32:25 +08:00
MMEXA 6ecfe0f0a1 fix(gateway): reconcile auth mismatch formats on key update 2026-07-30 22:14:08 +08:00
ZheFox 12057db476 Merge pull request #701 from zhefox/main
Persist OpenAI Responses continuation history across instances
2026-07-30 21:08:34 +08:00
ZheFox ff47d8d48a fix(gateway): route response history through ai seam 2026-07-30 20:34:41 +08:00
ZheFox ef5f36cc2b fix(ai): satisfy response history clippy checks 2026-07-30 20:13:36 +08:00
ZheFox 84022c4d48 Merge upstream/main into main 2026-07-30 19:40:39 +08:00
ZheFox 118f441029 feat(gateway): persist OpenAI Responses continuation history 2026-07-30 19:26:52 +08:00
elky 20399b004d Merge PR #700: fix admin pool batch update body buffering
Preserve main's failover and usage metadata fixes, restore default tunnel regression coverage, and satisfy current Clippy.
2026-07-30 17:56:37 +08:00
elky 050eb77508 fix(ai): harden responses replay and failure diagnostics 2026-07-30 17:19:54 +08:00
elky 1ab4f079c9 fix(gateway): restore failover and usage diagnostics 2026-07-30 09:12:11 +08:00
MMEXA 6c733f7590 fix(usage): preserve request diagnostics in event seeds 2026-07-30 06:44:59 +08:00
MMEXA d7d8db45ba test(gateway): align tunnel error fixture with failover policy 2026-07-30 06:44:59 +08:00
MMEXA 8cf9af79da fix(ci): remove redundant usage policy update 2026-07-30 05:45:38 +08:00
MMEXA e55793c765 fix(ci): satisfy gateway clippy on upstream baseline 2026-07-30 05:14:34 +08:00
MMEXA d8902ea612 fix(gateway): buffer admin pool batch update bodies 2026-07-30 05:14:34 +08:00
elky a04673a90d feat(gateway): harden failover and payload handling
Retry pre-response transport failures across candidates with an explicit stop policy, and propagate end-to-end timing into usage records and UI diagnostics.

Remove legacy body, import, cookie, PII, and tunnel replay caps while preserving optional operator-configured gateway limits.
2026-07-30 01:03:27 +08:00
ZheFox a97acc07fc Merge pull request #698 from zhefox/main
fix(ci): stabilize cross-platform workflow checks
2026-07-29 22:16:17 +08:00
zhefox f8000012f7 fix(ci): stabilize cross-platform workflow checks 2026-07-29 21:55:43 +08:00
worker-2 6080f8cc88 fix(gateway): stabilize worker task records
Key worker boot records by task so process restarts update the existing
row instead of registering another row for each gateway instance.

Closes #693
Confidence: high
Scope-risk: narrow
2026-07-29 17:27:51 +08:00
ZheFox 37df5b93b1 Merge pull request #696 from zhefox/main
Fix client metadata handling across formats
2026-07-28 18:12:59 +08:00
ZheFox e53abdaec2 Merge branch 'fawney19:main' into main 2026-07-28 18:12:28 +08:00
ZheFox 2db32ea97e Merge branch 'main' of https://github.com/zhefox/Aether 2026-07-28 17:40:46 +08:00
ZheFox 581897ee74 fix(formats): ignore responses client metadata across targets 2026-07-28 17:40:41 +08:00
ZheFox 9a88f966d8 Merge pull request #695 from zhefox/main
Enhance provider capabilities and clean up OAuth keys
2026-07-28 13:58:33 +08:00
ZheFox 9d9316e434 Merge branch 'fawney19:main' into main 2026-07-28 13:56:34 +08:00
ZheFox 1b697b1111 feat(providers): support FedRAMP Codex agent identity registration 2026-07-28 13:29:30 +08:00
ZheFox 3043982486 fix(providers): derive Codex primary quota label from window 2026-07-28 12:57:45 +08:00
ZheFox 0bf92ffffc feat(providers): advertise responses API agent capability 2026-07-28 12:02:03 +08:00
ZheFox f0f87b56a3 feat(providers): add credential-fenced OAuth key cleanup 2026-07-28 11:32:11 +08:00
elky 4148ab1931 fix(routing): harden routed pool scheduling 2026-07-27 22:06:28 +08:00
elky 550cc36760 feat(providers): expand OAuth account management
Add Claude Code manual and cookie authorization, including redacted batch tasks. Harden OAuth imports, duplicate replacement, provider dialogs, and related account-management tests.
2026-07-27 15:53:28 +08:00
elky 531cf11025 feat(gateway): harden provider request execution
Preserve exact request payloads and model client surface and API operation explicitly.

Add Anthropic compatibility profiles, bounded stream commitment, and scoped OAuth retry behavior across provider transports.
2026-07-27 09:36:31 +08:00
elky 79b70f7b5c fix(frontend): align sidebar collapse button 2026-07-26 15:07:51 +08:00
elky 10d369f59c feat(providers): add provider transfer limits 2026-07-26 15:06:56 +08:00
elky 2ef7ac79bc feat(frontend): add collapsible navigation sidebar
Persist the desktop sidebar state, provide accessible compact navigation tooltips, and cover the collapsed navigation markup with a focused component test.
2026-07-25 21:28:51 +08:00
elky 778cfb1a5c feat(data): complete portable SQL backend parity
Align MySQL and SQLite schemas, migrations, usage, stats, export, and backfill behavior with the shared data contracts. Extend gateway startup and maintenance support across all SQL drivers.
2026-07-25 21:28:21 +08:00
elky 764e9fd131 feat(frontend): improve provider detail drawer and pool actions 2026-07-25 11:16:59 +08:00
elky 387134ca87 fix(models): correct fast pricing and online sync 2026-07-24 01:45:38 +08:00
elky a0767d957c fix(frontend): synchronize pool account state 2026-07-23 16:20:18 +08:00
ZheFox b94ef91d07 Merge pull request #692 from zhefox/main
Sync global model prices and track online pricing sources
2026-07-23 16:02:17 +08:00
ZheFox e7910751d9 Merge branch 'fawney19:main' into main 2026-07-23 15:19:48 +08:00
ZheFox 1d2655432d feat(models): track online pricing sources and unsupported fields 2026-07-23 15:18:08 +08:00
ZheFox 323273ff30 feat(models): sync global model prices from online catalog 2026-07-23 13:29:28 +08:00
ZheFox fb2009c65b Merge pull request #691 from zhefox/main
fix(formats): ignore Responses client transport metadata
2026-07-23 12:18:40 +08:00
ZheFox e186cc6848 Merge branch 'main' of https://github.com/zhefox/Aether 2026-07-23 12:17:46 +08:00
ZheFox 615ac99ad7 fix(formats): ignore Responses client transport metadata 2026-07-23 12:16:44 +08:00
ZheFox ec36cfbf75 Merge pull request #690 from zhefox/main
fix(provider): classify deleted Codex agent runtime as invalid
2026-07-23 11:22:30 +08:00
ZheFox 7bf228a33c fix(provider): classify deleted Codex agent runtime as invalid 2026-07-23 11:21:55 +08:00
elky 3606290ac8 fix(provider): harden Agent Identity OAuth lifecycle 2026-07-23 09:33:00 +08:00
elky e49024d33b fix(frontend): shorten Agent Identity tab label 2026-07-22 20:26:49 +08:00
elky fdbc2607ec feat(provider): add dedicated Codex Agent Identity flow 2026-07-22 20:19:29 +08:00
elky c7cc8fd7db test(provider): simplify agent identity assertions 2026-07-22 14:19:58 +08:00
elky 07efcb5146 fix(data): repair legacy active flag synchronization 2026-07-22 14:19:34 +08:00
elky 856605defa fix(model-directives): harden suffix configuration 2026-07-22 14:19:09 +08:00
elky 713010fa0a fix(gateway): restore auth role refresh and Rust checks
Refresh the resolved user role without bypassing owner group and key policies. Resolve Rust 1.95 Clippy failures and make the pending persistence bound test scheduler-independent.
2026-07-22 11:25:24 +08:00
ZheFox cd2fbeeead Merge pull request #689 from AAEE86/feat/agent-identity-support
feat(codex): enroll agent identity from session token
2026-07-22 10:32:06 +08:00
AAEE86 a4350a482a feat(codex): enroll agent identity from session token 2026-07-22 10:21:11 +08:00
ZheFox c825375367 Merge pull request #688 from AAEE86/feat/agent-identity-support
feat(codex): support agent identity accounts
2026-07-22 09:20:11 +08:00
elky fc92c4f431 perf(gateway): scale request hot paths for 20k streams
Shard and singleflight hot-path caches, batch and prioritize candidate and usage lifecycle persistence, and extend database and pressure-test instrumentation for 20k concurrent streams.
2026-07-22 02:11:08 +08:00
AAEE86 b61c590bdb feat(codex): support agent identity accounts 2026-07-21 20:58:49 +08:00
ZheFox 7756c0913f Merge pull request #685 from zhefox/main
fix(gateway): apply group policy to admin-owned keys
2026-07-20 15:52:06 +08:00
ZheFox c34ec7c1ee fix(gateway): apply group policy to admin-owned keys 2026-07-20 15:51:01 +08:00
elky f8778c4a23 feat(gateway): configure cyber policy failover 2026-07-19 23:27:19 +08:00
elky e0dbb233f7 fix(frontend): avoid misleading cache TTL fallback label 2026-07-19 22:21:46 +08:00
elky 9725f9abae fix(frontend): clarify processing tier pricing 2026-07-19 22:00:12 +08:00
elky 5d575f1590 test(stats): treat bulk API key snapshots as authoritative 2026-07-19 19:07:04 +08:00
elky d562c594c3 fix(frontend): preserve compact scope and detail badge 2026-07-19 16:42:32 +08:00
MMEXA ce226a3010 Merge 0c3f51bcec into 644ae9c1bf 2026-07-19 16:12:37 +08:00
elky 644ae9c1bf feat(pool): add table-driven account batch actions 2026-07-19 16:09:20 +08:00
elky 95053f9502 Merge PR #672: 支持账号批量配置与可用模型管理 2026-07-18 22:06:32 +08:00
elky 8fbda84acb fix(data): preserve API key history end to end 2026-07-18 21:58:21 +08:00
elky 03b7d573e0 Merge PR #683: decouple API key historical identity 2026-07-18 21:20:35 +08:00
MMEXA 0c3f51bcec merge(main): 解决 usage 模型展示契约冲突 2026-07-18 19:26:14 +08:00
elky e3d97b573b fix(usage): align fast-tier pricing and model metadata 2026-07-18 16:57:04 +08:00
MMEXA ac3796af84 fix(gateway): 恢复响应边界并统一格式入口 2026-07-18 06:45:37 +08:00
MMEXA f9c343eb07 fix(gateway): 适配 Rust 1.95 整除检查 2026-07-18 05:55:06 +08:00
MMEXA e31df5989a merge(main): 解决 usage 展示与生命周期同步冲突 2026-07-18 05:38:38 +08:00
MMEXA 98fbf029fc fix(data): 解耦 API Key 历史统计身份 2026-07-18 04:42:45 +08:00
MMEXA 4d9a648202 test(gateway): 统一流错误测试的格式层入口 2026-07-18 03:37:31 +08:00
MMEXA 405ca3e66a fix(ci): 恢复非流式错误体边界并适配新版 Clippy 2026-07-18 03:26:53 +08:00
MMEXA 0355c28683 fix(data): 解耦候选记录的 API Key 历史身份 2026-07-18 02:40:34 +08:00
fawney19 6c33b8d8fb Merge pull request #682 from MMEXA/codex/codex-prompt-cache-identity-20260717
fix(codex): 统一通用缓存键与原生会话身份
2026-07-18 00:33:20 +08:00
elky a6c6f14b09 style(frontend): align pool cycle stats values 2026-07-18 00:13:35 +08:00
elky e558f55cd9 style(frontend): refine badges and cycle stats 2026-07-18 00:05:22 +08:00
elky 88a057b8d9 fix(usage): force fast badge background transparent 2026-07-17 22:56:54 +08:00
elky ed27d404ac style(usage): make fast badge background transparent 2026-07-17 22:52:47 +08:00
elky 5dda34c66e style(usage): give fast tier an amber accent 2026-07-17 22:36:55 +08:00
elky 373ebf26d6 fix(pricing): default zero tier ratios to one 2026-07-17 21:33:21 +08:00
elky f65ed2795c fix(codex): support dynamic quota windows 2026-07-17 20:18:04 +08:00
elky 664c063a06 feat(usage): enrich audit metadata and detail views 2026-07-17 19:20:16 +08:00
MMEXA 75795c6fbc test(codex): 对齐 Compact 确定性缓存身份 2026-07-17 08:49:35 +08:00
MMEXA 5b332da7d7 fix(codex): 补齐缓存身份终态请求头 2026-07-17 08:11:16 +08:00
MMEXA d9796d502b fix(codex): 统一通用缓存键与原生会话身份 2026-07-17 06:13:09 +08:00
MMEXA 3b0d87b0fd Merge remote-tracking branch 'origin/main' into codex/pool-key-bulk-management-20260714 2026-07-17 00:17:02 +08:00
MMEXA 3c348dff3a Merge remote-tracking branch 'origin/main' into codex/usage-pending-reasoning-reset-expiry-20260712
# Conflicts:
#	frontend/src/features/usage/components/__tests__/UsageRecordsTable.spec.ts
2026-07-17 00:16:58 +08:00
MMEXA ec1783a35c Merge remote-tracking branch 'origin/main' into codex/pool-key-bulk-management-20260714
# Conflicts:
#	apps/aether-gateway/src/handlers/admin/request/provider/tasks.rs
#	frontend/src/api/endpoints/pool.ts
2026-07-16 23:43:04 +08:00
MMEXA 427030c5de Merge remote-tracking branch 'origin/main' into codex/usage-pending-reasoning-reset-expiry-20260712
# Conflicts:
#	crates/aether-ai-formats/src/formats/openai/responses/mod.rs
#	crates/aether-usage/runtime/src/runtime.rs
#	frontend/src/features/usage/components/UsageRecordsTable.vue
#	frontend/src/features/usage/components/__tests__/UsageRecordsTable.spec.ts
2026-07-16 23:41:58 +08:00
elky 0be380243b feat(pricing): support processing tier multipliers 2026-07-16 23:30:42 +08:00
fawney19 312583f055 Merge pull request #680 from Kayphoon/codex/s3-backup-user-agent
feat(admin): configure S3 backup User-Agent
2026-07-16 23:30:30 +08:00
fawney19 33f49ea9b0 Merge pull request #678 from AAEE86/fix
fix: map Developer role to "system" in OpenAI Chat Completions output
2026-07-16 23:29:55 +08:00
fawney19 470cef17cf Merge pull request #676 from MMEXA/codex/sync-capture-envelope-finalize-20260716
修复同步 finalize 的 Responses 流聚合与转换
2026-07-16 23:29:38 +08:00
ZheFox 3f5f65eb9a Merge pull request #681 from zhefox/main
Codex 重置功能和显示缓存修复以及批量key的导入和管理功能
2026-07-16 19:48:33 +08:00
ZheFox 6664c2dbb8 feat(pool): 支持批量导入 Key 和选择性更新设置 2026-07-16 19:31:07 +08:00
ZheFox 0099167a6d fix(codex): 避免重置机会缺失触发配额刷新 2026-07-16 19:13:09 +08:00
ZheFox f009fb73c3 缓存问题修复 2026-07-16 18:55:28 +08:00
ZheFox 715a5ed626 修复重置次数缓存问题 2026-07-16 18:09:10 +08:00
ZheFox 5cf38d1b35 Codex 重置功能和显示修复 2026-07-16 17:23:41 +08:00
Kayphoon 6b707f29a2 feat(admin): configure S3 backup User-Agent 2026-07-16 08:56:56 +00:00
elky 9ea84f9748 fix(frontend): show service tier transitions 2026-07-16 16:38:30 +08:00
elky c32d043afb fix(frontend): preserve fetched model preset pricing 2026-07-16 16:38:30 +08:00
elky 8fe4d24408 fix(usage): canonicalize cached token totals 2026-07-16 16:38:30 +08:00
elky e369e4aab1 fix(formats): preserve chat-backed Responses metadata 2026-07-16 16:38:30 +08:00
ZheFox 7dc919e8e3 Merge pull request #679 from zhefox/main
fix(frontend): 优化移动端弹窗并完善提供商配额刷新
2026-07-16 15:48:49 +08:00
ZheFox 1333efdad5 fix(frontend): 优化移动端弹窗并完善提供商配额刷新 2026-07-16 15:21:11 +08:00
AAEE86 cd8de1aa13 fix: map Developer role to "system" in OpenAI Chat Completions output 2026-07-16 14:29:08 +08:00
elky d6215d9dec ci(tunnel): reduce artifact retention 2026-07-16 13:12:30 +08:00
elky 9a47267545 fix(usage): bound terminal event persistence
Add end-to-end terminal admission, bounded database fallback, and observable overload handling. Preserve first-byte lifecycle state across asynchronous runtime and frontend updates.
2026-07-16 13:12:30 +08:00
MMEXA 7851503fbc fix(finalize): 严格聚合并投影同步 Responses 流 2026-07-16 12:50:00 +08:00
ZheFox c6d373e6aa Merge pull request #677 from zhefox/main
fix(codex): 移除 Responses Lite 请求中的 context_management
2026-07-16 12:28:34 +08:00
ZheFox 71fcb9c168 fix(codex): 服务端压缩使用标准 Responses 合约 2026-07-16 12:02:33 +08:00
ZheFox 3976652942 fix(codex): 移除 Responses Lite 请求中的 context_management 2026-07-16 11:25:11 +08:00
MMEXA 7b56546e21 fix(finalize): 聚合同步流捕获包装 2026-07-16 10:24:32 +08:00
fawney19 85854e4476 Merge pull request #675 from fawney19/fix/pr-669-tail
feat(codex): complete PR #669 protocol follow-up
2026-07-16 08:54:29 +08:00
elky b50242ab9f fix(test): handle absent empty testkit bin directory 2026-07-16 01:29:51 +08:00
MMEXA 20b27a13b2 feat(codex): 按操作语义路由 Responses V2 压缩
(cherry picked from commit 2fc604e047)
2026-07-16 00:34:42 +08:00
MMEXA 598b2fb374 fix(auth): 授权 Responses Compact 伴随端点
(cherry picked from commit e8afa03e45)
2026-07-16 00:32:14 +08:00
MMEXA ff7988430d fix(openai): encode tool errors in Responses output
(cherry picked from commit f127b67e73)
2026-07-16 00:31:23 +08:00
MMEXA 25da99fac2 fix(codex): preserve reset consume request body
(cherry picked from commit fc2dfb82d2)
2026-07-16 00:27:28 +08:00
elky 8616fe6ee2 refactor(workspace): enforce layered crate boundaries 2026-07-15 23:47:19 +08:00
MMEXA 9b8724453b test(pool): 使用正式 Gemini API 格式 2026-07-14 08:39:41 +08:00
MMEXA 01e104d86a fix(pool): 对齐账号批量配置语义 2026-07-14 08:07:28 +08:00
MMEXA 0acd1de29c fix(gateway): 保持密钥更新模块显式所有权 2026-07-14 05:19:04 +08:00
MMEXA a25fab371a feat(pool): add bulk key configuration management 2026-07-14 04:57:05 +08:00
MMEXA 93e2f95c47 fix(frontend): 按端点能力约束会话压缩映射 2026-07-14 02:05:10 +08:00
MMEXA f10d631a9c feat(frontend): 澄清模型映射适用范围 2026-07-14 00:30:39 +08:00
MMEXA cfc4894dab fix(usage): 保留最新进行态生命周期事件 2026-07-14 00:30:24 +08:00
MMEXA 3f86fdd6bc feat(usage): 展示压缩操作与进行态请求语义 2026-07-13 22:03:44 +08:00
MMEXA b09d1f1c33 fix(usage): expose pending reasoning and exact reset expiry 2026-07-13 22:03:44 +08:00
MMEXA 2fc604e047 feat(codex): 按操作语义路由 Responses V2 压缩 2026-07-13 22:03:34 +08:00
MMEXA e8afa03e45 fix(auth): 授权 Responses Compact 伴随端点 2026-07-13 06:07:54 +08:00
MMEXA fc2dfb82d2 fix(codex): preserve reset consume request body 2026-07-12 23:04:33 +08:00
elky a728c090a9 fix(gateway): scope concurrency helper to tests 2026-07-12 22:36:48 +08:00
elky e58621a735 Merge PR #669: align GPT-5.6 and Codex request protocols 2026-07-12 21:50:20 +08:00
MMEXA b1be370b2e fix(gateway): scope concurrency test helper to tests 2026-07-12 21:08:30 +08:00
MMEXA cf0d957ac7 Merge f127b67e73 into 7f61bb43c7 2026-07-12 20:34:39 +08:00
MMEXA f127b67e73 fix(openai): encode tool errors in Responses output 2026-07-12 20:34:31 +08:00
elky 7f61bb43c7 feat(security): harden gateway request and runtime controls 2026-07-12 14:10:54 +08:00
MMEXA 25c49dd804 fix(data): keep terminal usage state monotonic 2026-07-12 05:45:37 +08:00
MMEXA 72222d935c test(gateway): use valid tunnel relay envelopes 2026-07-12 05:45:32 +08:00
MMEXA 063d517306 test(gateway): compare timeout response numerically 2026-07-12 04:43:13 +08:00
MMEXA 02495ce28e fix(admin): preserve inactive endpoint key counts 2026-07-12 04:19:06 +08:00
MMEXA 63936aa110 fix(gateway): route format rules through serving facade 2026-07-12 03:56:53 +08:00
MMEXA 8d4d42a887 fix(auth): resolve group policy before key intersection 2026-07-12 03:35:42 +08:00
MMEXA 2316df5c9a feat(codex): align Search and execution protocol 2026-07-12 03:04:15 +08:00
MMEXA 59d37ae1dd fix(frontend): import structured models.dev pricing 2026-07-11 18:12:55 +08:00
MMEXA 3014fd50c6 fix(billing): preserve effective cache and tier facts 2026-07-11 18:12:55 +08:00
MMEXA 14c4e3a04e fix(codex): enforce provider request identity 2026-07-11 18:09:14 +08:00
MMEXA 8f1070a451 feat(frontend): expose processing tier pricing 2026-07-11 12:27:09 +08:00
MMEXA 0b30cc6b0f feat(openai): unify tier authorization and settlement 2026-07-11 12:27:05 +08:00
MMEXA b2f596b8f0 fix(gateway): route Codex header through serving facade 2026-07-11 09:43:12 +08:00
MMEXA 01a96fed74 fix(codex): simplify summary normalization 2026-07-11 09:20:07 +08:00
MMEXA 46a903aada fix(codex): align current reasoning request semantics 2026-07-11 09:15:08 +08:00
MMEXA dfa121dd5b feat(openai): align GPT-5.6 and Codex request contracts 2026-07-11 07:40:12 +08:00
elky bc1da3bf3f feat(security): harden client IP and admin controls 2026-07-10 15:13:12 +08:00
elky 6e0dc3b59e feat(frontend): refine global model pricing dialog 2026-07-10 15:13:12 +08:00
elky 4bf5d4c044 Fix cache token accounting and tiered pricing 2026-07-10 15:13:12 +08:00
fawney19 736fc76345 Merge pull request #668 from MMEXA/codex/antigravity-empty-output-retry-20260709
修复 Gemini 空输出按候选重试处理
2026-07-10 09:16:56 +08:00
MMEXA b6b2ca38f4 触发 CI 重跑 2026-07-10 00:41:22 +08:00
MMEXA f07eb25cfc 修复 usage 详情 body 引用解包 2026-07-10 00:23:31 +08:00
MMEXA d2ea437c1c 修复 Gemini 空输出按候选重试处理 2026-07-09 23:10:30 +08:00
fawney19 7bc7d0f8d8 Merge pull request #666 from xixiknow/main
Fix provider key response time counter overflow
2026-07-09 18:04:11 +08:00
fawney19 14cf639aba Merge pull request #667 from MMEXA/codex/antigravity-v1internal-query-20260709
修复 Antigravity v1internal 查询参数透传
2026-07-09 17:50:38 +08:00
yangrs 55cdab592c Remove redundant response time conversion 2026-07-09 16:26:09 +08:00
MMEXA ee0ec18283 修复 Antigravity v1internal 查询参数透传 2026-07-09 16:03:29 +08:00
Start f31c9e03e2 Merge branch 'fawney19:main' into main 2026-07-09 15:11:44 +08:00
fawney19 e50db10439 Merge pull request #663 from MMEXA/codex/gemini-interactions-antigravity-20260705
完善 Gemini Interactions 与 Antigravity 全链路兼容
2026-07-09 14:55:51 +08:00
yangrs 192dc6c20d Fix provider key response time overflow 2026-07-09 14:40:48 +08:00
elky 5e1d14f19b Fix timeline duration display from latency 2026-07-09 11:46:45 +08:00
MMEXA b8b89d21b7 fix: 同步提交本地 sync 错误上报 2026-07-08 23:49:18 +08:00
MMEXA 5eddf4f9ee 细化 Antigravity 测试模型项目元数据补全 2026-07-08 22:45:30 +08:00
MMEXA c7186e1720 完善 Antigravity 配额展示与 CI 断言 2026-07-08 22:34:29 +08:00
MMEXA 4866509938 移除 Antigravity 未知重置时间噪音 2026-07-08 22:34:29 +08:00
MMEXA 2122660a5c 对齐原生 Antigravity 控制面与显示模型 2026-07-08 22:34:29 +08:00
MMEXA 5c68ab896a 恢复历史 backfill 兼容 live 账本 2026-07-08 22:34:29 +08:00
MMEXA f9c8ec41f4 完善 Antigravity 与 Gemini 跨格式兼容 2026-07-08 22:34:29 +08:00
MMEXA b1ed6b24b0 触发 CI 复跑 2026-07-08 22:34:29 +08:00
MMEXA c17c78ad4b 修正 Antigravity Gemini 3.5 Flash 档位展示 2026-07-08 22:34:29 +08:00
MMEXA 9ec48ab6b9 优化 Antigravity 配额展示顺序 2026-07-08 22:34:29 +08:00
MMEXA accd250226 修正 Antigravity 配额模型标签 2026-07-08 22:34:29 +08:00
MMEXA 80a6579766 支持 Gemini Interactions 与 Antigravity 配额精细化 2026-07-08 22:34:29 +08:00
fawney19 1ca83ca3fb Merge pull request #664 from MMEXA/codex/wallet-auth-cache-delay-20260706
修复钱包余额变更后的鉴权缓存延迟
2026-07-07 01:59:44 +08:00
fawney19 a931da0764 Merge pull request #662 from MMEXA/codex/reset-credit-20260704
增加 Codex 重置次数功能
2026-07-07 01:58:44 +08:00
fawney19 a61374c595 Merge pull request #661 from MMEXA/codex/frontend-debug-20260704
修复前端调试与基础交互问题
2026-07-07 01:57:41 +08:00
MMEXA c3136126e5 修复钱包余额变更后的鉴权缓存延迟 2026-07-06 06:15:31 +08:00
MMEXA b23d299533 重跑 Codex 重置次数 CI 2026-07-05 01:37:31 +08:00
MMEXA b03aae18c3 修复 Codex 重置次数 CI 检查 2026-07-04 15:47:01 +08:00
MMEXA 99b6fe468f 简化 Codex 重置机会展示标签 2026-07-04 15:16:38 +08:00
MMEXA ef77ec04ca 增加 Codex 重置次数功能 2026-07-04 06:10:53 +08:00
MMEXA 242081433e 修复前端调试与基础交互问题 2026-07-04 05:24:40 +08:00
ZheFox b86d4e1f0c Merge pull request #660 from zhefox/main
refactor(frontend): unify mobile menu background styles
2026-07-03 13:17:59 +08:00
ZheFox a151f37d63 refactor(frontend): unify mobile menu background styles 2026-07-03 13:17:18 +08:00
ZheFox 42f7907740 Merge pull request #659 from zhefox/main
refactor(frontend): improve mobile overflow handling
2026-07-03 12:54:01 +08:00
ZheFox e72e25c59c refactor(frontend): improve mobile overflow handling 2026-07-03 12:53:22 +08:00
ZheFox 1b0440481b Merge pull request #658 from zhefox/main
修复管理端额度显示、节点表格显示与移动端滚动问题
2026-07-03 12:49:15 +08:00
ZheFox 26d85681f0 refactor(frontend): improve mobile overflow and proxy node table 2026-07-03 12:25:53 +08:00
ZheFox 1dcee77055 refactor(frontend): improve dialog and mobile overflow handling 2026-07-03 11:26:40 +08:00
elky 1ac16005f9 Stabilize usage worker autoscale tests 2026-07-02 17:28:25 +08:00
elky 2f1cdb6a0b Record exhausted usage failures synchronously 2026-07-02 16:08:04 +08:00
elky ac93851b2a Stabilize Gateway h2c transport test 2026-07-02 14:02:26 +08:00
elky 400b3125a4 Preserve terminal request candidate state 2026-07-02 01:40:57 +08:00
elky 2e5ff32e1a perf(frontend): 收敛导航预取并去重首屏请求
- 导航预取仅保留 pointerdown 触发,移除 mouseenter/focus,避免鼠标划过误触发
- 后台预取只做组件懒加载,不再预取各页业务数据,减少首屏资源争抢
- 版本状态检查增加 sessionStorage 缓存(正常 20 分钟 / 错误 5 分钟 TTL)
- fetchModules、必读公告拉取增加请求去重,避免并发重复请求
- 更新检查改用可清理的定时器,组件卸载时清理
- UsageRecordsTable 搜索防抖改为自定义实现,卸载时取消挂起 emit 并补充测试
2026-07-01 20:42:52 +08:00
elky a0f7074e59 chore: disable Redis persistence by default, document triage and policy 2026-07-01 14:15:16 +08:00
elky 7c32be46ca Mark sync usage active earlier 2026-07-01 02:21:20 +08:00
Entropy.Xu 6ed2f9bd0a fix: apply actual billing cost to wallet settlement 2026-07-01 01:12:40 +08:00
elky 778b106023 test: stabilize gateway nextest timing 2026-06-30 18:42:57 +08:00
elky f179ee72f9 chore: update gateway pressure observability 2026-06-30 17:01:39 +08:00
elky 974def5fef refactor(frontend): extract provider key identity block 2026-06-30 17:01:39 +08:00
elky e5351b7d9d refactor(frontend): extract provider key actions 2026-06-30 17:01:39 +08:00
elky ed83184d55 refactor(frontend): extract provider quota display components 2026-06-30 17:01:39 +08:00
elky 15b6606c82 refactor(frontend): extract pool key display panels 2026-06-30 17:01:39 +08:00
elky d7411a3104 refactor(frontend): extract pool header and theme toggle 2026-06-30 17:01:39 +08:00
elky 9f138d09e6 refactor(frontend): modularize i18n architecture 2026-06-30 17:01:39 +08:00
ZheFox bf29129a4b Merge pull request #654 from zhefox/main
Cancel upstream streams on client disconnect and void cancelled usage billing
2026-06-29 01:14:11 +08:00
zhefox f6293b6812 fix(usage): void cancelled usage and cancel dropped streams 2026-06-29 00:30:41 +08:00
elky 7e9424008f Add usage queue worker autoscaling 2026-06-26 14:02:57 +08:00
elky 6c5e70ccb1 fix monitoring error totals and counter health 2026-06-26 10:48:45 +08:00
elky 063834e95b Split admin operations dashboard route 2026-06-26 01:48:37 +08:00
elky c76d6b6396 Add admin operations dashboard and usage state fixes 2026-06-26 01:32:45 +08:00
elky 6f00e9fc67 Improve gateway transport and usage runtime 2026-06-25 22:36:27 +08:00
elky d336d1a7fa Improve gateway scheduling and runtime admission 2026-06-24 01:53:45 +08:00
ZheFox cf0af8fa1e Merge pull request #652 from zhefox/main
fix(usage): preserve token counts in body redaction
2026-06-23 14:40:59 +08:00
zhefox fd220b6c42 fix(usage): preserve token counts in body redaction 2026-06-23 14:38:39 +08:00
zhefox 3472bb75e7 ci: combine gateway clippy and nextest jobs 2026-06-23 14:06:43 +08:00
zhefox ba65c96c74 Merge branch 'main' of https://github.com/zhefox/Aether 2026-06-23 13:36:00 +08:00
zhefox c54b214657 ci: shard gateway tests and disable debug info in rust ci 2026-06-23 13:35:56 +08:00
ZheFox 4fcc17114f Merge pull request #651 from zhefox/main
fix(ai-formats): accept Claude context_management in responses conversion
2026-06-23 10:43:26 +08:00
zhefox deb5f55786 fix(ai-formats): clean up cross-format safety rules for Gemini requests 2026-06-23 10:30:13 +08:00
zhefox 1836c2b652 fix(ai-formats): accept Claude context_management in responses conversion 2026-06-23 10:03:55 +08:00
elky 5b7805181b perf: queue request candidate persistence 2026-06-22 02:49:17 +08:00
elky f75894acbb perf: reduce gateway db pressure under load 2026-06-22 00:08:48 +08:00
elky 541cc197c4 fix: preserve in-memory user export fallback 2026-06-22 00:08:48 +08:00
fawney19 363d1aba9a Merge pull request #615 from AAEE86/main
feat: 健康监控仪表盘与关联下钻优化,完善使用记录展示
2026-06-21 12:48:13 +08:00
fawney19 eb2cf662b7 Merge pull request #650 from stabey/pr/claude-system-responses-20260620
fix(ai-formats): preserve Claude in-message system guidance in Responses
2026-06-21 12:47:26 +08:00
elky 900f8a7163 fix(pool): allow zero cooldown settings 2026-06-21 12:20:08 +08:00
elky 61bdd304b7 Handle inactive PAT owner as invalid OAuth token 2026-06-21 11:39:20 +08:00
elky 279735ae7f Auto-size SQL pool defaults 2026-06-21 11:15:40 +08:00
elky cc2830f6ec Merge branch 'review/pr-639' 2026-06-21 10:48:49 +08:00
elky 8dbd730568 fix: respect imported oauth authorization headers 2026-06-21 02:27:06 +08:00
stabey bb6aa03485 fix(ai-formats): strip Claude billing headers from preserved guidance 2026-06-21 00:22:57 +08:00
stabey 6a22488698 fix(ai-formats): preserve Claude in-message system guidance in responses 2026-06-21 00:07:57 +08:00
elky f1c30439ff fix: preserve provider auth metadata 2026-06-20 22:11:42 +08:00
fawney19 1123095bb7 Merge pull request #624 from MMEXA/codex/fix-antigravity-oauth-quota
修复 Antigravity OAuth 导入后配额复检缺 project
2026-06-19 23:11:38 +08:00
MMEXA 938f11981d fix(ai-serving): route Antigravity auth enum through facade 2026-06-19 22:25:52 +08:00
MMEXA 6c4e730e60 修复 Antigravity OAuth 配额复检缺 project 2026-06-19 22:21:22 +08:00
elky 16584067d7 Add route-backed routing profile views 2026-06-18 02:06:34 +08:00
fawney19 6de0fe75a4 Merge pull request #641 from Kayphoon/codex/usage-cleanup-break-condition
fix(usage): align cleanup loop break conditions with candidate row count
2026-06-17 11:04:55 +08:00
fawney19 34f0913ed0 Merge pull request #645 from zhefox/main
修复 OpenAI Chat/Responses/Messages 转换兼容性并透传 Codex cyber_policy 错误
2026-06-17 11:03:29 +08:00
zhefox 5b305c64e1 fix(ai-formats): omit request tool call ids in OpenAI Responses input 2026-06-17 09:15:38 +08:00
zhefox 8ad97761e8 fix(ai-formats): preserve OpenAI Responses tool call item ids 2026-06-17 08:29:25 +08:00
zhefox 0f92ef664d fix(ai-formats): support OpenAI Responses custom tool/raw passthrough 2026-06-17 04:24:56 +08:00
zhefox 16a4fd3687 Merge branch 'main' of https://github.com/zhefox/Aether 2026-06-17 04:07:53 +08:00
zhefox 3a3fcbe46a fix(ai-formats): preserve OpenAI tool call item ids 2026-06-17 04:05:09 +08:00
zhefox 18d8ea2052 fix(ai-formats): preserve OpenAI tool call item ids 2026-06-17 04:03:40 +08:00
zhefox 6ab08f4014 fix(ai-formats): preserve Claude raw blocks, reasoning tokens, and test stack safety 2026-06-17 03:45:23 +08:00
zhefox 628a3a0d8d fix(ai-formats): support cyber policy failover and custom tool/audio passthrough 2026-06-17 02:33:14 +08:00
elky f52628e00b Handle OpenAI Responses keepalive stream events 2026-06-16 22:52:06 +08:00
zhefox f9d97ececb fix(ai-formats): ignore OpenAI Responses metadata events 2026-06-16 22:34:38 +08:00
fawney19 803e555022 Merge pull request #635 from zhefox/main
fix(gateway): 支持 OpenAI 图片编辑端点请求
2026-06-16 22:31:52 +08:00
zhefox b1bd727978 将 JSON 提示注入为 developer 输入 2026-06-16 21:40:15 +08:00
zhefox c2748dc868 忽略 OpenAI Responses keepalive 事件 2026-06-16 20:00:46 +08:00
ZheFox 302620cb94 Merge branch 'fawney19:main' into main 2026-06-16 12:17:28 +08:00
AAEE86 c255f29e98 Merge remote-tracking branch 'upstream/main' 2026-06-16 10:53:47 +08:00
Kayphoon 6d1b818414 fix(usage): align cleanup loop break conditions with candidate row count
The cleanup loop break condition used rows_affected() from the UPDATE
statement, but for rows that only had blob/audit refs (no inline
compressed body data), the UPDATE reported 0 affected rows. This caused
the loop to exit after the first batch, skipping the majority of
candidates.

Change the break condition in all 4 cleanup functions from:
  if cleaned == 0 || cleaned < batch_size
to:
  if rows.len() < batch_size

This ensures the loop continues as long as SELECT returns a full batch,
regardless of how many rows the UPDATE actually modified.

Affected functions:
- cleanup_usage_raw_body_fields
- cleanup_usage_compressed_body_fields
- cleanup_usage_header_fields
- cleanup_usage_stale_body_fields
2026-06-16 04:19:58 +08:00
elky 669636d3e4 Harden PII redaction format conversion 2026-06-14 20:36:57 +08:00
elky 68038c182b Distinguish expired OAuth token status 2026-06-12 19:43:59 +08:00
elky 308cc88ef7 Fix provider deletion cleanup 2026-06-12 16:25:11 +08:00
elky 30b545785f feat: improve failover rules and request timeline 2026-06-11 00:49:29 +08:00
elky 31fade82f6 Preserve OpenAI encrypted reasoning blocks 2026-06-10 20:02:03 +08:00
elky 0246ba93dd fix(transport): preserve safe accept encoding 2026-06-10 19:58:57 +08:00
elky ff7ec8575c fix(ai-formats): ignore null stream errors 2026-06-10 18:44:46 +08:00
elky aa58cb4a05 build: speed up release image linking 2026-06-10 18:37:23 +08:00
elky e9b4efc2d4 fix(ai-serving): preserve explicit request encoding 2026-06-10 18:16:55 +08:00
elky ea76f7bb0b Support OpenAI Responses builtin tool stream items 2026-06-10 14:49:13 +08:00
elky 8edcbdcb29 feat(usage): expose request timing details 2026-06-10 09:16:15 +08:00
ndllz 5249660e07 fix: respect oauth module disabled state 2026-06-09 18:13:07 +08:00
ndllz 84b99a641a fix: speed up usage activity heatmap render 2026-06-09 16:52:58 +08:00
AAEE86 4824e4a487 fix(frontend): 移除账号导入重复处理中提示 2026-06-09 16:40:45 +08:00
zhefox ba723ebe48 fix(usage): always use truncated body placeholder when limit exceeded 2026-06-09 09:59:10 +08:00
zhefox 04ba8cbe9e fix(gateway): support openai image accept negotiation 2026-06-08 20:40:30 +08:00
elky 84f41dae77 feat(format): audit same-format compatibility rewrites 2026-06-08 16:12:37 +08:00
zhefox 82040bfc21 fix(gateway): support OpenAI image edit requests 2026-06-08 13:22:08 +08:00
elky 6155ffefcc fix(format): avoid false cache-control conversion blocks 2026-06-08 00:52:48 +08:00
elky bf4279a590 Merge remote-tracking branch 'origin/main' into dev
# Conflicts:
#	crates/aether-ai-formats/src/formats/openai/chat/stream.rs
#	crates/aether-ai-formats/src/formats/openai/responses/response.rs
#	crates/aether-ai-formats/src/formats/shared/sync_products.rs
2026-06-08 00:17:30 +08:00
elky 77759fac54 feat: 新增提供商批量处理功能 2026-06-07 22:57:02 +08:00
elky 63a2fd4dcf Merge origin/main into dev 2026-06-06 03:11:38 +08:00
elky 7a19891c60 fix: distinguish unaudited conversion fields 2026-06-06 00:35:27 +08:00
AAEE86 85573d7980 Merge remote-tracking branch 'upstream/main' 2026-06-05 08:28:43 +08:00
zhefox ebd59246a8 fix(test): assert responses timestamps and output text in finalize tests 2026-06-04 17:35:10 +08:00
zhefox fd27f55fe5 fix(provider): normalize OpenAI Responses modern fields and stream events 2026-06-04 15:45:37 +08:00
fawney19 69b8b96fb8 Merge pull request #625 from stabey/pr/responses-call-items-cache-control-20260604
fix: 剥离 Codex cache_control 并完善 Responses 工具调用展示
2026-06-04 13:59:56 +08:00
fawney19 19d1d36043 Merge pull request #620 from zhefox/main
fix(provider): 修复 Chat reasoning_effort 值域与 Responses 扩展透传
2026-06-04 13:59:42 +08:00
stabey 9f19ca5754 fix(usage): keep streamed call args in responses completion
The response.completed fallback rebuilt every call item with responsesCallInput(), which returns '{}' for a function_call lacking arguments. Since '{}' is truthy, ensureToolCall overwrote arguments already collected from streamed delta events. Guard the completed branch with responsesCallHasInput (matching the output_item.done branch) so empty/default inputs no longer clobber streamed args, and align its dedupe key with the streaming phase to avoid duplicate tool-call rendering when an item has no id. Drop the now-dead '工具调用' fallbacks since responsesCallName never returns empty.
2026-06-04 13:30:20 +08:00
stabey 2de2a792f6 fix(codex): strip cache_control before responses upstream 2026-06-04 12:01:36 +08:00
stabey ada690624b fix(usage): render responses call items in conversation view 2026-06-04 11:06:36 +08:00
elky 465476985b fix: preserve provider schema drift safely 2026-06-03 22:29:24 +08:00
elky da5624c98e chore: add format field coverage generator 2026-06-03 21:44:33 +08:00
elky b2f68bbaf7 feat: enforce full format field coverage audit 2026-06-03 21:18:49 +08:00
elky 7507af5829 feat: audit strict format conversion contracts 2026-06-03 20:27:15 +08:00
zhefox 5e39801bba fix(provider): clamp reasoning effort and filter chat extensions 2026-06-03 10:52:26 +08:00
elky 5ac153a0bb Fix gateway nextest stack limit 2026-06-03 01:32:25 +08:00
elky c7a5155ce4 Fix sync CLI test stack overflow 2026-06-03 01:12:40 +08:00
elky 869c3d3037 Fix finalize local test stack overflow 2026-06-03 00:48:36 +08:00
elky ef6a11c146 fix(gateway): preserve heartbeat no-path fallback 2026-06-03 00:25:01 +08:00
elky 21432911de Merge remote-tracking branch 'origin/pr/605' 2026-06-03 00:16:22 +08:00
elky eb98340924 Merge remote-tracking branch 'origin/pr/604' 2026-06-02 23:34:59 +08:00
elky 746af0d93e Fix Kiro cache usage reporting 2026-06-02 23:06:29 +08:00
elky 08ac9c5c58 Merge remote-tracking branch 'origin/pr/614' 2026-06-02 22:10:41 +08:00
elky bce3bf2b6e Merge remote-tracking branch 'origin/pr/593' 2026-06-02 21:40:59 +08:00
elky 4ec9ca61cf Merge remote-tracking branch 'origin/pr/613'
# Conflicts:
#	apps/aether-gateway/src/tests/usage/direct.rs
2026-06-02 19:27:50 +08:00
elky 657e6aa672 Merge remote-tracking branch 'origin/pr/619' 2026-06-02 19:23:44 +08:00
AAEE86 7835840ebd feat(dashboard): 增加全站实时指标和自动刷新
- 管理员仪表盘新增全站 RPM/TPM 与在线/启用用户指标
- 合并今日请求/费用、全站 RPM/TPM、在线/启用用户卡片展示
- 在线用户按最近 5 分钟活跃请求去重统计
- 全站 RPM/TPM 按最近 60 秒请求与 Token 统计
- 新增仪表盘自动刷新按钮,开启后每 10 秒静默刷新数据
- 同步前端类型、空态占位和仪表盘测试
2026-06-02 18:16:24 +08:00
zhefox 6cabcd85aa fix(provider): preserve Claude messages defaults in responses conversion 2026-06-02 17:14:26 +08:00
elky 03e436707d Fix gateway usage nextest stack overflow 2026-06-02 16:59:13 +08:00
elky 781bc5ac58 Merge branch 'pr-617' 2026-06-02 10:40:34 +08:00
AAEE86 86f72da3d9 feat(health): 增加历史状态条指标 Tooltip
- 为健康监控时间轴返回 timeline_details 分段指标
- Hover 历史状态柱时展示总请求/成功/失败/可用率/状态
- 展示平均耗时/TTFB/速度和完整时间范围
- 修复历史状态柱 Tooltip 触发区域不可用的问题
- 补齐前端类型、详情抽屉透传和 mock 数据
2026-06-02 10:36:54 +08:00
elky 0a2c674ad8 Ignore tunnel release tags for app build version 2026-06-02 09:43:12 +08:00
zhefox 0daa8c196b fix(provider): preserve reasoning and Claude tool results in responses conversion 2026-06-02 09:04:55 +08:00
zhefox 98dc5925a5 fix(provider): preserve openai responses tool history in chat conversion 2026-06-02 00:35:19 +08:00
AAEE86 d5d3f09846 refactor(health): add dashboard overview and related drill-down
- Replace health monitor tabs with a dashboard layout
- Add related health drill-down for endpoint, model, and provider cards
- Render provider health as cards and hide empty monitors
2026-06-02 00:10:59 +08:00
AAEE86 0e6fc96eb1 test(gateway): run wallet usage settlement test on larger stack
Wrap the wallet settlement usage test with the large-stack async test helper to
avoid stack overflow in the default test thread.
2026-06-01 22:40:22 +08:00
AAEE86 b052f40ffb test(gateway): run base usage body capture test on larger stack
Wrap the request_record_level=base local gateway usage test with the existing
large-stack async test helper to avoid stack overflow in the default test thread.
2026-06-01 22:23:54 +08:00
AAEE86 2aef9d2478 Refine mobile usage record metadata layout 2026-06-01 21:59:45 +08:00
AAEE86 8627a18f2e test(gateway): run local usage report test on large stack
Wrap the local OpenAI chat sync usage-reporting test in the existing
large-stack harness to avoid stack overflows under nextest suite load.
2026-06-01 21:45:44 +08:00
AAEE86 21c478be22 fix(health): hide empty endpoint monitors
- Remove raw API format label from endpoint health cards
- Hide endpoint health cards with no requests
2026-06-01 21:24:20 +08:00
AAEE86 9d8f7d158b Refine mobile usage record details
- Move mobile usage actions into the card header
- Add compact user/provider metadata line on mobile
- Preserve hidden unknown toggle and auto refresh controls
2026-06-01 21:13:10 +08:00
AAEE86 c1649fe837 refactor(health): consolidate monitor components 2026-06-01 20:49:38 +08:00
AAEE86 d3c8317939 fix(health): align model health card layout 2026-06-01 18:54:56 +08:00
AAEE86 7ffe33f867 feat(health): refine health monitor metrics
- add TPS to model and provider health payloads

- exclude user-cancelled 499 requests from health statistics

- update model/provider health cards with average latency, average TTFB, TPS, and availability
2026-06-01 18:34:51 +08:00
elky 6c2a57f237 fix rust ci failures 2026-06-01 02:42:10 +08:00
github-actions[bot] 0f4141ef3f chore(tunnel): update download links for tunnel-v0.3.16 2026-05-31 17:46:42 +00:00
elky 37413c0211 Refactor tunnel stability protocol 2026-06-01 01:36:49 +08:00
Entropy.Xu d1b64b6748 修复:完善 Kiro 模拟缓存共享回收 2026-05-31 22:43:27 +08:00
Entropy.Xu c2bcfab7d4 修复:Kiro 模拟缓存接入共享运行时 2026-05-31 22:13:45 +08:00
elky 392353ffff Merge remote-tracking branch 'entropy-xu/codex/ccswitch-import' 2026-05-31 20:52:26 +08:00
Entropy.Xu 9734be31cf 修复:收敛 Kiro 模拟缓存断点语义 2026-05-31 20:45:16 +08:00
elky 905453d62b revert: remove usage elapsed clock calibration 2026-05-31 20:30:51 +08:00
Entropy.Xu a3b8a99709 修复:补齐 Kiro 模拟缓存 TTL 和消息级断点 2026-05-31 20:22:18 +08:00
Entropy.Xu 2e24e5f358 修复:扩大 Kiro 模拟缓存前缀读取范围 2026-05-31 19:52:18 +08:00
github-actions[bot] 40eb3cf6e1 chore(tunnel): update download links for tunnel-v0.3.15 2026-05-31 11:17:51 +00:00
elky 549463088c chore: bump aether-tunnel version to 0.3.15 2026-05-31 19:09:56 +08:00
elky f8b5651883 Support encoded tunnel node names 2026-05-31 16:33:06 +08:00
stabey de0a880ca6 test(gateway): 修复 usage wallet 测试栈溢出 2026-05-31 03:35:01 +08:00
stabey ba4e194cb5 test(gateway): 修复 usage base 记录测试栈溢出 2026-05-31 03:21:29 +08:00
stabey 1c05a722c1 test(gateway): 修复 usage local 同步测试栈溢出 2026-05-31 03:09:11 +08:00
stabey eda94913cf test(gateway): 修复 usage 同步测试栈溢出
CI 中 gateway_records_pending_usage_before_execution_runtime_sync_result_arrives 仍会在默认测试栈上溢出。

复用 large-stack tokio runtime 包装该测试,避免 gateway 全量测试在无业务失败时被 SIGABRT 中断。
2026-05-31 02:54:11 +08:00
stabey 3dfafbc379 fix(ai): 按 Responses 文本分片去重快照
upstream 已有 8abedecb 处理单个 OpenAI Responses 文本流中 delta 与 done/completed 快照重复输出的问题。

本提交保留该方向,并把去重状态从全局文本扩展为按 output_index/item_id 与 content_index 分片记录,避免多个 message item 或多个 text content part 共用同一段快照状态。
2026-05-31 02:54:11 +08:00
stabey 8d1e54eba6 fix(stream): 中途失败时不合成正常收尾
上游流式读取失败后,已经缓冲的局部转换状态可能是不完整的工具调用。

在 terminal failure 存在时跳过 normalizer 和 rewriter 的 finish 路径,避免把半截 tool_use 补成正常的 Claude message_stop。
2026-05-31 02:54:11 +08:00
stabey 6bfd56b54f fix(usage): 避免上游流式错误误记为成功
当上游流式响应中途失败时,sync error payload 可能同时包含合成错误体和部分上游流 body。

优先使用合成错误体生成 usage 终态,避免只因为上游先返回过 200 和部分 SSE 内容就把失败请求记录为 completed/settled。
2026-05-31 02:54:11 +08:00
elky 49f952692b Fix remaining sync chat stack overflows 2026-05-31 02:12:22 +08:00
elky fde15c9b60 Fix sync chat test stack overflow 2026-05-31 01:13:09 +08:00
elky 06f26cfacf Merge remote-tracking branch 'origin/pr/597' 2026-05-31 00:09:42 +08:00
elky 5360665432 test(gateway): avoid stack overflow in cors proxy test 2026-05-30 22:48:19 +08:00
elky a20ac1d31f fix(pool): align oauth status filter with visible state 2026-05-30 21:37:21 +08:00
elky 1bdd300606 Merge branch 'review-pr-612' 2026-05-30 21:26:18 +08:00
elky c56f0198ff Merge branch 'review-pr-611' 2026-05-30 21:26:12 +08:00
elky 02fc6bd4ef Merge branch 'review-pr-610' 2026-05-30 21:26:07 +08:00
elky c3a8352d76 Merge branch 'review-pr-603' 2026-05-30 21:25:58 +08:00
elky 463576915f Merge branch 'review-pr-602' 2026-05-30 21:25:52 +08:00
elky 1db6b9d307 Merge branch 'review-pr-596' 2026-05-30 21:25:46 +08:00
elky b5a02a118f Merge branch 'review-pr-595' 2026-05-30 21:25:39 +08:00
elky ae96d5d61b fix usage trace active key selection 2026-05-30 19:48:18 +08:00
cym ce1d532e3c fix(pool): align status filters with visible key state 2026-05-30 18:49:26 +08:00
Entropy.Xu f27485ec05 fix(kiro): 忽略图片 base64 token 估算 2026-05-30 02:59:04 +08:00
Entropy.Xu 9616f458de fix(kiro): 模拟缓存读取移动断点前缀 2026-05-30 00:42:14 +08:00
MMEXA 3455faf7da 修复格式转换优先级保持的首轮候选排序
让开启格式转换优先级保持的跨格式候选进入首轮候选页。

普通跨格式候选仍延后到后续页,保持原有兜底语义。
2026-05-29 22:01:11 +08:00
Entropy.Xu 7ed4b84654 feat(ccswitch): 添加一键导入和用量查询 2026-05-29 21:39:45 +08:00
github-actions[bot] 0d76a8e478 chore(tunnel): update download links for tunnel-v0.3.14 2026-05-29 13:32:35 +00:00
elky b9612fef9b chore: bump aether-tunnel version to 0.3.14 2026-05-29 21:21:30 +08:00
fawney19 92ae88f1be fix: avoid postgres migration version collision 2026-05-29 16:18:59 +08:00
ZheFox 91a5e58cec Merge branch 'fawney19:main' into main 2026-05-29 15:27:46 +08:00
fawney19 1658925f52 Disable key circuit breaker for pool providers 2026-05-29 15:16:06 +08:00
Entropy.Xu bb5a4454a5 feat(gateway): 添加标准文本非流式心跳 2026-05-29 14:35:16 +08:00
fawney19 9fb600df1b Fix PR 599 check regressions 2026-05-29 12:30:56 +08:00
ZheFox fff4fe4e20 Merge branch 'fawney19:main' into main 2026-05-29 12:27:38 +08:00
fawney19 3e4dfd2bac Merge branch 'pr-599' 2026-05-29 02:46:07 +08:00
fawney19 8bd82c8c95 Update endpoint base URL placeholders 2026-05-29 02:44:03 +08:00
fawney19 b59c724455 Normalize endpoint API root handling 2026-05-29 02:29:33 +08:00
ZheFox 3ee272fd53 Merge branch 'fawney19:main' into main 2026-05-29 01:48:50 +08:00
AAEE86 ab5d1f266f fix(usage): Optimize the billing layout of the request details page for mobile devices 2026-05-29 00:01:51 +08:00
Entropy.Xu 906742e3c4 fix(billing): 复用待支付套餐订单 2026-05-28 23:09:36 +08:00
AAEE86 0ee45f41e1 feat(mobile): Refine mobile usage record layout 2026-05-28 22:43:20 +08:00
RWDai 6d285410c2 Preserve dashboard daily breakdown rows 2026-05-28 22:02:42 +08:00
Entropy.Xu eaabfb83ed fix(tunnel): bound upstream clients and heartbeat deltas 2026-05-28 20:34:08 +08:00
fawney19 ef2953038e Fix usage records filtering and pool trace display 2026-05-28 20:24:48 +08:00
zhefox cc1a63bf01 fix: repair missing routing profiles snapshot 2026-05-28 18:59:04 +08:00
Novick Yuan 4b2d8cef3c fix(usage): calibrate active elapsed clock efficiently 2026-05-28 18:45:16 +08:00
fawney19 df518ad668 fix contracts usage server time header 2026-05-28 17:54:56 +08:00
fawney19 37b0c00701 Merge remote-tracking branch 'origin/main' 2026-05-28 17:19:04 +08:00
fawney19 88f03aaef2 Keep key circuit breaker out of pool scoring 2026-05-28 17:18:42 +08:00
fawney19 ffd8d273c4 Revert "Merge remote-tracking branch 'origin/pr/592'"
This reverts commit 3504875922, reversing
changes made to 5c3a1aecbe.
2026-05-28 17:10:27 +08:00
fawney19 ef2a96bcc4 Remove default hot pool size cap 2026-05-28 17:01:03 +08:00
Novick Yuan 734717899b Invalidate model routing cache after admin model writes 2026-05-28 16:57:28 +08:00
RWDai d2d28c30d9 Use bearer auth for OpenAI embedding passthrough 2026-05-28 16:53:00 +08:00
fawney19 10532e1a55 Merge pull request #594 from AAEE86/main
feat(usage): support output_config effort badge source
2026-05-28 16:48:02 +08:00
fawney19 47886abd2b Preserve streaming usage timing on refresh 2026-05-28 16:33:10 +08:00
fawney19 d076f64db3 Harden usage server timing header passthrough 2026-05-28 16:28:19 +08:00
AAEE86 60e3ffc402 feat(usage): support output_config effort badge source
- extract reasoning effort from provider request body output_config.effort
- include output_config.effort in usage list fallback SQL
- cover the new request body shape in usage metadata tests
2026-05-28 16:14:37 +08:00
fawney19 b21be24faa Merge remote-tracking branch 'origin/pr/591' 2026-05-28 16:07:08 +08:00
fawney19 3504875922 Merge remote-tracking branch 'origin/pr/592' 2026-05-28 16:07:07 +08:00
Entropy.Xu 0f6d4b9146 feat(embedding): 接入阿里云多模态向量端点 2026-05-28 16:05:36 +08:00
Mas0nShi 6ebd39ed0b Stabilize stream first-byte usage test 2026-05-28 15:33:28 +08:00
fawney19 5c3a1aecbe Merge remote-tracking branch 'origin/pr/591' 2026-05-28 15:23:26 +08:00
Mas0nShi 1a45ec9386 Fix Gemini CLI streaming policy for OpenAI chat 2026-05-28 15:05:05 +08:00
fawney19 97133f657f style: 突出路由策略选中标签样式 2026-05-28 15:03:14 +08:00
Novick Yuan b108dc5ea6 Assert usage server timing over HTTP 2026-05-28 15:01:59 +08:00
Novick Yuan 35cf44b38e Extract active usage elapsed clock 2026-05-28 14:30:15 +08:00
Novick Yuan 6412294262 Use header-only usage server timing 2026-05-28 14:30:15 +08:00
Novick Yuan 01c8592ca6 Align admin user usage timing samples 2026-05-28 14:30:15 +08:00
Novick Yuan 9b5c3ecd23 Use shared clock for active usage timers 2026-05-28 14:30:15 +08:00
Novick Yuan 6de684df59 Track server clock offset for usage data 2026-05-28 14:30:15 +08:00
Novick Yuan 8aca1f8b93 Add server time to usage responses 2026-05-28 14:30:15 +08:00
fawney19 bb2fc2ec00 Optimize health monitor database reads 2026-05-28 14:24:37 +08:00
fawney19 18566b5837 Merge remote-tracking branch 'origin/pr/587' 2026-05-28 13:56:04 +08:00
fawney19 069e1c1e60 Fix merged PR check regressions 2026-05-28 13:53:04 +08:00
fawney19 2c28d9979c Merge commit 'refs/pr/585'
# Conflicts:
#	apps/aether-gateway/src/ai_serving/planner/passthrough/provider/family/request.rs
#	apps/aether-gateway/src/tests/ai_execute/stream_provider_gemini/local_cli.rs
#	apps/aether-gateway/src/tests/ai_execute/sync/gemini/cli.rs
#	apps/aether-gateway/src/tests/control/admin/provider_query.rs
#	crates/aether-provider-transport/src/gemini_cli/mod.rs
#	crates/aether-provider-transport/src/gemini_cli/request.rs
#	crates/aether-provider-transport/src/gemini_cli/url.rs
#	crates/aether-provider-transport/src/lib.rs
2026-05-28 13:19:42 +08:00
fawney19 0efb3d340d Merge commit 'refs/pr/530' 2026-05-28 12:53:45 +08:00
fawney19 535039c29e Merge remote-tracking branch 'origin/pr/584' 2026-05-28 12:16:51 +08:00
fawney19 93d3de1644 feat: improve routing policy diagnostics 2026-05-28 12:11:43 +08:00
AAEE86 4ce056fe45 feat(health): add model and provider health monitoring
- Rename the original health monitor to endpoint health monitor
- Add tab navigation for endpoint, model, and provider health views
- Add model health monitor cards with availability, latency, first-byte latency, and 60-point history
- Add admin-only provider health monitor with collapsible active-provider sections
- Show per-provider model health cards after expanding a provider
- Add backend model health and provider health monitor payload builders
- Add admin endpoint for provider health monitoring
- Add provider-scoped usage breakdown filtering for per-provider model statistics
- Add frontend API types and request helpers for model/provider health data
- Add demo mock data for model and provider health monitoring
- Fix model health timeline time-unit handling so request history segments render correctly

Verification:
- cargo fmt
- npm run type-check
- npm run build
- cargo test -p aether-gateway health_models
- cargo test -p aether-gateway health_providers
- cargo test -p aether-gateway gateway_exposes_frontdoor_manifest_without_proxying_upstream
2026-05-28 12:00:20 +08:00
Mas0nShi 9ad9858ac2 Merge origin/main into fix/gemini-cli-v1internal 2026-05-28 11:58:00 +08:00
MMEXA adca142d1e fix: adapt gemini cli to v1internal endpoint 2026-05-28 00:24:56 +08:00
stabey 739e39e1ca fix: 修复缓存 token usage 转换语义
统一 OpenAI、Gemini、Claude 之间缓存 token 的 usage 语义,避免 Claude 侧重复统计缓存输入 token。

同时补充 stream 合并逻辑、字段注释和覆盖转换链路的测试。
2026-05-27 23:49:07 +08:00
fawney19 14ad6e9b75 Merge remote-tracking branch 'origin/pr/583' 2026-05-27 18:42:57 +08:00
fawney19 d46d225a90 暗色模式下交换流式徽章填充与描边样式 2026-05-27 18:38:59 +08:00
ZheFox c05d227df2 Merge branch 'fawney19:main' into main 2026-05-27 17:06:31 +08:00
fawney19 42e723ff7c Clarify API key concurrency skip reasons 2026-05-27 17:00:09 +08:00
ZheFox b02d62642a Merge branch 'fawney19:main' into main 2026-05-27 16:18:21 +08:00
zhefox 8abedecb16 fix(ai): dedupe OpenAI responses text snapshot deltas 2026-05-27 16:11:02 +08:00
fawney19 d77a572dc7 fix: cast usage provider body before jsonb type checks 2026-05-27 15:48:12 +08:00
fawney19 8606455355 Merge pull request #581 from zhefox/main
fix(gateway): preserve JSON mode chat hints in responses normalization
2026-05-27 15:40:17 +08:00
fawney19 21e52722e6 Prefer provider request body for usage badges 2026-05-27 15:39:39 +08:00
fawney19 6673ab6d4a Add fast model directive service tier 2026-05-27 15:08:23 +08:00
fawney19 d488b1a680 fix auth refresh request body 2026-05-27 15:06:51 +08:00
fawney19 b9ac97ebc3 Simplify request detail cost overview 2026-05-27 14:24:57 +08:00
zhefox e09d3199c1 fix(gateway): update codex prompt cache key test 2026-05-27 13:57:48 +08:00
fawney19 ccfc4cbddc Cache provider catalog lookups 2026-05-27 13:56:39 +08:00
zhefox 41ad422002 fix(gateway): preserve JSON mode chat hints in responses normalization 2026-05-27 13:35:16 +08:00
fawney19 674cc85005 Stabilize stream runtime nextest timing 2026-05-27 11:00:04 +08:00
fawney19 dd2da69361 Merge pull request #580 from AAEE86/main
fix(mobile): improve usage and pool management layouts
2026-05-27 10:23:17 +08:00
fawney19 0ee6e393ce Record stream first byte on upstream event 2026-05-27 10:19:49 +08:00
fawney19 433a4d3c7d test(gateway): run claude pii redaction cases on large stack 2026-05-27 09:21:57 +08:00
AAEE86 049f26c03b fix(mobile): improve usage and pool management layouts
- Fix pool account batch dialog scrolling on mobile
- Rework usage records mobile filters into clearer rows
- Align user filter styling with other select filters
- Improve request detail drawer metric layout on mobile
2026-05-27 09:20:10 +08:00
fawney19 cf8372c8cb fix(usage): record visible stream first byte timing 2026-05-27 02:48:01 +08:00
fawney19 f03550415b style(usage): make fast badge white 2026-05-27 02:11:59 +08:00
fawney19 5a710c4f5e Merge remote-tracking branch 'origin/pr/578' 2026-05-27 02:07:52 +08:00
fawney19 56901f91ce Merge remote-tracking branch 'origin/pr/576' 2026-05-27 02:06:22 +08:00
fawney19 1109c3547c Merge branch 'pr-577' 2026-05-27 01:35:25 +08:00
fawney19 d24ead234d Merge branch 'pr-575'
# Conflicts:
#	apps/aether-gateway/src/ai_serving/planner/passthrough/provider/family/payload.rs
#	apps/aether-gateway/src/ai_serving/planner/passthrough/provider/family/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/family/payload.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/family/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/chat/decision/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/responses/decision/payload.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/responses/decision/request.rs
2026-05-27 01:34:19 +08:00
fawney19 d816ae5c88 Merge remote-tracking branch 'origin/pr/573' 2026-05-27 01:06:57 +08:00
fawney19 8c6e586063 Merge remote-tracking branch 'zhefox/main' 2026-05-27 01:01:34 +08:00
fawney19 c632ec616d Merge remote-tracking branch 'origin/pr/564' 2026-05-27 00:52:15 +08:00
fawney19 bd71a46c25 Merge pull request #561 from Kayphoon/codex/s3-integrated-backup 2026-05-27 00:48:59 +08:00
fawney19 e2b5c3acc8 docs: remove simple query inventory 2026-05-27 00:45:03 +08:00
AAEE86 e27ca671fd feat(usage): show reasoning and fast badges in usage records
- extract provider reasoning effort from request body metadata
- extract priority service tier and expose it as service_tier
- show reasoning level and fast badges after model names
- include badges in active request updates and usage list payloads
- add targeted backend and frontend coverage
2026-05-27 00:36:52 +08:00
fawney19 614c999871 feat(admin): expose s3 backup as module 2026-05-27 00:30:37 +08:00
fawney19 42693c2c52 chore: remove s3 backup docs 2026-05-27 00:07:37 +08:00
fawney19 e21cd72181 fix: preserve pool scan budget for exhausted accounts 2026-05-26 23:49:26 +08:00
MMEXA a9e6a7d644 fix(frontend): recover login redirect navigation 2026-05-26 23:22:30 +08:00
yangrs ba72770cab fix: wire windsurf oauth runtime scheduling 2026-05-26 22:40:46 +08:00
Kayphoon 7530bec7de test(gateway): cover chat pii redaction formats 2026-05-26 22:29:45 +08:00
zhefox 1173a4d9d5 fix(provider): split partial model fetch warnings from errors 2026-05-26 17:45:06 +08:00
zhefox aa409a8a9c fix(provider): refine endpoint default paths for openai and claude roots 2026-05-26 16:50:36 +08:00
AAEE86 949e251b2e fix(provider): 模型测试按 Key 模型权限过滤
测试模型前检查 provider key 的 allowed_models:
- 空权限视为允许所有模型
- 非空权限需匹配请求模型或映射后的实际模型
- 不匹配的 key 标记为跳过,避免发起测试请求

同时补充相关单测和前端跳过原因文案。
2026-05-26 16:46:21 +08:00
fawney19 4933ae9014 Merge pull request #572 from AAEE86/main
fix(admin): add User-Agent for Done-hub provider ops
2026-05-26 16:19:02 +08:00
AAEE86 1793443b09 fix(admin): add User-Agent for Done-hub provider ops
- Done-hub Cookie 请求增加浏览器 User-Agent
- 覆盖认证验证和余额查询的共享请求头
- 补充请求头测试,确认 Cookie 与 User-Agent 同时发送
2026-05-26 16:07:02 +08:00
ZheFox 23a36e37bb Merge branch 'fawney19:main' into main 2026-05-26 15:56:17 +08:00
zhefox c9cf1d458a fix(provider): factor model fetch route test type alias 2026-05-26 15:56:03 +08:00
zhefox d28a389a93 fix(provider): support unversioned API roots in model fetch 2026-05-26 15:39:56 +08:00
fawney19 7e76c9763d Clarify stream first byte timeout message 2026-05-26 15:02:50 +08:00
Kayphoon 9e029462aa test(gateway): stabilize stream timeout regression 2026-05-26 14:41:56 +08:00
Kayphoon 4523a2c67b fix(data): cast MySQL usage aggregates 2026-05-26 14:41:56 +08:00
Kayphoon 84c8bc960e feat(admin): add configurable S3 backups 2026-05-26 14:41:56 +08:00
zhefox c4927162b7 Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-26 13:57:27 +08:00
zhefox 1ebe0aeadf fix(users): allow clearing explicit admin group memberships 2026-05-26 13:57:22 +08:00
ZheFox 992c58f2bd Merge branch 'fawney19:main' into main 2026-05-26 13:08:59 +08:00
zhefox 0bf63cc80e fix(provider): support multi-key selection in model tests 2026-05-26 13:08:35 +08:00
zhefox 5fc6dc8019 Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-26 12:31:03 +08:00
zhefox 96184caa48 fix(codex): strip unsupported OpenAI responses body fields 2026-05-26 12:30:58 +08:00
fawney19 12ff87949d fix(ai): preserve combined Gemini builtin tools 2026-05-26 11:14:24 +08:00
fawney19 b75953bf4c Merge remote-tracking branch 'origin/pr/569' 2026-05-26 11:11:13 +08:00
MMEXA c733139091 fix(ai): normalize Gemini search grounding tools 2026-05-26 05:33:34 +08:00
fawney19 331d37be26 test: run sub2api balance provider ops on larger stack 2026-05-26 02:30:39 +08:00
fawney19 57ccd44b89 fix: fill provider quota execution timeout defaults 2026-05-26 02:11:10 +08:00
fawney19 d60b6e7454 test: run gemini image bridge case on larger stack 2026-05-26 01:52:01 +08:00
fawney19 50e4f27276 Merge remote-tracking branch 'origin/pr/567' 2026-05-26 01:21:03 +08:00
fawney19 a0f22ae659 fix: tighten pr 566 claude and deepseek handling 2026-05-26 00:57:28 +08:00
fawney19 235f32e10e Merge remote-tracking branch 'origin/pr/566' into review/pr-566-fix 2026-05-26 00:43:45 +08:00
fawney19 e7b3acdec3 fix(provider): format oauth import tests 2026-05-25 23:48:04 +08:00
fawney19 f3a367b02d Merge commit 'refs/pull/563/head' of github-fawney19:fawney19/Aether into review/pr-562 2026-05-25 23:44:02 +08:00
fawney19 c03aebba3f Merge branch 'pr-562' into review/pr-562 2026-05-25 23:10:29 +08:00
fawney19 4fb8955bc2 fix(provider): move key model auto-match into dialog 2026-05-25 23:03:52 +08:00
Novick Yuan 5dfccdec3e Fix stream candidate watchdog timeout semantics 2026-05-25 21:43:13 +08:00
fawney19 8b386b0aac Merge remote-tracking branch 'origin/pr/555' 2026-05-25 21:14:45 +08:00
hemo94931 9010f0806a fix(ai): sanitize Claude Read pages passthrough 2026-05-25 21:01:10 +08:00
hemo94931 495795327c fix(ai): sanitize empty Read pages for Claude tools 2026-05-25 21:01:10 +08:00
root 686311eabf test: align OpenAI image stream keepalive expectation 2026-05-25 21:01:10 +08:00
root e68b843875 Add DeepSeek thinking compatibility 2026-05-25 21:00:08 +08:00
root b46028cb85 refactor: clarify OpenAI SSE control policy 2026-05-25 21:00:08 +08:00
root fa172ecb95 fix: avoid synthetic keepalive for OpenAI streams 2026-05-25 21:00:08 +08:00
root 431311979a fix: handle split streaming terminal events 2026-05-25 20:58:17 +08:00
fawney19 d3249485fa Fix stream timeout semantics 2026-05-25 20:09:37 +08:00
zhefox 4c22a819f9 fix(provider): always show batch assign models action 2026-05-25 20:05:13 +08:00
zhefox f4d66021e4 Merge remote-tracking branch 'upstream/main'
# Conflicts:
#	crates/aether-data/src/repository/usage/mysql.rs
2026-05-25 17:29:12 +08:00
fawney19 54c5d5803b fix: cast mysql usage aggregate counters 2026-05-25 15:36:13 +08:00
fawney19 ae138ddb56 feat: update admin config import and runtime handling 2026-05-25 15:23:02 +08:00
zhefox b72abec2fc fix(gateway): spawn oauth account refresh asynchronously 2026-05-25 15:09:41 +08:00
zhefox db4f3fd210 fix(usage): cast mysql usage aggregates to numeric types 2026-05-25 13:56:12 +08:00
zhefox 230ce5df5f Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-25 13:38:38 +08:00
zhefox ec681335e8 fix(usage): treat empty body_state as missing terminal event 2026-05-25 13:38:18 +08:00
Entropy.Xu aaad113190 feat(admin-users): 支持按创建时间排序 2026-05-25 12:21:58 +08:00
calida-tec 63681b4be3 fix(provider): accept common OAuth token JSON aliases 2026-05-25 10:02:47 +08:00
MMEXA b347f1816d Fix native Antigravity stream envelope handling 2026-05-25 09:39:55 +08:00
ZheFox e9efc5c42a Merge branch 'fawney19:main' into main 2026-05-25 09:15:13 +08:00
MMEXA 28c3a5dbe4 Add Antigravity v1internal gateway adapter 2026-05-25 06:58:15 +08:00
fawney19 505d9fd8bc Merge remote-tracking branch 'origin/main' 2026-05-25 01:56:34 +08:00
fawney19 932397d1b3 fix(gateway): defer ChatGPT web image quota decrement 2026-05-25 01:56:25 +08:00
fawney19 1be445423c Merge remote-tracking branch 'origin/pr/558' 2026-05-25 01:22:03 +08:00
fawney19 2df9615fb9 Merge pull request #560 from Kayphoon/codex/remove-install-migration
fix(install): remove pg single-node migration entrypoint
2026-05-25 01:21:30 +08:00
fawney19 fe7fb17ff5 fix(gateway): align admin key health circuit summary 2026-05-25 01:18:07 +08:00
Kayphoon 72d43878ef fix(install): remove pg single-node migration entrypoint 2026-05-25 01:14:37 +08:00
fawney19 cc3ce8b5d7 Merge remote-tracking branch 'origin/pr/557' 2026-05-25 01:08:22 +08:00
fawney19 d4ae6e0e64 fix: read imported usage aggregates in dashboards 2026-05-25 00:51:46 +08:00
MMEXA 480579a0d5 fix(gateway): expire provider key circuit cooldowns 2026-05-25 00:40:38 +08:00
ZheFox ba188aea92 Merge branch 'fawney19:main' into main 2026-05-24 23:40:40 +08:00
fawney19 40b4e52508 Filter format-scoped key fields on import 2026-05-24 22:44:12 +08:00
fawney19 e2d5fc9dfb Preserve usage data in system imports 2026-05-24 21:40:48 +08:00
zhefox c92bdfba16 Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-24 18:51:28 +08:00
ZheFox 83a2609344 Merge branch 'fawney19:main' into main 2026-05-24 18:51:00 +08:00
fawney19 18d9004f22 fix docker app logging permissions 2026-05-24 18:50:39 +08:00
Codex 74c8bfc59f 调整 ChatGPT Web 生图 token 估算口径 2026-05-24 18:50:14 +08:00
Codex 3bf7469d30 修复 ChatGPT Web 生图 usage 估算 2026-05-24 18:50:14 +08:00
Codex 66837b7d7f 修复 ChatGPT Web 生图发起即扣额度 2026-05-24 18:50:14 +08:00
Codex 14b182d09b 修复 ChatGPT Web 生图调用起始预扣额度 2026-05-24 18:50:14 +08:00
Codex 9dba6ec1d9 修复 ChatGPT Web 生图预扣与 Free 限额继承 2026-05-24 18:50:14 +08:00
Codex a52b513533 调整 ChatGPT Web 生图请求预扣额度 2026-05-24 18:50:14 +08:00
Codex 218ca8e6eb 修复 ChatGPT Web 生图额度递减显示 2026-05-24 18:50:14 +08:00
Codex 2b2754b779 修复 ChatGPT Web 生图成功后额度同步 2026-05-24 18:50:13 +08:00
Codex 952d1c840d 修复 ChatGPT Web 额度刷新 403 误判 2026-05-24 18:50:13 +08:00
zhefox 2207b60834 fix(usage): preserve failed status for active request refreshes 2026-05-24 18:48:50 +08:00
ZheFox c09bb28d16 Merge branch 'fawney19:main' into main 2026-05-24 18:07:09 +08:00
github-actions[bot] 13e0759d5a chore(tunnel): update download links for tunnel-v0.3.13 2026-05-24 08:43:41 +00:00
fawney19 80054276eb chore: bump aether-tunnel version to 0.3.13 2026-05-24 16:36:57 +08:00
fawney19 517c9e5108 Merge pull request #552 from RWDai/issue-549-fix
Bound models route data reads
2026-05-24 16:17:41 +08:00
fawney19 576918daa5 Optimize provider scheduler database hotspots 2026-05-24 15:47:56 +08:00
zhefox bbd4338e8e Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-24 15:21:27 +08:00
zhefox e6423a91aa feat(provider): auto-match batch assign models from key 2026-05-24 15:19:37 +08:00
fawney19 78523f122d Limit pool score interest feedback writes 2026-05-24 12:35:55 +08:00
fawney19 e1df06f06c revert compose data layout to legacy paths 2026-05-24 00:37:03 +08:00
fawney19 ecfe04f48a Merge pull request #554 from zhefox/main 2026-05-24 00:15:47 +08:00
ZheFox ff7f27d4f8 Merge branch 'fawney19:main' into main 2026-05-23 23:46:18 +08:00
fawney19 dd07425d21 Merge remote-tracking branch 'origin/pr/550'
# Conflicts:
#	frontend/src/features/usage/components/UsageRecordsTable.vue
2026-05-23 23:40:10 +08:00
ZheFox ba9aa7c1bd Merge branch 'fawney19:main' into main 2026-05-23 23:32:40 +08:00
zhefox 92fd253cae Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-23 22:24:09 +08:00
zhefox 9f6fac418e feat(usage): normalize provider stats across usage and cost views 2026-05-23 22:22:37 +08:00
fawney19 e53a2757eb Merge branch 'pr-548' 2026-05-23 22:19:51 +08:00
fawney19 db32b1d982 fix(usage): ignore truncated stream captures for terminal inference 2026-05-23 22:16:02 +08:00
fawney19 6b1c1e4f50 Merge remote-tracking branch 'origin/pr/547' 2026-05-23 22:03:17 +08:00
fawney19 3eb9614e68 Merge pull request #546 from novcky/fix/pool-scheduler-skip-invalid-oauth-accounts
修复 Provider Pool 热池反复调度已失效 OAuth 账号的问题
2026-05-23 21:38:18 +08:00
fawney19 8087a98c9d Merge remote-tracking branch 'origin/pr/475'
# Conflicts:
#	apps/aether-gateway/src/handlers/admin/provider/oauth/dispatch/refresh/execution.rs
#	apps/aether-gateway/src/handlers/admin/provider/oauth/dispatch/refresh/response.rs
#	apps/aether-gateway/src/handlers/admin/provider/oauth/errors.rs
#	apps/aether-gateway/src/handlers/admin/provider/oauth/quota/shared.rs
#	apps/aether-gateway/src/state/oauth.rs
#	apps/aether-gateway/src/tests/control/admin/oauth.rs
#	crates/aether-admin/src/provider/quota.rs
2026-05-23 21:26:22 +08:00
fawney19 5643b2c901 feat: add compose data layout migration helper 2026-05-23 20:51:02 +08:00
fawney19 18eac2dd7a feat: clarify deployment update strategies 2026-05-23 20:14:26 +08:00
ZheFox 0f2a96554f Merge branch 'fawney19:main' into main 2026-05-23 19:54:24 +08:00
RWDai 0655e868a2 Bound models route data reads 2026-05-23 19:36:20 +08:00
fawney19 4b66cadf15 Merge remote-tracking branch 'origin/pr/544' 2026-05-23 18:41:52 +08:00
Kayphoon 4ee64339ba fix(usage): show retry marker with fallback 2026-05-23 17:54:48 +08:00
Kayphoon babe328565 fix(usage): include embedding formats in filters 2026-05-23 16:19:23 +08:00
fawney19 6447fda852 fix: harden tunnel security and timeout handling 2026-05-23 14:17:04 +08:00
fawney19 74f7348529 Merge remote-tracking branch 'origin/pr/535' 2026-05-23 13:00:18 +08:00
wzw bf456450d7 feat: 代理池均衡分发&批量添加代理节点 2026-05-23 10:40:25 +08:00
zhefox 91cb2bbbcc fix(usage): refine stream terminal capture gating for OpenAI responses 2026-05-23 03:31:15 +08:00
Novick Yuan c0252387b4 修复热池调度已失效 OAuth 账号 2026-05-23 03:17:09 +08:00
zhefox bd1e155332 fix(provider): normalize OpenAI chat tool history for Claude messages 2026-05-23 02:45:04 +08:00
fawney19 ab048b8a03 fix monitoring trace lookup fallback 2026-05-23 01:22:24 +08:00
zhefox e5ce2ac7a4 fix(usage): detect missing terminal events in stream reporting 2026-05-23 00:47:36 +08:00
fawney19 c7641dad0a fix: propagate build version through local deploy 2026-05-23 00:26:07 +08:00
fawney19 b5e942ca9d fix: increase postgres shared memory for dashboard queries 2026-05-23 00:21:34 +08:00
fawney19 74c82d9948 Merge remote-tracking branch 'origin/main' 2026-05-22 23:59:05 +08:00
fawney19 d18b13a91a fix: harden frontdoor and usage ingestion 2026-05-22 23:57:38 +08:00
Mas0nShi 0d80db8a8d Refactor Gemini CLI v1internal planner request builder 2026-05-22 18:57:03 +08:00
Mas0nShi 8cc6888c5b Fix Gemini CLI OpenAI conversion envelope 2026-05-22 18:40:11 +08:00
fawney19 9af1507238 Merge pull request #545 from RWDai/feat/expand-client-types
feat: expand client type recognition
2026-05-22 18:36:53 +08:00
Mas0nShi c67818ee86 Fix Gemini CLI standard conversion envelope 2026-05-22 18:24:40 +08:00
fawney19 6ef6cbade2 Fallback dashboard aggregate reads on schema mismatch 2026-05-22 17:52:36 +08:00
Mas0nShi 8df0e1790d Fix Gemini CLI batch import parse error entry 2026-05-22 17:25:13 +08:00
Mas0nShi 8b7643e150 Merge remote-tracking branch 'origin/main' into fix/gemini-cli-v1internal
# Conflicts:
#	apps/aether-gateway/src/ai_serving/transport.rs
#	apps/aether-gateway/src/handlers/admin/provider/oauth/dispatch/batch/parse.rs
#	apps/aether-gateway/src/handlers/shared/catalog.rs
#	crates/aether-admin/src/provider/quota.rs
#	crates/aether-model-fetch/src/strategy.rs
#	crates/aether-provider-pool/src/lib.rs
#	crates/aether-provider-pool/src/service.rs
#	crates/aether-provider-transport/src/provider_types.rs
#	frontend/src/features/providers/components/ProviderDetailDrawer.vue
#	frontend/src/utils/__tests__/providerKeyQuota.spec.ts
#	frontend/src/utils/providerKeyQuota.ts
#	frontend/src/views/admin/PoolManagement.vue
2026-05-22 17:13:57 +08:00
fawney19 ef04f4b0fb Add automatic B/T compact unit formatting 2026-05-22 17:13:16 +08:00
Mas0nShi ce02f1ae8c Add Gemini CLI v1internal quota support 2026-05-22 16:58:04 +08:00
RWDai c2d0f60784 fix(gateway): accept header sessions for unknown clients 2026-05-22 16:10:27 +08:00
zhiqicloud 781830a202 fix: resolve gateway clippy regressions 2026-05-22 15:54:51 +08:00
RWDai 9dd545353c Preserve omitted tunnel security in CLI mode 2026-05-22 15:54:10 +08:00
zhiqicloud 4c7ebf8b8d style: format admin update settings 2026-05-22 15:32:49 +08:00
zhiqicloud a4a5f70a10 Merge upstream/main into feat/one-click-update 2026-05-22 15:28:27 +08:00
RWDai 9633bce2e1 feat(frontend): centralize client family labels 2026-05-22 15:28:19 +08:00
RWDai ca341703bc feat(usage): infer additional client families 2026-05-22 15:27:39 +08:00
RWDai cb2ff61bbc feat(gateway): expand client session family detection 2026-05-22 15:26:56 +08:00
RWDai 08b27806a7 Respect explicit tunnel security off 2026-05-22 15:20:36 +08:00
zhiqicloud b59c3a9e3b feat: support admin online update and deploy flow 2026-05-22 15:15:17 +08:00
fawney19 ecf6019ccb Merge pull request #543 from zhefox/main
Accept Claude message bodies in OpenAI chat endpoints
2026-05-22 14:53:25 +08:00
zhefox 2a298de971 fix(provider): serialize Claude tool results as JSON strings 2026-05-22 14:41:27 +08:00
RWDai 33633637e5 Fix secure tunnel session handling 2026-05-22 14:35:43 +08:00
fawney19 8ede01ad4e Merge remote-tracking branch 'origin/main' 2026-05-22 14:16:07 +08:00
fawney19 8966fd6aac Protect background workers under DB pool pressure 2026-05-22 14:11:47 +08:00
ZheFox 2b8ff8a743 Merge branch 'fawney19:main' into main 2026-05-22 14:11:00 +08:00
zhefox 97de4ff8a3 fix(gateway): accept Claude Messages bodies on OpenAI chat endpoints 2026-05-22 14:09:51 +08:00
fawney19 f6c3ebf7d3 Merge pull request #542 from zhefox/main
Handle OpenAI chat body responses and SSE passthrough
2026-05-22 12:05:36 +08:00
ZheFox 56abfdf39d Merge branch 'fawney19:main' into main 2026-05-22 11:51:53 +08:00
zhefox 9b95fa4d95 fix(gateway): handle responses-shaped OpenAI chat bodies and SSE passthrough 2026-05-22 11:39:38 +08:00
fawney19 f341c573eb Merge pull request #541 from AAEE86/main
feat(notification): add Bark push support
2026-05-22 11:24:11 +08:00
fawney19 b227669985 Merge pull request #540 from zhefox/main
fix(usage): treat stream terminal failures as failures on HTTP 200
2026-05-22 11:23:53 +08:00
AAEE86 ce44d35eb6 feat(notification): add Bark push support
Add Bark as a notification-service delivery channel, including encrypted Device Key configuration, server URL/template settings, module status integration, and admin UI support.
2026-05-22 11:08:00 +08:00
RWDai b05f2a270a Fix gateway secure tunnel Clippy warning 2026-05-22 10:13:06 +08:00
RWDai 2e701a90c9 Complete secure tunnel encryption support 2026-05-22 09:47:28 +08:00
zhefox 507f2f8250 fix(usage): treat stream terminal failures as failures on HTTP 200 2026-05-22 09:34:22 +08:00
Mas0nShi 9533bd7043 fix: route Gemini CLI generateContent through stream 2026-05-22 09:30:55 +08:00
fawney19 2b32b9a445 Fix system data import export flows 2026-05-22 02:46:49 +08:00
fawney19 3714c211dc Merge pull request #534 from RWDai/fix/issue-528-cache-affinity-health
fix(gateway): preserve cache affinity during health updates
2026-05-22 02:10:45 +08:00
fawney19 504c1ccb37 feat: restructure notification services 2026-05-22 01:45:46 +08:00
fawney19 d5e64d6ad9 Merge branch 'pr-503'
# Conflicts:
#	apps/aether-gateway/src/handlers/admin/provider/summary/value.rs
#	apps/aether-gateway/src/lib.rs
#	apps/aether-gateway/src/maintenance/mod.rs
#	apps/aether-gateway/src/maintenance/runtime/workers.rs
#	frontend/src/api/endpoints/types/provider.ts
2026-05-22 00:19:23 +08:00
RWDai 9859aec16c fix(gateway): evict pooled affinity after sibling key failures 2026-05-21 23:50:34 +08:00
fawney19 0e6d7539ad Merge remote-tracking branch 'origin/pr/538' 2026-05-21 23:12:42 +08:00
fawney19 d6eb41aa78 Merge remote-tracking branch 'origin/pr/536'
# Conflicts:
#	apps/aether-gateway/src/execution_runtime/stream/execution.rs
2026-05-21 23:03:14 +08:00
fawney19 97997685b5 Merge remote-tracking branch 'origin/pr/498' 2026-05-21 22:56:43 +08:00
fawney19 ab0a90de97 fix(runtime-state): govern redis connections 2026-05-21 22:53:37 +08:00
ZheFox eeb7995214 Merge branch 'fawney19:main' into main 2026-05-21 22:27:07 +08:00
zhefox 68d8f86dc6 fix(gateway): stop stream polling on downstream disconnect and preserve Codex cache keys 2026-05-21 22:26:14 +08:00
RWDai 18fe5a4f11 fix(gateway): preserve affinity during adaptive retries 2026-05-21 21:56:26 +08:00
RWDai 26ee1a9958 fix(gateway): preserve affinity during quota telemetry 2026-05-21 21:56:26 +08:00
zhefox 77c2d91eb0 fix(gateway): drain downstream-disconnected streams and stop inferring cancelled usage 2026-05-21 20:30:44 +08:00
fawney19 b8a65cbdec Merge pull request #532 from RWDai/opencode/cosmic-nebula
fix: raise group rate limits by access tier
2026-05-21 19:39:50 +08:00
zhefox 71f9afc526 fix(gateway): move openai responses helper import to ai_serving module 2026-05-21 18:09:57 +08:00
zhefox 8ca4a10f24 fix(gateway): require terminal events for OpenAI responses streams 2026-05-21 17:46:21 +08:00
Mas0nShi 66f21de50e fix: unwrap Gemini CLI model test envelopes 2026-05-21 17:39:10 +08:00
Mas0nShi 3e6ce6cf4a fix: hydrate Gemini CLI project metadata 2026-05-21 17:10:17 +08:00
stabeyandClaude Opus 4.7 cadc45c5b8 fix(data): cleanup uses failed candidate status instead of 504
The stale-pending cleanup task previously hardcoded status_code=504 and a
generic timeout message for every usage row it finalized. When a request
had already been observed as failing — e.g. upstream Connection reset by
peer, watchdog 504, or an authenticated 4xx — the cleanup overwrote that
context with a misleading "服务器超时" outcome and 504 status, hiding the
real cause from the dashboards and customer.

Pull the most recent failed/cancelled candidate per stale request_id and,
if present, finalize the usage row with the candidate's status_code
(defaulting to 502 when none was recorded) and error_message. Requests
that have no terminal candidate (truly stuck pending/streaming) keep the
existing 504 + timeout-message behavior, since they really are timeouts
from the cleanup's perspective. Applied to all three SQL backends with
parameterized UPDATE statements.

The Postgres failed-candidate lookup orders by
COALESCE(finished_at, started_at, created_at) DESC, matching the MySQL
and SQLite ORDER BY clauses so the three backends pick the same
"most recent terminal candidate" under every NULL combination of timing
columns.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-21 16:56:56 +08:00
zhefox b7b7b4f718 fix(gateway): report terminal stream failure errors consistently 2026-05-21 16:15:03 +08:00
RWDai 4f49dd5943 Document tunnel security MVP config 2026-05-21 16:09:38 +08:00
RWDai 888414c41b Forward proxy tunnel security aliases 2026-05-21 16:09:00 +08:00
RWDai 8f41bc1558 Generate secure tunnel install sessions 2026-05-21 16:08:37 +08:00
RWDai a543ca9e07 Add tunnel installer security envs 2026-05-21 16:08:14 +08:00
RWDai 40b9db3545 Add tunnel security setup fields 2026-05-21 16:07:56 +08:00
RWDai bd4f6b9206 Add tunnel security config fields 2026-05-21 16:07:27 +08:00
RWDai 776f95b1ab chore: format auth rate limit tests for rustfmt 1.95 2026-05-21 15:53:40 +08:00
zhefox 12abde2aeb fix(usage): handle terminal stream failures and preserve usage updates 2026-05-21 15:52:51 +08:00
RWDai 965a8c79af fix(gateway): preserve cache affinity during health updates 2026-05-21 15:51:56 +08:00
RWDai d33043288d fix(frontend): clarify user group policy help 2026-05-21 15:38:53 +08:00
RWDai a2ad556f2b fix(gateway): raise group rate limits by tier 2026-05-21 15:38:42 +08:00
Mas0nShi e53d5f07e8 feat: support Gemini CLI v1internal quota 2026-05-21 15:38:10 +08:00
stabeyandClaude Opus 4.7 40434005c0 fix(gateway): use total_ms for non-stream upstream watchdog
When the endpoint forces upstream_stream_policy=force_non_stream while
the client streams, the local stream candidate watchdog still preferred
timeouts.first_byte_ms — a non-stream upstream produces no early first
byte, so the watchdog fired before the HTTP request_timeout and aborted
otherwise-healthy attempts at ~300s.

Read upstream_is_stream from report_context and invert the priority:
non-stream upstreams use total_ms first, falling back to first_byte_ms
and then the default; streaming upstreams keep the previous order.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-21 15:26:42 +08:00
stabeyandClaude Opus 4.7 3330b2ac4c refactor(report-context): extract UPSTREAM_IS_STREAM_KEY constant
The "upstream_is_stream" JSON key flows from the AI execution report
context producer (aether-ai-serving::report_context) through several
consumers — usage runtime metadata copy/move, gateway watchdog, sync
execution decision, observability handlers, and the per-driver usage
repositories. Each site spelled the key as a bare string literal, so a
producer-side rename would silently degrade every consumer to its
fallback (typically assuming streaming) with no compile-time signal.

Introduce a single pub const UPSTREAM_IS_STREAM_KEY in
aether-ai-formats (the lowest crate every consumer already depends on),
re-export from the crate root, and route producer + all map-style
consumers through it. The change is purely a string-literal → constant
swap; behaviour is identical.

Sites left as literals (intentional):
- `json!({"upstream_is_stream": ...})` macro keys, which must be string
  literals at the macro layer; these are also API-response payload
  field names (an external contract that should not silently track
  internal report-context renames).
- SQL column accessors (`try_get::<...>("upstream_is_stream")`), which
  refer to the database schema column, not the JSON key.
- Test fixtures and assertions, which validate the on-the-wire contract
  and should keep verifying the actual string.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-21 15:26:42 +08:00
zhefox be6e49b9c2 Merge branch 'main' of https://github.com/zhefox/Aether 2026-05-21 12:30:48 +08:00
zhefox e59e6c3797 fix(gateway): sanitize Claude thinking and handle missing stream finish 2026-05-21 12:30:42 +08:00
Entropy.Xu 6b04a0a3a6 fix(windsurf): 修复 native 工具流式回程 2026-05-21 02:55:05 +08:00
Entropy.Xu 4112a8b2ea fix(provider): 修复 Windsurf PR CI 失败 2026-05-21 02:21:35 +08:00
fawney19 b84e4a96e2 chore: tune default postgres settings for 2c4g 2026-05-21 01:36:05 +08:00
fawney19 e7f8b259ac Revert "Merge pull request #517 from zhiqicloud/feat/provider-balance-query"
This reverts commit 7e95e769d5, reversing
changes made to 490306c242.
2026-05-21 01:24:17 +08:00
Entropy.Xu 65c361115a fix(provider): 修复 Windsurf PR 冲突残留 2026-05-21 01:02:02 +08:00
Entropy.Xu 129c7c90c0 fix(provider): 修复 Windsurf 原生工具桥接 2026-05-21 01:02:02 +08:00
Entropy.Xu 82637ad882 fix(provider): 修复 Windsurf Connect 请求与端点计数 2026-05-21 01:02:02 +08:00
Entropy.Xu 931c577345 fix(provider): 接入 Windsurf 模型测试链路 2026-05-21 01:02:02 +08:00
Entropy.Xu 9466d92a7a fix(provider): 接入 Windsurf 模型拉取和格式转换 2026-05-21 01:02:02 +08:00
Entropy.Xu 0a0a8b31c7 fix(provider): 隐藏 Windsurf refresh token 刷新入口 2026-05-21 01:02:02 +08:00
Entropy.Xu 208c77a062 fix(provider): 对齐 Windsurf PostAuth 登录链路 2026-05-21 01:02:02 +08:00
Entropy.Xu 02d1343436 feat(provider): 补充 Windsurf 邮箱密码导入表单 2026-05-21 01:02:02 +08:00
Entropy.Xu 0226e14251 feat(provider): 原生接入 Windsurf provider 2026-05-21 01:02:02 +08:00
fawney19 923515ab28 fix: align image generation checks 2026-05-21 00:45:02 +08:00
fawney19 4d0c654822 Merge remote-tracking branch 'origin/pr/524' 2026-05-20 23:17:31 +08:00
ZheFox 779877acd0 Merge branch 'fawney19:main' into main 2026-05-20 22:49:49 +08:00
fawney19 d49b0a8a45 Make extension modules reorderable 2026-05-20 22:45:32 +08:00
ZheFox 5a9f19cbf2 fix(gateway): filter upstream SSE control-only blocks 2026-05-20 22:33:41 +08:00
zhiqicloud 9562295d8b feat: 添加在线更新功能 2026-05-20 22:29:11 +08:00
ZheFox 3c6924238f feat(usage): include cache token details in stream usage payloads 2026-05-20 21:20:10 +08:00
ZheFox 64ad0f694b feat(usage): include cache token details in stream usage payloads 2026-05-20 21:06:51 +08:00
fawney19 754f672ee2 Merge pull request #526 from MMEXA/fix/codex-responses-pending-recovery-20260520
修复 Codex Responses 工具字段和成功请求回收标记
2026-05-20 21:01:54 +08:00
fawney19 e50ceeba5a Merge pull request #525 from final0920/fix/issue-505-priority-order
fix: 修复优先级管理重新打开顺序回退
2026-05-20 21:00:06 +08:00
fawney19 57910f906d Preserve usage provider identity 2026-05-20 20:56:36 +08:00
ZheFox 8838e9289b fix(provider): remove stale image preview block from model test dialog 2026-05-20 20:53:13 +08:00
ZheFox d3355a8a09 fix(gateway): decode stream-encoded provider response JSON 2026-05-20 20:40:48 +08:00
fawney19 c972bbd397 Fix provider pool exhaustion scheduling 2026-05-20 20:16:57 +08:00
MMEXA fed9bdf01a Fix Codex Responses tool schema and pending recovery 2026-05-20 12:02:00 +00:00
ZheFox ab2287202d feat(provider): show image previews in model test dialog 2026-05-20 19:59:20 +08:00
流云 b47282fe4c fix: 修复优先级管理重新打开顺序回退
优先级管理弹窗依赖 grouped-by-format 接口回显格式优先级,但该接口此前读取 summary key 行。summary 查询会清空 global_priority_by_format 和 internal_priority 等路由字段,导致保存后的数据库顺序存在,重新打开页面却回退为前端占位顺序。

改为使用完整 key 查询,并增加 summary 字段被清空时仍能回显真实优先级的回归测试。

Fixes #505

Constraint: grouped-by-format 是优先级管理弹窗的数据源,必须返回真实 per-format priority 字段。
Rejected: 修改前端继续猜测顺序 | 无法区分真实数据库优先级与占位回退。
Confidence: high
Scope-risk: narrow
Tested: cargo fmt --check; git diff --check
Not-tested: cargo test on local Windows blocked by missing NASM for boring-sys2
2026-05-20 19:44:05 +08:00
ZheFox a31e237cc3 Merge upstream main 2026-05-20 19:28:49 +08:00
ZheFox cfa32a2b4d fix(gateway): preserve openai image 200 responses and sync success reporting 2026-05-20 19:17:42 +08:00
ZheFox 2cacf66a37 fix(gateway): route openai image streams with images surface 2026-05-20 18:13:34 +08:00
fawney19 d0981c2fd5 Merge pull request #522 from RWDai/fix/admin-users-server-pagination
Fix admin users server-side pagination
2026-05-20 18:03:11 +08:00
RWDai 3e12c06627 Fix user group options cache busting 2026-05-20 17:51:29 +08:00
RWDai 74a3df3f1e Fix user group options cache invalidation 2026-05-20 17:46:16 +08:00
fawney19 cb894208a1 Merge pull request #521 from Avilianb/fix/provider-query-responses-compact-body
Fix provider model test compact request bodies
2026-05-20 17:42:53 +08:00
fawney19 76752beca6 chore(postgres): 支持通过环境变量配置 PG 性能参数 2026-05-20 17:41:52 +08:00
RWDai e80e7b0cfa Fix admin users pagination review issues 2026-05-20 17:41:06 +08:00
RWDai 77051e6245 Fix admin users pagination follow-ups 2026-05-20 17:32:47 +08:00
ZheFox de7be4f15b fix(gateway): route openai image api requests with mapped models 2026-05-20 17:16:38 +08:00
RWDai 44fb1af287 Fix admin user count clippy lint 2026-05-20 17:06:33 +08:00
fawney19 e7a76b0510 chore(docker): postgres 启用 pg_stat_statements 扩展 2026-05-20 17:00:44 +08:00
fawney19 2881ff097a feat(usage): 管理员用量统计支持筛选刷新与失败保留旧数据
- 用量统计/聚合接口新增 skipCache 选项与 120s 超时,便于强制绕过缓存
- loadStats 增加 force/preserveOnFailure 选项,背景刷新失败时保留旧数据
- 管理员页面在用户/模型/Provider 筛选变化时强制刷新统计,并将筛选条件传入统计接口
- 手动刷新与自动刷新分离,自动刷新不再重载长期热力图相关聚合
2026-05-20 16:51:52 +08:00
RWDai 462c3dde79 Load admin users with server-side pagination 2026-05-20 16:51:46 +08:00
RWDai 1be703b56e Track admin users pagination in frontend data layer 2026-05-20 16:51:46 +08:00
RWDai 5130da9710 Return paginated admin users metadata 2026-05-20 16:51:46 +08:00
RWDai 8440846bae Expose admin user export counts through gateway state 2026-05-20 16:51:46 +08:00
RWDai 831554f11d Add admin user export count queries 2026-05-20 16:51:46 +08:00
Avilianb 97fb588a4a Apply rustfmt to compact provider test 2026-05-20 16:47:30 +08:00
Avilianb cd994d57d2 Fix provider model test compact request bodies 2026-05-20 16:29:46 +08:00
fawney19 70f2882a43 refactor(api-keys): 将独立余额 Key 表单的额度、IP 限制、敏感信息保护移至左侧列 2026-05-20 16:28:58 +08:00
fawney19 fa5d26ce38 Merge remote-tracking branch 'origin/main' 2026-05-20 16:14:09 +08:00
fawney19 f76bbaab52 feat: support api key ip restriction rules 2026-05-20 16:11:49 +08:00
ZheFox cde2062618 chore(gateway): make openai image intent import local 2026-05-20 16:09:24 +08:00
ZheFox 9673fc4c01 fix(codex): trigger image override only on explicit tool_choice 2026-05-20 15:34:31 +08:00
fawney19 19ae7c8902 Merge pull request #519 from RWDai/feat/aether-tunnel-ip-family-options
feat(tunnel): add tunnel IP family controls
2026-05-20 15:30:17 +08:00
RWDai eb63838f6a fix(proxy): keep legacy installer URLs working 2026-05-20 14:43:24 +08:00
RWDai 232006f71d fix(tunnel): restore IP family flag test builds 2026-05-20 14:42:52 +08:00
fawney19 b6bdc08267 Merge branch 'pr-501' 2026-05-20 14:05:00 +08:00
fawney19 7e95e769d5 Merge pull request #517 from zhiqicloud/feat/provider-balance-query
支持 Provider Key 余额查询与自动刷新
2026-05-20 13:59:24 +08:00
RWDai f4c79c80ac fix(tunnel): allow explicit false IP family flags 2026-05-20 13:50:58 +08:00
RWDai edded777e7 docs(tunnel): document IP family controls 2026-05-20 13:43:22 +08:00
RWDai 7284165f39 feat(tunnel): add tunnel IP family controls 2026-05-20 13:42:55 +08:00
RWDai 1604a6d87d fix(gateway): allow clearing API key IP whitelists 2026-05-20 13:41:14 +08:00
ZheFox 7d5ad1e70e chore(gateway): silence dead code warnings in openai image bridge 2026-05-20 13:32:14 +08:00
ZheFox 89860bec97 fix(codex): restrict openai image routing to codex responses 2026-05-20 13:17:24 +08:00
zhiqicloud ebc1774300 修正 new_api 验证测试断言 2026-05-20 13:04:22 +08:00
zhiqicloud 122daf0f87 支持 Provider Key 余额查询与自动刷新 2026-05-20 12:33:31 +08:00
RWDai 149651f831 test(gateway): expect image bridge tools 2026-05-20 11:22:45 +08:00
fawney19 490306c242 Merge commit 'refs/pull/511/head' of github-fawney19:fawney19/Aether 2026-05-20 11:19:58 +08:00
RWDai 316b1e3207 Merge remote-tracking branch 'upstream/main' into feat/500-api-key-ip-whitelist 2026-05-20 11:14:43 +08:00
RWDai 84c4c2f9c2 fix(gateway): preserve image generation tools 2026-05-20 11:02:41 +08:00
fawney19 4d856f3deb Merge remote-tracking branch 'origin/pr/516' 2026-05-20 10:54:32 +08:00
fawney19 61bcbe826a fix: tighten OAuth auto cleanup signals 2026-05-20 10:34:22 +08:00
RWDai bdc848b19e Merge upstream main into feat/500-api-key-ip-whitelist 2026-05-20 10:26:56 +08:00
mayrain 65e3b6f3da fix(codex): trigger image override only on explicit tool_choice
The codex `apply_codex_openai_responses_special_body_edits` override
previously triggered whenever the `tools` array contained an
`image_generation` entry, regardless of whether the caller actually
asked to use it. Codex CLI advertises `image_generation` alongside
~20 other tools under `tool_choice: "auto"`, so every routine codex
conversation was being rewritten into image-generation-only form:

  - `model` forced to `gpt-5.4-mini` (CODEX_OPENAI_IMAGE_INTERNAL_MODEL)
  - `stream` forced to `true`
  - `tools` truncated to a single `image_generation` entry
  - `tool_choice` overwritten to `{"type":"image_generation"}`

The upstream ChatGPT codex backend then rejected the request with
`400 Tool choice 'image_generation' not found in 'tools' parameter`,
which the gateway surfaced as a retryable 503 to clients. The bug
reproduced on every codex CLI session that included the image tool
in its tool catalogue, even though the user never requested image
generation.

Narrow the trigger to the caller's actual selection. The new helper
`codex_openai_responses_tool_choice_references_image_generation`
matches only the explicit string `"image_generation"` or the object
form `{"type":"image_generation"}`. The pre-existing
`is_openai_image_request(provider_api_format)` branch still handles
genuine `openai:image` traffic, so true image-generation flows are
unaffected.

Tests:
  - lock the regression: `tool_choice: "auto"` with image_generation
    in tools must not trigger the override (model/tools preserved)
  - lock variants: string `"image_generation"` and object form both
    still trigger; other tool_choice values and an absent
    `tool_choice` do not
2026-05-20 09:52:39 +08:00
zhiqicloud 97b05e744f 支持官方直连支付、退款配置与套餐联动 2026-05-20 08:16:52 +08:00
fawney19 fbda210b84 Merge pull request #511 2026-05-20 01:22:55 +08:00
fawney19 ed75ae6d56 Merge pull request #509 from beilo/feat/key-ranking-usage
Add paginated API key usage leaderboard
2026-05-20 01:06:58 +08:00
fawney19 d1ad1815f7 Merge pull request #513 from mayrainnn/fix/request-body-content-encoding
feat: normalize compressed request bodies
2026-05-20 01:05:59 +08:00
fawney19 b1a3a26815 Merge pull request #512 from Entropy-Xu/codex/fix-wallet-overdraft-settlement
[codex] 修复钱包余额不足后重复消费
2026-05-20 01:05:26 +08:00
fawney19 94760dbc14 refactor(tunnel): rename aether-proxy to aether-tunnel 2026-05-20 01:02:01 +08:00
zhiqicloud 3a318a86b2 支持官方直连支付、退款配置与套餐联动 2026-05-20 00:21:56 +08:00
fawney19 f4d0d5904a Remove image_generation tools from OpenAI image bridges 2026-05-20 00:20:56 +08:00
fawney19 25c7bb935e chore(frontend): simplify concurrent limit hint text 2026-05-19 23:52:59 +08:00
fawney19 f5deed8709 refactor(proxy): improve tunnel throughput and observability 2026-05-19 23:49:36 +08:00
beilo fe3a848eb5 feat(admin): add paginated API key usage leaderboard 2026-05-19 23:17:10 +08:00
mayrain 8f4f4d2d82 refactor(gateway): route decoded body access through ai_serving 2026-05-19 22:58:03 +08:00
mayrain 66a54cc39e feat(gateway): normalize compressed request bodies 2026-05-19 22:57:45 +08:00
Entropy.Xu 7ace958710 fix(wallet): 修复余额不足后重复消费
- 有限钱包结算允许扣成负数,先扣充值余额再扣赠送余额,缺口回写到充值余额
- 日额度钱包补扣路径在存在钱包时不再把余额不足标成 insufficient_quota
- 补充内存和 SQLite 结算回归覆盖,三种数据库实现保持一致
验证:
- cargo fmt --all --check
- cargo clippy -p aether-data --all-targets -- -D warnings
- cargo clippy -p aether-gateway --all-targets -- -D warnings
- cargo clippy --workspace --exclude aether-gateway --exclude aether-data --all-targets -- -D warnings
- cargo nextest run -p aether-gateway
- cargo nextest run -p aether-data
- cargo nextest run --workspace --exclude aether-gateway --exclude aether-data
- cargo test -p aether-data sqlite --lib
- Postgres/MySQL data_db_smoke commands from rust-ci.yml
2026-05-19 19:29:22 +08:00
fawney19 57655bdb25 Hide capability tags from UI 2026-05-19 19:10:30 +08:00
fawney19 124077a0a1 Merge branch 'pr-506' 2026-05-19 18:36:36 +08:00
fawney19 1b570daf72 Revert "Merge PR #504"
This reverts commit d216a9e219, reversing
changes made to 21e82abd54.
2026-05-19 17:23:57 +08:00
fawney19 8bcd5b8189 Revert "Merge remote-tracking branch 'origin/pr-473'"
This reverts commit f2cdb74ed8, reversing
changes made to 3f0fd15395.
2026-05-19 16:31:58 +08:00
fawney19 63202a63ef Merge branch 'codex/pool-hot-trace-fix'
# Conflicts:
#	apps/aether-gateway/src/ai_serving/planner/candidate_materialization.rs
2026-05-19 14:47:37 +08:00
fawney19 7e9ca88e00 fix: restore provider key circuit breaker backoff 2026-05-19 13:57:36 +08:00
beilo 75aa3dc0cc fix: refine oauth auto-removal behavior 2026-05-19 13:41:18 +08:00
fawney19 6a1da6a5ff fix: speed up admin pool loading 2026-05-19 12:05:36 +08:00
mayrain d88f092dd1 feat(kiro): add simulated cache provider toggle 2026-05-19 10:47:17 +08:00
mayrain b4d17a392a feat(kiro): simulate prompt cache usage accounting 2026-05-19 10:47:17 +08:00
fawney19 c6a408d4e8 Fix local gateway Docker native deps 2026-05-19 10:43:04 +08:00
RWDai d19bf71343 Refresh migration cutoff expectations 2026-05-19 10:25:06 +08:00
RWDai 02f09c2056 Regenerate API key schema baselines 2026-05-19 10:24:47 +08:00
RWDai 6a104d5736 Add allowed IPs to data schema sources 2026-05-19 10:24:27 +08:00
RWDai 95af482ffe Update auth snapshot observability fixture 2026-05-19 10:23:54 +08:00
RWDai b05264ff74 Stabilize wallet today usage test timing 2026-05-19 10:23:32 +08:00
RWDai 2aab1ea97b Clean up gateway API key whitelist handlers 2026-05-19 10:23:15 +08:00
RWDai f1687017e6 Fix provider endpoint format normalization path 2026-05-19 10:22:50 +08:00
fawney19 052de6b96e test: stabilize merged PR checks 2026-05-19 10:18:25 +08:00
fawney19 d2b42e91d2 test: align cancelled sync billing status 2026-05-19 08:25:39 +08:00
fawney19 d8a9d7eb5e chore: format merged PR changes 2026-05-19 08:13:56 +08:00
fawney19 d216a9e219 Merge PR #504 2026-05-19 08:10:57 +08:00
fawney19 21e82abd54 Merge PR #502 2026-05-19 08:10:48 +08:00
fawney19 18ed9a57c2 Merge PR #499 2026-05-19 08:10:37 +08:00
fawney19 702dc3ceb4 Merge PR #489 (ours: keep current main) 2026-05-19 03:38:26 +08:00
fawney19 3180ca2bf9 Merge PR #488 (ours: keep current main) 2026-05-19 03:38:22 +08:00
fawney19 69af74b1e0 Merge PR #488 and #489 2026-05-19 03:16:35 +08:00
MMEXA ef9c0ebbd4 Merge latest origin/main into codex/gemini-embedding-batch
# Conflicts:
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/mod.rs
#	apps/aether-gateway/src/execution_runtime/fallback.rs
2026-05-18 19:11:42 +00:00
MMEXA ca4d0dc819 Merge origin/main into codex/gemini-embedding-batch
# Conflicts:
#	apps/aether-gateway/src/ai_serving/api.rs
#	apps/aether-gateway/src/ai_serving/planner/passthrough/provider/family/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/family/request.rs
#	apps/aether-gateway/src/ai_serving/transport.rs
#	apps/aether-gateway/src/handlers/admin/provider/query/models/model_test/summary.rs
#	apps/aether-gateway/src/handlers/admin/provider/query/models/model_test/tests.rs
#	crates/aether-data/src/repository/candidate_selection/postgres.rs
#	crates/aether-model-fetch/src/strategy.rs
2026-05-18 19:02:19 +00:00
fawney19 cd1aa92931 Merge branch 'merge-pr-483' 2026-05-19 02:28:43 +08:00
fawney19 9fc9c334c9 Merge remote-tracking branch 'origin/pr/487'
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
#	crates/aether-data/src/repository/oauth_providers/postgres.rs
#	crates/aether-data/src/repository/oauth_providers/sqlite.rs
#	frontend/src/views/admin/OAuthSettings.vue
2026-05-19 02:27:39 +08:00
fawney19 c563c192b7 Merge remote-tracking branch 'origin/pr-483' into merge-pr-483
# Conflicts:
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/chat/decision/payload.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/chat/decision/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/responses/decision/payload.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/responses/decision/request.rs
#	apps/aether-gateway/src/execution_runtime/chatgpt_web_image.rs
2026-05-19 02:23:14 +08:00
fawney19 afedd90c80 Merge commit 'refs/pull/481/head' of github-fawney19:fawney19/Aether
# Conflicts:
#	apps/aether-gateway/src/ai_serving/api.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/family/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/chat/decision/request.rs
#	apps/aether-gateway/src/ai_serving/planner/standard/openai/responses/decision/request.rs
2026-05-19 01:46:41 +08:00
MMEXA be2e8e594c fix(frontend): align Vertex Gemini endpoint controls 2026-05-18 17:36:04 +00:00
fawney19 d392681c58 Merge branch 'pr-478'
# Conflicts:
#	apps/aether-gateway/src/data/state/mod.rs
#	apps/aether-gateway/src/handlers/admin/mod.rs
#	apps/aether-gateway/src/handlers/admin/routes.rs
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
#	crates/aether-data/src/repository/announcements/postgres.rs
#	frontend/src/features/auth/components/RegisterDialog.vue
2026-05-19 01:27:42 +08:00
MMEXA 5ed8325592 fix(gateway): use Vertex model garden catalog endpoint 2026-05-18 17:06:42 +00:00
fawney19 ed1d9fdb57 Fix postgres bigint counter decoding 2026-05-19 00:58:09 +08:00
ZheFox c58ce63fc3 Merge branch 'fawney19:main' into main 2026-05-19 00:50:52 +08:00
fawney19 5eed329916 Rootfix usage counter outbox 2026-05-19 00:44:37 +08:00
fawney19 19c8688eb1 Merge pull request #477 from RWDai/feat/356-usage-record-columns
Add configurable usage record columns
2026-05-19 00:35:30 +08:00
MMEXA 4317ff78b1 Expose local scheduling failures in usage UI 2026-05-18 16:28:37 +00:00
yangrsandClaude Opus 4.7 6c16f399d4 feat: 重要通知模块、Server 酱独立配置与额度提醒
- 新增重要通知统一模块(邮件 + Server 酱)作为后台任务通知出口
- 拆出独立的 Server 酱 配置页(SendKey + Markdown 模板,支持 {title}/{body} 变量替换),通过仪表盘内置工具入口进入
- 新增提供商额度提醒后台 worker:余额低于阈值时通过重要通知推送,提供商配置页加入额度提醒开关与阈值
- 重要通知页加入配置可用性守卫:未配置任一通道时禁用总开关,未配置邮件/SendKey 时禁用对应通道开关
- 测试通知端点支持 channel 过滤(all/email/server_chan),并绕过总开关与通道开关,便于配置阶段先验证通道
- 修复:测试通知路由未在 buffered-body 白名单导致 channel 参数丢失、测试时邮件分支被误触发
- 修复:sub2api 验证响应中 username 为 null 时正确回退到 email,避免误报"验证响应缺少: 用户信息"

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-18 23:51:14 +08:00
MMEXA b480f3aaff fix(gateway): normalize Gemini Vertex embedding transport 2026-05-18 15:39:29 +00:00
MMEXA 84f312fa4c fix(gateway): close dropped sync attempts 2026-05-18 14:15:22 +00:00
ZheFox 61d5fdb0ec fix(frontend): preserve fixed provider model test key inheritance 2026-05-18 22:06:27 +08:00
ZheFox 995ab302be Merge remote-tracking branch 'upstream/main'
# Conflicts:
#	apps/aether-gateway/src/handlers/admin/provider/endpoints_admin/payloads.rs
#	apps/aether-gateway/src/handlers/admin/provider/endpoints_admin/reads.rs
#	apps/aether-gateway/src/handlers/admin/provider/endpoints_admin/update.rs
#	apps/aether-gateway/src/tests/control/admin/endpoints/routes.rs
#	frontend/src/features/models/components/GlobalModelFormDialog.vue
#	frontend/src/features/providers/components/ProviderModelFormDialog.vue
#	frontend/src/features/providers/components/provider-tabs/__tests__/model-test-request.spec.ts
#	frontend/src/features/providers/components/provider-tabs/model-test-request.ts
2026-05-18 22:02:31 +08:00
RWDai c6ee558180 Add admin user key IP whitelist UI 2026-05-18 20:48:28 +08:00
RWDai 460cd63d3a Add user API key IP whitelist UI 2026-05-18 20:48:21 +08:00
RWDai 2a3593d9c5 Update planner auth snapshot fixtures 2026-05-18 20:48:14 +08:00
RWDai bcca8d295a Update auth context test fixtures 2026-05-18 20:48:05 +08:00
RWDai d8f68c1d9a Preserve allowed IP defaults in admin key imports 2026-05-18 20:47:57 +08:00
RWDai ab53326865 Support allowed IPs in admin user key endpoints 2026-05-18 20:47:46 +08:00
RWDai 96b857f642 Support allowed IPs in user API key endpoints 2026-05-18 20:47:37 +08:00
RWDai fc12cc8a36 Enforce API key IP restrictions in proxy auth 2026-05-18 20:47:28 +08:00
RWDai 276d19b63c Persist allowed IPs in auth repositories 2026-05-18 20:47:17 +08:00
RWDai 437024cdb1 Add API key allowed IP data types 2026-05-18 20:47:03 +08:00
RWDai 64b41434ce Add API key allowed IP migrations 2026-05-18 20:46:53 +08:00
RWDai 24129d7f12 Merge upstream/main into feat/356-usage-record-columns 2026-05-18 20:33:05 +08:00
fawney19 d51b44d642 Merge remote-tracking branch 'origin/pr-485'
# Conflicts:
#	crates/aether-data/src/repository/provider_catalog/sqlite.rs
2026-05-18 19:44:18 +08:00
RWDai e8c55e8f1f Label OpenAI JS SDK in cache monitoring 2026-05-18 19:30:39 +08:00
RWDai 9179516b19 Label OpenAI JS SDK in usage records 2026-05-18 19:30:19 +08:00
RWDai 37abfe66f0 Label OpenAI JS SDK in user usage 2026-05-18 19:29:59 +08:00
RWDai ae9d4038b1 Prefer typed admin usage client family 2026-05-18 19:29:36 +08:00
RWDai b6f558d10b Project usage client family in Postgres reads 2026-05-18 19:29:10 +08:00
RWDai 6d994917a0 Add usage client family read model field 2026-05-18 19:28:36 +08:00
fawney19 b99f43783c Merge remote-tracking branch 'origin/pr-482'
# Conflicts:
#	.github/workflows/release.yml
2026-05-18 18:55:11 +08:00
fawney19 7f76eff827 Merge remote-tracking branch 'origin/pr-484' 2026-05-18 18:01:57 +08:00
fawney19 be939f7e63 Merge remote-tracking branch 'origin/pr-493' 2026-05-18 18:01:43 +08:00
fawney19 a8a87d2b41 Merge remote-tracking branch 'origin/pr-494' 2026-05-18 18:00:47 +08:00
RWDai 0a26accca4 Merge remote-tracking branch 'upstream/main' into feat/356-usage-record-columns 2026-05-18 17:49:35 +08:00
fawney19 40e925680c Merge remote-tracking branch 'origin/pr-480' 2026-05-18 16:46:35 +08:00
fawney19 f2cdb74ed8 Merge remote-tracking branch 'origin/pr-473'
# Conflicts:
#	frontend/src/features/providers/components/ProviderModelFormDialog.vue
2026-05-18 16:46:22 +08:00
Codex 7ac8159728 fix: use Vertex Model Garden models endpoint 2026-05-18 08:17:09 +00:00
fawney19 3f0fd15395 Merge remote-tracking branch 'origin/pr-491' 2026-05-18 16:14:07 +08:00
fawney19 90dbc279fd Merge remote-tracking branch 'origin/pr-476' 2026-05-18 16:13:45 +08:00
fawney19 3723f165bc Merge remote-tracking branch 'origin/pr-496' 2026-05-18 16:13:03 +08:00
fawney19 3bffecddf2 fix: route access log sanitizer through gateway api 2026-05-18 15:47:02 +08:00
fawney19 a818af7833 Merge remote-tracking branch 'origin/pr-492' 2026-05-18 14:52:29 +08:00
fawney19 187b3a08fb Merge remote-tracking branch 'origin/pr-495' 2026-05-18 14:51:50 +08:00
fawney19 2405ccc0f2 Merge remote-tracking branch 'origin/pr-490' 2026-05-18 14:51:14 +08:00
fawney19 0fcfcae9c8 Merge remote-tracking branch 'origin/pr-486' 2026-05-18 14:50:48 +08:00
ZheFox 2de0cbbafc Merge branch 'fawney19:main' into main 2026-05-18 13:13:37 +08:00
ZheFox a4012ad353 Merge upstream/main 2026-05-18 13:11:11 +08:00
fawney19 e4315fbbf0 fix: refine frontend admin and auth UI 2026-05-18 12:41:57 +08:00
ZheFox a10c02ef63 feat(billing): add image output range pricing support 2026-05-18 11:52:01 +08:00
MMEXA 66f154a251 fix(gateway): avoid low OpenAI-compatible test token cap 2026-05-18 03:06:46 +00:00
fawney19 92813e6122 feat: add routing profile scheduling policies 2026-05-18 11:03:49 +08:00
MMEXA f50f26e599 fix(gateway): cover Google OpenAI-compatible roots 2026-05-18 02:15:23 +00:00
ZheFox a3094fda53 fix(gateway): ignore OpenAI tools for image intent routing 2026-05-18 09:54:10 +08:00
MMEXA b004a02e4a fix(gateway): harden Gemini endpoint routing 2026-05-18 00:53:34 +00:00
MMEXA 9586f5158e Fix fixed-provider endpoint key counts 2026-05-17 20:55:36 +00:00
ZheFox f5ace4fd6d fix(frontend): stabilize image generation overrides in model forms 2026-05-18 03:39:55 +08:00
ZheFox a05ae94cea fix(frontend): update checkbox bindings to checked events 2026-05-18 03:27:31 +08:00
ZheFox dff17b6cb1 feat(billing): support image output pricing in model forms and details 2026-05-18 03:19:58 +08:00
ZheFox 0b2a8fafce feat(billing): add image output pricing and usage tracking 2026-05-18 02:49:56 +08:00
ZheFox 691ccaaa04 fix(usage): track OpenAI image SSE completion and usage estimates 2026-05-18 01:32:45 +08:00
RWDai 26900c8c9e Show only client family in usage client column 2026-05-18 00:46:41 +08:00
MMEXA 226ce45d0d fix: pass explicit local build version 2026-05-17 16:38:22 +00:00
Kayphoon ae472d6744 fix(data): require provider id for provider usage aggregation 2026-05-18 00:36:29 +08:00
MMEXA 89d08d9953 fix: align provider model fetch state 2026-05-17 16:17:47 +00:00
ZheFox c024c782e4 feat(billing): add image quality pricing and usage tracking 2026-05-18 00:11:30 +08:00
MMEXA 84fc1e35e2 fix(frontend): support login autofill and reliable redirect 2026-05-17 16:03:57 +00:00
MMEXA 995be3781a fix(frontend): align usage APIs with Rust routes 2026-05-17 15:37:52 +00:00
MMEXA ed570155a1 fix(gateway): redact credential query values in access logs 2026-05-17 15:26:56 +00:00
ZheFox 680b617b00 feat(gateway): route OpenAI image streams through chat bridge 2026-05-17 23:09:50 +08:00
RWDai 0a62e4bc77 Remove usage request path column 2026-05-17 23:05:02 +08:00
RWDai 96d40dd21d Infer usage client family from User-Agent 2026-05-17 23:04:31 +08:00
MMEXA ff83b54c3b fix(gateway): reject empty Gemini success responses 2026-05-17 14:55:33 +00:00
MMEXA 81ff375bfd fix(gateway): route Gemini embedding batches correctly 2026-05-17 14:24:32 +00:00
dalamudx b0fc6e68ef fix: add icon_url to mysql generated baseline 2026-05-17 22:00:28 +08:00
dalamudx e1a73be2e1 fix: update schema baselines, logical schema, and snapshot cutoff for icon_url 2026-05-17 21:52:36 +08:00
dalamudx cf2c74e8e8 fix: add new migration version to test whitelist 2026-05-17 21:42:56 +08:00
dalamudx 2cb01f7a69 fix: add missing icon_url arg in test helper 2026-05-17 21:40:54 +08:00
dalamudx 48deff15c4 fix(oauth): fix login failures and add provider icon_url config
- Fix FIND_OAUTH_LINKED_USER_SQL missing allowed_providers_mode columns
- Fix TOUCH_OAUTH_LINK_SQL json/jsonb type mismatch in COALESCE
- Add icon_url field to OAuth provider config (DB, API, frontend)
- Fix admin OAuth test: accept 404 as reachable, use system proxy
2026-05-17 21:33:02 +08:00
ZheFox d6c8c14de7 feat(gateway): route OpenAI image intents through image bridge 2026-05-17 21:19:17 +08:00
RWDai b2266b588e Preserve usage origin metadata in Postgres lists 2026-05-17 21:09:59 +08:00
RWDai fcaafb3939 chore(ci): retrigger flaky sqlite smoke 2026-05-17 20:38:55 +08:00
fawney19 7d569127ae Fix active probe pool fallback tracing 2026-05-17 20:34:06 +08:00
RWDai 981020a5ab fix(ci): apply rustfmt to usage metadata handlers 2026-05-17 20:28:08 +08:00
ZheFox d9c8119bda fix(routing): match provider model names in admin routing counts 2026-05-17 18:59:41 +08:00
ZheFox 485d166912 fix(provider): count inherited endpoint formats for model tests 2026-05-17 18:04:16 +08:00
Kayphoon 5060532c51 fix: package release sqlite compose template 2026-05-17 16:33:35 +08:00
Kayphoon f29cca72ba refactor(data): limit query abstraction to postgres and sqlite 2026-05-17 15:40:44 +08:00
Kayphoon 77640d51a6 refactor(data): add select query abstraction 2026-05-17 15:04:35 +08:00
HsungKayphoon f0a6fffa87 refactor(data): introduce simple query helper 2026-05-17 14:07:23 +08:00
Entropy.Xu 1b24e1c22a fix(wallet): 修复额度耗尽后仍可消费 2026-05-17 11:49:08 +08:00
ZheFox ab0d766f47 fix(usage): stop inferring cache reads from prompt_cache_key 2026-05-17 03:04:33 +08:00
ZheFox 41ffb18604 fix(billing): avoid double counting cache read in OpenAI cache hit context 2026-05-17 02:40:01 +08:00
ZheFox b903f8ff7d fix(usage): estimate cache read tokens for cancelled requests 2026-05-17 02:13:13 +08:00
ZheFox 0483d001b4 fix(usage): bill cancelled terminal usage and preserve total token estimates 2026-05-17 01:11:42 +08:00
HsungKayphoon d290a1fdb9 fix: satisfy rust 1.95 clippy 2026-05-17 00:00:46 +08:00
ZheFox 2803e9317d fix(usage): bill cancelled terminal usage and preserve terminal telemetry 2026-05-16 23:43:40 +08:00
HsungKayphoon c5e26a1ed6 fix: align sqlite repositories with postgres behavior 2026-05-16 23:43:40 +08:00
fawney19 a2f91b4108 Cancel upstream stream on downstream disconnect 2026-05-16 22:04:11 +08:00
fawney19 664bd98056 Merge pull request #479 from Entropy-Xu/codex/fix-balance-cost-estimate
fix(gateway): 修复额度预检误判余额不足
2026-05-16 21:23:23 +08:00
mayrain 5bf236957e feat(grok): add runtime image surfaces 2026-05-16 21:15:38 +08:00
mayrain 936e1ae37b feat(grok): add admin oauth and quota support 2026-05-16 21:15:38 +08:00
mayrain cbfe1d378f feat(grok): add provider pool and transport support 2026-05-16 21:15:38 +08:00
mayrain e5f1f52759 feat(model-test): wire image previews into provider tests 2026-05-16 21:15:27 +08:00
mayrain edacc5a7d0 feat(model-test): add image-aware request helpers 2026-05-16 21:15:27 +08:00
fawney19 ed9267562b Adjust provider detail key pagination 2026-05-16 21:08:25 +08:00
Entropy.Xu bddae47454 fix(gateway): 修复额度预检误判余额不足 2026-05-16 20:45:40 +08:00
fawney19 3a5922d4ee fix: use codex quota refresh for account checks 2026-05-16 20:16:23 +08:00
fawney19 56994d4c29 fix(frontend): relax system import limits 2026-05-16 19:00:20 +08:00
Entropy.Xu 973eb1a614 feat(referrals): 添加邀请返利和注册确认功能 2026-05-16 17:41:52 +08:00
HsungKayphoon f9f1fa928a refactor: drive pg sqlite copy from target schema 2026-05-16 17:29:14 +08:00
fawney19 328ac721ce Merge pull request #472 from Kayphoon/codex/manual-usage-cleanup-origin-main
feat: add scoped manual usage cleanup
2026-05-16 16:25:40 +08:00
HsungKayphoon 527feb69db fix: cover portable migration tables in logical schema 2026-05-16 15:58:41 +08:00
RWDai ba661f1b3c Add usage record column controls 2026-05-16 15:55:38 +08:00
RWDai 4f584a71df Add usage metadata frontend plumbing 2026-05-16 15:55:38 +08:00
RWDai 97cd92a1a2 Expose usage record metadata fields 2026-05-16 15:55:38 +08:00
RWDai df7b2824a6 Persist client family usage metadata 2026-05-16 15:55:38 +08:00
RWDai 03ba1f94d7 Show execution failure reasons in request details 2026-05-16 15:42:13 +08:00
RWDai 6dd5d2fe14 Add failure notice resolver for usage records 2026-05-16 15:42:13 +08:00
RWDai a2649718ea Expose scheduling failure details in usage payload 2026-05-16 15:42:13 +08:00
RWDai 9325c2ad9d fix(providers): remove unreachable manual model add flow 2026-05-16 15:35:26 +08:00
RWDai 590151f40b feat(models): allow manual global model creation 2026-05-16 15:27:34 +08:00
HsungKayphoon 4b12ec8913 feat: add postgres to single-node migration 2026-05-16 15:26:43 +08:00
HsungKayphoon 74a1e5ad7d feat: add scoped manual usage cleanup 2026-05-16 15:23:44 +08:00
fawney19 75b7319465 Merge pull request #471 from Kayphoon/fix/usage-provider-id-null 2026-05-16 15:19:30 +08:00
fawney19 6a608b8e3f fix: recover finalized usage provider links safely 2026-05-16 15:15:57 +08:00
beilo 2ca4b486ec fix: auto-remove invalid oauth pool keys 2026-05-16 15:08:16 +08:00
fawney19 c2cdcefdc8 fix(public): 恢复 support.rs 的 system_config_bool 引入 2026-05-16 14:28:19 +08:00
fawney19 893ac18d60 Merge PR #469: 修复用户可见性、额度、验证与 Codex 探测 2026-05-16 14:28:19 +08:00
fawney19 74abb50bdc fix(public): keep original GitHub links visible
(cherry picked from commit 7c93552697c9c35b77df35e117900ff8e9b62994)
2026-05-16 13:50:46 +08:00
Kayphoon f817f856c8 fix: allow upsert to backfill NULL provider link fields after billing finalizes
The ON CONFLICT update guard for provider_id, provider_endpoint_id, and
provider_api_key_id previously required billing_status = 'pending'. Once a
usage row left pending state with these fields still NULL, subsequent upserts
could never fill them. Add an OR IS NULL clause so missing provider links are
always recoverable regardless of billing status.
2026-05-16 13:11:59 +08:00
fawney19 3a23eaa572 Merge pull request #470 from yao177/fix/provider-keep-priority-on-conversion-load-balance
fix: preserve conversion priority beneath scheduler modes
2026-05-16 12:56:57 +08:00
fawney19 a7fdce493b feat: paginate provider keys from backend 2026-05-16 12:52:49 +08:00
yao177 232976c14a fix: keep conversion priority below load balance 2026-05-16 04:41:07 +00:00
yao177 dc1009798d test: cover cache affinity conversion priority ordering 2026-05-16 04:32:56 +00:00
fawney19 b6d74249a4 fix(public): keep original GitHub links visible
(cherry picked from commit 7c93552697c9c35b77df35e117900ff8e9b62994)
2026-05-16 11:38:49 +08:00
fawney19 f72ab383c9 Refine OAuth auto cleanup conditions
(cherry picked from commit 78826eae47ab18ace6931dd190a41070416b5e10)
2026-05-16 11:33:45 +08:00
fawney19 f59cf1090d feat(admin): unify system data management and aggregate import/export 2026-05-16 02:16:53 +08:00
Entropy.Xu 1a50c5e112 merge: 同步主线并解决用户侧验证冲突 2026-05-16 01:25:42 +08:00
fawney19 48da062251 Merge pull request #468 from fawney19/revert-pr-466
Revert PR #466
2026-05-16 01:01:44 +08:00
Entropy.Xu bbd3c30b0e fix(public): 修复用户可见性、额度、验证与 Codex 探测 2026-05-16 00:51:44 +08:00
fawney19 d6c320bf06 Revert "Merge remote-tracking branch 'origin/pr/466'"
This reverts commit 0e0a24862f, reversing
changes made to b09fd48eee.
2026-05-16 00:50:00 +08:00
fawney19 d53546d56f Make pool probing request-driven 2026-05-16 00:18:47 +08:00
fawney19 43d891bee1 Remove legacy Python tests 2026-05-16 00:06:21 +08:00
fawney19 0e0a24862f Merge remote-tracking branch 'origin/pr/466' 2026-05-15 22:49:35 +08:00
ZheFox 2345df0e38 fix(usage): bill cancelled terminal usage 2026-05-15 22:38:34 +08:00
fawney19 b09fd48eee Merge remote-tracking branch 'origin/pr/462' 2026-05-15 22:28:40 +08:00
fawney19 c916e76bd2 Merge branch 'pr-461'
# Conflicts:
#	.github/workflows/rust-ci.yml
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
2026-05-15 21:52:37 +08:00
fawney19 8eb4c029b2 Merge pull request #467 from AAEE86/main
fix(usage): prevent detail view from overriding active request status
2026-05-15 21:08:48 +08:00
fawney19 87e44479cc Add proxy tunnel diagnostics and default logging 2026-05-15 19:43:59 +08:00
AAEE86 9c7757f801 fix(usage): prevent detail view from overriding active request status
- Keep pending/streaming lifecycle status authoritative for active requests
- Prevent detail status code or trace state from misclassifying in-flight requests as stream/failed
- Add regression coverage for status resolution and timeline state emission
2026-05-15 19:04:50 +08:00
fawney19 1503986d40 Fix compose installer defaults 2026-05-15 18:43:47 +08:00
fawney19 0bd3e2fa88 Merge remote-tracking branch 'origin/main' 2026-05-15 18:13:26 +08:00
fawney19 0f0b9a6118 chore: add db maintenance make targets 2026-05-15 18:10:49 +08:00
fawney19 e4f427f921 Merge remote-tracking branch 'entropy-xu/payment-billing-plans'
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
#	frontend/src/views/admin/Users.vue
2026-05-15 16:37:17 +08:00
fawney19 2006a3e678 Merge pull request #464 from RWDai/fix/issue-455-codex-compact-include
fix: strip include from codex compact requests
2026-05-15 16:10:33 +08:00
fawney19 38240f77ff chore: replace dev script with make targets 2026-05-15 16:09:11 +08:00
RWDai 6014925e60 style: format codex compact regression tests 2026-05-15 14:40:27 +08:00
RWDai 1dc9b505e4 fix: strip include from codex compact requests 2026-05-15 14:20:16 +08:00
fawney19 b0ba1f250d Merge pull request #463 from AAEE86/main
chore(deps): update npm lockfiles
2026-05-15 14:13:14 +08:00
fawney19 2e9feaa60a Add local dev.sh script 2026-05-15 13:51:52 +08:00
AAEE86 8538364930 chore(deps): update npm lockfiles 2026-05-15 13:46:32 +08:00
fawney19 fb2662b877 Merge pull request #460 from RWDai/manual-provider-model-save-20260515
Allow manual provider model save without online discovery
2026-05-15 13:33:32 +08:00
fawney19 170218bb26 Merge pull request #458 from RWDai/opencode/sunny-orchid
Add one-click proxy node installation
2026-05-15 12:49:08 +08:00
fawney19 582504d11a Allow deploy to pull missing images 2026-05-15 11:29:16 +08:00
RWDai 88d8a8b79f Pin Rust CI component installs to 1.95.0 2026-05-15 10:14:50 +08:00
RWDai 8e4fc40be5 Refresh Rust CI for audit admin enum fix 2026-05-15 10:02:01 +08:00
fawney19 01a982bdf6 fix: preserve codex include fields 2026-05-15 03:25:06 +08:00
fawney19 daf33a82a6 deploy: restore local build and sqlite compose 2026-05-15 03:17:14 +08:00
fawney19 cc5a44e373 Merge branch 'pr-453' 2026-05-15 02:25:39 +08:00
fawney19 8e0f8003a9 fix: simplify account self-check config 2026-05-15 02:21:11 +08:00
fawney19 54a8312e46 feat: add adaptive pool metrics and self-check 2026-05-15 02:21:11 +08:00
Entropy.Xu 85f48123b6 feat(auth): 添加 Turnstile 注册防护 2026-05-15 01:54:20 +08:00
RWDai fdea51c7b1 Simplify proxy install command copy 2026-05-15 01:21:31 +08:00
RWDai 870bb19798 Update proxy installer branch references 2026-05-15 01:21:08 +08:00
RWDai dbec85344d Fix Rust CI toolchain setup 2026-05-15 01:20:36 +08:00
RWDai 4db01244ec Update Postgres bootstrap for audit admin role 2026-05-15 01:20:29 +08:00
RWDai 6021110fb2 Add audit admin Postgres userrole migration 2026-05-15 01:20:29 +08:00
RWDai 1216fa940e Allow manual provider model save without online discovery 2026-05-15 00:51:30 +08:00
RWDai bcf4adf944 Merge branch 'main' into opencode/sunny-orchid
# Conflicts:
#	README.md
2026-05-14 20:22:36 +08:00
RWDai 07ea745f56 Document proxy one-click installation 2026-05-14 19:35:41 +08:00
RWDai 3e122c84ef Add proxy node script install UI 2026-05-14 19:35:16 +08:00
RWDai 98a54b4633 Add proxy install session client API 2026-05-14 19:34:56 +08:00
RWDai 2db85b1c53 Add proxy one-click installer scripts 2026-05-14 19:34:33 +08:00
RWDai 91545cf906 Add admin proxy install session creation 2026-05-14 19:34:09 +08:00
RWDai 5a15822ce0 Add public proxy install session delivery 2026-05-14 19:33:47 +08:00
ZheFox eef21b6c34 Merge branch 'fawney19:main' into main 2026-05-14 17:47:47 +08:00
zhefox 498b3b1226 fix(auth): correct postgres api key create field ordering 2026-05-14 17:41:13 +08:00
zhefox 3b77686cee fix(auth): cast standalone api key expires_at update to timestamptz 2026-05-14 16:28:38 +08:00
fawney19 bf511f9f8c Merge pull request #449 from RWDai/feat/audit-admin-readonly
Add read-only audit administrator role
2026-05-14 15:51:27 +08:00
ZheFox a0eed2cc51 fix: split Codex chat and responses defaults 2026-05-14 15:33:03 +08:00
fawney19 e61aa46dad deploy: limit installer to supported modes 2026-05-14 15:26:58 +08:00
fawney19 d2831ec6f0 deploy: remove local build compose path 2026-05-14 14:40:25 +08:00
fawney19 a7e71624e3 ci: align release channels and speed rust checks 2026-05-14 14:09:27 +08:00
RWDai bc0017a1b4 Use audit admin permissions in proxy auth 2026-05-14 13:39:57 +08:00
RWDai 4cb7b2d494 Add audit admin internal read permissions 2026-05-14 13:39:57 +08:00
RWDai eb7c8a3ad5 Format audit admin role helpers 2026-05-14 13:39:57 +08:00
RWDai 6e2f90aba8 Keep audit admins on read-only admin views 2026-05-14 13:39:57 +08:00
RWDai 9c59c0e1a2 Add audit role to user management UI 2026-05-14 13:39:57 +08:00
RWDai 6a9a54cad6 Show audit admin labels in shared layouts 2026-05-14 13:39:57 +08:00
RWDai e768145961 Update public auth redirects for audit admins 2026-05-14 13:38:45 +08:00
RWDai a4e040a7d7 Route audit admins into admin console 2026-05-14 13:38:45 +08:00
RWDai e818443841 Add frontend audit admin auth semantics 2026-05-14 13:38:45 +08:00
RWDai 337d0af136 Accept audit role in user administration 2026-05-14 13:38:45 +08:00
RWDai 4d2667764f Enforce read-only admin route permissions 2026-05-14 13:38:45 +08:00
RWDai feb676b66f Allow audit admins through backend admin auth 2026-05-14 13:38:45 +08:00
RWDai 14871c2255 Add audit administrator role helpers 2026-05-14 13:38:45 +08:00
fawney19 48fe52b207 Update guide setup commands 2026-05-14 13:02:25 +08:00
fawney19 509bd30252 Redesign sensitive info protection settings 2026-05-14 11:14:20 +08:00
fawney19 91955ad1e0 Merge remote-tracking branch 'origin/pr/451' into aether-rust-pioneer 2026-05-14 02:10:37 +08:00
fawney19 b41a4a000f Merge remote-tracking branch 'origin/pr/435' into aether-rust-pioneer 2026-05-14 02:02:26 +08:00
fawney19 d29d1cf63b Remove deprecated Python source tree 2026-05-14 01:58:05 +08:00
fawney19 1f8ff7f6d2 Fix PR 434 check failures 2026-05-14 01:51:23 +08:00
fawney19 e251a63cb3 Merge remote-tracking branch 'pr-434/fix-provider-model-test-compat' into aether-rust-pioneer 2026-05-14 01:28:33 +08:00
fawney19 eb654c2fe7 Merge pull request #441 from zhefox/aether-rust-pioneer
fix: add codex reasoning defaults and stream rewrite tests
2026-05-14 00:35:45 +08:00
fawney19 697b6e0653 fix: stabilize openai responses reasoning streams 2026-05-14 00:27:10 +08:00
fawney19 acd44328d6 feat: add version update flow 2026-05-13 22:28:05 +08:00
fawney19 8b813f645f Merge branch 'pr-438' into aether-rust-pioneer 2026-05-13 22:28:05 +08:00
zhefox 2d354fa294 fix: preserve passthrough for same-format stream rewrites 2026-05-13 19:42:22 +08:00
fawney19 cfb22d3f06 Merge pull request #439 from Kayphoon/feat/macos-one-click-install
feat(compose): add solo sqlite compose deployment
2026-05-13 19:03:49 +08:00
fawney19 1a196c8cf2 Merge pull request #443 from AAEE86/rust
feat(admin): add Done-hub provider ops template
2026-05-13 18:42:28 +08:00
fawney19 f847a71747 Merge pull request #448 from RWDai/fix/disable-hidden-user-policies
Disable hidden per-user policy fields
2026-05-13 18:40:46 +08:00
RWDai 87c0a915b8 Align rate limit monitoring test with group policy 2026-05-13 18:27:56 +08:00
Kayphoon 2958041dc7 feat(gateway): add reversible chat pii redaction 2026-05-13 18:25:13 +08:00
fawney19 5d1460e051 refactor: extract provider pool abstractions 2026-05-13 18:19:15 +08:00
RWDai 6a9017bfce Cover disabled user policy API behavior 2026-05-13 17:56:18 +08:00
RWDai a1b0db38f5 Reject hidden user policy payloads 2026-05-13 17:56:18 +08:00
RWDai a99546b390 Disable user policy during auth resolution 2026-05-13 17:56:17 +08:00
zhefox 1adbf23be4 feat(ai-formats): add reasoning summary boundaries for streams 2026-05-13 17:41:34 +08:00
AAEE86 afbb656510 feat(admin): add Done-hub provider ops template
- 新增 Done-hub Cookie 认证架构
- 使用 /api/user/profile 查询余额,按 quota / 500000 换算
- 补充余额解析、认证头和校验相关测试
2026-05-13 16:10:59 +08:00
zhefox 1726df1169 fix: add codex reasoning defaults and stream rewrite tests 2026-05-13 14:34:16 +08:00
Kayphoon d69d862034 feat(compose): add solo sqlite compose deployment 2026-05-13 11:21:35 +08:00
RWDai a03cab4a09 Show update status on admin dashboard 2026-05-13 10:28:18 +08:00
RWDai c90ed14a24 Fetch Aether releases for update checks 2026-05-13 10:27:45 +08:00
RWDai e00df5f7a2 Add admin update payload builder 2026-05-13 10:27:21 +08:00
fawney19 3c2497f019 Fix admin pool sorting and OAuth refresh 2026-05-13 09:16:52 +08:00
Entropy.Xu 3fb007a56d Merge remote-tracking branch 'origin/aether-rust-pioneer' into payment-billing-plans
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
2026-05-13 01:29:18 +08:00
Entropy.Xu 10285c5eb9 feat: add payment gateway and billing plans 2026-05-13 01:18:38 +08:00
Kayphoon 15800d7a80 feat(admin): manual request-records cleanup with typed confirmation 2026-05-13 00:36:43 +08:00
fawney19 4387a9cdd5 Sync Cargo.lock for release build 2026-05-13 00:07:55 +08:00
fawney19 8714d93d4b Stabilize Codex OAuth import tests 2026-05-12 23:36:43 +08:00
fawney19 68256ece2d Fix Kiro manual OAuth refresh test stack 2026-05-12 23:09:09 +08:00
fawney19 339808d55b 更新aether-proxy版本号 2026-05-12 22:46:49 +08:00
fawney19 e7471f44b0 fix provider oauth endpoint reconciliation 2026-05-12 22:46:20 +08:00
fawney19 d1a47c068e feat: update gateway pool and usage flows 2026-05-12 21:05:11 +08:00
mayrain 43c476d54a fix: refine provider model test dialog 2026-05-12 20:11:37 +08:00
mayrain 9f26383de5 fix: route provider model tests through candidates 2026-05-12 20:11:16 +08:00
mayrain 8f082674d7 fix: align embedding provider request formats 2026-05-12 20:10:52 +08:00
mayrain fca3f24d91 fix: preserve legacy admin config imports 2026-05-12 20:10:27 +08:00
fawney19 38012c62ff Allow full management tokens to access token management 2026-05-12 19:53:31 +08:00
fawney19 63149fe281 Add tunnel overload protection 2026-05-12 18:36:41 +08:00
fawney19 5509f70ad4 Bump proxy version to 0.3.9 2026-05-12 18:36:41 +08:00
github-actions[bot] 110cb147d1 chore(proxy): update download links for proxy-v0.3.9 2026-05-12 09:04:17 +00:00
fawney19 92e4066977 Merge pull request #432 from RWDai/fix/cli-install-copy
Add admin API key CLI install sessions
2026-05-12 16:50:24 +08:00
fawney19 ba5e802174 Merge pull request #433 from Kayphoon/feat/macos-one-click-install
feat(install): add macOS native one-click install with launchd
2026-05-12 16:49:50 +08:00
fawney19 4b2507b155 Fix score popover duplication 2026-05-12 16:04:00 +08:00
Kayphoon 6d2fcf12cd feat(install): add macOS native one-click install with launchd
- Release workflow builds macos-amd64/macos-arm64 tarballs on native runners
- install.sh detects Darwin, downloads macos-* assets, installs LaunchDaemon
- Dedicated _aether service account (dscl), env root:_aether 0640
- Launchd stdout/stderr in /var/log/aether (root-owned dir, service-writable files)
- Wrapper script parses env literally without shell expansion
- Auto-prune old releases (default keep 3)
- README documents macOS launchd commands
2026-05-12 15:36:29 +08:00
RWDai f60fc1cd7a fix: improve api key install copy flow 2026-05-12 15:34:16 +08:00
RWDai 9fc270fe69 feat: add admin api key install dialog 2026-05-12 15:34:08 +08:00
RWDai a8ff050518 fix: narrow api key install session types 2026-05-12 15:33:53 +08:00
RWDai a7bab0ebd2 feat: add admin api key install sessions 2026-05-12 15:33:45 +08:00
RWDai 8eda0932b0 fix: preserve install base URLs 2026-05-12 15:33:30 +08:00
fawney19 4e563e3385 Adjust proxy tunnel sizing defaults 2026-05-12 15:30:55 +08:00
fawney19 9c05b5f4e0 Unify pool score display as account health 2026-05-12 15:27:21 +08:00
fawney19 fa73655134 refactor: isolate dispatch scheduling core 2026-05-12 13:15:41 +08:00
fawney19 81ee27cdea Merge remote-tracking branch 'origin/codex/pool-member-scores' into aether-rust-pioneer 2026-05-12 09:17:25 +08:00
fawney19 fad28eee2c Improve pool score probing rules 2026-05-12 09:08:10 +08:00
fawney19 c578689356 Fix pool score CI regressions 2026-05-12 02:17:19 +08:00
fawney19 b9e62d1667 Implement generic pool member scoring and probing 2026-05-12 01:46:22 +08:00
fawney19 09146f8cdd fix: tighten model candidate matching scopes 2026-05-12 00:52:19 +08:00
fawney19 0fa97595bf Fix reasoning model directive response identity 2026-05-11 23:03:11 +08:00
fawney19 7ae38b6c43 Merge pull request #429 from AAEE86/rust
fix(usage): 统一用户排行榜 Token 统计口径
2026-05-11 22:46:56 +08:00
AAEE86 bfbf7a4663 fix(usage): 统一用户排行榜 Token 统计口径
- 将用户排行榜 Tokens 调整为与仪表盘今日 Token 一致
- 统一 effective input、cache creation fallback、cache read 计算规则
- 补齐 Postgres 聚合、admin 内存构建和内存仓库回退路径
- 增加缓存命中场景的统计口径测试
2026-05-11 22:34:37 +08:00
fawney19 cb0ccb9cdb fix postgres user role enum casts in migrations 2026-05-11 20:31:47 +08:00
fawney19 ce70780851 Fix proxy node uptime bucket rendering 2026-05-11 20:22:40 +08:00
fawney19 30b18d3310 fix: use gateway healthcheck flag in compose 2026-05-11 19:21:46 +08:00
fawney19 1d33e2c51b Fix Gemini model denial test fixture 2026-05-11 18:40:28 +08:00
fawney19 fed676f54f fix: exempt admins from default user group limits 2026-05-11 18:20:01 +08:00
fawney19 9bed5e9f83 fix: use intersection for user group policies 2026-05-11 17:00:07 +08:00
fawney19 e16a225eb3 Merge pull request #428 from AAEE86/rust
fix(gateway): 修复正则模型映射未生效到出站请求
2026-05-11 16:04:56 +08:00
AAEE86 67ca47afd4 fix(gateway): 修复正则模型映射未生效到出站请求
修复 key allowed_models 通过 global_model_mappings 正则命中时,
候选选择仍使用 provider model mapping 作为出站模型的问题。

正则命中后将 allowed model 写入 selected_provider_model_name,
确保最终 execution runtime 请求体中的 model 使用映射后的模型。

补充三层回归测试:
- scheduler-core 正则映射解析
- gateway candidate/data 候选选择输出
- gateway ai_execute 出站请求 model 捕获
2026-05-11 15:58:13 +08:00
fawney19 9057537ab8 fix: preserve model associations on refresh 2026-05-11 14:52:50 +08:00
fawney19 247ea9d1bd Restrict scheduler affinity to cache affinity mode 2026-05-11 14:06:49 +08:00
fawney19 e91c874863 fix(migrations): explicitly set timestamps in system_configs insert
老库的 created_at/updated_at 没有 DEFAULT now(),依赖默认值会写入 NULL
触发 NOT NULL 约束失败,改为显式传入 now() 与 sqlite/mysql 版本保持一致
2026-05-11 03:22:46 +08:00
github-actions[bot] 40470d8ca5 chore(proxy): update download links for proxy-v0.3.8 2026-05-10 19:10:49 +00:00
fawney19 94c0076778 chore(proxy): bump version to 0.3.8 2026-05-11 03:03:04 +08:00
fawney19 b813498e40 feat(proxy): surface node resource diagnostics 2026-05-11 02:59:18 +08:00
fawney19 cc4512fbbb Refine load balance candidate ranking 2026-05-11 02:32:11 +08:00
fawney19 ef4cc55c9a Record per-candidate upstream error bodies 2026-05-11 02:30:45 +08:00
fawney19 e3574e1918 Track scheduler affinity epochs and key sorting 2026-05-11 01:45:49 +08:00
fawney19 7b81c77424 Fix regex model mapping direction 2026-05-11 01:43:14 +08:00
fawney19 c9c968c2e9 fix: avoid duplicate user policy migration version 2026-05-11 00:39:34 +08:00
fawney19 e3f8fef30c Merge remote-tracking branch 'origin/pr/425' into aether-rust-pioneer 2026-05-11 00:19:50 +08:00
fawney19 ceda0635e4 Merge remote-tracking branch 'origin/pr/424' into aether-rust-pioneer
# Conflicts:
#	crates/aether-provider-transport/src/vertex/auth.rs
#	crates/aether-provider-transport/src/vertex/mod.rs
#	crates/aether-provider-transport/src/vertex/policy.rs
2026-05-11 00:19:04 +08:00
fawney19 bacb14e5f0 refactor: lazy pool key scheduling 2026-05-11 00:12:05 +08:00
Entropy.Xu 9e705ff603 Fix legacy user policy modes for empty lists 2026-05-10 19:35:09 +08:00
fawney19 1a0f1a7b72 Merge branch 'codex/pr-416-420-integration' into aether-rust-pioneer 2026-05-10 19:22:28 +08:00
fawney19 3201851667 Merge remote-tracking branch 'origin/pr/416' into codex/pr-416-420-integration 2026-05-10 19:13:42 +08:00
Codex 75df21932a 修复 Gemini 工具结果透传 Vertex 格式 2026-05-10 19:08:35 +08:00
Codex fb96771b56 修复 Vertex AI 服务账号访问 2026-05-10 19:08:27 +08:00
fawney19 c4b484fb43 fix: satisfy vertex transport lint checks 2026-05-10 19:07:51 +08:00
fawney19 37fb79fb87 fix: drop stray auth modules migration 2026-05-10 18:57:58 +08:00
fawney19 f03039d846 Merge remote-tracking branch 'origin/pr/420' into codex/pr-416-420-integration 2026-05-10 18:44:29 +08:00
fawney19 69476f69b9 Merge remote-tracking branch 'origin/pr/416' into codex/pr-416-420-integration
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
2026-05-10 18:44:19 +08:00
Entropy.Xu bb22978574 Merge remote-tracking branch 'upstream/aether-rust-pioneer' into fix-management-token-oauth-jsonb
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
2026-05-10 18:36:53 +08:00
fawney19 228253c166 Merge pull request #410 from Entropy-Xu/codex/codex-image-progress-heartbeat
Add Codex image progress heartbeat
2026-05-10 17:57:33 +08:00
fawney19 d26321006b Merge remote-tracking branch 'origin/aether-rust-pioneer' into aether-rust-pioneer 2026-05-10 17:48:54 +08:00
fawney19 377dd52805 style: polish multi select dropdown 2026-05-10 17:40:32 +08:00
fawney19 d246f6b42c fix: align user group access controls 2026-05-10 17:28:23 +08:00
Entropy.Xu 59d16ebb4b Merge remote-tracking branch 'upstream/aether-rust-pioneer' into fix-management-token-oauth-jsonb
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
2026-05-10 16:21:09 +08:00
Entropy.Xu 948a173f39 fix: avoid management token migration version collision 2026-05-10 16:16:11 +08:00
Entropy.Xu 56857280d3 fix: tolerate legacy management token json columns 2026-05-10 15:59:13 +08:00
Codex 7e804c408f 修复 Vertex AI 服务账号访问 2026-05-10 15:51:47 +08:00
Entropy.Xu 5268f62a64 feat: add sync image heartbeat toggle 2026-05-10 12:40:52 +08:00
github-actions[bot] 7f101431c5 chore(proxy): update download links for proxy-v0.3.7 2026-05-10 03:27:15 +00:00
fawney19 a8ac944959 Merge remote-tracking branch 'entropy-xu/codex/user-groups-default-permissions' into aether-rust-pioneer 2026-05-10 11:26:29 +08:00
fawney19 e8259a76f0 chore(proxy): bump version to 0.3.7 2026-05-10 11:19:26 +08:00
fawney19 3983b5a5cf Merge pull request #419 from wendaochangsheng/aether-rust-pioneer
修复 Vertex AI 服务账号 JSON 导入入口
2026-05-10 09:27:22 +08:00
fawney19 79b3a76dc1 Merge pull request #411 from buniakinazach-dev/feat/codex-spark-quota
新增 Codex Spark 额度展示
2026-05-10 09:25:44 +08:00
fawney19 c2bf17b4dd Fix Codex Spark quota formatting 2026-05-10 09:14:19 +08:00
Codex be3afbd279 修复 Vertex AI 服务账号 JSON 导入入口 2026-05-10 04:21:29 +08:00
fawney19 18690ceed2 Merge pull request #414 from stabey/fix/claude-tool-results-openai-chat
fix: preserve Claude tool results in OpenAI chat conversion
2026-05-10 02:50:36 +08:00
fawney19 0f42a6ed82 Fix Codex image progress heartbeat merge regressions 2026-05-10 02:10:23 +08:00
Entropy.Xu 545299fc62 fix: align management token oauth permissions and jsonb schema 2026-05-10 01:47:48 +08:00
fawney19 3c1456706a Merge pull request #405 from Entropy-Xu/codex/async-cleanup-records
Add async request body cleanup records
2026-05-10 01:12:12 +08:00
fawney19 a81053e6ff fix(admin): run destructive purges as cleanup tasks 2026-05-10 00:41:21 +08:00
fawney19 391c2fbe5d Merge remote-tracking branch 'origin/aether-rust-pioneer' into codex/async-cleanup-records
# Conflicts:
#	apps/aether-gateway/src/maintenance/mod.rs
#	apps/aether-gateway/src/maintenance/runtime/runners.rs
2026-05-10 00:28:39 +08:00
Entropy.Xu 121bdbd614 fix(data): avoid duplicate user group migration version 2026-05-09 22:26:31 +08:00
stabey dcfdba0a97 fix: preserve claude tool results in openai chat conversion
Preserve all Claude tool_result blocks when emitting OpenAI Chat messages, including multimodal image/file content and is_error markers.
2026-05-09 22:16:27 +08:00
Entropy.Xu a68c690874 Merge remote-tracking branch 'upstream/aether-rust-pioneer' into codex/user-groups-default-permissions 2026-05-09 22:04:09 +08:00
Entropy.Xu 3a814f3d1f feat: add user groups and inherited access policies 2026-05-09 21:47:33 +08:00
fawney19 209322b499 feat(gateway): unify background task runtime and storage 2026-05-09 21:19:29 +08:00
fawney19 4a64d078f3 Merge pull request #407 from stabey/pr/provider-stream-policy
fix: enforce provider upstream stream policy
2026-05-09 13:43:36 +08:00
stabey 5f4fa4ce1f fix: preserve upstream stream accept negotiation 2026-05-09 12:54:47 +08:00
stabey 7e5a08e09d fix: enforce provider upstream stream policy 2026-05-09 12:30:56 +08:00
fawney19 8958bf5e08 Merge pull request #409 from RWDai/feat/issue-373-cli-install
feat: add API key CLI install flow
2026-05-09 12:06:58 +08:00
fawney19 8b67120964 Merge pull request #402 from AAEE86/rust
feat(model-fetch): fetch Kiro models from upstream
2026-05-09 12:03:15 +08:00
fawney19 79d07ac79a Improve request trace upstream diagnostics 2026-05-09 10:46:04 +08:00
Alice 757c264e3e 新增 Codex Spark 额度展示
解析 GPT-5.3-Codex-Spark 额度窗口,并在号池页面和提供商账号抽屉中展示 Spark 周额度与 5H 额度。Spark 额度仅用于展示,不影响普通 Codex 周额度和 5H 额度的调度与筛选逻辑。
2026-05-09 10:44:45 +08:00
RWDai a72bf19454 fix(users): buffer API key install session body 2026-05-09 10:29:36 +08:00
Entropy.Xu eda3738d59 Merge branch 'aether-rust-pioneer' of https://github.com/fawney19/Aether into codex/codex-image-progress-heartbeat
# Conflicts:
#	frontend/src/features/usage/components/__tests__/HorizontalRequestTimeline.spec.ts
2026-05-09 01:26:44 +08:00
Entropy.Xu 3b4f27f767 Add Codex image progress heartbeat 2026-05-09 01:21:26 +08:00
fawney19 4cf0de681a feat(proxy): add node metrics expansion panel 2026-05-08 23:20:50 +08:00
RWDai 2c865ede35 feat(users): add CLI install modal 2026-05-08 23:16:16 +08:00
RWDai 46f44f4ea7 feat(users): add install session client 2026-05-08 23:16:16 +08:00
RWDai da46eb4791 feat(users): add API key install sessions 2026-05-08 23:16:16 +08:00
fawney19 e21fb58479 chore(proxy): guard metrics retention cleanup 2026-05-08 22:57:00 +08:00
fawney19 a703acd1fe feat(proxy): record tunnel stability metrics 2026-05-08 22:03:17 +08:00
fawney19 9a84a6ff6c refactor(ai-formats): group formats by provider
Move protocol/request/response format modules under provider-oriented formats modules and update registry, transport, and architecture paths.
2026-05-08 15:51:14 +08:00
fawney19 84a84e3f31 Merge pull request #404 from RWDai/fix/python-v063-user-import-compat
Preserve Python user import compatibility
2026-05-08 14:20:08 +08:00
Entropy.Xu 3f29335fd6 Add async request body cleanup records 2026-05-08 13:50:18 +08:00
RWDai 141a81d4a3 Preserve Python user import compatibility 2026-05-08 13:37:32 +08:00
AAEE86 2543676437 feat(model-fetch): fetch Kiro models from upstream
- add Kiro ListAvailableModels request planning and headers
- route Kiro model refresh through upstream fetch
- normalize Kiro model payloads and default model metadata
- remove profileArn requirement from ListAvailableModels
2026-05-08 13:24:27 +08:00
fawney19 fa22384f24 Merge remote-tracking branch 'origin/pr/399' into aether-rust-pioneer
# Conflicts:
#	crates/aether-data/src/lifecycle/bootstrap/postgres.rs
#	crates/aether-data/src/lifecycle/migrate/tests.rs
2026-05-08 02:52:13 +08:00
fawney19 2a603fa1e4 Merge remote-tracking branch 'origin/pr/397' into aether-rust-pioneer 2026-05-08 02:47:13 +08:00
fawney19 31d142effc Merge remote-tracking branch 'origin/pr/395' into aether-rust-pioneer 2026-05-08 02:47:07 +08:00
fawney19 66d8e90647 Merge remote-tracking branch 'origin/pr/394' into aether-rust-pioneer 2026-05-08 02:46:35 +08:00
fawney19 080784cd9f Refactor provider transport modules 2026-05-08 02:34:45 +08:00
fawney19 c52ef1993f Improve provider OAuth device flow 2026-05-08 01:33:23 +08:00
fawney19 6247ac3edc refactor: extract runtime state backends 2026-05-08 00:18:12 +08:00
Entropy.Xu cc5cb3475e feat: add management token permissions 2026-05-07 23:48:29 +08:00
RWDai 71742d5ad2 Fix pool cycle usage display 2026-05-07 22:25:52 +08:00
RWDai 1bc0822ce6 fix(users): align batch selection checkbox 2026-05-07 22:22:55 +08:00
RWDai fcefa4d198 feat(users): refine batch action dialog 2026-05-07 22:22:55 +08:00
RWDai 1e2ff26e86 fix(users): harden batch role and quota updates 2026-05-07 22:22:55 +08:00
Entropy.Xu dd8c2ebec6 fix(admin): repair system maintenance controls
Implement server-side searchable user filtering for usage records, including backend search parameters and a shared frontend selector with loading, empty, and pinned-selected states.

Fix announcement deletion by cascading announcement read rows through a Postgres migration and defensive repository cleanup across supported backends.

Wire admin system purge and cleanup endpoints to real data deletion/maintenance flows, improve DataManagement messages, rebuild stats after stats purge, and add runtime OAuth token refresh maintenance when enabled.

Verified with rust-ci equivalent checks: cargo fmt, split clippy, split cargo tests, SQLite/Postgres/MySQL smoke tests, plus frontend npm ci, build, pages build, type-check, and targeted usage selector tests.
2026-05-07 21:26:40 +08:00
fawney19 6f620d92be Skip release checksum verification in installer 2026-05-07 19:32:39 +08:00
RWDai 61473bfb77 feat(users): wire batch actions into users page 2026-05-07 19:16:49 +08:00
RWDai b7172092e8 feat(users): add batch action dialog 2026-05-07 19:16:49 +08:00
RWDai ea014e0d89 feat(users): add batch API store methods 2026-05-07 19:16:49 +08:00
RWDai 8fa90e8ecf refactor(users): share access control options 2026-05-07 19:16:49 +08:00
RWDai 9eb17eca32 refactor(pool): share batch selection helpers 2026-05-07 19:16:49 +08:00
RWDai 18b247de4c test(users): cover batch admin endpoints 2026-05-07 19:16:49 +08:00
RWDai 94852ce60e feat(users): classify batch admin routes 2026-05-07 19:16:49 +08:00
RWDai 59312ebe73 feat(users): add batch admin handlers 2026-05-07 19:16:49 +08:00
RWDai 738031696b fix(users): thread rate limit presence through gateway 2026-05-07 19:16:49 +08:00
RWDai a44667d2a9 fix(users): preserve nullable rate limit updates 2026-05-07 19:16:49 +08:00
fawney19 b8fc36033b Improve installer language and download source prompts 2026-05-07 18:53:09 +08:00
fawney19 6ef0bd9488 fix(ci): clean stale release drafts before publish 2026-05-07 16:30:39 +08:00
fawney19 f3d9523502 Expose upstream proxy in proxy setup TUI 2026-05-07 16:20:20 +08:00
fawney19 2b439094c5 Fix endpoint condition source handling 2026-05-07 15:56:28 +08:00
fawney19 6317587a6b Merge pull request #391 from RWDai/fix/affinity-list-display
Fix affinity list display
2026-05-07 15:54:53 +08:00
github-actions[bot] 7fdbd0c808 chore(proxy): update download links for proxy-v0.3.6 2026-05-07 07:26:05 +00:00
RWDai 44c2534f46 Fix affinity monitoring CI checks 2026-05-07 15:24:17 +08:00
fawney19 af8f9e9057 chore(proxy): bump version to 0.3.6 2026-05-07 15:18:43 +08:00
RWDai cb9d9944e3 Improve affinity list display 2026-05-07 15:08:30 +08:00
RWDai 2cfeb14d88 Add session-scoped affinity deletion 2026-05-07 15:08:14 +08:00
RWDai d5d93eda11 Fix affinity key parsing and counters 2026-05-07 15:08:07 +08:00
fawney19 9bbdf889d4 feat(proxy): support upstream egress proxy 2026-05-07 15:01:18 +08:00
fawney19 96f9be26da polish usage output rate and installer progress 2026-05-07 13:39:25 +08:00
fawney19 0c03c188f1 fix codex window usage stats 2026-05-07 13:35:25 +08:00
fawney19 64af7b1d8a fix: preserve selected keys for quota refresh 2026-05-07 12:55:39 +08:00
github-actions[bot] a345bb8c0d chore(proxy): update download links for proxy-v0.3.5 2026-05-07 04:01:32 +00:00
fawney19 624733e874 fix(compose): rely on postgres database url fallback 2026-05-07 11:53:25 +08:00
fawney19 fd44906bb7 fix provider pool quota status handling 2026-05-07 11:28:44 +08:00
fawney19 01c9f9be49 ci: rename release workflow 2026-05-07 09:46:35 +08:00
fawney19 3d9f189f0e ci: publish prerelease images to pre tag 2026-05-07 09:12:59 +08:00
fawney19 a83aa928a0 ci: parallelize rust checks and guard prereleases 2026-05-07 03:51:58 +08:00
github-actions[bot] 0386e0426b chore(proxy): update download links for proxy-v0.3.4 2026-05-06 19:35:24 +00:00
fawney19 10e679bb2f Add provider key cycle stats reset handling 2026-05-07 03:23:20 +08:00
fawney19 6c0ac2e8da chore: omit deprecated python quota changes 2026-05-07 01:39:48 +08:00
fawney19 2a124318b9 Merge branch 'pr-390' into aether-rust-pioneer 2026-05-07 01:34:38 +08:00
fawney19 47a755a585 Merge branch 'pr-389' into aether-rust-pioneer 2026-05-07 01:34:30 +08:00
fawney19 40314aa7e5 Merge branch 'pr-388' into aether-rust-pioneer 2026-05-07 01:34:22 +08:00
fawney19 df9d30340d Revert "feat: combine usage quota and pool stats updates"
This reverts commit 1f5b294bd0.
2026-05-07 01:33:40 +08:00
fawney19 1f5b294bd0 feat: combine usage quota and pool stats updates 2026-05-07 01:26:42 +08:00
fawney19 012f8bcdf7 feat: scope model mappings by endpoint 2026-05-07 00:48:15 +08:00
Kayphoon 79cb9b502a feat(pool): add codex cycle stats mode 2026-05-07 00:07:32 +08:00
fawney19 4e9f063385 use original request sources for endpoint conditions 2026-05-06 23:48:31 +08:00
Codex 41f75a3f19 feat: show chatgpt web image quota 2026-05-06 22:58:47 +08:00
fawney19 e2bffbbaca Remove body name style rule support 2026-05-06 22:12:21 +08:00
fawney19 9525a68719 Reduce endpoint rules JSON editor height 2026-05-06 21:45:37 +08:00
fawney19 a6b45702e8 Add JSON editor for endpoint rules 2026-05-06 21:30:13 +08:00
fawney19 7b11d43466 Allow request body model editing 2026-05-06 20:35:49 +08:00
fawney19 4bd49d0d7a Add unified install script and release packaging 2026-05-06 20:03:07 +08:00
mayrain 54afe35fcc feat(usage): show output speed in usage records
Display completed request output speed in the usage records table while keeping active requests focused on first-byte latency and live total duration.

The visible table keeps the compact tps label, while the tooltip expands the same value as tokens/s alongside first-byte, total, and generation durations.

Constraint: Synced against aether-rust-pioneer at 77c04749 with no upstream diff

Confidence: high

Scope-risk: narrow

Tested: npm run test:run -- src/features/usage/components/__tests__/UsageRecordsTable.spec.ts src/features/usage/__tests__/performance.spec.ts

Tested: npm run type-check

Tested: npm run build

Not-tested: Full backend test suite
2026-05-06 17:32:23 +08:00
fawney19 77c0474947 Fix usage records table layout 2026-05-06 16:20:09 +08:00
fawney19 33ed7c0737 Fix admin import export version checks 2026-05-06 16:06:45 +08:00
fawney19 bfda9b3e78 Capture TLS fingerprints in report context 2026-05-06 15:02:44 +08:00
fawney19 621eb55ae1 Merge pull request #387 from wendaochangsheng/codex/chatgpt-web-access-token-import
feat: 支持 ChatGPT Web Access Token 导入
2026-05-06 15:02:32 +08:00
Codex 7eaf3e7d03 feat: support chatgpt web access token import 2026-05-06 14:45:39 +08:00
fawney19 68216bf868 Clean up transport fingerprint configuration
Remove legacy tls_profile handling, keep header fingerprint under transport profiles, and drop the duplicate auth_modules migration.
2026-05-06 13:54:33 +08:00
fawney19 6fbb867f5f Exclude unknown providers from usage aggregations 2026-05-06 13:40:18 +08:00
fawney19 94a6f315ca Merge pull request #385 from Entropy-Xu/codex/chatgpt-web-image-proxy
feat(image): 接入 ChatGPT Web 生图反代
2026-05-06 12:01:21 +08:00
Entropy.Xu 4baee436ba feat(image): 接入 ChatGPT Web 生图反代 2026-05-06 11:52:33 +08:00
fawney19 beee7a76d2 Fix postgres export smoke import compatibility 2026-05-06 09:29:12 +08:00
fawney19 887a58d639 Fix usage provider stats and performance analysis UI 2026-05-06 09:29:12 +08:00
fawney19 110b02a213 Merge pull request #386 from fawney19/codex/pr-376-377-383-384-combined
Combine PR #376 #377 #383 #384
2026-05-06 03:20:05 +08:00
fawney19 6219491389 chore: squash inactive mysql sqlite migrations 2026-05-06 03:09:53 +08:00
fawney19 d7c2232062 fix: reconcile combined usage aggregation helpers 2026-05-06 02:47:52 +08:00
fawney19 2eb4afbec4 Merge remote-tracking branch 'origin/pr/384' into codex/pr-376-377-383-384-combined
# Conflicts:
#	frontend/src/views/admin/PerformanceAnalysis.vue
2026-05-06 02:39:53 +08:00
fawney19 b03509a5f4 Merge remote-tracking branch 'origin/pr/383' into codex/pr-376-377-383-384-combined 2026-05-06 02:37:09 +08:00
fawney19 b8a6feef4a Merge remote-tracking branch 'origin/pr/377' into codex/pr-376-377-383-384-combined
# Conflicts:
#	apps/aether-gateway/src/tests/frontdoor/public_support/dashboard.rs
#	crates/aether-data/src/lifecycle/backfill.rs
#	crates/aether-data/src/repository/usage/postgres/mod.rs
2026-05-06 02:36:57 +08:00
fawney19 4bf86f85b4 Merge remote-tracking branch 'origin/pr/376' into codex/pr-376-377-383-384-combined 2026-05-06 02:33:48 +08:00
fawney19 be580c35bb chore: resolve pr 377 checks 2026-05-06 02:22:32 +08:00
fawney19 cf27773582 feat: improve admin performance analysis 2026-05-06 02:01:32 +08:00
fawney19 6083461822 Add editable response header rules 2026-05-06 01:29:42 +08:00
fawney19 3264857e4a fix(usage): repair aggregate usage statistics 2026-05-06 01:13:23 +08:00
fawney19 f1358dd845 Merge pull request #375 from RWDai/feat/303-client-aware-scheduler
feat: add session-aware scheduler affinity
2026-05-06 00:21:00 +08:00
fawney19 8043cca126 fix: align postgres core export schema 2026-05-06 00:20:06 +08:00
fawney19 6dd7464793 Merge remote-tracking branch 'origin/aether-rust-pioneer' into pr-375-session-scope 2026-05-05 22:34:47 +08:00
fawney19 79c272f0fd Merge pull request #378 from fawney19/codex/transport-profile-routing
Implement transport profile routing
2026-05-05 22:24:48 +08:00
fawney19 f959f02d40 Implement transport profile routing 2026-05-05 22:21:23 +08:00
fawney19 98421126f2 refactor: carry normalized client session affinity 2026-05-05 21:34:17 +08:00
fawney19 ddf3fb6f63 refactor: introduce client session scope abstraction 2026-05-05 20:58:42 +08:00
fawney19 aacab1a90c Merge remote-tracking branch 'origin/aether-rust-pioneer' into aether-rust-pioneer
# Conflicts:
#	crates/aether-data-contracts/src/repository/usage/mod.rs
#	crates/aether-data/src/repository/global_models/postgres.rs
#	crates/aether-data/src/repository/usage/postgres/mod.rs
2026-05-05 18:53:14 +08:00
fawney19 fce7e959e5 Add multi-database data layer
Introduce aether-data-schema and driver-specific schema generation for Postgres, MySQL, and SQLite.

Split data backends, lifecycle, repositories, and gateway runtime integration across database drivers.

Verified with cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace.
2026-05-05 18:27:36 +08:00
mayrain 6b6d32b4a3 feat(usage): 新增输出速度统计与请求详情性能分析
- 把 upstream_is_stream 物化到 usage 与 billing facts,避免 Provider 聚合回连 public.usage
- 统一前端标准/流式 TPS 计算与显示,流式按首字后生成耗时计算
- 新增请求详情抽屉展示单请求输出速度与 Provider 聚合 TPS
2026-05-05 17:34:12 +08:00
mayrain f0197b685f fix(data): 同步模型调用次数至 global_models 读模型
- 模型列表改为读取 global_models.usage_count,避免每次扫描 usage 明细表
- 通过历史 backfill 与 usage upsert delta 维护该读模型
- 详情页保留实时 facts 兜底,统一排除 pending/streaming 状态
2026-05-05 17:34:02 +08:00
mayrain 53fa33b0c5 fix(dashboard): 修复仪表盘与明细统计数值不一致并重建 cost_savings 聚合
- 统一 dashboard 聚合与 raw 查询的 token 计算,拆分今日节省和周期节省
- cache savings 改用 input price 估算未命中成本,修复历史 cost_savings 偏高
- raw 查询 total_tokens 改用 effective_input + output + cache_creation + cache_read 公式
- 新增独立 backfill 重建历史 cost_savings 聚合表,保留已发布 backfill 不变
2026-05-05 17:33:16 +08:00
mayrain 2f915b33c7 perf(usage): 重构多天聚合查询为数据库端按天分组计算
- 多天范围的模型、供应商和 API 格式分析复用已有用户日聚合表
- 只对未完成窗口回退明细查询,再合并排序截断
- 避免长范围直接扫 usage_billing_facts 视图
2026-05-05 17:33:04 +08:00
mayrain fd8230121c fix(usage): 修复多天统计 NUMERIC→f64 解码失败导致记录清空
- 多天统计从日聚合表 SUM numeric 字段后返回 NUMERIC,代码按 f64 解码失败
- 显式添加 ::DOUBLE PRECISION 类型转换
- 管理员统计加载失败时不再清空已加载的记录列表
2026-05-05 17:32:50 +08:00
fawney19 825c1b496d Merge pull request #374 from Entropy-Xu/codex/provider-performance-stats
feat(stats): 添加 Provider 性能统计分析
2026-05-05 13:16:06 +08:00
fawney19 ccde3d4045 style(gateway): format secret masking changes 2026-05-05 13:04:56 +08:00
fawney19 f7071967c0 fix(gateway): mask catalog secrets on char boundaries 2026-05-05 12:48:19 +08:00
RWDai 657f9f0be1 fix: carry session affinity through Gemini files 2026-05-05 12:25:07 +08:00
RWDai 628329e493 fix: scope scheduler effects by session 2026-05-05 12:25:07 +08:00
RWDai 82e7a0080e fix: prefer client session adapters 2026-05-05 12:25:07 +08:00
Entropy.Xu ff1ed8b57d feat(stats): 添加 Provider 性能统计分析 2026-05-05 11:24:16 +08:00
RWDai 17b06bd4d6 refactor: abstract model fetch strategies 2026-05-05 11:23:54 +08:00
RWDai 5fcb49b08e feat: wire specialized planner session affinity 2026-05-05 11:23:54 +08:00
RWDai 044ef59d81 feat: wire OpenAI responses session affinity 2026-05-05 11:23:54 +08:00
RWDai 51b191ad2c feat: wire OpenAI chat session affinity 2026-05-05 11:23:54 +08:00
RWDai e890a5c2f1 feat: wire standard family session affinity 2026-05-05 11:23:54 +08:00
RWDai 591dcf5cf2 feat: carry session affinity through candidate materialization 2026-05-05 11:23:54 +08:00
RWDai 58d6b2add6 feat: scope planner affinity cache by session 2026-05-05 11:23:54 +08:00
RWDai bcd542f7ee feat: pass session affinity through planner inputs 2026-05-05 11:23:54 +08:00
RWDai 37da4e245a feat: scope tunnel affinity by client session 2026-05-05 11:23:54 +08:00
RWDai 9ef7952da5 fix: keep scheduler affinity out of runtime checks 2026-05-05 11:23:54 +08:00
RWDai eb8878695a feat: thread session affinity through scheduler selection 2026-05-05 11:23:54 +08:00
RWDai c596d82dec feat: pass session scope into scheduler cache 2026-05-05 11:23:54 +08:00
RWDai 96dd3fa4ca feat: add gateway session affinity adapters 2026-05-05 11:23:54 +08:00
RWDai 14eeff8f75 fix: keep affinity out of scheduler eligibility 2026-05-05 11:23:54 +08:00
RWDai 2cbbd9ca36 feat: add session-aware scheduler keys 2026-05-05 11:23:54 +08:00
fawney19 627eb4f335 Merge pull request #371 from Kayphoon/feature/embedding-model-support
feat: add embedding and rerank support
2026-05-04 12:56:11 +08:00
fawney19 8dd4135f7c Fix provider key insert placeholder order 2026-05-04 12:21:54 +08:00
Kayphoon 6bdd7792eb fix: repair embedding rerank CI checks 2026-05-04 11:33:00 +08:00
fawney19 099653f732 Cap Codex pool cooldowns and add key circuit breaker 2026-05-04 02:15:32 +08:00
fawney19 1d62722d47 Improve Codex model fetching 2026-05-04 01:16:56 +08:00
fawney19 fb642f7d62 feat: improve OAuth provider configuration 2026-05-04 00:18:05 +08:00
fawney19 cb5b8ee1ee Use Rust build metadata for system version 2026-05-03 22:06:16 +08:00
fawney19 b4a8c5dde2 Format execution runtime transport imports 2026-05-03 21:52:15 +08:00
fawney19 53a5e18b20 Merge pull request #370 from yao177/aether-rust-pioneer
fix(proxy): record manual proxy node traffic
2026-05-03 21:40:20 +08:00
fawney19 6f00cabe96 Lazy load requested model candidates 2026-05-03 20:56:31 +08:00
fawney19 a24e4a793d Refactor pool candidate scheduling 2026-05-03 20:14:29 +08:00
fawney19 8ebee9922c fix: return auth mismatch key setting in provider key list 2026-05-03 20:11:49 +08:00
Yao177 c3d97b8c16 fix(proxy): record manual proxy node traffic
- add manual proxy traffic mutations to the data layer
- record sync and stream outcomes for manual proxy requests
- prevent tunnel proxy nodes from receiving manual traffic updates
2026-05-03 18:38:04 +08:00
Kayphoon 5abe664d65 feat: add embedding and rerank support 2026-05-03 17:32:41 +08:00
jiuwovo-aiandroot 3e2eca4fd0 fix: 修复手动代理节点请求不计数及延迟/心跳不显示的问题 (#368)
* fix: 修复手动代理节点请求不计数及延迟/心跳不显示的问题

问题描述:
- 手动添加的代理节点请求数始终为0,不会递增
- 手动代理节点的延迟和最后心跳时间不显示

根因:
Rust 重写版中缺失了 Python 版的手动代理节点请求计数逻辑。
隧道节点通过心跳上报计数,但手动节点没有心跳机制,
需要在 usage recording 路径中递增计数并更新延迟信息。

修复方案:
1. 在 ExecutionPlan 中提取 proxy 信息,注入到 request_metadata
2. 在 request_metadata 白名单中添加 proxy 字段
3. 新增 INCREMENT_MANUAL_PROXY_NODE_REQUESTS_SQL,
   递增 total_requests/failed_requests,
   同时更新 avg_latency_ms 和 last_heartbeat_at
4. 在 ProxyNodeWriteRepository trait 新增 increment_manual_node_requests 方法
5. 在 write_event_record 中解析 request_metadata 中的 proxy 信息,
   对非隧道模式的手动节点调用递增方法
6. 为 GatewayDataState 实现 ManualProxyNodeCounter trait

影响范围:
- 仅影响手动代理节点的统计数据
- 隧道节点不受影响(继续通过心跳计数)
- 不影响请求转发逻辑

* style: cargo fmt

---------

Co-authored-by: root <[email protected]>
2026-05-03 14:50:36 +08:00
fawney19 c4ea042eb4 feat: add model directive management 2026-05-03 14:48:25 +08:00
Kayphoonandfawney19 fe27fb17fb feat: add provider-key concurrent limit (#352)
* feat: add provider-key concurrent limit

* Fix provider key concurrent limit checks

---------

Co-authored-by: fawney19 <[email protected]>
2026-05-03 01:55:23 +08:00
fawney19 11c5884d4f Remove API format rate multiplier field 2026-05-03 00:55:59 +08:00
fawney19 e3ea2d1451 feat: configure auth channel mismatch formats 2026-05-03 00:49:22 +08:00
fawney19 3a770306cc refactor: separate request auth channel from route kind
Extract request_auth_channel as a distinct routing dimension to distinguish
between api_key and bearer_like authentication methods. This improves routing
clarity by decoupling authentication mechanism from route classification.
2026-05-02 21:23:33 +08:00
fawney19 47ee8b9c13 refactor: fold ai surfaces into formats 2026-05-02 18:19:39 +08:00
bfd1ea72b3 fix: codex free accounts incorrectly marked as quota exhausted (#366)
* fix: codex free accounts incorrectly marked as quota exhausted when skip_exhausted_accounts enabled

- Cross-validate stored exhausted flag against window used_ratio in snapshot reader
- Guard has_credits:false check with window data presence in upstream_metadata fallback
- Add windows.is_empty() precondition to exhausted_by_credits in snapshot builder

Free codex accounts have has_credits:false (they use window-based quotas, not credits),
but the old logic treated this as credits depleted, skipping actual window usage checks.
This caused all free accounts to be incorrectly marked exhausted.

* test: cover codex free quota windows

---------

Co-authored-by: root <[email protected]>
Co-authored-by: fawney19 <[email protected]>
2026-05-02 14:11:29 +08:00
fawney19 c130d0e2c9 refactor ai serving modules and crates 2026-05-02 13:23:54 +08:00
fawney19 4fc7cecf30 Fix OAuth token import and table filters 2026-05-01 02:14:49 +08:00
fawney19 9570e5c2c1 Fix legacy Claude and Gemini auth migration 2026-04-30 18:01:58 +08:00
fawney19 33aa70c22b fix scheduler affinity candidate selection 2026-04-30 16:27:24 +08:00
fawney19 558abfcfa3 Route request origin through ai pipeline facade 2026-04-30 12:15:51 +08:00
fawney19andRWDai 7fcb9e6292 feat(usage): record request origin metadata (#362)
Record client IP and User-Agent in usage request_metadata for local execution requests without adding schema, API, or table fields.

References #357 and supersedes #358.

Co-authored-by: RWDai <[email protected]>
2026-04-30 11:26:56 +08:00
RWDaiandfawney19 2b5247b9a8 fix(data): repair request candidate epoch created_at (#343)
Co-authored-by: fawney19 <[email protected]>
2026-04-30 09:23:21 +08:00
fawney19 fa47e8a3c0 feat: add select all to access limit dropdowns 2026-04-30 00:45:53 +08:00
Entropy.Xu 4d59d518d1 fix: correct API key expiry and dashboard savings (#361) 2026-04-30 00:35:32 +08:00
fawney19 9e2faa7e5b Tighten local auth allow-list matching 2026-04-30 00:26:33 +08:00
fawney19 37392d8774 fix(ai): allow null OpenAI Responses error fields 2026-04-29 21:12:14 +08:00
fawney19 a16550249b fix: allow empty provider secrets to attempt requests 2026-04-29 18:55:52 +08:00
fawney19 d6de917878 Fix key API format auth help 2026-04-29 17:35:16 +08:00
fawney19 e751289dfb Support per-format provider key auth 2026-04-29 15:46:50 +08:00
fawney19 07a319259b Normalize canonical API formats 2026-04-29 10:20:41 +08:00
fawney19 02ad67fe33 fix(test): update openai format identifier from cli to responses 2026-04-28 21:04:38 +08:00
github-actions[bot] 655e369f21 chore(proxy): update download links for proxy-v0.3.3 2026-04-28 12:04:24 +00:00
fawney19 a9b809a32c chore(proxy): bump version to 0.3.3 2026-04-28 19:57:29 +08:00
fawney19 230e7d6259 Normalize management token prefixes 2026-04-28 19:46:38 +08:00
fawney19 3d20c05ef7 Fix OAuth refresh through tunnel proxy 2026-04-28 19:45:34 +08:00
fawney19 9194f78e56 OAuth refresh 隧道代理统一启用 follow-redirects 2026-04-28 17:47:28 +08:00
fawney19 70f747d406 Add shared OAuth flows 2026-04-28 15:46:21 +08:00
AAEE86 712b484bc8 fix(scheduler): 恢复正则模型映射作为上游模型 (#355)
保留 provider_model_mappings 的优先级选择逻辑,但恢复
GlobalModel model_mappings 正则命中后的行为:使用命中的
allowed_model 作为实际上游 mapped_model。

修复了 #354 请求绕过基于正则表达式的模型映射的问题
2026-04-28 10:36:50 +08:00
fawney19 0c9b5ddd77 Fix usage terminal state regression 2026-04-28 10:31:16 +08:00
fawney19 3f1abb6906 Format OAuth refresh consistency changes 2026-04-28 09:54:35 +08:00
fawney19 29fc0be121 Tighten OAuth refresh consistency 2026-04-28 09:36:42 +08:00
Kayphoon 5311eb0da1 Fix provider model mapping selection (#354) 2026-04-28 09:23:39 +08:00
fawney19 321f21ad49 Fix provider OAuth refresh token reuse 2026-04-28 02:00:35 +08:00
fawney19 639f26ed5d fix: force http1 for tunnel oauth refresh 2026-04-28 01:37:40 +08:00
fawney19 032dcf40e7 fix: filter candidate rows to resolved global model name only 2026-04-28 01:12:27 +08:00
fawney19 9a67497d8c fix: align oauth tunnel refresh transport 2026-04-28 01:10:34 +08:00
fawney19 6426b5d80e chore: add oauth refresh diagnostics 2026-04-28 00:18:51 +08:00
fawney19 83e1f99ccf fix: invalidate oauth transport cache after refresh 2026-04-27 23:35:05 +08:00
fawney19 05b8b8a442 fix: bypass oauth cache for forced refresh 2026-04-27 23:04:31 +08:00
fawney19 ed17147281 fix: align provider oauth refresh redirects 2026-04-27 22:08:02 +08:00
fawney19 a7a3c9f023 Fix OAuth import proxy handling 2026-04-27 20:05:43 +08:00
fawney19 4c16b11cb4 Align runtime miss tests with skipped candidates 2026-04-27 18:09:56 +08:00
fawney19 17f09fc8c1 Expose ranking metadata in request trace 2026-04-27 17:24:32 +08:00
fawney19 3e0293ac28 Rename local candidate resolution entrypoints 2026-04-27 17:24:32 +08:00
fawney19 07fba70a90 Guard pool scheduler ranking boundary 2026-04-27 17:24:32 +08:00
fawney19 da0e968975 Update transport ranking architecture guard 2026-04-27 17:24:32 +08:00
fawney19 98e4e91a98 Rename transport ordering facts 2026-04-27 17:24:32 +08:00
fawney19 218e73e324 Use core affinity helpers directly 2026-04-27 17:24:32 +08:00
fawney19 51c3beb614 Narrow core ranking public surface 2026-04-27 17:24:32 +08:00
fawney19 9da47ceb22 Remove duplicate capability candidate sorting 2026-04-27 17:24:32 +08:00
fawney19 e9f03d8d29 Remove ranked minimal selection compatibility helper 2026-04-27 17:24:32 +08:00
fawney19 e5e09b49f4 Remove ranked candidate data read wrappers 2026-04-27 17:24:32 +08:00
fawney19 dda6f34a07 Rename ranked minimal candidate reads 2026-04-27 17:24:32 +08:00
fawney19 44930532dc Align ranked candidate selection test names 2026-04-27 17:24:32 +08:00
fawney19 f10f5f071d Clarify ranked minimal candidate selection 2026-04-27 17:24:32 +08:00
fawney19 517e84e5ae Move local ranking test helper into tests 2026-04-27 17:24:31 +08:00
fawney19 2101a4ecc7 Drop legacy candidate ordering helpers 2026-04-27 17:24:31 +08:00
fawney19 8d2cbf32df Remove legacy tunnel-only candidate ordering 2026-04-27 17:24:31 +08:00
fawney19 a23758808d Isolate core candidate selection helpers 2026-04-27 17:24:31 +08:00
fawney19 91db4eefd0 Persist ranking metadata through request traces 2026-04-27 17:24:31 +08:00
fawney19 f52220a8ec Clean up planner candidate modules 2026-04-27 17:24:31 +08:00
fawney19 1d23bf4ecb Isolate standalone key wallet handling 2026-04-27 17:24:31 +08:00
fawney19 3b542434a2 Unify candidate ranking pipeline 2026-04-27 17:24:31 +08:00
Kayphoon 9b866a6d17 fix(auth): 修复独立密钥继承用户访问限制 (#347) 2026-04-27 12:32:21 +08:00
Entropy.Xu d8f66b14a8 Handle Codex image stream rate limits (#346) 2026-04-27 12:23:48 +08:00
Entropy.Xu 6e1eaf8aec fix(kiro): 修复 Kiro WebSearch MCP 调用 (#344)
* fix(kiro): 接入 Kiro MCP web_search 工具调用

* fix(kiro): 对齐 Kiro IDE MCP 鉴权与 profileArn 头部

* fix(responses): 保留 Responses 嵌套与自定义工具参数

* test(ci): 通过 Rust Action 三项检查
2026-04-27 12:20:26 +08:00
fawney19 488bd08f04 Stabilize Kiro Claude CLI sync tests 2026-04-27 00:45:48 +08:00
fawney19 3991d47166 Fix format checks and local error messages 2026-04-27 00:27:03 +08:00
fawney19 0c73f245ab shrink formats conversion to compatibility facades 2026-04-26 23:59:53 +08:00
fawney19 989b27426b refactor formats adapters and matrix ownership 2026-04-26 23:59:53 +08:00
fawney19 c10cd8240e refactor ai formats registry adapters 2026-04-26 23:59:53 +08:00
fawney19 5cd2244bc2 fix frontend openai responses alias display 2026-04-26 23:59:53 +08:00
fawney19 0ec4b4c8a4 guard legacy openai alias compatibility 2026-04-26 23:59:53 +08:00
fawney19 4ec591fbf2 centralize openai responses alias handling 2026-04-26 23:59:53 +08:00
fawney19 ea3dc3257e clean up legacy openai cli adapter names 2026-04-26 23:59:53 +08:00
fawney19 5b914aa78c migrate ai format conversion to responses adapters 2026-04-26 23:59:53 +08:00
Entropy.Xu e36fb8c07a Honor global format conversion override (#341)
Apply the global enable_format_conversion setting when building provider transport snapshots so enabled global conversion overrides provider-level settings at runtime.

Keep the cached provider snapshot unchanged so disabling the global switch restores each provider's original conversion configuration.
2026-04-26 23:54:52 +08:00
Kayphoon 1af9d00abd fix(gateway): 修复仅配置 provider retry 时 429 不重试 (#338)
- attempt.rs: 从 failover_rules/endpoint/provider 三级读取 max_retries 生成 attempt slots,支持配置 provider retry 后 429 自动重试,限制最大 99 并兼容 legacy 默认值 2
2026-04-26 23:50:14 +08:00
Entropy.Xu e8763b3bbd fix(kiro): 放行可刷新 OAuth 调度候选 (#340) 2026-04-26 23:49:55 +08:00
AAEE86 f3a9fd4c82 feat(gateway): 补齐号池主动探测后台任务 (#342)
- 解析 pool_advanced 主动探测开关与间隔配置
- 对齐 Python 版本的探测间隔默认值和范围限制
- 新增号池配额主动探测 worker
- 支持 codex、kiro、antigravity provider
- 使用 Redis 时间戳和 provider 级锁避免重复探测
- 接入 gateway 后台任务并补充测试
2026-04-26 23:48:31 +08:00
fawney19 baa0ddd787 Improve runtime miss usage diagnostics 2026-04-26 01:44:54 +08:00
fawney19 1d3ea3232d Add unified candidate failure diagnostics 2026-04-26 01:38:37 +08:00
Entropy.Xu d784c540b6 fix(kiro): 隐藏 OAuth 刷新重试流程 (#339) 2026-04-25 21:29:45 +08:00
fawney19 429fdb47e6 fix(usage): 简化本地执行错误提示 2026-04-25 20:46:22 +08:00
fawney19 912a92cd1a refactor(rules): 规则引擎容错优化,无效规则条目跳过而非中止整个规则集
- header/body rules 的 _are_locally_supported 简化为仅检查是否为数组
- apply 逻辑中遇到格式错误/不支持的规则条目改为 continue 跳过,而非 return false
- 允许非字符串 header value,自动序列化为 JSON 字符串
- 宽松处理无效 regex flag,不再拒绝整条规则

fix(gateway): Claude CLI 路由仅检查 bearer 头,不排斥同时携带 x-api-key 的请求

feat(observability): 监控链路候选展示解密 auth_config 的账号标签和 OAuth 计划类型
2026-04-25 19:46:52 +08:00
fawney19 00744c0ce5 feat(observability): 引入错误链路 error_flow 元数据并区分上游/客户端错误
- 网关在本地 failover 时构建 error_flow 元数据(分类/决策/传播策略),写入 report_context
- scheduler-core 解析并透传 error_flow 至候选 extra_data
- admin usage 详情拆分 request/upstream/client/failure_summary 错误域,敏感上游错误标记为 suppressed
- 前端 RequestDetailDrawer 拆出"返回客户端"与"上游响应"双错误卡片
- HorizontalRequestTimeline 节点详情展示真实请求错误及 error_flow 标签
2026-04-25 17:01:07 +08:00
Entropy.Xu bc97e383d3 fix(gateway): 对齐号池调度预设行为 (#332)
- 保持 sticky session 优先于分配模式排序,并支持 sticky_session_ttl_seconds=0 禁用粘滞绑定
- 移除旧版 free_team_first 预设入口,统一使用 free_first / team_first 新模式
- 对齐 plus_first 对 Plus/Pro 账号的优先级处理
- 调整负载均衡排序种子,避免同一分钟内固定命中同一 Key
- multi-score 预设下记录延迟样本,并补充号池调度/配置/运行时测试
- 收敛 request candidate extra_data 构造入参,修复 clippy too_many_arguments
2026-04-25 16:41:51 +08:00
RWDai b8205b5a09 Fix/usage json entity escaping (#333)
* fix(usage): 修复 JSON 视图实体转义显示

* fix(usage): 解码 OpenAI 工具参数实体
2026-04-25 16:41:25 +08:00
Entropy.Xu bb7fe9fe37 fix(kiro): 修复 Kiro OAuth 过期懒刷新 (#334)
* fix(kiro): 修复 OAuth 过期懒刷新

* fix(kiro): 修复 OAuth 过期懒刷新

* fix(scheduler): 修复 candidate extra data clippy 告警
2026-04-25 16:40:45 +08:00
fawney19 cef7dcac71 refactor(usage): 抽取 JsonContentPanel 通用 JSON 展示组件
请求详情抽屉与时间线统一复用工具栏(展开/收缩、复制)和 JSON 视图,时间线的额外信息也由 pre 改为可交互面板
2026-04-25 11:44:01 +08:00
fawney19 61e12e2c17 feat(scheduler): candidate report 携带 header_rules 与 body_rules
各 planner 在构造 report context 时附带端点的 header/body 改写规则,scheduler 解析后写入 candidate extra_data,便于追踪请求实际应用的透传规则
2026-04-25 11:43:53 +08:00
fawney19 65e915fd1d refactor(api-keys): 网关 API Key 改为 32 位字母数字随机串
由 UUID 拼接(48 位 hex)改为 base62 字母数字,缩短长度并提升可读性
2026-04-25 10:04:04 +08:00
fawney19 1761921670 fix(usage): 移动端缓存 token 拆分为读/写两列展示
cache_read 与 cache_creation 原本被合并为单值,现分别显示并补充测试
2026-04-25 09:53:32 +08:00
fawney19 e5bbc797e0 chore(gateway): 调小 Postgres 连接池默认值
- max_connections 100 → 20
- idle_timeout 60s → 30s
- .env.example 补充连接池相关变量注释
2026-04-25 09:53:21 +08:00
fawney19 f30b1f3f6b fix(usage): 流式终端 usage 以更完整值为准,Codex CLI 显式选择 response 解析器
- 新增 StandardizedUsage::signal_score/is_more_complete_than/choose_more_complete,流式合并与终端落库均按信号完整度择优
- OpenAI Chat/CLI 解析器支持仅 usage 的终结 chunk 与 response.completed usage
- Codex provider 注入 provider_stream_event_api_format=openai:cli,解析器选择改由 report_context 显式决定
- usage_mapper 扩展嵌套 response/message/item 兼容 Claude message_start/message_delta 及 Gemini stream chunks
- usage SQL upsert 在终态(completed/failed/cancelled)时按 GREATEST 写入 token/费用镜像列
2026-04-25 00:57:01 +08:00
fawney19 2c6209277f fix(usage): Claude 流式合并 message_start/delta usage,前端移动端缓存 token 合并展示
- Claude provider state 在 message_start 记录基础 usage,message_delta 合并 output_tokens
- 前端 UsageRecordsTable 移动端缓存列改为 cache_creation + cache_read 合计展示
2026-04-24 22:28:04 +08:00
fawney19 5b3593038d fix(usage): OpenAI 缺失 input_tokens 时从 total 回推
- 同步与流式两条路径均在 input_tokens 为 0 且 total > output 时用 total - output 兜底
- 补充对应单元测试
2026-04-24 22:08:23 +08:00
Entropy.Xu 67b092253d fix(gateway): 兼容 OpenAI 图像 multipart boundary 大小写 (#331) 2026-04-24 21:41:20 +08:00
fawney19 db1059e73b fix(usage): 修复 Gemini total 重复累加 cache_read 并补充测试
- write.rs: Gemini usage 提取 total 时不再叠加 cachedContentTokenCount
- 补充 Claude 大 cache_read 场景下 input_tokens 不被扣减的测试
- 补充前端 getEffectiveInputTokens 对 Claude 格式不减 cache_read 的测试
2026-04-24 21:20:45 +08:00
fawney19 e265475c17 fix(usage): 修正 input_tokens 统计与显示
- SQL 查询不再用 settlement snapshot 的 billing_input_tokens 覆盖原始 input_tokens
- usage_mapper 针对 OpenAI 格式在 input_tokens 缺失时从 total_tokens - output_tokens 推导
- 前端轮询合并优先采用最新 record 的 input_tokens / effective_input_tokens
2026-04-24 20:59:12 +08:00
Entropy.Xu 343345529d fix(kiro): 修复 Claude CLI 跨格式 Responses 流式转换 (#329) 2026-04-24 20:23:02 +08:00
fawney19 657fcd595c refactor(rust): 拆分 usage sql 仓储为模块并外置 SQL 文件
- 将 usage/sql.rs 拆分为 sql/mod.rs 与 sql/tests.rs
- 将内联 SQL 抽离到 sql/queries/ 目录下的独立 .sql 文件
- 前端用量记录表格移除缓存创建 token 展示,保留 cache read 列
2026-04-24 20:22:24 +08:00
fawney19 d0d71aa51a fix(gateway): OpenAI 图像 finalize 统一返回 b64_json 并对齐 codex CLI 候选流式上游
- finalize 移除 response_format=url 的 data URL 分支,统一输出 b64_json
- image planner decision 依据请求体中的 stream 字段决定 upstream_is_stream
- openai chat sync plan 对 codex+openai:cli 候选强制上游流式
- 扩充 conversion registry 与 standard matrix 的全量 surface 对端测试
2026-04-24 19:01:11 +08:00
AAEE86 4618184516 fix: 修复账号批量操作快捷多选不生效和今日 Token 不准确 (#328)
* fix(pool): 修复账号批量操作快捷多选不生效

- 解析号池账号列表接口的 quick_selectors 查询参数
- 列表查询复用现有快捷筛选匹配逻辑
- 支持冷却列表与普通列表的快捷筛选后分页
- 更新架构测试中的 pool admin helper 边界断言

* fix(dashboard): 今日 Token 改为按分项汇总口径展示

- /api/dashboard/stats 的 today.tokens 改为 输入+输出+写缓存+读缓存
- “今日 Token”卡片 value 与 subValue 口径保持一致
- 同步更新 dashboard 相关测试断言
2026-04-24 18:36:38 +08:00
fawney19 3cc54deb9c fix(gateway): SSE 流跳过成功探测与 prefetch,统计聚合 SQL 显式 CAST BIGINT
- stream: event-stream 响应跳过 direct finalize prefetch 与成功失败转移探测,避免消费流首帧
- stream_pump: 仅在已知 content-length 时缓冲非 SSE 响应
- maintenance: usage_billing_facts 聚合列显式 CAST 为 BIGINT,避免类型不匹配
2026-04-24 18:14:59 +08:00
fawney19 f695238e8a feat(billing): 引入 billing v3 settlement snapshot 及 usage_billing_facts 视图
- 新增迁移 20260424000000:为 usage_settlement_snapshots 添加 settlement_snapshot、
  billing_dimensions、billing_input/output/cache tokens、billing_total_cost_usd 等列,
  并创建 usage_billing_facts 视图(从 settlement snapshot 覆盖原始 usage token/cost 字段)
- event_enrichment:构建结构化 settlement_snapshot(含 pricing_snapshot、billing_plan_snapshot、
  resolved_dimensions、cost_breakdown 等),写入 request_metadata
- pricing:新增 pricing_source() 方法区分 provider_override / global_default / unpriced
- sql.rs:settlement snapshot 写入新列;用量汇总查询改用 usage_billing_facts 视图
- maintenance/runtime:所有统计查询的 FROM usage 替换为 FROM usage_billing_facts
- admin observability:在 settlement 响应中暴露 settlement_snapshot / billing_dimensions,
  并从 metadata 中剥离对应字段
- request_metadata:允许 settlement_snapshot / billing_dimensions 字段传播
- stream execution:在首个数据帧到达时记录 TTFB,补全流式请求的 streaming 事件
2026-04-24 17:52:33 +08:00
Entropy.Xu fb46fcd80f feat(rust): 对齐 SMTP 测试与注册邮件 TLS 链路 (#325)
* feat(rust): 对齐 SMTP 连接测试链路

* fix(rust): 修复注册邮件 TLS provider 初始化
2026-04-24 14:59:00 +08:00
fawney19 e0c928fbe8 feat(auth): /me 接口返回 has_password 字段,前端 Profile 类型同步调整 2026-04-24 14:52:49 +08:00
fawney19 780f09c1a2 feat(billing): 引入 model_id 精确计费查找路径,传播模型 ID 至用量事件与 report context
- UsageEventData 新增 model_id / global_model_id 字段,write.rs seed 结构体同步补充
- report_context 新增 model_id / global_model_id / global_model_name,各 payload 构建时从 candidate 传入
- event_enrichment 优先按 model_id 精确查找计费上下文,回退为按名称多轮查找(保留 NoRule 结果降级逻辑)
- BillingReadRepository 新增 find_model_context_by_model_id,memory/sql 分别实现;SQL 查询重构支持按 provider_model_name 和 mappings 匹配并按优先级排序
- pricing.rs 修复:model_tiered_pricing 为空 tiers 时回退到 default_tiered_pricing
- request_metadata 允许字段列表补充 model_id / global_model_id / global_model_name
- admin usage 路由信息输出及脱敏字段列表同步新增三个 model 相关字段
2026-04-24 14:40:28 +08:00
fawney19 f3c9835759 feat(pool): 引入 pro_first 调度预设、Pool 候选持久化跳过与诊断信息优化
- 新增 pro_first 调度预设(Pro 优先),更新 plus_first 仅针对 Plus 计划,移除 free_team_first
- Pool 内部候选(pool_key_index 不为空)跳过 DB 持久化(available/skipped/unused 均适用)
- LRU 排序新增 catalog_lru_score 回退:runtime 无记录时使用 last_used_at_unix_secs
- 执行路径 miss 诊断消息细化为中文,按 reason 分类输出可读说明
- build_local_request_candidate_status_record 补充 extra_data 和 created_at_unix_ms 字段
- OpenAI CLI 计划构建流程补充候选评估进度跟踪与 terminal reason 设置
- 前端 PoolSchedulingDialog 增加 pro_first 预设展示,修复 LRU 默认预设检测逻辑
2026-04-24 13:29:05 +08:00
fawney19 f29649e3a8 feat(stream): 在流式执行中引入 StreamingStandardTerminalObserver 用量采集
- 在 execute_stream_from_frame_stream 中集成 StreamingStandardTerminalObserver,对流式响应逐行观察并在结束时合并终态摘要
- 新增 observe_stream_usage_bytes / finalize_stream_usage_observer / merge_stream_terminal_summary 辅助函数
- 更新 codex-cli 流式集成测试:补充 response.completed 含完整 usage 字段的 mock 数据,并断言用量写入 usage_repository
2026-04-24 11:34:41 +08:00
Dalamudandfawney19 35400b0c2d fix(aether-ai-pipeline): fix same-format Responses finalize output reconstruction (#323)
- rebuild final Responses output from streamed SSE events when terminal output is empty
- preserve authoritative completed output, multipart content ordering, reasoning/non-text parts, and final annotations
- update gateway finalize tests to match reconstructed output behavior

Co-authored-by: fawney19 <[email protected]>
2026-04-24 10:32:39 +08:00
fawney19 5d710d9d10 feat(rust): 支持 image 同步转流式 SSE、free_team_first 调度预设及账户错误自动重试
- openai:image 同步响应桥接为流式 SSE(image_generation.completed / image_edit.completed 事件)
- image 请求解析与前置校验移除模型白名单,支持任意自定义模型名
- 调度器新增 free_team_first 预设(mode: both / free_only / team_only)
- pool config 解析重构:新增 POOL_ALLOWED_SCHEDULING_PRESETS 白名单,规范化 mode 字段
- 错误分类器新增账户/账单错误模式集,升级为 RetryUpstreamFailure 而非 StopSemanticClientError
- 修复 stream execution 中上游 headers 与输出 headers 混用导致 content-type 判断错误的问题
- codex image 工具始终写入 action 字段(generate/edit),仅 generate 操作填充默认 size/quality
- 前端:pool 节点组只展示实际执行过的候选节点,全部 skipped 时折叠为最后一个节点
- Redis 测试就绪检测从 TCP 连接改为 PING/PONG 协议验证
2026-04-24 10:07:24 +08:00
AAEE86 a9d10163af fix(oauth): 修复 OAuth 刷新后 Token 有效期不更新问题 (#324)
- 持久化刷新后的 `expires_at` 到 Provider Key SQL 更新链路
- 手动刷新接口优先返回本次刷新得到的过期时间
- 按当前 Key 字段重建 OAuth 状态快照,避免旧快照覆盖
- 前端刷新后防止旧列表数据回退覆盖新有效期
2026-04-24 09:35:25 +08:00
Entropy.Xu 31e871fe1b fix(kiro): 修复流响应识别与号池配额刷新 Token 续期问题 (#322)
* fix(kiro): 兼容 application/json 头下的 eventstream 流响应

- stream_pump: 对带 kiro:generateAssistantResponse envelope 的上游响应按流处理,不再仅依赖 text/event-stream 头判断\n- tests: 补充 application/json 头下仍识别为 Kiro stream 的回归测试

* fix(kiro): 修复号池配额刷新误判封禁
2026-04-24 09:34:34 +08:00
fawney19 5148370253 fix(usage): 修复含失败信号的 pending 请求被误判为活跃的问题
- pending/streaming 状态含 status_code >= 400 或 error_message 时视为 failed
- 候选状态派生改为 failed 优先于 stale pending/streaming
- 活跃请求轮询时保留失败信号字段的更新
- 图片流支持 partial_image 事件转发及从 response.completed 读取最终图片
2026-04-24 03:07:11 +08:00
fawney19 581bc03d4e feat(ai-pipeline): 支持 reasoning signature 及媒体内容块的跨格式流式转换
- 在 CanonicalStreamEvent 中新增 ReasoningSignature 和 ContentPart 变体,CanonicalContentPart 枚举覆盖图片/文件/音频
- Gemini 流解析器提取 thoughtSignature,并对 inlineData/fileData 等非文本 part 生成 ContentPart 事件
- Claude 流聚合支持 thinking_delta / signature_delta,输出 thinking block 携带 signature 字段
- Gemini 同步响应聚合重写,支持媒体 part 和 reasoning signature 的完整还原
- 跨格式矩阵(gemini↔claude↔openai)补全图片块双向转换及 reasoning signature 传递
- 请求转换层(to/from openai_chat)补全 Claude/Gemini 的 thinking、图片、工具调用字段映射
- 新增/扩展测试:inline image 双向重写、thinking signature 聚合、跨格式 sync product 媒体字段
2026-04-24 02:18:34 +08:00
Entropy.Xu 0f94f92c37 fix(provider): 将rust分支的gemini cli端点行为对齐到python分支 (#321)
* fix(provider): 对齐 Vertex/Gemini 上游发包与 Python master

- provider-transport: 为 custom+aiplatform 推断 Vertex API key 上下文并统一 URL 构建顺序,复用共享 request_url 构建最终上游地址
- ai-pipeline/gateway: Vertex Gemini 路径改为仅使用 URL query key,不再向上游附带 x-goog-api-key header;同步对齐 standard/admin/test-connection/runtime miss 摘要中的最终 URL
- gemini conversion: 按 Python master 输出 Gemini 请求体,补齐 system_instruction / generation_config / tool_config / function_declarations 形态,并移植 Gemini schema 清洗逻辑
- scheduler/executor: 将最终 upstream_url、mapped_model、key_name 写入候选 extra_data,运行时 miss 诊断优先展示真实展开后的上游 URL 便于服务器排障

* fix(provider): 修复 Vertex provider 测试与本地调度链路

* fix(provider): 对齐 Vertex 本地执行与 Rust CI
2026-04-23 23:01:06 +08:00
AAEE86 ccec46eddd fix(admin-api-keys): 独立余额 Key 统计与时间字段改为直读 api_keys 表 (#319)
- 列表/详情/更新响应中的 `total_requests`、`total_tokens`、`created_at`、`last_used_at` 统一直接读取 `api_keys` 导出记录字段
- 保持“已消费”继续走钱包字段(不改消费口径)
- 在 auth 导出记录中补充 `last_used_at_unix_secs`、`created_at_unix_secs`、`updated_at_unix_secs`
- 扩展 auth SQL 查询与 RETURNING 字段映射,确保时间字段完整回传
- 移除 admin api-keys 路由中对 usage summary token 聚合的依赖
- 更新 admin api-keys 控制层测试,校验 token/时间字段来源与格式
2026-04-23 22:32:00 +08:00
Entropy.Xu ad58f4e852 fix(kiro): 修复 Kiro thinking 流重写按字节截断导致的 UTF-8 边界 panic (#320)
- events.rs: 保留 <thinking>/</thinking> 尾部窗口时改为按 UTF-8 字符边界拆分,避免中文流式 chunk 在 String 切片时 panic
- tests.rs: 抽取 Kiro report_context 构造并补充普通文本/思维块内多字节内容回归测试
2026-04-23 22:21:27 +08:00
fawney19 bb312ff0cf chore(tsconfig): 移除冗余的 baseUrl 配置 2026-04-23 22:05:02 +08:00
fawney19 342d4a268c feat(stream-bridge): 支持上游 sync 响应转 SSE 流式输出,记录 client/upstream 流模式差异
- 新增 sync_to_stream 桥接模块,将非 SSE 上游响应转换为 SSE 格式回传给流式客户端
- stream_pump 检测非 SSE 响应头后缓冲整包并通过桥接逻辑重写为 SSE 帧
- proxy handler 同步支持 sync→stream 聚合与转换(覆盖 openai/claude/gemini 四种格式)
- sync_products 补全 openai:cli 的完整流式事件聚合(text delta、reasoning、tool call 等)
- usage runtime 写入 client_requested_stream / upstream_is_stream 到 request_metadata
- SQL 查询层将两个布尔字段从 request_metadata jsonb 中提取并回传给前端
- 前端 status.ts 新增 resolveUsageStreamLabelSegments,优先读取 client_requested_stream
- RequestDetailDrawer 在流式转换场景下显示"客户端→上游"两段 Badge
2026-04-23 21:53:50 +08:00
fawney19 40282c3447 chore(migrations): 合并 ephemeral cache 迁移到 20260423000000,移除冗余迁移文件并同步 baseline cutoff 2026-04-23 14:53:54 +08:00
fawney19andEntropy.Xu fa328e18a1 feat: provider api_formats 可空继承、OpenAI 图片 edit/variation 与用量配额多项补强
- 鉴权: provider_api_keys.api_formats 改为可空,OAuth 托管 key 自动继承 provider endpoints 激活格式,相关 handler/测试同步更新
- 图片 planner: OpenAI 图片路由新增 edit/variation 操作并完善参数校验、响应合并与流式处理
- 用量: user me usage 返回区分 client_requested_stream/upstream_is_stream,前端 usage 列表筛选与展示增强
- 统计: stats_daily_model 新增 cache_creation_ephemeral_5m/1h tokens 字段与回填链路
- 配额/observability: quota repository 新增内存与 SQL 扩展,admin observability usage 字段扩充
- 其它: OAuth 导入/轮询收敛、provider 汇总与 pool admin 读写链路小修、新增 system_config 缓存与 provider template handler

Closes #318

Co-authored-by: Entropy.Xu <[email protected]>
2026-04-23 14:42:51 +08:00
Entropy.Xu f55f22d2e8 feat(codex-image): 封装 GPT Image 2 图片接口并收紧错误处理
- 新增 openai:image 路由、planner 与 finalize,内部通过 Codex responses image_generation tool 执行生图

- 补充 Codex OAuth/header 兼容、图片 success report 本地处理与相关前后端/集成测试

- 禁止 chat/completions 使用 gpt-image-2,图片接口限制 n=1,并移除 Provider 模型页的图片能力开关
2026-04-22 22:46:28 +08:00
fawney19 4374f53315 fix(adaptive): 修复 429 计数归零写入及 reset 语义
- sql.rs: UPDATE 时对 concurrent_429_count / rpm_429_count 加 COALESCE(, 0),避免 None 覆写为 NULL
- effects.rs: rpm_429_count 始终写入 Some(projection.rpm_429_count),不再过滤零值;新增 unknown-429 场景下零值持久化的单元测试
- adaptive.rs: reset 接口将两个计数字段置为 Some(0) 而非 None,与数据库默认语义对齐;同步修正集成测试断言
- docker-compose.build.yml: 补充 AETHER_GATEWAY_AUTO_PREPARE_DATABASE 默认开启
- README.md: 同步说明 build compose 也已默认开启自动迁移
2026-04-22 22:16:51 +08:00
fawney19 c8d7dbd8d6 refactor(gateway): 将 CF 头剥离中间件下移至各 router 构建函数并重构为前缀匹配 2026-04-22 21:10:16 +08:00
fawney19 cf6228f525 feat(stats,rules): api_key 用量统计(total_tokens 字段 + 回填)与 body rules 新增 append/insert/regex_replace/name_style 操作
- 新增迁移 20260422120000_add_api_key_usage_stats.sql,为 api_keys 表添加 total_tokens 列
- 新增回填 20260422120000_backfill_api_key_usage_stats.sql,按历史 usage 重建 api_key 维度汇总
- 在 UsageWriteRepository trait 中添加 rebuild_api_key_usage_stats,补齐 SQL/内存实现及上层调用链
- 内存实现中新增 apply_usage_stats_delta,支持增量更新 api_key 统计快照
- dev.sh:改进临时日志目录管理,并在网关异常退出时输出错误提示
- frontend EndpointFormDialog:将 append 操作从 insert 分支拆分,提供独立 path/value 输入 UI
- rules.rs:扩展 body rules 支持,新增 append/insert/regex_replace/name_style 操作及 WildcardSlice 路径段
2026-04-22 20:16:57 +08:00
fawney19 62153d7d36 feat(gateway): 新增 --auto-prepare-database 启动选项并在 docker-compose 默认开启
- aether-gateway 启动时可自动执行挂起的 migration 与 backfill
- docker-compose.yml 默认开启 AETHER_GATEWAY_AUTO_PREPARE_DATABASE
- 同步更新 .env.example、README 与 deploy.sh 的部署说明
2026-04-22 18:20:24 +08:00
fawney19 a5e6bd3b62 feat(stats): 新增聚合读路径与回填机制并重构 dashboard/usage 读取链路
- 新增 stats_user_summary 及 user_daily_provider/api_format/cost_savings 等聚合表
- 扩展 stats_daily/hourly 有效 token 与响应时间等字段,maintenance runtime 同步写入
- 新增 backfill 模块与 --apply-backfills 命令补齐历史聚合数据
- 重写 dashboard_filters、usage_heatmap、user_rollups 查询改走聚合表
- 同步更新 baseline_v2.sql 与 migration 集,README/dev.sh 补充回填用法
2026-04-22 17:29:39 +08:00
fawney19 063ef02306 fix(oauth): 系统导入时清理失效标记并强制刷新 refresh_token,前端徽章优先使用 account_id
- 后端 admin 系统导入 OAuth 凭据时重置 expires_at 与失效标记,并在存在 refresh_token 时触发一次本地刷新
- 前端 OAuth 徽章改为优先展示 account_id,采用去前缀后 5 字符的紧凑格式
2026-04-21 18:06:39 +08:00
fawney19 25a2b417be refactor: 优化调度候选排序与用量写入链路并改进 Fernet 缓存与前端批量列表 2026-04-21 16:19:07 +08:00
fawney19 c5c56ff92f feat(oauth): 允许替换已失效的活跃 OAuth 账号并同步 status_snapshot
- 活跃但 token 已过期或刷新失败的重复账号视为可替换
- 清除失效标记、刷新配额时同步更新 status_snapshot.oauth
- oauth_invalid 清除接口同时识别 invalid_at 与 invalid_reason 两种标记
- 批量导入任务状态区分 created_count / replaced_count,前端据此展示新增/替换统计
- 补充重复替换场景的集成测试,用量测试等待超时从 10s 提升到 30s 以适应并行压力
2026-04-20 22:59:02 +08:00
RWDai cf7d129595 fix(proxy-nodes): allow management tokens with json null allowed_ips (#317)
* fix(proxy-nodes): allow management tokens with json null allowed_ips

* style(proxy-nodes): format regression test
2026-04-20 21:50:48 +08:00
fawney19 226a6e58d5 refactor(gateway): 重构格式转换候选资格检查并优化测试基础设施
- 将 format_conversion_disabled 的过滤提前到候选遴选阶段,替代原来的 skip_reason 标记机制
- 为测试添加 execution_runtime_sync_override 直接注入支持,避免启动额外 HTTP 服务器
- 将 submit_terminal_event 改为 async record_terminal_event 确保失败用量事件可靠入库
- 新增 test_support 模块封装可重试的 loopback 端口绑定逻辑
- 前端:poolTrace 将 skipped 从隐藏状态移除,新增 buildPoolParticipatedCandidates 统一新旧链路逻辑,并将 skipped 状态色改为 foreground
2026-04-20 16:59:18 +08:00
fawney19 87afe4898e fix(frontend): 为配额自动刷新加入 5 分钟冷却并优化 OAuth org 徽章显示
- 新增 quotaAutoRefreshCooldown 工具,按 provider 维度限制后台自动刷新频率,手动触发(刷新 token、变更 key)通过 ignoreCooldown 绕过
- OAuth org 徽章去除 org- 前缀后再缩略,避免 org:org-xx 的重复前缀,并缩小字号高度以适配紧凑布局
2026-04-20 15:56:59 +08:00
fawney19 d100c934c0 fix(frontend): 优化 OAuth 身份徽章优先级并避免 Kiro 订阅标签重复
- getOAuthOrgBadge 优先展示 organization id, 其次回落到 account_id 或 account_user_id, 不再使用 account_name 作为徽章文案
- Kiro provider 的订阅标签仅在与 OAuth plan 标签不一致时显示, 避免重复徽章
- 补充 getOAuthOrgBadge 的单元测试覆盖组织优先与回落场景
2026-04-20 15:00:17 +08:00
fawney19 ffe5d16094 fix(oauth): 兼容 Kiro device token 的 snake_case 字段并保留 error 响应体
- device poll 同时接受 accessToken/access_token、refreshToken/refresh_token、expiresIn/expires_in
- post_kiro_device_oidc_json 在解析失败响应时保留完整 JSON 并附加 _error 标记, 避免丢失 authorization_pending 等详细信息
- 补充 authorization_pending 时 device session 保持 pending 状态的测试
2026-04-20 15:00:02 +08:00
Dalamud 078347b722 fix(aether-crypto): fix provider key decryption for legacy Python Fernet secrets (#316) 2026-04-20 14:31:54 +08:00
fawney19 9b48a008dc fix(frontend): 合并 OAuth 状态展示并移除 Codex 积分信息
- providerKeyStatus: 合并 snapshot 与 legacy OAuth 状态,按严重程度取优并补全失效原因
- ProviderDetailDrawer: 移除 Codex 积分摘要与相关字段显示
2026-04-20 14:27:48 +08:00
AAEE86 b2ff7d2c28 fix(wallet): 删除与过期删除 API Key 时同步禁用关联钱包 (#315)
- 在 delete_user_api_key 和 delete_standalone_api_key 事务中,先将关联 wallets.status 更新为 disabled
- 在过期 Key 自动清理流程中,删除前同步禁用对应钱包
- 保留 wallet 与交易流水,避免出现 orphaned 且 active 的钱包状态
2026-04-20 14:03:17 +08:00
fawney19 13bf222b8d refactor(oauth): 抽出 Kiro 刷新逻辑为共享模块并在 device-poll 复用
- 将 batch/kiro_import 中的 Kiro refresh/IDC helpers 抽到 dispatch/kiro.rs 共享
- device-poll 新增 IDC refresh 复核、JWT 缺失时通过 usage 接口回填 email、代理节点写入 key.proxy
- 补充对应集成测试覆盖 refresh 复核与 email 回填路径
2026-04-20 01:06:51 +08:00
fawney19 c2b7b6c231 refactor(proxy-nodes): 移除节点列表的 rollout 升级标签展示 2026-04-20 00:31:30 +08:00
fawney19 c1b9d94c84 feat(adaptive): 完善自适应 RPM 学习并将 Pool 调度状态与健康分解耦
- orchestration 新增 AdaptiveSuccess 效果,在成功回报路径上根据利用率窗口扩张 learned_rpm_limit
- 429 路径改用 429_observation/adjustment 记录以及基于历史的置信度评估,新增 last_rpm_peak 边界字段
- Pool 调度状态不再因 health_score 低或熔断而降级/拦截,前端同步移除相关按钮与文案兜底
- 新增前端 poolTrace 工具(附测试)承接原 HorizontalRequestTimeline 内的候选合并逻辑
2026-04-20 00:27:48 +08:00
fawney19 1f74e660de feat(transport): 暴露同格式 provider 的细分失败原因并在时间线展示
- claude_code/kiro/vertex policy 新增 transport_unsupported_reason_with_network 变体,返回具体失败码
- planner candidate_metadata 在 request_pair 写入同格式 provider 的细分 transport 原因
- 前端请求时间线优先展示 transport_diagnostics 的细分原因而非通用 transport_unsupported
2026-04-19 21:45:57 +08:00
fawney19 77aac74590 feat(oauth): 完善账号异常识别并在调度/展示层拦截失效 OAuth 密钥
- 新增 aether-admin provider status 模块,统一解析账号状态(禁用/工作区停用等)
- 调度器 runtime 增加 oauth_invalid 判定,跳过刷新失败或已撤销的 OAuth 密钥(REQUEST_FAILED 保留可选)
- gateway state 在 local oauth 刷新返回 4xx 时持久化失败原因并同步状态快照
- admin pool 列表/详情回填 account 状态与 scheduling 阻塞原因(account_blocked)
- 共享 catalog 的 status_snapshot payload 附加 account 字段
2026-04-19 20:50:31 +08:00
fawney19 d719a1329c fix(usage): cache-affinity 间隔查询将 interval_minutes 强制转为 DOUBLE PRECISION 2026-04-19 16:32:29 +08:00
fawney19 c302dfe42d fix(usage): 完善 cache-affinity 时间线的用户名回退与模型空值兜底
- 当 auth 用户查询失败时回退到历史 username,避免时间线丢失用户信息
- SQL 投影对 usage.model 使用 COALESCE 兜底空值
- IntervalTimelineCard 在刷新间隔为 0 时跳过定时刷新
- Usage 页面关闭时间线卡片的自动轮询
2026-04-19 16:07:01 +08:00
fawney19 41b51f10a9 perf: 并行化 admin 聚合路由并完善前端缓存预取
- gateway: usage detail / provider summary / pool overview / users list 改为 tokio join 并行拉取依赖数据
- usage: interval timeline 支持自动刷新并按查询区间动态展示,取消服务端 120 分钟过滤并在 ScatterChart 统一封顶
- frontend: 新增管理端导航预取工具及 SidebarNav/MainLayout 触发,admin 读接口统一走 cachedRequest 的短期缓存
- dashboard: request detail 支持短 TTL 缓存并在 UsageRecordsTable mousedown 时预取
- data: migrate 测试在 wait_for_postgres 失败时清理子进程,避免遗留
2026-04-19 15:17:25 +08:00
fawney19 97cd877ce5 style(proxy-nodes): 统一缩进并移除未使用的 confirmWarning 引用 2026-04-19 13:46:21 +08:00
fawney19 7007d7a556 refactor(proxy-nodes): 简化批量升级为直接写入 upgrade_to 目标
- 后端移除分波 rollout/探测逻辑,改为一次性给所有合格 tunnel 节点写入升级目标,并自动取消活动中的 rollout
- 前端移除 rollout 进度、阶段筛选、重试/跳过/冲突清理等相关 UI 和交互
- 同步更新批量升级接口类型与测试用例
2026-04-19 13:38:59 +08:00
fawney19 e6fd95453d test(proxy-nodes): 将离线隧道用例的连通性探测指向 Cloudflare 以避免环境抖动 2026-04-19 04:03:37 +08:00
fawney19 0329b7b784 feat(usage): 在 CanonicalUsage 中追加缓存 token 字段并在标准化流程透传
- 扩展 CanonicalUsage 支持 cache_creation/cache_read 及 5m/1h 明细
- OpenAI/Claude/Gemini 解析器提取缓存 token 并计入 total
- usage_mapper 补齐 Claude 缓存字段映射
- 新增 gateway pricing 集成测试覆盖三家同步/流式缓存计费
2026-04-19 03:55:55 +08:00
fawney19 2f487fda66 fix(usage): 修复缓存创建 token 展示并在总量中计入缓存组件
- gateway/admin 用量载荷在 cache_creation_input_tokens 为 0 时回填 ephemeral_5m/1h 合计
- 标准化用量写入时将 cache_creation/cache_read token 计入 total_tokens
- 前端列表与轮询同步新增分类字段,修复缓存 token 列显示
2026-04-19 02:46:28 +08:00
fawney19 e82c9c5104 fix(usage): 容错缺列投影以支持旧版和部分升级场景
map_usage_row 中对 http_audit/routing/settlement 等新增投影列改用
row_try_get_optional,将 ColumnNotFound 视同 NULL,避免仅选取核心字段
的读取路径在列缺失时整行失败。同时补充 UPSERT 和 LIST 前缀的占位列
断言。
2026-04-19 01:48:16 +08:00
github-actions[bot] c31261789c chore(proxy): update download links for proxy-v0.3.2 2026-04-18 17:30:37 +00:00
AAEE86 6666992d01 feat(admin): 将用户管理的用量统计改为全量累计口径 (#314)
- 新增按用户 ID 批量汇总 usage 的后端查询
- 在 /api/admin/users 中返回 request_count 和 total_tokens
- 移除用户管理页面额外的 usage 聚合请求
- 为管理端用户列表补充累计统计回归测试
2026-04-19 01:26:22 +08:00
fawney19 ab54881eb0 ci: 固定 macOS runner 版本并确保 Rust target 安装 2026-04-19 01:21:21 +08:00
fawney19 f8545c5141 chore: bump aether-proxy version to 0.3.2 2026-04-19 01:04:55 +08:00
AAEE86andfawney19 bafb63a665 持久化 provider key 使用统计并移除 usage 汇总覆盖 (#313)
* 持久化 provider key 使用统计并移除 usage 汇总覆盖

- 在 usage upsert 时按请求前后差值同步 provider_api_keys 统计
- 增加基于保留 usage 记录的 provider key 统计重建能力
- 号池管理列表直接读取 provider_api_keys 统计字段

* fix: harden provider key usage stats sync

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-19 01:02:09 +08:00
RWDaiandfawney19 425227509a Fix/provider query cli model tests (#312)
* fix(admin): support openai cli provider-query model tests

* fix(admin): support claude and gemini cli provider-query model tests

* fix(admin): preserve provider-query prompts on cli fallback

* test(usage): relax async status wait for local usage checks

* fix(gateway): align provider-query CLI auth and headers

* fix(gateway): resolve provider-query clippy lint

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-18 23:05:39 +08:00
fawney19 f2a3836877 Improve tunnel proxy diagnostics and request body spooling 2026-04-18 21:13:37 +08:00
fawney19 3363592751 Refactor usage body capture and stream terminal reporting 2026-04-18 17:48:21 +08:00
fawney19 569242d72f refactor gateway orchestration and failover effects 2026-04-18 11:35:11 +08:00
fawney19 3321bb3ccc Implement independent provider pool scheduling runtime 2026-04-17 23:05:49 +08:00
RWDaiandfawney19 dd4641d618 Fix/provider query non kiro tests (#311)
* fix(admin): enable local provider-query tests for non-kiro providers

* test(admin): cover non-kiro provider-query model execution

* fix(admin): preserve provider-query test errors and failover retries

* fix(admin): handle provider-query test alias and HTTP retry edges

* fix(admin): extend provider-query failover coverage and fallback behavior

* fix(admin): prefer supported endpoints for provider-query tests

* fix(admin): prefer provider-query endpoints with compatible keys

* fix(admin): fall back to compatible provider-query endpoints

* fix(admin): align provider-query local tests with transport policy

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-17 19:45:20 +08:00
fawney19 0ce61bc91c fix(gateway): align provider restrictions with provider catalog 2026-04-17 18:59:13 +08:00
fawney19 cb647d95f9 Hide format_conversion_disabled skips when exact-format candidate shares the same key
Also switch failover test to surface an auth failure instead of a 502 upstream error.
2026-04-17 18:24:36 +08:00
fawney19 7eae1f90f6 Unify quota snapshots and oauth refresh handling 2026-04-17 18:22:41 +08:00
RWDaiandfawney19 b8702ae124 Fix/api key concurrency runtime miss (#309)
* test(cli): 覆盖 API key 并发等待与超时路径

* feat(scheduler): API key 并发饱和时等待可用槽位

* fix(proxy): 区分 API key 并发受限与真正的 runtime miss

* fix(outcome): runtime miss 仅归因真实执行候选

* feat(api-keys): 统一 concurrent_limit 默认值与校验辅助

* feat(admin): 独立 Key 接口支持 concurrent_limit

* feat(admin): 用户 API Key 路由支持 concurrent_limit

* feat(public): 自助 API Key 路由支持 concurrent_limit

* feat(import): 导入与存储层持久化 concurrent_limit

* feat(frontend): 同步 API Key concurrent_limit 类型定义

* feat(frontend): 独立 Key 表单支持 concurrent_limit

* feat(frontend): 管理员用户 API Key 表单支持 concurrent_limit

* feat(frontend): 自助 API Key 页面支持 concurrent_limit

* chore(fmt): 统一 runtime 归因相关 Rust 格式

* chore(fmt): 统一 admin API key 路由 Rust 格式

* chore(fmt): 统一 public 路由与相关测试 Rust 格式

* fix(test): 对齐 no-execution usage 归因断言

* test(middleware): 固定 access log tracing 用例线程模型

* fix(frontend): 提取用户 API Key payload 默认并发辅助

* fix(frontend): 保留用户 Key 的 concurrent_limit 默认值

* fix(api-keys): remove hardcoded concurrent limit default

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-17 14:21:43 +08:00
AAEE86andfawney19 e5d3722adf fix(admin): force manual oauth refresh to bypass local cache (#308)
Co-authored-by: fawney19 <[email protected]>
2026-04-17 14:12:15 +08:00
RWDaiandfawney19 ff4e853fd3 Fix/usage transfer filter (#307)
* fix(usage): 恢复 usage 列表中的 fallback 路由信号

* fix(admin): 支持 usage 记录展示和筛选 fallback 转移

* fix(usage): 在用户 usage 记录中暴露 fallback 标记

* fix(usage): 共享 usage 页面支持 fallback 筛选

* fix(usage): 对齐 fallback 筛选相关前端类型

* fix(usage): propagate has_fallback through active polling

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-17 13:38:23 +08:00
fawney19 654a41c3b0 Revert "fix(admin): force manual oauth refresh to bypass local cache"
This reverts commit 85a630cfa9.
2026-04-17 13:36:51 +08:00
Entropy.Xuandfawney19 c6af4791f8 fix(oauth): 对齐 OpenAI 回调 state 解析与提交流程 (#306)
Co-authored-by: fawney19 <[email protected]>
2026-04-17 13:26:23 +08:00
fawney19 85a630cfa9 fix(admin): force manual oauth refresh to bypass local cache 2026-04-17 13:23:51 +08:00
Entropy.Xuandfawney19 ac1a126756 fix(kiro,pool,model): 对齐 Kiro 管理链路并修复全局模型删除行为 (#305)
* feat(pool): 号池支持跳过额度耗尽账号

- 新增 pool_advanced.skip_exhausted_accounts 开关及高级设置 UI, 默认关闭并兼容旧配置
- 为 Codex/Kiro 增加额度耗尽判定, 接入请求侧候选跳过并新增 account_quota_exhausted skip reason
- 号池列表将额度耗尽账号标记为 blocked/额度耗尽, 并补充前后端相关测试

* fix(kiro): 对齐账号管理与 provider-query 的 Rust 行为

- 修复 Kiro 单条导入误走 import-refresh-token 的前端分流, 并为误用路径返回明确错误提示
- 为 Kiro 导入与本地请求链补齐 bearer 兼容, 同步放开账号启停等 Key 更新操作的 auth_type 校验
- 实现 Kiro provider-query 本地模型测试与 failover 执行链, 并修复结果弹窗在无 trace 时无法展示 attempts/响应体的问题

* fix(model): 删除全局模型时级联清理关联提供商模型

- 对齐 Python 版本删除逻辑, GlobalModel 删除前先在事务内清理关联的 Provider Model 记录
- 修复已绑定 Provider 的模型在 Rust SQL 仓库下会被外键约束拦住、无法正常删除的问题
- 增加管理端回归测试, 覆盖绑定 Provider Model 的 GlobalModel 删除场景

* fix(kiro,ci): 恢复 Kiro OAuth 持久化并修复 Rust CI

* Fix oauth-managed provider key semantics

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-17 12:57:06 +08:00
RWDaiandfawney19 96a25d058b Fix OpenAI family local auth to use bearer (#302)
* Fix OpenAI family local auth to use bearer

* test(gateway): fix bearer auth assertions for openai local flows

* test(usage): make local usage status wait resilient

* style(gateway): apply rustfmt to usage test helper

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-17 11:10:05 +08:00
Entropy.Xuandfawney19 6964729cb7 feat(payments): 增加兑换码与支付适配框架 (#299)
* feat(payments): 增加兑换码与支付适配框架

* fix(ci): 对齐 Rust 1.95 lint 与格式要求

* fix(payments): harden redeem code wallet credits

---------

Co-authored-by: fawney19 <[email protected]>
2026-04-17 10:07:52 +08:00
fawney19 54d77598ae ci(rust): 将环境变量提升至全局,各 job 添加 toolchain 显示步骤 2026-04-17 01:58:12 +08:00
fawney19 0bcfc7d352 fix(ci): 将 rustfmt/clippy 组件安装改为独立 rustup 步骤 2026-04-17 01:53:59 +08:00
fawney19 faaaec28e1 fix(clippy): 修复冗余 into_iter、sort_by 及 match 嵌套 if 等 lint 警告 2026-04-17 01:43:45 +08:00
fawney19 dde02e6111 fix(ci): 恢复 rust-toolchain@stable,版本由 rust-toolchain.toml 控制 2026-04-17 01:17:29 +08:00
fawney19 aadc6b665c chore: 锁定 Rust 版本为 1.95.0,统一本地与 CI 环境 2026-04-17 01:15:16 +08:00
fawney19 6730e821b2 fix(clippy): 修复 collapsible_match 和 unnecessary_sort_by 警告 2026-04-17 01:12:43 +08:00
fawney19 05ab09c469 fix(data): 修复 global_models 插入时缺少 usage_count 字段导致的列数不匹配问题 2026-04-17 01:09:14 +08:00
fawney19 1e0bc61526 feat(gateway/data): 新增 usage 关键词搜索、缓存命中摘要、结算成本摘要及 dashboard 聚合查询能力
- 新增 UsageAuditKeywordSearchQuery,支持多关键词、用户名、API Key 交叉过滤
- 新增 UsageCacheHitSummaryQuery/StoredUsageCacheHitSummary,统计请求缓存命中率
- 新增 UsageSettledCostSummaryQuery/StoredUsageSettledCostSummary,汇总结算成本
- 新增 UsageDashboardSummaryQuery、UsageBreakdownSummaryQuery 等 dashboard 聚合类型
- SQL 层实现对应查询方法,含 list_by_ids、list_usage_audits_by_keyword_search、count_usage_audits_by_keyword_search
- 将上述能力通过 GatewayDataState / AdminAppState 暴露给 handler 层
- user_me_usage 及 dashboard_filters 切换为新查询接口,移除旧的内存过滤逻辑
- 同步更新 memory 层及测试
2026-04-17 01:00:38 +08:00
github-actions[bot] 6e5af5ef70 chore(proxy): update download links for proxy-v0.3.1 2026-04-16 12:41:35 +00:00
fawney19 d312c397e1 chore: bump aether-proxy version to 0.3.1 2026-04-16 20:34:02 +08:00
fawney19 0dce667019 feat(tunnel/usage): proxy writer 双优先级队列、hub 连接压力感知选择、usage 请求记录级别控制及 trace 页面 proxy timing 增强 2026-04-16 20:14:50 +08:00
fawney19 65cd9dc3e5 fix(admin): 修复 usage 观测全表扫描并下推聚合查询
- 收紧 admin usage/stats 默认时间范围和 active 轮询查询
- 将 summary/records/aggregation/time-series/leaderboard 下推到 SQL 侧
- 统一前端时间参数并补齐 admin usage/stats 回归测试
2026-04-16 17:46:19 +08:00
AAEE86 10605d9fb0 fix(admin): 同步 OAuth 刷新后的状态快照有效期 (#300)
手动刷新 OAuth Token 后同步回写 status_snapshot.oauth
修复号池管理页面仍显示旧有效期的问题
补充刷新后快照更新的回归测试
2026-04-16 14:37:59 +08:00
AAEE86 af9711c2a2 fix(pool): 修复 Codex 配额倒计时不准确 (#297)
* fix(pool): 修复 Codex 配额倒计时不准确

- 后端号池 keys payload 增加 upstream_metadata 透传
- 前端补充 PoolKeyDetail.upstream_metadata 与 Codex reset_after_seconds 类型
- 号池页倒计时优先使用 reset_at/reset_seconds/updated_at 结构化数据计算
- 仅在缺少结构化字段时回退 account_quota 文案解析

* fix(pool): 将已重置的 Codex 配额恢复为 100%

- reset_after_seconds/reset_seconds 会按 updated_at 扣减经过时间(不再当作静态值)
- 若窗口已重置(剩余秒数 <= 0),该窗口 used_percent 按 0 处理
2026-04-16 13:34:41 +08:00
fawney19 9a41b2c0dc feat: 候选排序引入 API 格式偏好, 追踪页面展示完整格式转换信息
- 候选排序优先同 kind (chat/cli) 再同 family, 替代原有固定顺序
- 不再隐藏 format_conversion_disabled 候选, 保留完整追踪链路
- DecisionTrace 新增 provider/endpoint/key 格式转换相关字段
- 前端追踪面板新增 Key 支持端点和转换策略展示
2026-04-16 13:00:44 +08:00
fawney19 d805a28c9a feat: 请求候选追踪添加 proxy 元数据, 修复 usage 状态回退, 优化前端轮询
- 在各 planner decision payload 中注入 proxy trace 信息 (node_id, node_name, url, source)
- request_candidate 报告上下文支持 proxy 字段, extra_data 合并逻辑改为 merge 而非覆盖
- SQL/内存仓库防止 usage status 从 streaming 回退到 pending
- 前端移除活跃请求完成时的全表刷新, active discovery 尊重 globalAutoRefresh 开关
2026-04-16 11:57:42 +08:00
fawney19 ffe34120c1 tune(tunnel): 放宽 ping 间隔和超时参数, 增大出站队列容量
- gateway ping 间隔 500ms -> 15s, 出站队列 128 -> 512
- proxy ping 间隔 1s -> 10s, 连接超时 1.5s -> 3s, stale 超时 5s -> 30s
2026-04-16 11:05:44 +08:00
fawney19 47a11ee0b5 feat(tunnel): 添加流帧分发超时机制和结构化请求日志
- dispatcher: 为 stream handler 的帧接收添加超时保护, 防止单个
  handler 阻塞 WebSocket 读循环; 超时后发送 StreamError 并清理流
- stream_handler: 在所有请求完成和错误路径添加结构化日志, 记录
  method/host/path/status/duration 等关键信息
- 将常规隧道连接/断开日志从 info 降级为 debug, 减少日志噪音
2026-04-16 10:32:05 +08:00
fawney19 28a489acbe fix(tunnel): writer 退出时立即标记连接关闭, 防止后续请求堵塞
- ws_tx.send 添加 15s 超时, 防止 TCP 背压导致 writer 无限阻塞
- writer 退出时立即 request_close(), 不等 reader 结束即从路由表摘除连接
- ws_tx.close 添加 5s 超时, 防止 close 握手阻塞
2026-04-16 02:29:36 +08:00
fawney19 7597caa3a5 修改日志格式 2026-04-16 02:13:27 +08:00
fawney19 586d2cc42b fix(tunnel): 为 hub reader body 推送添加超时, 防止流间 head-of-line blocking
push_body_chunk 原先使用无超时的 channel send, 当消费端慢时会阻塞
整条 proxy 连接的 reader, 导致同连接上其他 stream 的帧无法路由。
添加 5 秒超时后, 单个 stream 的背压不再影响其他 stream。
2026-04-16 02:11:32 +08:00
fawney19 1fd1f8216d fix(gateway): 补充 example 中缺失的 idle_timeout 字段 2026-04-16 01:14:49 +08:00
fawney19 0c3f730852 fix(testkit): 补充 TunnelConnConfig 缺失的 idle_timeout 字段 2026-04-16 01:09:53 +08:00
fawney19 b678132176 fix(tunnel): 默认禁用 proxy idle timeout (设为 0)
保留 idle timeout 机制但默认不启用,避免误断连。
2026-04-16 00:56:24 +08:00
fawney19 e4be1d6b56 Revert "refactor(tunnel): 移除 proxy 连接 idle timeout 机制"
This reverts commit b004551fe5.
2026-04-16 00:53:40 +08:00
fawney19 5f0fba1807 diag(tunnel): writer/reader 帧计数诊断日志
- writer task 记录发送帧数、Binary 帧大小、send 失败原因
- reader 记录接收帧总数,disconnect 时输出
- 用于定位 gateway→proxy 数据流是否正常
2026-04-16 00:31:10 +08:00
fawney19 205f78b39c style: cargo fmt 2026-04-15 23:54:14 +08:00
fawney19 5aa8883865 fix(tunnel): gateway 回复 WebSocket Pong 防止 proxy stale 断连
gateway reader 之前用 `_ => {}` 忽略了 proxy 发来的 WebSocket Ping,
导致 proxy 的 stale_timeout (5s) 持续触发断连重连循环,
所有隧道请求都因等待 response headers 超时而失败。

同时补充 hub 诊断日志和连续请求集成测试。
2026-04-15 23:45:22 +08:00
fawney19 1e2c3fc4fc fix(tunnel): 移除 testkit 和 example 中残留的 idle_timeout 引用 2026-04-15 22:18:41 +08:00
fawney19 b004551fe5 refactor(tunnel): 移除 proxy 连接 idle timeout 机制
idle timeout 在实际使用中容易误断活跃连接,移除该逻辑并简化 reader 循环。
2026-04-15 22:10:30 +08:00
fawney19 704858390d feat(gateway): 流式执行支持 local tunnel 传输并改进 SSE 错误通知
- 流式执行优先尝试 local tunnel 路径,不可用时降级到直连
- stream_pump 适配 Reqwest 和 LocalTunnel 双响应类型
- SSE passthrough 流中断时向下游发送 aether.error 终端事件
- 提取 frame 编码辅助函数消除重复代码
- 新增本地隧道流式场景的集成测试
2026-04-15 21:10:16 +08:00
fawney19 c569081340 fix(usage): streaming 状态在首字节到达前显示为 pending
新增 resolveDisplayRequestStatus 函数,当记录状态为 streaming 但
first_byte_time_ms 为空时回退显示为 pending,避免误导用户。
2026-04-15 17:50:45 +08:00
fawney19 77d413d777 tune(tunnel): 调整 tunnel 超时与连接池参数,增加冗余连接下限
- Gateway: 增大 idle timeout 至 2s、ping interval 至 500ms
- Proxy: 统一 reconnect/ping/stale 超时为常量,降低 reconnect_max 至 250ms
- 自动连接池引入 redundant floor(2),保证低负载时也有冗余连接
2026-04-15 16:27:42 +08:00
fawney19 43e7ad112f perf(usage): user-me usage 查询避免全量加载,默认限制 7 天范围
- UsageAuditListQuery 新增 limit 字段,支持在 SQL 层 LIMIT
- user-me usage 端点无时间范围时默认查最近 7 天
- active usage 端点无 ids 过滤时限制查最近 1 小时
- InMemory 实现同步支持 limit truncate
2026-04-15 16:25:57 +08:00
fawney19 fbb8249c0d fix(gateway): 改进流式传输稳定性
- stream_pump 读取错误时记录完整错误链并输出 warn 日志
- hub body 转发从 try_send 改为 async send,支持背压避免丢帧
- tunnel_stale_timeout 默认值从 900ms 提升到 10s,减少误判过期
2026-04-15 14:26:57 +08:00
fawney19 707d9ac274 perf(gateway): 流式处理代码层性能优化
- 将 spawn 内 buffered_body 从 Vec<u8> 改为 VecDeque<u8>, drain 前端从 O(n) 变 O(1)
- 消除 prefetched body/chunks 的不必要 clone, 改为直接 move
- Pending/Streaming 非终态候选状态写入改为 tokio::spawn fire-and-forget
- 连接池默认 max_connections 从 30 提升到 50
2026-04-15 11:55:45 +08:00
fawney19 026a77306c fix(schema): 移除 api_keys.concurrent_limit 的默认值 5
之前 DEFAULT 5 导致所有新建 API Key 自动带上并发限制,
改为 NULL (不限制) 以匹配预期行为。
2026-04-15 10:43:38 +08:00
fawney19 4087e096f2 fix(usage): 为 InMemory 仓库补充 status 过滤逻辑 2026-04-15 09:53:52 +08:00
fawney19 8827c46c33 perf(usage): 将 status 过滤下推到 SQL 查询层
为 UsageAuditListQuery 新增 statuses 字段,支持在数据库端按状态
筛选用量记录。admin usage summary 路由不再全量拉取后内存过滤,
改为直接查询 pending/streaming 状态的记录。
2026-04-15 09:30:42 +08:00
fawney19 adde9ff237 chore(proxy): bump version to 0.3.0 2026-04-15 08:46:19 +08:00
fawney19 cfb4f4582b perf(gateway): 限制流式响应体缓冲区大小为 256KB
流式转发过程中 provider 和 client 的 body buffer 可能无限增长,
对长响应造成内存压力。添加 256KB 上限, 超出时截断早期数据,
并在上报日志中跳过已截断的 body 以避免输出不完整内容。
2026-04-15 08:46:19 +08:00
github-actions[bot] 8bb637962f chore(proxy): update download links for proxy-v0.3.0 2026-04-14 19:11:01 +00:00
fawney19 67a2ca6e31 perf(gateway): 为 dashboard 接口添加短时响应缓存
新增 DashboardResponseCache,对 stats/daily_stats/provider_status
三个接口按用户维度缓存 15-30 秒,减少重复数据库查询。
2026-04-15 03:03:20 +08:00
fawney19 98ad1172b0 perf(usage): heatmap 改为数据库端按天聚合查询
将 admin 和 user heatmap 从逐条加载 usage audit 记录后在应用层聚合,
改为通过 SQL GROUP BY DATE 在数据库端直接按天汇总, 大幅减少数据传输量。

新增 UsageDailyHeatmapQuery / StoredUsageDailySummary 类型,
在 trait、SQL、内存实现中均补齐 summarize_usage_daily_heatmap 方法。

同时优化 docker-compose: postgres 增加空闲事务超时与 keepalive 参数,
gateway 增加健康检查配置。
2026-04-15 02:11:32 +08:00
fawney19 05fbbac493 feat(proxy): 支持配置 private 目标地址放行 2026-04-15 00:31:36 +08:00
fawney19 a4e7ac1df6 feat(proxy): 重构 Proxy 节点管理与隧道系统
- 重构 proxy_nodes 管理端,支持节点注册、心跳、隧道生命周期管理
- 增强 tunnel 嵌入式 hub 和隧道协议
- 重构 aether-proxy 配置、隧道客户端、心跳和调度机制
- 调整 admin OAuth/配额/导入等处理器的参数传递
- 扩展数据迁移模块
- 补充 proxy nodes、OAuth、配额、系统导入等测试
- 更新前端 proxy nodes 视图和 API
2026-04-14 22:51:02 +08:00
AAEE86 fb31928e44 fix(mapping): 对齐全局模型映射的正则匹配行为与范围 (#296)
- scheduler_core: `matches_model_mapping` 改为大小写不敏感且整串匹配,并补充单测
- global model routing 预览:
  - Key 过滤增加 `allowed_models + model_mappings` 校验
  - `all_keys_whitelist` 改为收集全站活跃 Provider 的活跃 Key 白名单
- provider mapping-preview:
  - 优先使用 admin 全量 GlobalModel(含非激活)参与映射
  - admin 数据为空时回退 public 模型,保持兼容
- public models 匹配逻辑统一复用 scheduler_core 实现,避免行为分叉
- 更新网关测试,覆盖未关联 Provider 的 Key 也进入 whitelist 的场景
2026-04-14 21:58:52 +08:00
fawney19 47bf1d04a1 fix(admin): 补齐 key 自动获取模型的即时刷新场景 (#295)
- 新增 key 且开启自动获取时立即抓取并写回 allowed_models
- 自动获取已开启时修改包含/排除规则后立即刷新 allowed_models
- 补充创建与过滤规则变更场景的控制层回归测试
2026-04-14 14:56:22 +08:00
AAEE86 b70f32c6c2 fix(dashboard): 使用有效输入口径展示今日 Token (#294)
将仪表盘“今日 Token”卡片中的输入项改为有效输入口径。
对 OpenAI/Gemini 从 input_tokens 中扣除 cache_read_tokens,
Claude 保持原始输入口径不变。

同步更新 dashboard 集成测试,覆盖 OpenAI 与 Claude 的差异化行为。
2026-04-14 14:53:59 +08:00
fawney19andAAEE86 21ac1825f3 fix(gateway): 修正 Codex 实时配额同步
Close #293

Co-authored-by: AAEE86 <[email protected]>
2026-04-14 14:51:26 +08:00
AAEE86 0081622f90 fix(gateway): 补齐 Codex 普通请求的实时配额同步
在 usage reporting 的 sync/stream 终态处理链中补充 Codex 配额回写,
将普通请求响应头中的 x-codex-* 实时同步到 key.upstream_metadata.codex。

- 复用 parse_codex_usage_headers 解析配额头
- 仅对 codex provider 执行 best-effort 回写
- 增加 30s TTL 指纹去重,忽略 reset 倒计时等波动字段
- 补充 realtime helper 单测与 sync/stream 集成测试
2026-04-14 14:40:48 +08:00
AAEE86 d089ed22c7 fix(admin): 补齐 key 自动获取模型的即时刷新场景
- 新增 key 且开启自动获取时立即抓取并写回 allowed_models
- 自动获取已开启时修改包含/排除规则后立即刷新 allowed_models
- 补充创建与过滤规则变更场景的控制层回归测试
2026-04-14 14:33:45 +08:00
fawney19 861ae81ff0 feat(admin): 完善代理节点与 OAuth 授权管理 2026-04-14 14:09:24 +08:00
fawney19 593640ac19 feat(gateway): 增强候选路由策略与可观测信息 2026-04-14 11:50:52 +08:00
fawney19 62e0a0338d refactor: 移除 shadow results 相关模块和接口 2026-04-14 09:46:16 +08:00
fawney19 5fd3240fcf fix(admin): 同步 key 自动获取模型与相关测试
Closes #292
Co-authored-by: AAEE86 <[email protected]>
2026-04-14 02:34:36 +08:00
AAEE86 563dd44957 test(gateway): 稳定本地 usage 终态断言
- 为 usage local 测试添加 wait_for_usage_status 辅助方法
- 轮询直到 usage 记录进入 completed/failed 终态
- 避免异步 usage runtime 先落 pending/streaming 导致测试抖动
2026-04-14 02:22:23 +08:00
AAEE86 23233a3243 fix(admin): 修复 Key 自动获取模型时 allowed_models 同步逻辑
- 关闭自动获取上游模型时清空 allowed_models
- 开启自动获取上游模型时立即拉取并覆盖 allowed_models
- 增加模型覆盖提示并补充相关回归测试
2026-04-14 02:21:46 +08:00
fawney19 1000b706be fix(gateway): 收紧本地 usage 测试轮询条件 2026-04-14 01:55:13 +08:00
fawney19 53acfbabf6 refactor(gateway): 重构 ai pipeline 规划链路 2026-04-14 01:27:04 +08:00
RWDai 37bb120d20 feat(aether-proxy): 支持 Alpine 主机服务安装与 musl 发布 (#291)
* feat(aether-proxy): 支持 Alpine 主机服务安装与 musl 发布

* fix(aether-proxy): address alpine support review findings
2026-04-13 17:20:22 +08:00
fawney19 4fd2b4a014 fix(gateway): 修复 balance 刷新去重键冲突、usage 状态回退与测试竞态
- balance_cache: 引入实例级 refresh key 防止多实例共享进程级 HashSet 冲突
- InMemoryUsageRepo: 阻止 pending/streaming 状态覆盖已终结(completed/failed/cancelled)记录
- usage 同步测试: 等待条件从 is_some() 改为检查 status=="completed" 避免竞态
- wallet 测试: 增加轮询等待 wallet 扣款完成
- 整理 import 语句与 tests 模块位置
2026-04-13 16:48:05 +08:00
fawney19 9a376e4223 refactor(admin): 将排行榜排序与排名函数移至 tests 模块之前 2026-04-13 16:13:18 +08:00
AAEE86 b2d85d70ca refactor(runtime): 优化管理端摘要查询与维护聚合链路
- 为 provider catalog key 和 video task 列表增加 summary/page 查询与排序能力,减少列表场景读取重字段
- 将多处 SQL 结果读取改为流式收集,降低 `fetch_all` 的内存占用
- 把日/小时统计、钱包日用量等维护任务改为数据库侧 `CTE + upsert` 聚合
- 修复视频任务轮询更新时从本地 snapshot 回填稀疏字段,避免 `prompt` 和请求体信息丢失
2026-04-13 15:44:58 +08:00
fawney19 6aa16ec792 feat(data): 废弃 usage 表 HTTP/结算列,迁移至 settlement_snapshots 与 http_audits
- 新增迁移 20260413030000:标记 billing_status、finalized_at、request_headers 等列为 DEPRECATED
- 更新 baseline_v2.sql 同步废弃注释,BASELINE_V2_CUTOFF_VERSION 升至 20260413030000
- usage/sql.rs:inline body 阈值归零,强制所有 body 走 blob 存储;upsert 时清空 legacy header/output_price 列
- 查询层优先读 usage_settlement_snapshots 的 billing_status、finalized_at、output_price_per_1m
- runtime.rs:stale usage 处理同步写入 usage_settlement_snapshots;SELECT FOR UPDATE 改为 FOR UPDATE OF usage
- 前端:PerformanceAnalysis 页面重构为实时面板,新增 prometheus 工具函数与 monitoring API
2026-04-13 14:53:46 +08:00
fawney19 e46629d11a fix(frontend): 统一 Usage 页面默认时间范围为 today 2026-04-13 14:14:07 +08:00
fawney19 5bb08e6aa4 feat(gateway): 重构 usage 数据层、迁移系统与系统导入
数据库迁移:
- 引入 baseline v2 bootstrap,空库首次启动自动初始化
- 服务启动不再自动执行迁移,需显式 `--migrate` 运行
- 新增 pending migration 检测,schema 落后时拒绝启动

Usage 数据层:
- usage body 存储外部化为独立 blob 表
- 新增 HTTP audit 表拆分存储请求/响应头与 body ref
- 后台清理任务支持 legacy body ref 元数据迁移
- usage runtime 写入迁移到专用 tokio runtime(独立线程池, 8MB 栈)

系统导入/导出:
- 支持用户、API Keys、钱包数据的完整导入
- 兼容 legacy 与 v1.3+ 两种导出格式

其他改进:
- executor outcome 增加 runtime miss 诊断上下文
- 主 tokio runtime 栈大小调整为 8MB
- 前端 provider 管理支持 base URL 配置
- dev.sh 支持 --migrate 参数
2026-04-13 14:01:22 +08:00
fawney19 3698e5a833 fix(test): 修复 MutexGuard 跨 await 点的 Clippy 警告 2026-04-12 16:22:29 +08:00
fawney19 f84febbf89 fix(proxy): 修复 Clippy dead_code 与 too_many_arguments 警告 2026-04-12 16:19:23 +08:00
fawney19 e029012f73 Merge pull request #289 from AAEE86/rust
fix(usage): 统一使用记录与仪表盘的缓存命中率计算口径
2026-04-12 16:12:22 +08:00
fawney19 9703840a36 feat(proxy): 实现代理节点批量升级回滚、隧道重定向跟随及远程配置管理
核心功能:
- 新增代理节点批量升级回滚工作流,支持分批升级、健康探针、跳过/重试/取消等操作
- proxy 隧道流处理器支持 HTTP 重定向跟随(最多 10 跳),区分 307/308 可重播与不可重播请求体
- proxy 协议新增 follow_redirects / http1_only 字段,网关侧同步支持
- 新增代理节点远端配置变更接口(名称、允许端口、调度状态、升级目标等)
- 新增代理节点注册/反注册/心跳的 Admin API,及节点过期清理维护任务
- gateway 隧道 owner-relay 支持流式代理大请求体,新增 5 MiB 默认限制
- 新增 ProxyNodeRegistrationMutation / ProxyNodeRemoteConfigMutation 数据类型
- proxy 配置新增重定向重播预算、心跳间隔等参数,TUI 安装向导同步更新
- 前端 ProxyNodes 页面新增批量升级操作面板及滚动进度展示
2026-04-12 16:02:38 +08:00
AAEE86 c24a29fa65 fix(model-fetch): 提取获取计划请求,并使网关运行时与共享构建器保持一致
- 将 build_execution_plan 的散参数收敛为 ModelFetchExecutionPlanRequest,消除 clippy too_many_arguments
- 在 aether-gateway 的 model_fetch runtime 中显式依赖 build_models_fetch_execution_plan
- 补充共享 models fetch plan builder 的运行时测试覆盖
2026-04-12 11:57:06 +08:00
AAEE86 3fcb2b1514 fix(dashboard): 修复今日统计口径并对齐每日统计日期显示
- 前端请求 /api/dashboard/stats 时传递 timezone 和 tz_offset_minutes
- 后端仪表盘汇总过滤 pending/streaming 和占位 provider,修正今日请求/Token/费用统计
- 今日 Token 卡片增加 K/M 单位显示,并补充写缓存/读缓存 Token 信息
- 修复每日统计 YYYY-MM-DD 被按 UTC 解析导致的“今天/昨天”串天问题
- 补充前后端回归测试,覆盖统计口径和日期解析场景
2026-04-12 11:20:39 +08:00
AAEE86 ab82841426 feat(model-fetch): 对齐 Rust 上游模型抓取行为到 Python 语义
将 Rust 版上游可用模型抓取逻辑收敛到 Python 版行为,统一后台自动抓模
与管理员 provider-query 的模型发现路径,消除标准 /models、固定模型目录、
Antigravity、Vertex AI 等 provider 在两端实现上的分叉。

核心变更:
- 在 aether-model-fetch 中引入统一抓模策略层
- 覆盖标准 /models、Vertex API Key、Vertex Service Account、
  Antigravity fetchAvailableModels、固定模型目录五类抓模路径
- 将 provider-query 与后台自动抓模都切换到共享抓模入口,避免重复拼接
  URL、headers 和 provider 特判逻辑

标准模型抓取对齐:
- 按 Python 语义调整抓模优先级:
  openai:chat > openai:cli > openai:compact
  claude:chat > claude:cli
  gemini:chat > gemini:cli
- 从抓模候选中移除 openai:responses
- 为 openai:cli/openai:compact、claude:cli、gemini:* 补齐 Python 同款
  User-Agent / 浏览器指纹请求头
- Claude 抓模保留 after_id 分页语义
- Gemini 抓模统一为 v1beta/models?key=... 语义

provider-query 对齐:
- 返回结果改为按 model id 聚合,并合并/排序 api_formats
- 最终模型列表按 model id 排序,行为与 Python 保持一致
- 固定目录 provider(codex/kiro/claude_code/gemini_cli)不再依赖活跃
  endpoint,即使无 endpoint 也能返回预设模型目录
- Antigravity 多 key 查询改为按账户可用性 + tier 排序,首个成功结果即
  停止,并接入 provider 级缓存
- 仅配置 openai:responses 的 provider 不再被视为抓模成功路径

自动抓模对齐:
- 自动抓模成功时写入 allowed_models、upstream_models cache,并同步
  upstream_metadata
- upstream_metadata 合并逻辑对齐 Python,对 quota_by_model 做模型级合并,
  并保留已有 reset_time
- 自动抓模失败时不覆盖已有 allowed_models
- 固定目录 provider 在无 endpoint 场景下也可成功更新 allowed_models

Antigravity 对齐:
- 使用 POST /v1internal:fetchAvailableModels 抓取可用模型
- 按 Python 规则处理 URL fallback 和 429/404/408/5xx fallback 状态
- 强制要求 auth_config.project_id
- 过滤 Python 黑名单模型
- 解析并持久化 upstream_metadata.antigravity.quota_by_model

Vertex AI 对齐:
- API Key 模式仅抓取 publishers/google/models
- Service Account 模式新增 JWT token exchange,并按 Python region 顺序
  抓取 google + anthropic publishers
- 模型 owned_by / display_name / api_format 推断与 Python 对齐
- 软 404 处理行为与 Python 收敛

Gemini CLI / 固定目录对齐:
- Gemini CLI 改为返回 Python 预设模型目录
- 在可用时通过 loadCodeAssist 补充 plan_type/project_id 元数据
- Codex/Kiro/Claude Code 改为共享固定模型目录实现

测试:
- 扩展 aether-model-fetch 单元测试,覆盖格式优先级、openai:responses 排除、
  请求头、Claude 分页、Gemini query auth、固定目录与 metadata 合并
- 调整 provider-query 控制面测试到 Python 语义
- 新增自动抓模运行时测试,覆盖固定目录成功、Antigravity metadata 合并、
  失败保留旧 allowed_models

验证:
- cargo nextest run -p aether-model-fetch --lib
- cargo nextest run -p aether-gateway control::admin::provider_query model_fetch::runtime::tests
2026-04-12 10:14:29 +08:00
AAEE86 def8135118 fix(compact): 移除 OpenAI Compact 请求中不受支持的 store 参数
- 在 OpenAI Compact 请求规范化流程中剥离 `store`
- 将 compact 默认 body rules 与 Codex CLI 默认规则拆分
- 补充 same-format 和转换链路的 compact 回归测试
2026-04-12 01:52:25 +08:00
AAEE86 335e440cc5 fix(usage): 统一使用记录与仪表盘的缓存命中率计算口径
- 新增归一化总输入上下文计算逻辑,按 provider 区分 OpenAI/Gemini 与 Claude 的 cache token 语义
- 将管理端使用聚合、用户使用记录、仪表盘缓存统计、缓存亲和性分析统一为 token 级缓存命中率
- 修正 total_input_context 字段,避免 cache_read 在部分 provider 上被重复计入分母
- 同步更新相关 Rust 单元测试与网关集成测试断言
- 调整前端 dashboard mock 中 cache_hit_rate 的单位为百分比
2026-04-12 00:57:40 +08:00
fawney19 7c5bb7f383 fix(admin): 修复系统配置导入的多项兼容性问题
- 引入 serde_path_to_error,反序列化失败时返回精确字段路径
- 为所有浮点字段新增数字字符串兼容反序列化器,支持 Python 序列化格式
- 修复 OAuth provider key 导入:正确写入加密的 api_key 和 auth_config
- 优化代理节点跳过提示,区分手动 URL 代理与 node_id 引用代理
- 新增测试覆盖上述场景
2026-04-11 23:10:25 +08:00
fawney19 a9f610fa69 feat(admin): 实现系统数据导入导出功能,支持提供商和模型批量配置 2026-04-11 21:39:04 +08:00
fawney19 801e16c988 refactor(gateway): 统一 AETHER_GATEWAY_BIND 为 APP_PORT,新增 API Key 前缀配置和启动自举管理员
- 绑定地址固定 0.0.0.0,仅通过 APP_PORT 控制端口,简化 CLI/Docker/systemd/dev.sh/前端代理全链路
- 新增 API_KEY_PREFIX 环境变量,抽取 handlers/shared/api_keys.rs 消除 admin/public 重复逻辑
- 新增 bootstrap_admin.rs,启动时通过 ADMIN_* 环境变量在无管理员时自动创建首个本地管理员
- 前端密码输入改用 type=password,API Key 占位符改为动态前缀
- 删除过时的 pyproject.toml/uv.lock 和旧部署文档
- 更新 .env.example/README 反映新配置项
2026-04-11 17:39:02 +08:00
fawney19 a570a77cca fix(data): baseline 迁移末尾恢复 search_path 为 public,确保 sqlx 记账正常 2026-04-11 16:17:02 +08:00
fawney19 b0f068cc5d ci(rust): 移除冗余的 build_release job,测试已覆盖编译 2026-04-11 15:33:04 +08:00
fawney19 fff82de933 fix(data): baseline 迁移 search_path 改为事务级作用域,修复 sqlx 迁移表访问问题 2026-04-11 15:29:13 +08:00
fawney19 8a4a41fcef fix(docker): 容器以 root 运行,解决日志目录写入权限问题 2026-04-11 15:07:44 +08:00
fawney19 aa5761954e refactor(data): baseline 迁移改为完整幂等建库脚本,放宽 checksum 校验
- baseline migration 从占位 SELECT 1 替换为完整的幂等 DDL,支持全新数据库从零建库
- 迁移校验从 checksum 严格报错改为 warn-only,仅按版本匹配
- 移除过时的 schema 导出文件(README/TSV)
- Dockerfile 移除 :nonroot 标签,docker-compose 移除 user 指令,统一以默认用户运行
2026-04-11 14:31:28 +08:00
fawney19 8cd2c4d5bd fix(gateway): 默认监听端口改为 8084 以支持非 root 运行
- Dockerfile 及 main.rs 默认 bind 从 0.0.0.0:80 改为 0.0.0.0:8084
- docker-compose 以 UID:GID 非 root 用户启动 app 容器
- compose 端口映射与容器内监听端口保持一致,通过 AETHER_GATEWAY_BIND 注入
2026-04-11 13:55:11 +08:00
fawney19 940a28cff4 chore: 清理 .env.example 中过时的 Python 运行时配置项 2026-04-11 13:12:28 +08:00
fawney19 c95feea286 chore: 清理过时的 Python 遗留配置和文档
- .dockerignore 移除 Python 相关忽略规则和废弃的 dist 放行条目
- README 更新密钥生成为 shell 脚本,移除过时的 systemd 部署章节
- deploy.sh 从哈希计算中移除已删除的 entrypoint.sh
2026-04-11 13:00:04 +08:00
fawney19 1d896467dc fix(build): .dockerignore 放行 dist/aether-gateway-* 和 dist/frontend/ 2026-04-11 12:49:40 +08:00
fawney19 3a655440b9 chore: 移除过时 Python 脚本,密钥生成改用 shell 脚本
- generate_keys.py 替换为 generate_keys.sh (无需 Python 依赖)
- 移除已废弃的 backfill_provider_key_status_snapshot.py
2026-04-11 12:43:19 +08:00
fawney19 80bae8bc2a fix(build): ldap3 切换到 rustls 后端以消除 openssl-sys 依赖
musl 交叉编译时 openssl-sys 找不到系统 OpenSSL, 改用 tls-rustls feature
2026-04-11 12:36:35 +08:00
fawney19 f68c67021c refactor(build): 切换到 musl 交叉编译 + distroless 镜像方案
- docker-publish.yml: 改为 cross 交叉编译 amd64/arm64 musl 静态二进制,
  前端在 CI 独立构建, buildx 组装多架构镜像
- Dockerfile.app: 从 139 行容器内编译简化为 24 行纯 COPY 打包
- Dockerfile.app.local: 移除 jemalloc 动态链接 (LD_PRELOAD/libjemalloc2)
- aether-gateway: 引入 tikv-jemallocator 静态链接 jemalloc
2026-04-11 12:21:33 +08:00
fawney19 e37a32c83d chore(deploy): 默认镜像标签切换为 pre 以跟踪预发布版本 2026-04-11 10:48:34 +08:00
fawney19 ec0bde819a refactor: 简化 outcome clippy lint 并移动 usage 测试模块至文件末尾 2026-04-11 09:35:31 +08:00
fawney19 848f99d3e5 merge(pr287): absorb remaining rust branch changes
Absorb the remaining changes from PR #287 into aether-rust-pioneer after resolving conflicts locally and preserving the admin fixes already landed in this branch.

Closes #287
Co-authored-by: AAEE86 <[email protected]>
2026-04-11 03:10:16 +08:00
fawney19 34d295c1e0 fix(admin): align global model responses with repository counts 2026-04-11 02:55:02 +08:00
fawney19 a54ac76688 fix(admin): restore pool key usage aggregates 2026-04-11 02:54:54 +08:00
fawney19 cf02a10050 chore: 调整 Postgres 连接池默认参数:增大最大连接数、缩短获取超时 2026-04-11 02:26:33 +08:00
fawney19 6144473ebe feat: 新增 frontdoor 执行回环守卫与多项可观测性增强
- 新增 frontdoor_loop_guard 模块,检测并拒绝 execution runtime 回环到本地网关的请求(HTTP 508)
- candidate loop 引入 span tracking、执行尝试日志与流式看门狗超时
- 本地故障转移策略支持从 report_context 加载,新增 append_local_failover_policy_to_value
- runtime tracing 美化:移除 identity 前缀,按 span 深度树形缩进,target 固定宽度展示
- Codex OpenAI CLI 补齐 chatgpt-account-id/x-client-request-id/session_id/conversation_id 请求头
- OpenAI CLI same/cross-format 聚合规则放宽以支持 openai:compact 客户端格式,并过滤 error-like 响应体
- auth/proxy/finalize 日志补充 user_id/api_key_id/api_key_name/balance_remaining 等字段
- 启动日志拆分为 starting/ready/config 三段,新增 resolve_bind_http_base_url
- access_log middleware 将生成的 trace_id 回注到下游请求头
- Cargo.toml 启用 serde_json preserve_order 特性
2026-04-11 01:50:24 +08:00
AAEE86 174f11604a fix(data): 统一 usage 内存仓储时间戳单位并通过格式检查
- 将 usage 内存仓储的秒级过滤条件转换为毫秒后再比较
- 汇总 provider api key 最近使用时间时将毫秒转换为秒
- upsert 在缺少 created_at 时默认写入毫秒时间戳
- 补充时间戳单位回归测试
- 调整 registry.rs 格式以通过 cargo fmt --all --check
2026-04-10 22:23:26 +08:00
fawney19 3f057628b7 fix: 修正用量失败状态判断与 Codex 会话头 2026-04-10 21:33:34 +08:00
fawney19 46f1507d44 feat: 提前记录 pending 用量、优化流遥测时序与前端活跃请求发现机制
- 将 record_pending 调用移至执行开始前(sync/stream 两路),确保请求在执行前即有 pending 记录
- stream_pump 在收到第一个数据块前优先 yield 遥测帧,保证 ttfb 早于 data 帧到达
- stream execution 增加 should_refresh_stream_usage_telemetry,在遥测帧携带新 ttfb/elapsed 时及时更新 record_stream_started
- access_log 对高频轮询路径(usage/active、usage/records 等)降级为 TRACE 日志,减少日志噪音
- 前端新增 reconcileActiveRequestDiscovery 工具函数及 discoverActiveRequests 逻辑,活跃请求发现与全局自动刷新解耦,空闲时降频为 5 秒扫描
- RequestDetailDrawer 调整:进行中请求不再自动开启轮询,由用户手动触发;刷新按钮 title 动态适配状态
2026-04-10 20:58:01 +08:00
fawney19 d5b8583d6b fix: 修正时间戳精度、TTL 定价匹配逻辑及账单快照展示
- 将 created_at_unix_ms 从 current_unix_secs 改为 current_unix_ms,修正候选尝试和跳过记录的时间戳精度
- formula_engine: TTL 定价从「<=上限」模糊匹配改为精确匹配,null 值改为回退到基础价格而非透传
- 新增 ttl_pricing_requires_exact_match 和 ttl_pricing_null_value_falls_back_to_base_tier_value 测试用例
- service: 新增 5min/1h cache TTL 的端到端计费验证测试
- RequestDetailDrawer: 优先从 billing_snapshot 读取已解析的价格和费用,正确展示当前 TTL 对应的缓存创建/读取价格,修正输入/输出/缓存费用列的数据来源
2026-04-10 18:50:59 +08:00
AAEE86 b1f6fff0a5 fix(gateway): 修复 local finalize 与 usage 聚合相关测试失败
- 修正内存 usage 仓库的时间单位处理,恢复 stats/monitoring/dashboard/wallet/pool 聚合结果
- 允许 openai:compact 走 CLI 响应转换与本地 finalize cross-format 路径
- 更新 cross-format 测试夹具,显式启用 enable_format_conversion
- 调整 failover 测试优先级,消除候选平局导致的顺序不稳定
- 补齐 ai_pipeline 架构断言所需导入并清理残留 runtime 空目录
2026-04-10 18:27:02 +08:00
fawney19 010ab127e2 feat: 扩展 cache creation token 细分统计与 effective_input_tokens 计费逻辑
- 新增 cache_creation_ephemeral_5m/1h_input_tokens 字段,区分不同 TTL 的缓存写入 token
- 引入 effective_input_tokens(扣除 cache read 后的有效输入 token),暴露给 usage 接口
- billing 规则生成器支持 5m/1h ephemeral cache 独立定价与分级计费
- usage_mapper 增加 Claude/Anthropic 格式映射,修复 OpenAI responses 格式字段兼容性
- 迁移逻辑增强:支持 checksum 容错、applied/pending 数量日志、逐步执行信息输出
- executor 抽离 LocalExecutionRequestOutcome 类型,统一 sync/stream 路径返回语义
- provider-transport auth 层新增 complete passthrough headers 构建逻辑
- 前端 usage 类型全面补充 effective_input_tokens、cache_creation_tokens、total_input_context 字段
2026-04-10 17:44:55 +08:00
AAEE86 82c3c33610 fix(build): 修复网关构建失败并收口候选选择与 finalize 回归
- 补齐 DecisionTraceCandidate 新增字段,修复审计测试构造
- 修正 usage 内存仓库的 created_at_unix_ms 字段引用与秒/毫秒换算
- 将 build_minimal_candidate_selection 重构为输入对象,消除 clippy 参数过多问题
- 修复 admin global model created_at 旧字段残留引用
- 修复 openai:cli 与 openai:compact 同家族 finalize 在 needs_conversion=true 时的成功回落逻辑
- 清理 aether-gateway 中的 derive/default 与 needless borrow 等 clippy 问题
2026-04-10 17:23:40 +08:00
AAEE86 677b8f5acb Merge remote-tracking branch 'upstream/aether-rust-pioneer' into rust 2026-04-10 15:08:32 +08:00
AAEE86 b8ce02b4f7 refactor(core): 重构 Provider Ops 架构注册、校验链路与余额缓存流程
- 将 provider ops 纯逻辑下沉到 aether-admin,拆分 architectures、actions、verify 模块
- 用统一的 architecture spec 驱动 verify、query_balance、checkin 行为,替换分散的条件分支
- 新增 sub2api / anyrouter / cubence / yescode 等架构的请求头构建、校验解析与余额解析实现
- 引入 provider balance Redis 缓存、异步刷新、pending 响应以及配置变更后的缓存清理
- 补充 provider ops 的控制面测试、Redis 缓存测试和架构边界测试

- 影响说明:统一了 Provider Ops 的扩展方式与运行时行为,降低后续新增架构的接入成本
- 影响说明:余额查询从“实时阻塞返回”扩展为“缓存命中即返回并后台刷新”的模式,前端需要兼容 pending 状态
2026-04-10 15:06:42 +08:00
fawney19 5014e2f5fd refactor: 抽离 AI pipeline 与调度共享能力逻辑 2026-04-10 01:46:14 +08:00
AAEE86 87b433e290 fix(gateway): 管理端 provider query helper 改用 AdminAppState
将 admin provider query 路由相邻层 helper 中直接使用的 AppState 参数改为 AdminAppState,并通过 state.app() 访问底层状态,满足 admin_shared 架构约束。
2026-04-09 16:23:47 +08:00
AAEE86 5581f7a085 fix(gateway): 对齐 admin global models Rust 响应与 Python 字段口径
- 为 /api/admin/models/global 补齐 usage_count 字段
- 直接使用仓储层 provider_count 和 active_provider_count 统计结果
- 修复 handler 层重复统计导致的 active_provider_count 口径偏差
- 移除列表接口额外的 provider models 查询
- 补充列表、详情和 payload 组装的回归测试
2026-04-09 15:57:32 +08:00
AAEE86 0b3f619280 fix: 号池列表统计与最后使用时间显示
- 为 usage 仓储新增按 provider_api_key_id 汇总请求数、Token、费用和最后使用时间的能力
- 在 /api/admin/pool/{provider_id}/keys 中优先使用 usage 汇总结果覆盖 request_count、total_tokens、total_cost_usd、last_used_at
- 补充数据层与网关侧回归测试,避免号池管理页统计全为 0 且最后使用为空
2026-04-09 15:10:21 +08:00
fawney19 b901a6ffc7 chore: 移除废弃的 build-hub workflow,格式化测试代码 2026-04-09 14:01:37 +08:00
fawney19 fe81eafe2c feat: 增强 provider query 端点选择逻辑,支持 openai:responses 格式与 key 级 api_formats 过滤
- provider_query_selected_fetch_endpoints 支持按 key.api_formats 过滤可用端点
- 端点选择增加 endpoint_supports_rust_models_fetch 检查
- 支持优先级列表之外的自定义 api_format 端点
- 新增 openai:responses 端点的 provider query 集成测试
- 各 crate 补充缺失的 tracing instrument 和小修正
- 前端 Pool 管理页面补充状态逻辑
2026-04-09 13:55:18 +08:00
fawney19 b0b40c16ff feat: 全栈功能增强 - 扩展 provider/pool 管理、完善调度与数据层、重构前端 Pool 页面
后端:
- 扩展 pool_admin payloads 和 provider query models,增强 endpoint key 管理
- 完善 scheduler-core 候选排序与请求候选逻辑
- 增强 usage-runtime 写入、provider-transport 网络层与 OAuth 刷新
- 改进 AI pipeline 响应转换与流式处理
- 扩展 global_models/provider_catalog 数据层查询能力
- 增强 video-tasks-core 多 provider 支持
- 新增大量集成测试覆盖 pool/keys/provider_query/frontdoor

前端:
- 重构 PoolManagement 页面,拆分状态管理/对话框逻辑到独立模块
- 新增 poolAdvancedDialog/poolSchedulingDialog/poolManagementState/poolMobilePresentation 工具函数及测试
- 改进 Dialog 组件与 provider tabs 显示

部署:
- 更新 Rust CI workflow 和 Dockerfile 构建配置

Closes #275
Co-authored-by: AAEE86 <[email protected]>
2026-04-09 13:51:50 +08:00
fawney19 4fc95adfb9 refactor: 大规模模块拆分与重组,新增 aether-admin crate
- 新建独立 aether-admin crate 承载 admin 相关共享契约与纯辅助函数
- 拆分 ai_pipeline 下 kiro/private_envelope/conversion/planner 等大文件为子模块目录
- 重组 admin handlers 各业务域(billing/oauth/provider/system/users 等)为目录结构,移除 shared.rs/builders.rs 等反模式
- 移除 ai_pipeline runtime adapters 旧实现(claude/openai/gemini/kiro/vertex/antigravity 等),改由 provider transport 统一承载
- 移除 control_facade/execution_facade/auth_snapshot_facade 等冗余 facade 层
- 拆分 query/billing 与 query/monitoring 模块、state/runtime/payments 与 security 模块
- 扩展架构测试覆盖 admin_billing/admin_model/admin_users 等新模块
- 删除 docs/architecture/refactor-execution-plan.md 已完成的执行计划文档
2026-04-09 00:10:38 +08:00
fawney19 4fb9882b54 refactor: 拆分 admin handler 大文件为模块目录,消除 shared.rs 反模式
- endpoint_keys/pool/architecture 等大文件拆分为独立模块目录
- 删除 crud/query/strategy/system 中的 shared.rs,内容归位到各自模块
- endpoints_admin/models/oauth/write 等模块拆出 payloads/responses/support 子文件
- system/core 下多个 *_routes.rs 合并到 system_routes.rs
- 更新 refactor-execution-plan 文档进度
2026-04-07 12:15:19 +08:00
fawney19 29055c575f refactor: 拆分 system.rs 大文件,将 email_templates/proxy_errors/system_config 下沉到 shared 层
- 删除 1666 行的 admin/system/shared/system.rs,按职责拆分到独立模块
- 新增 handlers/shared/email_templates.rs 和 system_config_values.rs 存放跨层共用的模板/配置工具函数
- 新增 admin/system/shared/email_templates.rs 存放 admin 专用的模板操作逻辑
- 新增 admin/shared/proxy_errors.rs 存放 build_proxy_error_response
- 清理 public/system_modules_helpers 中不属于 public 层的导出
- 新增架构测试守护模块归属边界
2026-04-07 08:19:26 +08:00
fawney19 5d96d6673b refactor: 大规模模块拆分与代码精简,新增 ai-pipeline/data-contracts 独立 crate
- 新增 aether-ai-pipeline 和 aether-data-contracts crate,将 pipeline 逻辑与数据契约从 gateway 中解耦
- 重构 admin handlers:拆分单体模块为 auth/billing/endpoint/features/model/observability/provider/system 等独立子模块
- 合并 chat/cli 重复代码路径:精简 conversion、finalize、planner 中的 sync/chat/cli 分支
- 重构 scheduler/executor/data 层,引入 facade 模式降低模块间耦合
- 移除冗余的 intent 模块,将 plan_fallback/policy/stream_path/sync_path 迁移至 executor
- 前端适配:调整 admin API 调用和 provider 模型测试对话框
2026-04-07 02:50:19 +08:00
fawney19 763ff03a7b refactor: 拆分 gateway 单体为独立 crate,新增 systemd 部署方案
将 gateway 内部的 model-fetch、provider-transport、scheduler-core、
usage-runtime、video-tasks-core 模块提取为独立 crate;重构 gateway
内部模块结构(state/router/cache/data/query 等);移除大量遗留模块
文件;新增 systemd 二进制部署骨架及相关文档;更新前端 usage 相关
API 和组件。
2026-04-05 20:23:16 +08:00
fawney19 cbc811f6ce refactor: 移除 Python upstream 依赖,清理全部 legacy/Python 兼容层
- 移除 upstream_base_url 参数及 AETHER_GATEWAY_UPSTREAM 环境变量,gateway 不再需要指向 Python 宿主
- 删除所有 LEGACY_*/PYTHON_* 常量、路由组、header 定义及 sunset/phaseout 机制
- 将 legacy_gateway_bridge 重命名为 internal_gateway,executor 相关命名统一为 execution_runtime
- dev.sh 新增 Postgres/Redis 预检查,移除 upstream 相关启动参数和提示
- 新增 ai_public 路由处理器
- 全量适配 handler、test、state、control 等模块的命名和接口变更
2026-04-04 01:40:24 +08:00
fawney19 1d9c77522a refactor: 移除 Python 后端源码,全面迁移至 Rust gateway 架构
- 删除全部 Python 源码 (src/) 及 Alembic 迁移脚本,归档至 _deprecated_py_src/
- 重构 Rust gateway ai_pipeline: 拆分 planner/finalize 模块,新增 contracts/adaptation 层
- 重组 handlers 模块为 admin/public/proxy/internal/shared 子模块结构
- 新增 executor 模块,引入 Rust 原生数据库迁移 (aether-data/migrations)
- 简化 CI/Docker 构建流程,移除 base image 二级构建,统一为单一 app image
- 移除 Python 相关基础设施文件 (entrypoint.sh, gunicorn_conf.py, Dockerfile.base)
2026-04-03 16:26:16 +08:00
fawney19 8f26e1a31f refactor: 移除独立 hub/proxy/executor/gateway crate,统一为 gateway tunnel 架构
- 删除 aether-hub、aether-proxy 独立项目及其 Dockerfile/配置
- 删除 crates/aether-executor 和 crates/aether-gateway 全部模块
- 新增 apps/ 目录作为应用入口
- 将 hub 概念重构为 gateway tunnel transport
- 将 executor 重构为 execution runtime
- 新增 tunnel.rs 合约定义和 testkit tunnel/execution_runtime 模块
- 更新 Python 服务层和测试适配新架构命名
2026-04-03 14:59:58 +08:00
fawney19 ddf18fed9a feat: 扩展 Rust gateway 全功能模块,新增 billing/crypto/wallet crate 及完整数据层
- 新增 aether-billing、aether-crypto、aether-wallet 独立 crate
- aether-data 扩展 repository 层:announcements、auth_modules、billing、
  candidate_selection、gemini_file_mappings、global_models、management_tokens、
  oauth_providers、proxy_nodes、quota、users、wallet 等模块
- aether-gateway 新增 api/auth/billing/control/middleware/scheduler/usage/
  video_tasks/hooks/maintenance/model_fetch/provider_transport 等功能模块
- 重构 executor decision 和 gateway state 为模块目录结构
- 新增 gateway router、frontdoor 路由层及对应测试
- Python 侧 API 路由重构,新增 compat/support 模块
- 前端 Logo 组件更新及 Provider 管理页面调整
2026-03-31 19:19:04 +08:00
fawney19 b5a0070023 feat: 引入 aether-runtime/cache/data/http/testkit 基础 crate,完善并发门控与审计系统
新增 crate:
- aether-runtime: 服务运行时基础设施(并发门控、分布式并发、指标、队列、优雅关闭、tracing)
- aether-cache: 通用 TTL 缓存与命名空间抽象
- aether-data: 数据访问层(PostgreSQL/Redis 后端、repository 模式)
- aether-http: HTTP 客户端封装(重试、配置)
- aether-testkit: 集成测试工具集(gateway/executor/hub/proxy fixture、等待、负载测试)

gateway 扩展:
- 引入 audit 模块(shadow 执行审计、决策链路追踪、请求审计 bundle)
- 引入 cache 模块(AuthContext 缓存、direct-plan bypass 缓存)
- 引入 data 模块(auth/candidates/config/usage/video_tasks 数据访问)
- 集成 ConcurrencyGate/DistributedConcurrencyGate 请求门控
- 新增本地 auth 拒绝、过载响应构建器
- 补充 control/auth_cache/video/concurrency 集成测试

aether-proxy 扩展:
- AppState 集成 stream_gate / distributed_stream_gate 并发门控
- 新增 ProxyAdmissionError 及准入拒绝流程
- stream_handler 补充门控饱和/不可用场景测试
- 配置与注册客户端逻辑完善

aether-hub 扩展:
- main.rs 引入运行时初始化、指标端点、健康检查
- local_relay 重构为 lib.rs 暴露公共接口
2026-03-24 15:12:56 +08:00
fawney19 eaf8475f9e refactor: 拆分 Rust gateway/executor 大文件为模块目录结构,拆分 Python gateway.py 为子模块
Rust 侧:
- executor.rs 拆分为 executor/ 目录 (plan_builders, stream, sync, submission)
- 新增 kiro_stream/, local_finalize/, local_stream/ 模块目录
- 新增 video_tasks.rs
- 测试文件 ai_execute/files/video 拆分为子目录
- handlers/headers/control/constants 扩展支持新模块

Python 侧:
- gateway.py 拆分为 24 个子模块 (routes, shared, contract, chat, cli, video, files, finalize, reporting 等)
- 新增 antigravity/gemini_cli/kiro 的 rust_http 适配层
- upstream_fetcher 增加 Rust sidecar 支持
- executor_plan/candidate/pipeline 适配调整

测试:
- 对应拆分 test_internal_gateway_routes 为子目录
- 新增 rust_http 相关测试
2026-03-23 17:19:15 +08:00
fawney19 455234e797 feat: 实现 executor 流式执行路径及完善 gateway 多格式 plan 构建
- Rust executor 新增流式 plan 支持,覆盖 openai/claude/gemini 的 chat/cli/video 格式
- Gateway 新增 finalize-sync、report-stream 端点及 stream report 模型
- 新增 video sync 操作(create/cancel/remix/delete)的 plan 构建
- 修正 OpenAI Responses(openai:cli) SSE 格式: 添加 event: 行、移除 [DONE] 哨兵
- 统一 OpenAI CLI normalizer 的 ID 生成方法
2026-03-21 15:51:37 +08:00
fawney19 53bb23b510 feat: 实现 executor sync 执行路径及 plan-sync/report-sync 端点
Rust gateway 新增同步执行模式,支持 AI 请求(OpenAI/Claude/Gemini chat/cli)
及 Gemini Files CRUD 通过 executor 直接同步执行,执行完成后通过 report-sync
端点向 control 回报结果。Python 侧新增 /plan-sync 和 /report-sync 内部端点,
构建同步执行计划并处理结果上报。包含完整测试覆盖。
2026-03-21 14:07:45 +08:00
fawney19 d735b6316f feat: 引入 Rust executor/gateway sidecar 及 Python 侧双后端适配
- 新增 Rust workspace crates: aether-contracts, aether-executor, aether-gateway
- aether-executor: 支持 Unix Socket/TCP 双传输模式,处理同步/流式上游请求
- aether-gateway: 作为本地主入口代理,集成 /api/internal/gateway/resolve 认证预解析
- Python 侧新增 ExecutionPlan 契约和 RustExecutorClient,各 handler 支持
  executor_backend=rust 时将可序列化请求转发给 Rust executor 执行
- 重构 dev.sh 支持 executor/gateway 进程编排与生命周期管理
- 新增 internal gateway 路由,提供 resolve/passthrough 端点
- handler 层(chat/cli/video/endpoint_checker 等)全面适配 Rust executor 回退逻辑
- pipeline 层支持 trusted auth context 跳过重复认证
- 新增 Rust CI workflow 及对应测试用例
2026-03-21 12:57:09 +08:00
fawney19 46737d32f8 feat: 引入 status_snapshot 统一 provider key 状态管理
- 新增 StatusSnapshot 模型,聚合 OAuth / 账号 / 配额三维状态
- 新增 StatusSnapshotStore 负责快照的持久化与查询
- 重构 response_builder / endpoint_models,基于 snapshot 输出状态字段
- 前端抽取 providerKeyStatus / oauthRefreshFeedback 工具函数,
  统一 PoolManagement、ProviderDetailDrawer、BatchDialog 的状态展示
- errorParser 增加已知 OAuth 错误的友好提示
- refresher 适配 snapshot 写入,account_state 扩展状态分类
- 新增 alembic 迁移及存量数据回填脚本
- 补充前后端单元测试
2026-03-20 19:16:52 +08:00
fawney19 25d38ae632 feat(oauth): 账号封禁前置 OAuth 验证、抽取 provider_context、完善账号状态分类
- 新增 verify_oauth_before_account_block:在标记账号封禁前先尝试刷新 token,
  区分 OAuth 过期与真正的账号级封禁,避免误标
- 抽取 provider_context.py 统一解析 provider_type,解决 ORM detached 访问问题
- account_state 新增 workspace_deactivated 分类和 auto-removable 状态集合,
  补充中文验证关键词匹配
- OAuth refresh 成功后仅清除可恢复的 token 错误,不再自动清除账号级 block
- deploy.sh 依赖指纹改用纯 shell 实现,移除对 Python tomllib 的依赖
- 前端 Pool 管理页面新增筛选和批量操作优化
- 补充对应测试用例
2026-03-20 16:50:59 +08:00
fawney19 aa83b4a7a7 fix: 加固续租失败处理、verify_auth 异常捕获及调度器注册追踪
- task_coordinator: 续租连续失败 5 次后主动触发 lock_lost 回调,失败间加指数退避
- proxy_nodes: lock_lost 回调由 lambda 改为具名 async 函数,确保异步停止逻辑正确执行
- provider_ops: 将 prepare_verify_config 纳入外层 try,捕获 ValueError 并返回失败响应
- maintenance_scheduler: 用 _registered_job_ids 动态追踪已注册任务,stop 时按列表清理
- stats_aggregator: 内联 _do_aggregate 为 for/range(2) 循环,消除内嵌函数
2026-03-20 01:22:07 +08:00
fawney19 913ce2dbcb Merge pull request #252 from AAEE86/nn
fix(startup): 收口 leader 失锁后的后台任务
2026-03-20 01:13:01 +08:00
fawney19 cae5e520ac fix(frontend): 修复 restoreOriginalPlaceholder 递归调用、优化日志参数格式,补全 tsconfig lib 配置 2026-03-20 01:06:15 +08:00
fawney19 772f2ea601 fix(vertex): SA 认证注入代理配置,细化 token 获取异常处理
- _auth_service_account 接收 endpoint 参数,通过 _get_proxy_config 解析代理
- vertex_auth 区分 TimeoutException/RequestError/通用异常,提供可读错误信息
- 新增测试覆盖代理传递和超时场景
2026-03-20 00:49:44 +08:00
fawney19 28fa03451c feat(provider): 模型测试支持自定义请求头,优化对话框布局与并发策略
- 前后端新增 request_headers 字段,测试时可自定义额外请求头
- ModelTestDialog 拆分为请求头/请求体并排双栏布局,增加格式化与重置按钮
- 区分 Pool 托管(并发5)和单 Key Provider(并发1)的测试并发数
- JsonImportInput 新增 multiple prop 支持单文件模式
- KeyFormDialog Service Account 输入改用 JsonImportInput,支持拖拽导入
2026-03-20 00:24:56 +08:00
fawney19 6984984c22 feat(provider): 重构模型测试对话框,加固 Vertex AI 传输层
模型测试:
- 将消息输入替换为完整 JSON 请求体编辑器,支持格式化和校验
- 新增端点选择面板,测试前可选择目标端点
- 新增调试检查器,可查看每次尝试的请求/响应头和体
- 结果视图改用 HorizontalRequestTimeline 组件展示请求追踪
- endpoint_checker 返回完整调试数据,通过 candidate extra_data 持久化

Vertex AI:
- 改进上下文检测逻辑,不再仅依赖 provider_type,支持从 base_url 推断
- Service Account 密钥现支持自动拉取模型(使用 auth_config 而非 api_key)
- 移除 Gemini Developer API 回退,API Key 仅走 Express 模式
- 端点表单为 Vertex AI 显示格式特定的默认路径模板
- 密钥格式校验仅在 auth_type/api_formats 变更时执行

其他:
- 禁用 ClaudeCode 提供商类型创建入口
- Dialog 组件新增 closeOnBackdrop 属性
2026-03-19 23:52:17 +08:00
AAEE86 1209c835c7 fix(startup): 收口 leader 失锁后的后台任务
- 为后台调度器注册失锁回调并只在 stop 成功后清空生命周期引用
- 停止调度器时移除定时 job,补充启动与任务协调器回归测试
- 降低多 worker 下重复调度风险,保持停机收口与统计聚合回归一致
2026-03-19 22:26:51 +08:00
fawney19 e4ebd5cca1 refactor(oauth): LinuxDo 备用端点回退、Basic Auth 认证,修复 session 外访问 ORM 对象
- LinuxDo provider: token/userinfo 请求增加 backup 端点自动回退
- LinuxDo provider: token 请求改用 HTTP Basic Auth 认证
- 授权 URL 构建: scope 为空时不再发送该参数
- OAuthService: 引入 OAuthAuthenticatedUser 快照,避免 DB session 关闭后访问 ORM 对象
- OAuthService: _handle_login_sync 设置 expire_on_commit=False 防止属性过期
- 新增 LinuxDo provider 单元测试(Basic Auth、端点回退)
- 新增 _handle_login_sync 返回快照的集成测试
2026-03-19 20:32:33 +08:00
fawney19 f573110725 fix(frontend): 用 CSS text-security 替代 password 输入框,简化配额进度条 UI
- Input 组件 masked 模式改用 WebkitTextSecurity: disc 替代 type=password,
  避免浏览器密码管理器自动填充干扰
- LoginDialog/UserFormDialog/Settings/ProxyNodes 密码字段统一迁移到 masked 属性
- PoolManagement 配额进度条布局从 grid 改为 flex,移除未使用的
  getQuotaProgressDisplayClass/getQuotaProgressTooltip 函数

Close #249
Co-authored-by: AAEE86 <[email protected]>
2026-03-19 20:04:56 +08:00
fawney19andAAEE86 a8620e133a fix(kiro): 加固 Kiro adapter 错误处理与请求构建逻辑
- 提取 request.py 统一 URL/headers/payload 构建,消除 handler_adapter_base 与 envelope 的重复逻辑
- 新增 error_enhancer 模块,分类 HTTP 状态码与连接错误,增强上游错误诊断信息
- envelope 实现 extract_error_text / on_http_status / on_connection_error,透传错误上下文到 eventstream rewriter
- KiroRequestContext 扩展错误状态字段,支持网络诊断信息传递
- provider_oauth_utils 改用 importlib 动态加载,避免 core 层对 services 的静态依赖
- idc auth_method 下跳过 profileArn,修复 usage 查询参数

Closes #247

Co-authored-by: AAEE86 <[email protected]>
2026-03-19 13:54:40 +08:00
RWDai ddd6adbcf7 feat(provider): support custom prompts for model tests (#242) 2026-03-19 13:11:54 +08:00
fawney19 b570aaac48 fix: 补全 OpenAI 工具参数 schema 中缺失的 properties 字段
OpenAI API 要求 type=object 的 schema 节点必须声明 properties,
否则会拒绝请求。在 request_from_internal 出口处对工具参数 schema
进行深拷贝并递归补全缺失的空 properties,不影响内部表示。
2026-03-19 02:36:22 +08:00
fawney19 086efe6efe refactor(usage-queue): ACK 后立即删除消息,清理 consumer 元数据
- 消费成功/移入 DLQ 后立即 XDEL,避免主流 Redis 保留已入库历史
- 缩小 usage_queue_stream_maxlen 默认值:200000 -> 2000(仅作短暂缓冲)
- 启动时清理 pending=0 且长期闲置的旧 consumer(防 consumer group 元数据累积)
- 停机时主动 XGROUP DELCONSUMER 移除自身
- 移除 cache_fingerprint 模块及其对 telemetry/recording_helpers 的引用
- 同步更新相关测试,验证 xdel 调用及 consumer 生命周期行为
2026-03-19 02:09:09 +08:00
fawney19 56f3c95763 fix: 统一 input_context_expr 计算口径,移除按 api_format 分支的 CASE 逻辑
input_context_expr() 原先按 OpenAI/Gemini 和 Claude 分支计算输入上下文,
现统一为 input_tokens + cache_read_input_tokens,与 usage 表展示口径一致。
同步更新 admin 和 user_me 路由中的缓存命中率注释,并新增单元测试。
2026-03-19 01:35:00 +08:00
fawney19 8a6a961900 feat: 增强 OAuth 标识展示与 Codex 调试日志
- 优化 OAuth badge 支持 account_id/account_user_id,tooltip 展示完整身份信息
- 重构池管理额度进度条布局,倒计时独立展示并增加样式区分
- Codex 插件增加 token 解析快照日志,便于排查 OAuth 透传问题

Closes #245
Co-Authored-By: kayphoon <[email protected]>
2026-03-19 00:50:18 +08:00
kayphoon 6e55968487 feat: Codex account_name 透传并优化池列表 OAuth 标识与额度重置倒计时展示
(cherry picked from commit 1b9176fc4a)
2026-03-18 23:58:42 +08:00
fawney19 8d8cddcef6 fix(replay): rerun model mapping on replay target 2026-03-18 23:54:19 +08:00
RWDai b90d5095f1 Fix replay fallback when model mapping is missing 2026-03-18 23:54:19 +08:00
RWDai a4505b1281 Fix usage replay model remapping 2026-03-18 23:54:19 +08:00
fawney19 1d72a8f9c1 feat: 流式空闲超时、健康监控查询优化、限流桶内存上限与维护清理修复
Close #233

Co-authored-by: AAEE86 <[email protected]>

- cli_monitor_mixin: 引入 STREAM_IDLE_TIMEOUT_SECONDS(可通过环境变量配置),
  流传输开始后若超出空闲窗口无新 chunk 则提前取消并返回 504,避免长时间挂起
- stream_context: 新增 managed_recorded_bodies 上下文管理器,确保 chunks 在
  telemetry 完成后及时释放;stream_telemetry 使用该接口统一管理 response body 构建
- health endpoint: 将状态聚合改为 GROUP BY 直接统计,事件列表按 api_format
  单独查询,避免单次 limit 拉取大量记录导致的遗漏与性能问题;同时过滤不活跃
  provider/endpoint,与公开健康接口保持一致
- endpoint health service: 修正时间线数据按 endpoint_id 而非 key_id 聚合
- token_bucket: 引入 max_buckets/bucket_expiry 上限与定时清理,防止内存无限增长;
  修复 refill_rate=0 时 get_reset_time 除零异常;新增 _is_unlimited_rate_limit 判断
- maintenance_scheduler: 调整清理顺序(先删整行再按窗口清理),新增 newer_than
  边界参数,避免同一行在同一轮中被重复改写
- sync_execute: 新增 create_pending_usage 开关,允许已预创建记录的调用方跳过重复创建
- quota_reader / provider_ops balance: 小幅修复与健壮性提升
- Dockerfile: 添加 MALLOC_ARENA_MAX=2 环境变量以降低 gunicorn worker RSS
- 补充相关测试覆盖
2026-03-18 23:38:26 +08:00
fawney19 3d5b6141a5 feat(codex): 引入 upstream_headers hook 机制,为 Codex 注入 session/conversation/account headers
- 新增 upstream_headers.py:可注册 provider+endpoint 维度的 extra headers 构建 hook
- Codex openai:cli 注入 session_id + conversation_id(由 prompt_cache_key sha256 派生)
- Codex openai:compact 注入 chatgpt-account-id(来自 auth_config)+ session_id,不注入 conversation_id
- 修复 prompt_cache:compact 格式现统一为 codex 策略,不再跳过注入
- chat_handler_base / cli_request_mixin 均在 extra_headers 阶段调用 build_upstream_extra_headers
2026-03-18 20:43:14 +08:00
fawney19 203cd5a9d5 fix(redis): 按事件循环隔离 Redis 连接,防止子线程 asyncio.run 导致连接泄漏
RedisClientManager 新增 _redis_by_loop 字典,按 event loop id 维护独立连接,
避免 usage consumer 在 asyncio.to_thread + asyncio.run 场景下复用主循环连接。

同步重构 consumer_streams 的写库路径:_apply_record_event 统一走
record_usage_batch,_apply_streaming_event 改为 to_thread 执行同步 DB 操作,
移除冗余的 session 传递和手动 rollback 逻辑。
2026-03-18 17:56:48 +08:00
fawney19 696ec65175 refactor(normalizer): 移除 request key reorder 机制,保持自然插入顺序
移除 OpenAI/OpenAI CLI normalizer 中的 _reorder_request_prefix_keys 及
基类 _reorder_request_keys 死代码,request_from_internal 直接返回构建
顺序的 dict,测试同步更新为验证自然插入顺序。
2026-03-18 14:01:56 +08:00
fawney19 cbb66a5667 refactor(task): 引入 MutableRequestBodyState 替代 request_body_ref 字典容器
将请求体可变状态从 {"body": dict} 字典容器重构为独立的
MutableRequestBodyState 类,统一管理 original_body / current_body /
build_attempt_body / rectify 等语义,消除各层通过 ref["body"] 间接
读写的隐式约定。

- 新增 src/services/task/request_state.py 定义 Protocol 与实现
- handler/executor/mixin 层改用 request_state 参数传递
- error_handler/state_transition 通过 request_state 判断整流状态
- 新增 request_state 单元测试与 chat/cli 请求体隔离测试
2026-03-18 13:43:39 +08:00
fawney19 53ef35ec80 refactor(cli): 提取 _build_upstream_request 统一流式/非流式的上游请求构建逻辑
将 cli_stream_mixin 和 cli_sync_mixin 中重复的上游请求构建代码
(provider behavior / stream policy / envelope / auth / RequestBuilder / URL 构建)
提取到 cli_request_mixin._build_upstream_request,返回 CliUpstreamRequestResult dataclass。
2026-03-18 13:13:47 +08:00
fawney19 1af3067303 refactor(codex): 移除 envelope/request_patching 层,用 context var 统一 compact 状态判断
- 删除 CodexOAuthEnvelope 和 request_patching 模块,Codex 不再需要 envelope 层
- 移除 _aether_compact 请求体内部标记,改用 is_codex_compact_request() 集中查询
- 简化 OpenAI CLI adapter,移除 Codex 专用的 get_cli_extra_headers/build_test_request_body 逻辑
- 移除 Codex behavior variant 注册(same_format/cross_format)
- normalizer patch_same_format_request 对 codex 变为 no-op
- 更新相关测试适配新的架构
2026-03-18 12:35:57 +08:00
fawney19 d026398bab refactor(body-rules): 移除 protected_body_keys 机制,允许 body_rules 自由修改所有请求体字段
删除 get_cache_sensitive_protected_body_keys 函数及相关常量、_is_protected_path
辅助函数,从 apply_body_rules、RequestBuilder、ProviderRequestResult 等处移除
protected_body_keys 参数,更新所有调用点和测试用例。
2026-03-18 10:52:07 +08:00
fawney19 684689a82b fix(usage): session touch 独立提交避免行锁阻塞 & 管理员页面顺序加载降低并发压力
后端: 将 session touch 的 commit 从请求事务中分离,防止管理员 usage
页面的长查询持有 user_sessions 行锁阻塞后续请求。touch_session 改为
返回 bool 以支持按需提交。

前端: 管理员 Usage 页面将并行 API 调用改为顺序加载,优先显示记录表格,
统计面板在后台异步刷新,避免瞬时并发打满后端 worker。loadRecords 支持
传入 dateRange 参数确保时间范围一致性。
2026-03-18 00:13:28 +08:00
github-actions[bot] eeb5f41bad chore(proxy): update download links for proxy-v0.2.5 2026-03-17 14:23:27 +00:00
3792 changed files with 1223951 additions and 274023 deletions
+6 -16
View File
@@ -1,24 +1,14 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
env/
venv/
ENV/
.venv
.uv/
*.egg-info/
dist/
!aether-hub/dist/aether-hub
# Build artifacts
build/
target/
*.so
*.egg
*.egg-info/
# Frontend
frontend/node_modules/
frontend/dist/
frontend/.vite/
# frontend/dist/ - 注释掉,因为我们需要预构建的dist文件
# Development
.git/
@@ -60,4 +50,4 @@ Dockerfile.*
# Deployment
deploy/
scripts/
scripts/
+63 -75
View File
@@ -1,19 +1,48 @@
# ==================== 必须配置(启动前) ====================
# 以下配置项必须在项目启动前设置
# 应用端口(默认 8084)
APP_PORT=8084
# 对外访问地址,用于一键安装、CC Switch 导入、支付回调等需要生成公网 URL 的场景。
# 生产环境建议显式配置为不带内部端口的公网域名,例如 https://aether.example.com
# AETHER_PUBLIC_BASE_URL=https://aether.example.com
# Docker Compose 镜像(默认正式版 latest;提前测试可改 rc/beta;也可固定具体版本)
# 示例:
# APP_IMAGE=ghcr.io/fawney19/aether:latest
# APP_IMAGE=ghcr.io/fawney19/aether:rc
# APP_IMAGE=ghcr.io/fawney19/aether:beta
# APP_IMAGE=ghcr.io/fawney19/aether:0.7.0-rc.1
# API Key 前缀(默认 sk)
API_KEY_PREFIX=sk
# Rust 日志过滤(默认 aether_gateway=info)
# 示例: aether_gateway=debug,sqlx=warn
RUST_LOG=aether_gateway=info
# CORS 配置(跨域带 Cookie 时不要写 *,必须显式列出前端源)
# 示例: http://localhost:5173,https://app.example.com
# CORS_ORIGINS=http://localhost:5173
# CORS_ALLOW_CREDENTIALS=true
# 如果前后端跨站并依赖登录刷新 Cookie,还要配合:
# AUTH_REFRESH_COOKIE_SAMESITE=None
# AUTH_REFRESH_COOKIE_SECURE=true
# 数据库配置
DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_NAME=aether
DB_PASSWORD=your_secure_password_here
DB_PASSWORD=aether
# Redis 配置
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=your_redis_password_here
REDIS_PASSWORD=aether
# JWT密钥(使用 python generate_keys.py 生成)
# JWT密钥(使用 ./generate_keys.sh 生成)
# 用于用户登录 token 签名,更换后所有用户需重新登录
JWT_SECRET_KEY=change-this-to-a-secure-random-string
@@ -21,82 +50,41 @@ JWT_SECRET_KEY=change-this-to-a-secure-random-string
# 注意:更换此密钥后需要在管理面板重新配置所有 Provider API Key
ENCRYPTION_KEY=change-this-to-another-secure-random-string
# 支付回调共享密钥(公开 /api/payment/callback/* 入口必须携带 x-payment-callback-token)
# 建议使用 32+ 位随机字符串
PAYMENT_CALLBACK_SECRET=change-this-to-a-secure-callback-secret
# 管理员账号(仅首次初始化时使用, 创建完成后可在系统内修改密码)
# 启动自举管理员(仅在当前库里还没有活动管理员时生效)
# 手动部署时取消注释并设置;install.sh 首次生成配置时会提示输入。
ADMIN_EMAIL=[email protected]
ADMIN_USERNAME=admin
ADMIN_PASSWORD=admin123456
ADMIN_USERNAME=admin123456
# ADMIN_PASSWORD=
# ==================== 可选配置(有默认值) ====================
# 以下配置项有合理的默认值,可按需调整
# 应用端口(默认 8084)
# APP_PORT=8084
# 可信反向代理 IP/CIDR,只有这些来源发送的 X-Real-IP / X-Forwarded-For 会被采用。
# 默认仅信任本机回环代理:127.0.0.0/8,::1/128。
# Docker/Nginx 位于独立容器时,请按实际容器网络设置,例如:172.16.0.0/12。
# AETHER_TRUSTED_PROXY_CIDRS=127.0.0.0/8,::1/128,172.16.0.0/12
# 生产部署镜像(deploy.sh 会读取)
# APP_IMAGE=ghcr.io/fawney19/aether:latest
# docker compose 下 app 启动前自动执行 pending migration/backfill(默认 true)
# AETHER_GATEWAY_AUTO_PREPARE_DATABASE=true
# Gunicorn Worker 数量(默认 2)
# Tunnel 请求统一经 Hub 转发,可安全使用多 worker。
# 非 Docker 运行时若使用 ProxyNode tunnel,请确保 aether-hub 可达(默认 ws://127.0.0.1:8085)。
# GUNICORN_WORKERS=2
# PostgreSQL 连接池配置(默认每核 4 条、总池至少 32 条且最多 100 条;多实例部署应显式分配每实例预算)
# AETHER_GATEWAY_DATA_POSTGRES_MIN_CONNECTIONS=12
# AETHER_GATEWAY_DATA_POSTGRES_MAX_CONNECTIONS=80
# AETHER_GATEWAY_MAX_IN_FLIGHT_REQUESTS=2048
# AETHER_GATEWAY_REQUEST_BODY_BUFFER_BUDGET_MB=256
# AETHER_GATEWAY_REQUEST_BODY_READ_TIMEOUT_MS=120000
# 可选的 Payload 上限(MiB);默认及 0 均表示不限制。
# AETHER_MAX_REQUEST_BODY_MB=0
# AETHER_GATEWAY_SECURITY_CACHE_TTL_MS=1000
# AETHER_MAX_REDACTED_SYNC_RESPONSE_BODY_MB=0
# AETHER_MAX_INTERNAL_BUFFERED_BODY_MB=0
# AETHER_TUNNEL_NODE_STATUS_QUEUE_CAPACITY=1024
# Gunicorn Max Requests(默认 4000)
# Worker 处理指定数量请求后自动重启,防止内存泄漏
# max-requests-jitter 会自动设置为 MAX_REQUESTS/20 (5%)
# MAX_REQUESTS=4000
# HTTP 连接池上限(默认总预算约 200,按 worker 平分)
# 如果容器内存偏高,可继续下调;例如 2 worker 时设为 80-100
# HTTP_MAX_CONNECTIONS=100
# HTTP 保活连接数(默认约为 max_connections 的 30%)
# HTTP_KEEPALIVE_CONNECTIONS=30
# HTTP 代理/Tunnel 客户端空闲清理(默认每 5 分钟扫描,空闲 600 秒即关闭)
# HTTP_CLIENT_IDLE_CLEANUP_INTERVAL_MINUTES=5
# HTTP_CLIENT_IDLE_CLEANUP_MAX_SECONDS=600
# curl_cffi session 池上限(默认 20,按 impersonate + proxy 组合缓存)
# CURL_CFFI_MAX_SESSIONS=20
# 流式响应块缓存上限(单位 MB,默认 2)
# 说明:
# - 这是单个流式请求可保留的“解析后响应块”内存上限,不是全局上限
# - 粗略峰值内存 ≈ 并发流数量 × RESPONSE_CHUNKS_MAX_SIZE_MB
# 例如:100 并发、2MB 上限,理论峰值约 200MB
# - 建议:
# - 内存敏感环境:1
# - 通用生产环境:2(默认)
# - 需要更多调试上下文:4
# RESPONSE_CHUNKS_MAX_SIZE_MB=2
# API Key 前缀(默认 sk)
# API_KEY_PREFIX=sk
# 日志级别(默认 INFO,可选:DEBUG, INFO, WARNING, ERROR)
# LOG_LEVEL=INFO
# CORS 配置(允许跨域的源,多个源用逗号分隔)
# 示例: http://localhost:3000,https://example.com
# 默认: * (允许所有源)
# CORS_ORIGINS=*
# 启动预热配置(默认启用,降低首请求冷启动延迟)
# 是否启用启动期预热任务(默认 true)
# STARTUP_WARMUP_ENABLED=true
# /readyz 是否等待预热完成(默认 true)
# STARTUP_WARMUP_GATE_READINESS=true
# 预热时优先 bootstrap 的 provider_type 列表(逗号分隔;留空表示自动探测)
# STARTUP_WARMUP_PROVIDER_TYPES=codex,kiro
# ==================== 计费系统(可选) ====================
# Video/Image/Audio 缺失 billing_rule 时是否拒绝请求(默认 false:允许请求但 cost=0 并告警)
# BILLING_REQUIRE_RULE=false
#
# required 维度缺失时是否拒绝请求/标记任务失败(默认 false:cost=0 + 标记 incomplete)
# BILLING_STRICT_MODE=false
#
# PostgreSQL 容器调优:docker-compose.yml 已内置通用默认值,通常不用配置。
# 只有在 Postgres 独占大内存、或压测显示 DB 缓存/排序/维护任务成为瓶颈时再覆盖。
# 内置默认:shared_buffers=1GB, effective_cache_size=3GB, shm_size=512mb,
# work_mem=16MB, maintenance_work_mem=256MB。
# POSTGRES_SHARED_BUFFERS=8GB
# POSTGRES_EFFECTIVE_CACHE_SIZE=24GB
# POSTGRES_SHM_SIZE=2gb
# POSTGRES_WORK_MEM=16MB
# POSTGRES_MAINTENANCE_WORK_MEM=1GB
-91
View File
@@ -1,91 +0,0 @@
name: Build aether-hub
on:
push:
tags: ['hub-v*']
workflow_dispatch:
permissions:
contents: write
jobs:
build:
name: ${{ matrix.name }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- name: linux-amd64
target: x86_64-unknown-linux-gnu
use_cross: true
- name: linux-arm64
target: aarch64-unknown-linux-gnu
use_cross: true
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.target }}
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
workspaces: aether-hub -> target
key: ${{ matrix.target }}
- name: Install cross
if: matrix.use_cross
uses: taiki-e/install-action@cross
- name: Build
working-directory: aether-hub
shell: bash
run: |
if [ "${{ matrix.use_cross }}" = "true" ]; then
cross build --release --target ${{ matrix.target }}
else
cargo build --release --target ${{ matrix.target }}
fi
- name: Package
shell: bash
run: |
cd aether-hub/target/${{ matrix.target }}/release
chmod +x aether-hub
tar czf ../../../../aether-hub-${{ matrix.name }}.tar.gz aether-hub
- name: Upload artifact
uses: actions/upload-artifact@v5
with:
name: aether-hub-${{ matrix.name }}
path: aether-hub-*.tar.gz
if-no-files-found: error
release:
needs: build
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- name: Download all artifacts
uses: actions/download-artifact@v5
with:
merge-multiple: true
path: artifacts
- name: Generate checksums
working-directory: artifacts
run: sha256sum aether-hub-* > SHA256SUMS.txt
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
name: "${{ github.ref_name }}"
generate_release_notes: true
files: |
artifacts/aether-hub-*
artifacts/SHA256SUMS.txt
fail_on_unmatched_files: true
-227
View File
@@ -1,227 +0,0 @@
name: Build aether-proxy
on:
push:
tags: ['proxy-v*']
workflow_dispatch:
permissions:
contents: write
packages: write
env:
REGISTRY: ghcr.io
GHCR_IMAGE: fawney19/aether-proxy
DOCKERHUB_IMAGE: fawney19/aether-proxy
jobs:
build:
name: ${{ matrix.name }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- name: linux-amd64
target: x86_64-unknown-linux-gnu
os: ubuntu-latest
use_cross: true
- name: linux-arm64
target: aarch64-unknown-linux-gnu
os: ubuntu-latest
use_cross: true
- name: macos-amd64
target: x86_64-apple-darwin
os: macos-latest
use_cross: false
- name: macos-arm64
target: aarch64-apple-darwin
os: macos-latest
use_cross: false
- name: windows-amd64
target: x86_64-pc-windows-msvc
os: windows-latest
use_cross: false
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.target }}
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
workspaces: aether-proxy -> target
key: ${{ matrix.target }}
- name: Install cross
if: matrix.use_cross
uses: taiki-e/install-action@cross
- name: Build
working-directory: aether-proxy
shell: bash
run: |
if [ "${{ matrix.use_cross }}" = "true" ]; then
cross build --release --target ${{ matrix.target }}
else
cargo build --release --target ${{ matrix.target }}
fi
- name: Package (Unix)
if: runner.os != 'Windows'
shell: bash
run: |
cd aether-proxy/target/${{ matrix.target }}/release
chmod +x aether-proxy
tar czf ../../../../aether-proxy-${{ matrix.name }}.tar.gz aether-proxy
- name: Package (Windows)
if: runner.os == 'Windows'
shell: bash
run: |
cd aether-proxy/target/${{ matrix.target }}/release
7z a ../../../../aether-proxy-${{ matrix.name }}.zip aether-proxy.exe
- name: Upload artifact
uses: actions/upload-artifact@v5
with:
name: aether-proxy-${{ matrix.name }}
path: |
aether-proxy-*.tar.gz
aether-proxy-*.zip
if-no-files-found: error
release:
needs: build
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- name: Download all artifacts
uses: actions/download-artifact@v5
with:
merge-multiple: true
path: artifacts
- name: Generate checksums
working-directory: artifacts
run: sha256sum aether-proxy-* > SHA256SUMS.txt
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
name: "${{ github.ref_name }}"
generate_release_notes: true
files: |
artifacts/aether-proxy-*
artifacts/SHA256SUMS.txt
fail_on_unmatched_files: true
docker:
needs: build
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/checkout@v5
- name: Download Linux artifacts
uses: actions/download-artifact@v5
with:
pattern: aether-proxy-linux-*
merge-multiple: true
path: artifacts
- name: Prepare binaries
run: |
mkdir -p aether-proxy/build/linux-amd64 aether-proxy/build/linux-arm64
tar xzf artifacts/aether-proxy-linux-amd64.tar.gz -C aether-proxy/build/linux-amd64
tar xzf artifacts/aether-proxy-linux-arm64.tar.gz -C aether-proxy/build/linux-arm64
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY }}/${{ env.GHCR_IMAGE }}
docker.io/${{ env.DOCKERHUB_IMAGE }}
tags: |
type=match,pattern=proxy-v(.*),group=1
type=match,pattern=proxy-v(\d+\.\d+),group=1
type=sha,prefix=
flavor: |
latest=auto
- name: Build and push
uses: docker/build-push-action@v6
with:
context: ./aether-proxy
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: linux/amd64,linux/arm64
update-readme:
needs: release
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/checkout@v5
with:
ref: master
- name: Update README download links
env:
TAG: ${{ github.ref_name }}
run: |
VERSION="${TAG#proxy-v}"
BASE="https://github.com/fawney19/Aether/releases/download/${TAG}"
cd aether-proxy
TABLE="| Platform | Download |\n|----------|----------|\n"
TABLE+="| Linux x86_64 | [aether-proxy-linux-amd64.tar.gz](${BASE}/aether-proxy-linux-amd64.tar.gz) |\n"
TABLE+="| Linux ARM64 | [aether-proxy-linux-arm64.tar.gz](${BASE}/aether-proxy-linux-arm64.tar.gz) |\n"
TABLE+="| macOS x86_64 | [aether-proxy-macos-amd64.tar.gz](${BASE}/aether-proxy-macos-amd64.tar.gz) |\n"
TABLE+="| macOS ARM64 | [aether-proxy-macos-arm64.tar.gz](${BASE}/aether-proxy-macos-arm64.tar.gz) |\n"
TABLE+="| Windows x86_64 | [aether-proxy-windows-amd64.zip](${BASE}/aether-proxy-windows-amd64.zip) |"
# Replace content between markers
if grep -q '<!-- DOWNLOAD_TABLE_START -->' README.md; then
awk -v table="$TABLE" '
/<!-- DOWNLOAD_TABLE_START -->/ { print; printf "%s\n", table; skip=1; next }
/<!-- DOWNLOAD_TABLE_END -->/ { skip=0 }
!skip { print }
' README.md > README.tmp && mv README.tmp README.md
fi
- name: Commit and push
run: |
cd aether-proxy
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add README.md
git diff --cached --quiet && exit 0
TAG="${GITHUB_REF#refs/tags/}"
git commit -m "chore(proxy): update download links for ${TAG}"
git push
+239
View File
@@ -0,0 +1,239 @@
name: Build aether-tunnel
on:
push:
tags: ['tunnel-v*']
workflow_dispatch:
permissions:
contents: write
concurrency:
group: build-tunnel-${{ github.ref }}
cancel-in-progress: false
jobs:
preflight:
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/checkout@v5
- name: Ensure tunnel tag matches Cargo version
shell: bash
run: |
TAG="${GITHUB_REF_NAME}"
EXPECTED="${TAG#tunnel-v}"
ACTUAL="$(cargo metadata --manifest-path apps/aether-tunnel/Cargo.toml --locked --no-deps --format-version 1 | jq -r '.packages[] | select(.name == "aether-tunnel") | .version')"
echo "tag version: ${EXPECTED}"
echo "cargo version: ${ACTUAL}"
if [ -z "${ACTUAL}" ]; then
echo "Could not resolve aether-tunnel package version" >&2
exit 1
fi
if [ "${EXPECTED}" != "${ACTUAL}" ]; then
echo "tunnel tag ${TAG} does not match apps/aether-tunnel/Cargo.toml version ${ACTUAL}" >&2
exit 1
fi
build:
needs: preflight
if: always() && (needs.preflight.result == 'success' || needs.preflight.result == 'skipped')
name: ${{ matrix.name }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- name: linux-amd64
target: x86_64-unknown-linux-gnu
os: ubuntu-latest
use_cross: true
- name: linux-arm64
target: aarch64-unknown-linux-gnu
os: ubuntu-latest
use_cross: true
- name: linux-musl-amd64
target: x86_64-unknown-linux-musl
os: ubuntu-latest
use_cross: true
- name: linux-musl-arm64
target: aarch64-unknown-linux-musl
os: ubuntu-latest
use_cross: true
- name: macos-amd64
target: x86_64-apple-darwin
os: macos-15-intel
use_cross: false
- name: macos-arm64
target: aarch64-apple-darwin
os: macos-15
use_cross: false
- name: windows-amd64
target: x86_64-pc-windows-msvc
os: windows-latest
use_cross: false
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.target }}
- name: Ensure Rust target is installed
run: rustup target add ${{ matrix.target }}
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
workspaces: apps/aether-tunnel -> target
key: ${{ matrix.target }}
- name: Install cross
if: matrix.use_cross
uses: taiki-e/install-action@cross
- name: Build
working-directory: apps/aether-tunnel
shell: bash
run: |
if [ "${{ matrix.use_cross }}" = "true" ]; then
cross build --release --locked --target ${{ matrix.target }}
else
cargo build --release --locked --target ${{ matrix.target }}
fi
- name: Package (Unix)
if: runner.os != 'Windows'
shell: bash
run: |
cd target/${{ matrix.target }}/release
chmod +x aether-tunnel
tar czf ../../../aether-tunnel-${{ matrix.name }}.tar.gz aether-tunnel
- name: Package (Windows)
if: runner.os == 'Windows'
shell: bash
run: |
cd target/${{ matrix.target }}/release
7z a ../../../aether-tunnel-${{ matrix.name }}.zip aether-tunnel.exe
- name: Upload artifact
uses: actions/upload-artifact@v5
with:
name: aether-tunnel-${{ matrix.name }}
path: |
aether-tunnel-*.tar.gz
aether-tunnel-*.zip
if-no-files-found: error
retention-days: 1
release:
needs: build
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- name: Download all artifacts
uses: actions/download-artifact@v5
with:
merge-multiple: true
path: artifacts
- name: Generate checksums
working-directory: artifacts
run: sha256sum aether-tunnel-* > SHA256SUMS.txt
- name: Delete stale draft releases for tag
env:
GH_TOKEN: ${{ github.token }}
RELEASE_TAG: ${{ github.ref_name }}
REPOSITORY: ${{ github.repository }}
shell: bash
run: |
set -euo pipefail
draft_ids="$(gh api "repos/${REPOSITORY}/releases" --paginate --jq '.[] | select(.tag_name == env.RELEASE_TAG and .draft == true) | .id')"
if [[ -z "${draft_ids}" ]]; then
echo "No stale draft releases for ${RELEASE_TAG}"
exit 0
fi
while IFS= read -r release_id; do
[[ -z "${release_id}" ]] && continue
echo "Deleting stale draft release ${release_id} for ${RELEASE_TAG}"
gh api -X DELETE "repos/${REPOSITORY}/releases/${release_id}"
done <<< "${draft_ids}"
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
name: "${{ github.ref_name }}"
generate_release_notes: true
files: |
artifacts/aether-tunnel-*
artifacts/SHA256SUMS.txt
fail_on_unmatched_files: true
update-readme:
needs: release
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/')
steps:
- uses: actions/checkout@v5
with:
ref: main
- name: Update README download links
env:
TAG: ${{ github.ref_name }}
run: |
VERSION="${TAG#tunnel-v}"
BASE="https://github.com/fawney19/Aether/releases/download/${TAG}"
if [ -d apps/aether-tunnel ]; then
TUNNEL_DIR="apps/aether-tunnel"
else
TUNNEL_DIR="aether-tunnel"
fi
cd "$TUNNEL_DIR"
TABLE="| Platform | Download |\n|----------|----------|\n"
TABLE+="| Linux x86_64 (GNU) | [aether-tunnel-linux-amd64.tar.gz](${BASE}/aether-tunnel-linux-amd64.tar.gz) |\n"
TABLE+="| Linux ARM64 (GNU) | [aether-tunnel-linux-arm64.tar.gz](${BASE}/aether-tunnel-linux-arm64.tar.gz) |\n"
TABLE+="| Linux x86_64 (musl) | [aether-tunnel-linux-musl-amd64.tar.gz](${BASE}/aether-tunnel-linux-musl-amd64.tar.gz) |\n"
TABLE+="| Linux ARM64 (musl) | [aether-tunnel-linux-musl-arm64.tar.gz](${BASE}/aether-tunnel-linux-musl-arm64.tar.gz) |\n"
TABLE+="| macOS x86_64 | [aether-tunnel-macos-amd64.tar.gz](${BASE}/aether-tunnel-macos-amd64.tar.gz) |\n"
TABLE+="| macOS ARM64 | [aether-tunnel-macos-arm64.tar.gz](${BASE}/aether-tunnel-macos-arm64.tar.gz) |\n"
TABLE+="| Windows x86_64 | [aether-tunnel-windows-amd64.zip](${BASE}/aether-tunnel-windows-amd64.zip) |"
# Replace content between markers
if grep -q '<!-- DOWNLOAD_TABLE_START -->' README.md; then
awk -v table="$TABLE" '
/<!-- DOWNLOAD_TABLE_START -->/ { print; printf "%s\n", table; skip=1; next }
/<!-- DOWNLOAD_TABLE_END -->/ { skip=0 }
!skip { print }
' README.md > README.tmp && mv README.tmp README.md
fi
- name: Commit and push
run: |
if [ -d apps/aether-tunnel ]; then
TUNNEL_DIR="apps/aether-tunnel"
else
TUNNEL_DIR="aether-tunnel"
fi
cd "$TUNNEL_DIR"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add README.md
git diff --cached --quiet && exit 0
TAG="${GITHUB_REF#refs/tags/}"
git commit -m "chore(tunnel): update download links for ${TAG}"
git push
+28
View File
@@ -15,7 +15,35 @@ concurrency:
cancel-in-progress: false
jobs:
preflight:
runs-on: ubuntu-latest
outputs:
deploy_pages: ${{ steps.classify.outputs.deploy_pages }}
steps:
- name: Ensure stable Pages release tag
id: classify
shell: bash
run: |
set -euo pipefail
echo "deploy_pages=false" >> "${GITHUB_OUTPUT}"
if [[ "${GITHUB_REF_TYPE}" != "tag" ]]; then
echo "Manual Pages deployment."
echo "deploy_pages=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
tag="${GITHUB_REF_NAME}"
if [[ ! "${tag}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "Skipping Pages deploy for non-stable release tag: ${tag}"
exit 0
fi
echo "deploy_pages=true" >> "${GITHUB_OUTPUT}"
build:
needs: preflight
if: needs.preflight.outputs.deploy_pages == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
-327
View File
@@ -1,327 +0,0 @@
name: Build and Publish Docker Image
on:
push:
tags: ['v*']
workflow_dispatch:
inputs:
build_base:
description: 'Rebuild base image'
required: false
default: false
type: boolean
env:
REGISTRY: ghcr.io
BASE_IMAGE_NAME: fawney19/aether-base
APP_IMAGE_NAME: fawney19/aether
GITHUB_REPO: fawney19/Aether
# Base image hash inputs:
# - Dockerfile.base
# - pyproject.toml (dependency fingerprint only; ignores tool/optional deps)
# - frontend/package-lock.json
jobs:
check-base-changes:
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
outputs:
base_changed: ${{ steps.check.outputs.base_changed }}
steps:
- uses: actions/checkout@v5
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Check if base image needs rebuild
id: check
run: |
if [ "${{ github.event.inputs.build_base }}" == "true" ]; then
echo "base_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
# Calculate current hash of base-related inputs (dependency-only fingerprint)
PY_FINGERPRINT=$(python3 - <<'PY'
import json
import pathlib
import tomllib
data = tomllib.loads(pathlib.Path("pyproject.toml").read_text("utf-8"))
project = data.get("project") or {}
build = data.get("build-system") or {}
fingerprint = {
"requires-python": project.get("requires-python"),
"dependencies": sorted(project.get("dependencies") or []),
"build-backend": build.get("build-backend"),
"build-requires": sorted(build.get("requires") or []),
}
print(json.dumps(fingerprint, sort_keys=True, separators=(",", ":")))
PY
)
CURRENT_HASH=$(
(
cat Dockerfile.base
printf '%s\n' "$PY_FINGERPRINT"
cat frontend/package-lock.json
) | sha256sum | cut -d' ' -f1
)
echo "Current base hash: $CURRENT_HASH"
# Try to get hash label from remote image config
# Pull the image config and extract labels
REMOTE_HASH=""
if docker pull ${{ env.REGISTRY }}/${{ env.BASE_IMAGE_NAME }}:latest; then
REMOTE_HASH=$(docker inspect ${{ env.REGISTRY }}/${{ env.BASE_IMAGE_NAME }}:latest --format '{{ index .Config.Labels "org.opencontainers.image.base.hash" }}' 2>/dev/null) || true
else
echo "WARN: failed to pull remote base image; forcing base rebuild."
echo "base_changed=true" >> $GITHUB_OUTPUT
exit 0
fi
if [ -z "$REMOTE_HASH" ] || [ "$REMOTE_HASH" == "<no value>" ]; then
# No remote image or no hash label, need to rebuild
echo "No remote base image or hash label found, need rebuild"
echo "base_changed=true" >> $GITHUB_OUTPUT
elif [ "$CURRENT_HASH" != "$REMOTE_HASH" ]; then
echo "Hash mismatch: remote=$REMOTE_HASH, current=$CURRENT_HASH"
echo "base_changed=true" >> $GITHUB_OUTPUT
else
echo "Hash matches, no rebuild needed"
echo "base_changed=false" >> $GITHUB_OUTPUT
fi
build-base:
needs: check-base-changes
if: needs.check-base-changes.outputs.base_changed == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v5
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Calculate base files hash
id: hash
run: |
PY_FINGERPRINT=$(python3 - <<'PY'
import json
import pathlib
import tomllib
data = tomllib.loads(pathlib.Path("pyproject.toml").read_text("utf-8"))
project = data.get("project") or {}
build = data.get("build-system") or {}
fingerprint = {
"requires-python": project.get("requires-python"),
"dependencies": sorted(project.get("dependencies") or []),
"build-backend": build.get("build-backend"),
"build-requires": sorted(build.get("requires") or []),
}
print(json.dumps(fingerprint, sort_keys=True, separators=(",", ":")))
PY
)
HASH=$(
(
cat Dockerfile.base
printf '%s\n' "$PY_FINGERPRINT"
cat frontend/package-lock.json
) | sha256sum | cut -d' ' -f1
)
echo "hash=$HASH" >> $GITHUB_OUTPUT
- name: Extract metadata for base image
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.BASE_IMAGE_NAME }}
tags: |
type=raw,value=latest
type=sha,prefix=
labels: |
org.opencontainers.image.base.hash=${{ steps.hash.outputs.hash }}
- name: Build and push base image
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile.base
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=base
cache-to: type=gha,mode=max,scope=base
platforms: linux/amd64,linux/arm64
download-hub:
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
hub_tag: ${{ steps.hub-tag.outputs.tag }}
steps:
- name: Get latest hub release tag
id: hub-tag
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
TAG=$(gh release list --repo "${{ env.GITHUB_REPO }}" --limit 50 --json tagName,isDraft,isPrerelease \
--jq '[.[] | select(.tagName | startswith("hub-v")) | select(.isDraft == false and .isPrerelease == false)] | .[0].tagName')
if [ -z "$TAG" ] || [ "$TAG" = "null" ]; then
echo "No hub release found"
exit 1
fi
echo "tag=$TAG" >> $GITHUB_OUTPUT
echo "Hub release tag: $TAG"
build-app:
needs: [check-base-changes, build-base, download-hub]
if: always() && (needs.build-base.result == 'success' || needs.build-base.result == 'skipped') && needs.download-hub.result == 'success'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v5
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to Container Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Extract metadata for app image
id: meta
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY }}/${{ env.APP_IMAGE_NAME }}
docker.io/fawney19/aether
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=raw,value=pre,enable=${{ contains(github.ref, '-') }}
type=raw,value=fix,enable=${{ contains(github.ref, '-fix') }}
type=sha,prefix=
flavor: |
latest=auto
- name: Extract version from tag
id: version
run: |
# 从 tag 提取版本号,如 v0.2.5 -> 0.2.5
VERSION="${GITHUB_REF#refs/tags/v}"
if [ "$VERSION" = "$GITHUB_REF" ]; then
# 不是 tag 触发,使用 git describe
VERSION=$(git describe --tags --always | sed 's/^v//')
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "Extracted version: $VERSION"
- name: Update Dockerfile.app to use registry base image
run: |
sed -i "s|FROM aether-base:latest AS builder|FROM ${{ env.REGISTRY }}/${{ env.BASE_IMAGE_NAME }}:latest AS builder|g" Dockerfile.app
- name: Generate version file
run: |
# 生成 _version.py 文件
cat > src/_version.py << EOF
# Auto-generated by CI
__version__ = '${{ steps.version.outputs.version }}'
__version_tuple__ = tuple(int(x) for x in '${{ steps.version.outputs.version }}'.split('.') if x.isdigit())
version = __version__
version_tuple = __version_tuple__
EOF
- name: Resolve hub release for build args
run: |
echo "Hub release tag: ${{ needs.download-hub.outputs.hub_tag }}"
- name: Build and push app image (amd64)
id: build-amd64
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile.app
labels: ${{ steps.meta.outputs.labels }}
no-cache-filters: builder
cache-from: type=gha,scope=app-amd64
cache-to: type=gha,mode=min,scope=app-amd64
build-args: |
HUB_RELEASE_REPO=${{ env.GITHUB_REPO }}
HUB_TAG=${{ needs.download-hub.outputs.hub_tag }}
platforms: linux/amd64
outputs: type=image,"name=${{ env.REGISTRY }}/${{ env.APP_IMAGE_NAME }},docker.io/fawney19/aether",push-by-digest=true,name-canonical=true,push=true
- name: Build and push app image (arm64)
id: build-arm64
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile.app
labels: ${{ steps.meta.outputs.labels }}
no-cache-filters: builder
cache-from: type=gha,scope=app-arm64
cache-to: type=gha,mode=min,scope=app-arm64
build-args: |
HUB_RELEASE_REPO=${{ env.GITHUB_REPO }}
HUB_TAG=${{ needs.download-hub.outputs.hub_tag }}
platforms: linux/arm64
outputs: type=image,"name=${{ env.REGISTRY }}/${{ env.APP_IMAGE_NAME }},docker.io/fawney19/aether",push-by-digest=true,name-canonical=true,push=true
- name: Create multi-arch manifest and push
run: |
# Extract digests
AMD64_DIGEST="${{ steps.build-amd64.outputs.digest }}"
ARM64_DIGEST="${{ steps.build-arm64.outputs.digest }}"
echo "amd64 digest: $AMD64_DIGEST"
echo "arm64 digest: $ARM64_DIGEST"
# For each tag, create multi-arch manifest on each registry
TAGS=$(echo "${{ steps.meta.outputs.tags }}" | tr '\n' ' ')
for FULL_TAG in $TAGS; do
# Determine which registry this tag belongs to
if [[ "$FULL_TAG" == ghcr.io/* ]]; then
REPO="${{ env.REGISTRY }}/${{ env.APP_IMAGE_NAME }}"
elif [[ "$FULL_TAG" == docker.io/* ]]; then
REPO="docker.io/fawney19/aether"
else
continue
fi
echo "Creating manifest for $FULL_TAG"
docker buildx imagetools create -t "$FULL_TAG" \
"$REPO@$AMD64_DIGEST" \
"$REPO@$ARM64_DIGEST"
done
+342
View File
@@ -0,0 +1,342 @@
name: Release Aether
on:
push:
tags: ['v*']
workflow_dispatch:
permissions:
contents: write
packages: write
concurrency:
group: release-aether-${{ github.ref }}
cancel-in-progress: false
env:
REGISTRY: ghcr.io
GHCR_IMAGE: fawney19/aether
DOCKERHUB_IMAGE: fawney19/aether
jobs:
preflight:
name: Release preflight
runs-on: ubuntu-latest
outputs:
publish: ${{ steps.classify.outputs.publish }}
version_tag: ${{ steps.classify.outputs.version_tag }}
prerelease: ${{ steps.classify.outputs.prerelease }}
make_latest: ${{ steps.classify.outputs.make_latest }}
steps:
- name: Classify release tag
id: classify
shell: bash
run: |
set -euo pipefail
echo "publish=false" >> "${GITHUB_OUTPUT}"
echo "version_tag=" >> "${GITHUB_OUTPUT}"
echo "prerelease=false" >> "${GITHUB_OUTPUT}"
echo "make_latest=false" >> "${GITHUB_OUTPUT}"
if [[ "${GITHUB_REF_TYPE}" != "tag" ]]; then
echo "Manual release build; publish jobs will be skipped."
exit 0
fi
tag="${GITHUB_REF_NAME}"
if [[ ! "${tag}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-(beta|rc)\.[0-9]+)?$ ]]; then
echo "Unsupported release tag: ${tag}" >&2
echo "Expected vX.Y.Z, vX.Y.Z-beta.N, or vX.Y.Z-rc.N." >&2
exit 1
fi
echo "version_tag=${tag}" >> "${GITHUB_OUTPUT}"
if [[ "${tag}" == *-* ]]; then
echo "prerelease=true" >> "${GITHUB_OUTPUT}"
else
echo "make_latest=true" >> "${GITHUB_OUTPUT}"
fi
if [[ "${GITHUB_EVENT_NAME}" == "push" ]]; then
echo "publish=true" >> "${GITHUB_OUTPUT}"
else
echo "Manual release build for ${tag}; publish jobs will be skipped."
fi
frontend:
name: Build frontend
needs: preflight
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install & build
working-directory: frontend
run: |
npm ci
npm run build
- name: Upload frontend artifact
uses: actions/upload-artifact@v5
with:
name: frontend-dist
path: frontend/dist/
if-no-files-found: error
retention-days: 1
build:
name: Build ${{ matrix.name }}
needs: preflight
runs-on: ${{ matrix.os }}
strategy:
fail-fast: true
matrix:
include:
- name: linux-amd64
target: x86_64-unknown-linux-musl
platform: linux
arch: amd64
os: ubuntu-latest
use_cross: true
- name: linux-arm64
target: aarch64-unknown-linux-musl
platform: linux
arch: arm64
os: ubuntu-latest
use_cross: true
- name: macos-amd64
target: x86_64-apple-darwin
platform: macos
arch: amd64
os: macos-15-intel
use_cross: false
- name: macos-arm64
target: aarch64-apple-darwin
platform: macos
arch: arm64
os: macos-15
use_cross: false
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ matrix.target }}
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: release-${{ matrix.target }}
workspaces: . -> target
- name: Install cross
if: matrix.use_cross
uses: taiki-e/install-action@cross
- name: Build
env:
AETHER_VERSION: ${{ needs.preflight.outputs.version_tag }}
AETHER_BUILD_TYPE: release
CARGO_TERM_COLOR: always
shell: bash
run: |
if [[ "${{ matrix.use_cross }}" == "true" ]]; then
cross build --release --locked -p aether-gateway --target ${{ matrix.target }}
else
cargo build --release --locked -p aether-gateway --target ${{ matrix.target }}
fi
- name: Upload binary artifact
uses: actions/upload-artifact@v5
with:
name: aether-gateway-${{ matrix.platform }}-${{ matrix.arch }}
path: target/${{ matrix.target }}/release/aether-gateway
if-no-files-found: error
retention-days: 1
docker:
name: Docker multi-arch
needs: [preflight, frontend, build]
if: needs.preflight.outputs.publish == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Download all artifacts
uses: actions/download-artifact@v5
with:
path: artifacts
- name: Prepare dist layout
run: |
mkdir -p dist
cp artifacts/aether-gateway-linux-amd64/aether-gateway dist/aether-gateway-amd64
cp artifacts/aether-gateway-linux-arm64/aether-gateway dist/aether-gateway-arm64
chmod +x dist/aether-gateway-amd64 dist/aether-gateway-arm64
cp -r artifacts/frontend-dist dist/frontend
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
with:
images: |
${{ env.REGISTRY }}/${{ env.GHCR_IMAGE }}
docker.io/${{ env.DOCKERHUB_IMAGE }}
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}},enable=${{ needs.preflight.outputs.make_latest == 'true' }}
type=raw,value=latest,enable=${{ needs.preflight.outputs.make_latest == 'true' }}
type=raw,value=beta,enable=${{ contains(github.ref_name, '-beta.') }}
type=raw,value=rc,enable=${{ contains(github.ref_name, '-rc.') }}
type=sha,prefix=
flavor: |
latest=false
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile.app
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: linux/amd64,linux/arm64
package:
name: Release tarballs
needs: [preflight, frontend, build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Download all artifacts
uses: actions/download-artifact@v5
with:
path: artifacts
- name: Build release packages
run: |
set -euo pipefail
if [[ "${GITHUB_REF_TYPE}" == "tag" ]]; then
VERSION="${GITHUB_REF_NAME}"
SOURCE_REF="${GITHUB_REF_NAME}"
else
VERSION="snapshot-${GITHUB_SHA::7}"
SOURCE_REF="${GITHUB_SHA}"
fi
mkdir -p package release-assets
for platform in linux macos; do
for arch in amd64 arm64; do
bundle="aether-${VERSION}-${platform}-${arch}"
root="package/${bundle}"
mkdir -p \
"${root}/bin" \
"${root}/frontend"
install -m 0755 "artifacts/aether-gateway-${platform}-${arch}/aether-gateway" "${root}/bin/aether-gateway"
cp -R artifacts/frontend-dist/. "${root}/frontend/"
sed \
-e "s/^SOURCE_REF=\"\${AETHER_SOURCE_REF:-main}\"/SOURCE_REF=\"\${AETHER_SOURCE_REF:-${SOURCE_REF}}\"/" \
-e "s/^VERSION=\"\${AETHER_VERSION:-}\"/VERSION=\"\${AETHER_VERSION:-${VERSION}}\"/" \
install.sh > "${root}/install.sh"
chmod 0755 "${root}/install.sh"
install -m 0755 update.sh "${root}/update.sh"
install -m 0644 docker-compose.yml "${root}/docker-compose.yml"
install -m 0644 docker-compose.single-node.yml "${root}/docker-compose.single-node.yml"
install -m 0644 .env.example "${root}/.env.example"
install -m 0755 generate_keys.sh "${root}/generate_keys.sh"
install -m 0644 README.md "${root}/README.md"
install -m 0644 LICENSE "${root}/LICENSE"
tar -C package -czf "release-assets/${bundle}.tar.gz" "${bundle}"
done
done
sed \
-e "s/^SOURCE_REF=\"\${AETHER_SOURCE_REF:-main}\"/SOURCE_REF=\"\${AETHER_SOURCE_REF:-${SOURCE_REF}}\"/" \
-e "s/^VERSION=\"\${AETHER_VERSION:-}\"/VERSION=\"\${AETHER_VERSION:-${VERSION}}\"/" \
install.sh > release-assets/install.sh
chmod +x release-assets/install.sh
(cd release-assets && sha256sum *.tar.gz > SHA256SUMS)
- name: Upload release package artifact
uses: actions/upload-artifact@v5
with:
name: release-assets
path: release-assets/*
if-no-files-found: error
retention-days: 7
github-release:
name: GitHub Release assets
needs: [preflight, docker, package]
if: needs.preflight.outputs.publish == 'true'
runs-on: ubuntu-latest
steps:
- name: Download release package artifact
uses: actions/download-artifact@v5
with:
name: release-assets
path: release-assets
- name: Delete stale draft releases for tag
env:
GH_TOKEN: ${{ github.token }}
RELEASE_TAG: ${{ github.ref_name }}
REPOSITORY: ${{ github.repository }}
shell: bash
run: |
set -euo pipefail
draft_ids="$(gh api "repos/${REPOSITORY}/releases" --paginate --jq '.[] | select(.tag_name == env.RELEASE_TAG and .draft == true) | .id')"
if [[ -z "${draft_ids}" ]]; then
echo "No stale draft releases for ${RELEASE_TAG}"
exit 0
fi
while IFS= read -r release_id; do
[[ -z "${release_id}" ]] && continue
echo "Deleting stale draft release ${release_id} for ${RELEASE_TAG}"
gh api -X DELETE "repos/${REPOSITORY}/releases/${release_id}"
done <<< "${draft_ids}"
- name: Publish GitHub Release assets
uses: softprops/action-gh-release@v2
with:
generate_release_notes: true
prerelease: ${{ needs.preflight.outputs.prerelease }}
make_latest: ${{ needs.preflight.outputs.make_latest }}
files: |
release-assets/*.tar.gz
release-assets/SHA256SUMS
release-assets/install.sh
+681
View File
@@ -0,0 +1,681 @@
name: Rust CI
on:
push:
branches:
- master
- main
paths:
- "Cargo.toml"
- "Cargo.lock"
- "crates/**"
- "apps/**"
- ".github/workflows/rust-ci.yml"
pull_request:
paths:
- "Cargo.toml"
- "Cargo.lock"
- "crates/**"
- "apps/**"
- ".github/workflows/rust-ci.yml"
concurrency:
group: rust-ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
CARGO_INCREMENTAL: 0
CARGO_PROFILE_DEV_DEBUG: 0
CARGO_PROFILE_TEST_DEBUG: 0
CARGO_TERM_COLOR: always
jobs:
fmt:
name: Format
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.95.0
components: rustfmt
- name: Format
run: cargo fmt --all --check
clippy_gateway:
name: Clippy (Gateway)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.95.0
components: clippy
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Clippy
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo clippy -p aether-gateway --lib --bins --examples -- -D warnings
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
clippy_data:
name: Clippy (Data)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.95.0
components: clippy
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Clippy
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo clippy -p aether-data --all-targets -- -D warnings
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
clippy_rest:
name: Clippy (Workspace Rest)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.95.0
components: clippy
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Clippy
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo clippy --workspace --exclude aether-gateway --exclude aether-data --exclude aether-integration-tests --all-targets -- -D warnings
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
clippy:
name: Clippy
runs-on: ubuntu-latest
needs:
- clippy_gateway
- clippy_data
- clippy_rest
if: ${{ always() }}
steps:
- name: Verify clippy jobs
run: |
if [ "${{ needs.clippy_gateway.result }}" != "success" ] || \
[ "${{ needs.clippy_data.result }}" != "success" ] || \
[ "${{ needs.clippy_rest.result }}" != "success" ]; then
echo "Clippy failed"
exit 1
fi
test_gateway:
name: Test (Gateway)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Show Rust toolchain
run: rustup show active-toolchain
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Setup mold
uses: rui314/setup-mold@v1
- name: Install nextest
uses: taiki-e/install-action@nextest
- name: Test lib
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
RUST_MIN_STACK: "16777216"
RUSTFLAGS: "-C link-arg=-fuse-ld=mold"
run: cargo nextest run -p aether-gateway --lib
- name: Test bins
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
RUST_MIN_STACK: "16777216"
RUSTFLAGS: "-C link-arg=-fuse-ld=mold"
run: cargo nextest run -p aether-gateway --bins
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
test_data:
name: Test (Data)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Show Rust toolchain
run: rustup show active-toolchain
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Install nextest
uses: taiki-e/install-action@nextest
- name: Test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo nextest run -p aether-data
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
check_data_features:
name: Check (Data Feature - ${{ matrix.feature }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
feature:
- postgres
- mysql
- sqlite
- all-drivers
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Check selected data driver
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo check -p aether-data --no-default-features --features ${{ matrix.feature }}
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
test_rest:
name: Test (Workspace Rest)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Show Rust toolchain
run: rustup show active-toolchain
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Install nextest
uses: taiki-e/install-action@nextest
- name: Test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo nextest run --workspace --exclude aether-gateway --exclude aether-data --exclude aether-integration-tests
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
test_data_adapters:
name: Test (Data Adapter - ${{ matrix.package }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
package:
- aether-data-postgres
- aether-data-mysql
- aether-data-sqlite
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Install nextest
uses: taiki-e/install-action@nextest
- name: Test adapter
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo nextest run -p ${{ matrix.package }}
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
check_integration_scenarios:
name: Test (Integration Scenarios)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Test scenario binaries and end-to-end suites
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo test -p aether-integration-tests --bins --tests
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
test:
name: Test
runs-on: ubuntu-latest
needs:
- test_gateway
- test_data
- check_data_features
- test_rest
- test_data_adapters
- check_integration_scenarios
if: ${{ always() }}
steps:
- name: Verify test jobs
run: |
if [ "${{ needs.test_gateway.result }}" != "success" ] || \
[ "${{ needs.test_data.result }}" != "success" ] || \
[ "${{ needs.check_data_features.result }}" != "success" ] || \
[ "${{ needs.test_rest.result }}" != "success" ] || \
[ "${{ needs.test_data_adapters.result }}" != "success" ] || \
[ "${{ needs.check_integration_scenarios.result }}" != "success" ]; then
echo "Tests failed"
exit 1
fi
data_db_smoke_sqlite:
name: Data DB Smoke (SQLite)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Show Rust toolchain
run: rustup show active-toolchain
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Run SQLite data smoke tests
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: cargo test -p aether-data --all-features sqlite --lib
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
data_db_smoke_postgres:
name: Data DB Smoke (Postgres)
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_DB: aether_test
POSTGRES_USER: aether
POSTGRES_PASSWORD: aether
ports:
- 5432:5432
options: >-
--health-cmd="pg_isready -h 127.0.0.1 -U aether -d aether_test"
--health-interval=5s
--health-timeout=5s
--health-retries=20
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Show Rust toolchain
run: rustup show active-toolchain
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Add PostgreSQL server binaries to PATH
run: echo "$(pg_config --bindir)" >> "$GITHUB_PATH"
- name: Run Postgres migration smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_POSTGRES_URL: postgres://aether:[email protected]:5432/aether_test
run: cargo test -p aether-data --all-features postgres_migrations_create_core_config_tables_when_url_is_set --lib -- --nocapture
- name: Run Postgres provider metadata migration smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_POSTGRES_URL: postgres://aether:[email protected]:5432/aether_test
run: cargo test -p aether-data --all-features postgres_provider_upstream_metadata_migration_preserves_json_when_url_is_set --lib -- --nocapture
- name: Run Postgres API key lifecycle tests
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_REQUIRE_LOCAL_POSTGRES_TESTS: "true"
run: |
cargo test -p aether-data --all-features lifecycle::migrate::tests::postgres_request_candidates_preserve_deleted_api_key_identity --lib -- --exact --nocapture
cargo test -p aether-data --all-features lifecycle::migrate::tests::postgres_request_candidate_migration_decouples_legacy_api_key_foreign_key --lib -- --exact --nocapture
cargo test -p aether-data --all-features lifecycle::migrate::tests::postgres_stats_daily_api_key_migration_decouples_legacy_foreign_key --lib -- --exact --nocapture
cargo test -p aether-data --all-features lifecycle::migrate::tests::postgres_expired_api_key_cleanup_preserves_historical_identity --lib -- --exact --nocapture
cargo test -p aether-data --all-features lifecycle::migrate::tests::postgres_api_key_leaderboard_user_filter_preserves_aggregate_history --lib -- --exact --nocapture
- name: Run Postgres core export smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_POSTGRES_URL: postgres://aether:[email protected]:5432/aether_test
run: cargo test -p aether-data --all-features postgres_core_export_reads_migrated_database_rows_when_url_is_set --lib -- --nocapture
- name: Run SQLite-to-Postgres import smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_POSTGRES_URL: postgres://aether:[email protected]:5432/aether_test
run: cargo test -p aether-data --all-features sqlite_core_export_reads_migrated_database_rows --lib -- --nocapture
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
data_db_smoke_mysql:
name: Data DB Smoke (MySQL)
runs-on: ubuntu-latest
services:
mysql:
image: mysql:8.0
env:
MYSQL_DATABASE: aether_test
MYSQL_USER: aether
MYSQL_PASSWORD: aether
MYSQL_ROOT_PASSWORD: aether_root
ports:
- 3306:3306
options: >-
--health-cmd="mysqladmin ping -h 127.0.0.1 -uaether -paether --silent"
--health-interval=5s
--health-timeout=5s
--health-retries=20
steps:
- uses: actions/checkout@v5
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Show Rust toolchain
run: rustup show active-toolchain
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: rust-ci-${{ runner.os }}
workspaces: . -> target
- name: Setup sccache
uses: mozilla-actions/[email protected]
- name: Run MySQL migration smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data --all-features mysql_migrations_create_core_config_tables_when_url_is_set --lib -- --nocapture
- name: Run MySQL usage write smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data-mysql mysql_usage_write_repository_upserts_and_flushes_counters_when_url_is_set --lib -- --nocapture
- name: Run MySQL usage read smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data-mysql mysql_usage_read_repository_reads_usage_contract_views_when_url_is_set --lib -- --nocapture
- name: Run MySQL provider catalog smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data-mysql mysql_provider_catalog_repository_round_trips_when_url_is_set --lib -- --nocapture
- name: Run MySQL core export smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data --all-features mysql_core_export_reads_migrated_database_rows_when_url_is_set --lib -- --nocapture
- name: Run MySQL wallet read smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data-mysql mysql_wallet_read_repository_reads_wallet_contract_views --lib -- --nocapture
- name: Run MySQL wallet daily usage aggregation smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data --all-features mysql_wallet_daily_usage_aggregation_uses_settlement_wallets_when_url_is_set --lib -- --nocapture
- name: Run MySQL stats aggregation smoke test
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
AETHER_TEST_MYSQL_URL: mysql://aether:[email protected]:3306/aether_test
run: cargo test -p aether-data --all-features mysql_stats_aggregation_runs_after_mysql_migrations_when_url_is_set --lib -- --nocapture
- name: Show sccache stats
if: always()
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
run: sccache --show-stats
data_db_smoke:
name: Data DB Smoke
runs-on: ubuntu-latest
needs:
- data_db_smoke_sqlite
- data_db_smoke_postgres
- data_db_smoke_mysql
if: ${{ always() }}
steps:
- name: Verify database smoke jobs
run: |
if [ "${{ needs.data_db_smoke_sqlite.result }}" != "success" ] || \
[ "${{ needs.data_db_smoke_postgres.result }}" != "success" ] || \
[ "${{ needs.data_db_smoke_mysql.result }}" != "success" ]; then
echo "Data DB smoke failed"
exit 1
fi
check:
name: check
runs-on: ubuntu-latest
needs:
- fmt
- clippy
- test
- data_db_smoke
if: ${{ always() }}
steps:
- name: Verify required jobs
run: |
if [ "${{ needs.fmt.result }}" != "success" ] || \
[ "${{ needs.clippy.result }}" != "success" ] || \
[ "${{ needs.test.result }}" != "success" ] || \
[ "${{ needs.data_db_smoke.result }}" != "success" ]; then
echo "Rust CI failed"
exit 1
fi
+10 -1
View File
@@ -1,12 +1,17 @@
# Created by https://www.toptal.com/developers/gitignore/api/python
# Edit at https://www.toptal.com/developers/gitignore?templates=python
*.rsa
*_rsa
# AI Assistant Configuration
.codex/
.claude/
.deepseek/
.serena/
.gemini*/
.plans
.playwright-mcp/
### Python ###
*.db
@@ -213,6 +218,10 @@ backups/
# Runtime lock files
.locks/
# Local Rust/Cargo configuration
.cargo/
# Demo and test files
frontend/public/*-demo.html
frontend/public/*-measure.html
@@ -238,4 +247,4 @@ src/_version.py
# Analysis folder (third-party code for reference)
analysis/
new-api/
/aether-proxy/target/
apps/aether-tunnel/aether-tunnel.toml
+2
View File
@@ -0,0 +1,2 @@
[tools]
rust = "latest"
-1
View File
@@ -1 +0,0 @@
3.13
Generated
+6402
View File
File diff suppressed because it is too large Load Diff
+159
View File
@@ -0,0 +1,159 @@
[workspace]
members = [
"apps/aether-tunnel",
"crates/aether-ai/formats",
"crates/aether-admin",
"crates/aether-admission-core",
"crates/aether-ai/serving",
"crates/aether-pool-core",
"crates/aether-provider/core",
"crates/aether-provider/pool",
"crates/aether-routing-core",
"crates/aether-data/contracts",
"crates/aether-data/adapters/postgres",
"crates/aether-data/adapters/mysql",
"crates/aether-data/adapters/sqlite",
"crates/aether-data/query",
"crates/aether-data/schema",
"crates/aether-dispatch-core",
"crates/aether-cache",
"crates/aether-billing",
"crates/aether-wallet",
"crates/aether-crypto",
"crates/aether-contracts",
"crates/aether-data/runtime",
"crates/aether-model-fetch",
"crates/aether-oauth",
"crates/aether-provider/transport",
"crates/aether-scheduler-core",
"crates/aether-runtime/state",
"crates/aether-task/runtime",
"crates/aether-task/core",
"crates/aether-gateway/frontdoor",
"crates/aether-gateway/control",
"crates/aether-gateway/execution",
"crates/aether-gateway/workers",
"crates/aether-gateway/tunnel",
"crates/aether-testing/loadtools",
"crates/aether-testing/integration",
"crates/aether-usage/core",
"crates/aether-testing/support",
"crates/aether-usage/runtime",
"crates/aether-video-tasks-core",
"apps/aether-gateway",
"crates/aether-http",
"crates/aether-runtime/base",
"crates/aether-testing/testkit",
]
default-members = [
"apps/aether-gateway",
]
resolver = "2"
[workspace.package]
edition = "2021"
license = "LicenseRef-Aether-NonCommercial"
repository = "https://github.com/fawney19/Aether.git"
[workspace.dependencies]
aether-admin = { path = "crates/aether-admin" }
aether-admission-core = { path = "crates/aether-admission-core" }
aether-ai-formats = { path = "crates/aether-ai/formats" }
aether-ai-serving = { path = "crates/aether-ai/serving" }
aether-pool-core = { path = "crates/aether-pool-core" }
aether-provider-core = { path = "crates/aether-provider/core" }
aether-provider-pool = { path = "crates/aether-provider/pool" }
aether-routing-core = { path = "crates/aether-routing-core" }
aether-data-contracts = { path = "crates/aether-data/contracts" }
aether-data-postgres = { path = "crates/aether-data/adapters/postgres" }
aether-data-mysql = { path = "crates/aether-data/adapters/mysql" }
aether-data-sqlite = { path = "crates/aether-data/adapters/sqlite" }
aether-data-query = { path = "crates/aether-data/query" }
aether-data-schema = { path = "crates/aether-data/schema" }
aether-dispatch-core = { path = "crates/aether-dispatch-core" }
aether-cache = { path = "crates/aether-cache" }
aether-billing = { path = "crates/aether-billing" }
aether-wallet = { path = "crates/aether-wallet" }
aether-crypto = { path = "crates/aether-crypto" }
aether-contracts = { path = "crates/aether-contracts" }
aether-data = { path = "crates/aether-data/runtime" }
aether-model-fetch = { path = "crates/aether-model-fetch" }
aether-oauth = { path = "crates/aether-oauth" }
aether-provider-transport = { path = "crates/aether-provider/transport" }
aether-scheduler-core = { path = "crates/aether-scheduler-core" }
aether-runtime-state = { path = "crates/aether-runtime/state" }
aether-task-runtime = { path = "crates/aether-task/runtime" }
aether-task-core = { path = "crates/aether-task/core" }
aether-gateway-frontdoor = { path = "crates/aether-gateway/frontdoor" }
aether-gateway-control = { path = "crates/aether-gateway/control" }
aether-gateway-execution = { path = "crates/aether-gateway/execution" }
aether-gateway-workers = { path = "crates/aether-gateway/workers" }
aether-gateway-tunnel = { path = "crates/aether-gateway/tunnel" }
aether-loadtools = { path = "crates/aether-testing/loadtools" }
aether-integration-tests = { path = "crates/aether-testing/integration" }
aether-test-support = { path = "crates/aether-testing/support" }
aether-usage-core = { path = "crates/aether-usage/core" }
aether-usage-runtime = { path = "crates/aether-usage/runtime" }
aether-video-tasks-core = { path = "crates/aether-video-tasks-core" }
aether-gateway = { path = "apps/aether-gateway" }
aether-http = { path = "crates/aether-http" }
aether-runtime = { path = "crates/aether-runtime/base" }
aether-testkit = { path = "crates/aether-testing/testkit" }
aes = "0.8"
aes-gcm = "0.10"
async-stream = "0.3"
async-trait = "0.1"
axum = "0.8"
base64 = "0.22"
bcrypt = "0.16"
brotli = "8"
bytes = "1"
cbc = "0.1"
chrono = { version = "0.4", features = ["serde"] }
chrono-tz = "0.10"
crypto_box = { version = "0.9", features = ["seal"] }
ed25519-dalek = { version = "2.2", features = ["pkcs8"] }
flate2 = "1"
futures-util = "0.3"
hmac = "0.12"
http = "1"
object_store = { version = "0.12", default-features = false, features = ["aws"] }
pbkdf2 = { version = "0.12", default-features = false, features = ["hmac"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "stream", "rustls-tls", "http2", "socks"] }
redis = { version = "0.28", default-features = false, features = ["tokio-comp", "script", "streams", "connection-manager"] }
regex = "1"
rustls = { version = "0.23", features = ["ring"] }
semver = "1"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1", features = ["preserve_order"] }
serde_path_to_error = "0.1"
sha2 = "0.10"
socket2 = "0.6"
tar = "0.4"
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio-rustls", "chrono"] }
thiserror = "2"
tokio = { version = "1", features = ["macros", "net", "rt-multi-thread", "signal", "sync", "time"] }
tokio-util = { version = "0.7", features = ["codec", "io-util"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
uuid = { version = "1", features = ["serde", "v4", "v5", "v7"] }
webpki-roots = "0.26"
wreq = { version = "6.0.0-rc.28", default-features = false, features = ["json", "stream", "socks", "webpki-roots", "ws"] }
wreq-util = "3.0.0-rc.10"
url = "2"
zstd = "0.13"
[profile.dev]
# Keep file/line information for backtraces while avoiding full debug info
# generation on very large crates during local development builds.
debug = "line-tables-only"
[profile.test]
# The gateway test target pulls in a very large in-crate test tree, so use the
# lighter debug format here as well to reduce rustc peak memory.
debug = "line-tables-only"
[profile.release]
lto = "thin"
strip = true
codegen-units = 8
+37 -272
View File
@@ -1,278 +1,43 @@
# syntax=docker/dockerfile:1
# 运行镜像:从 base 提取产物到精简运行时
# 构建命令: docker build -f Dockerfile.app -t aether-app:latest .
# 用于 GitHub Actions CI(官方源)
# Aether Gateway runtime image (cross-compilation)
# Binary and frontend assets are pre-built by CI; this Dockerfile only packages them.
# Usage: docker buildx build --platform linux/amd64,linux/arm64 -f Dockerfile.app .
#
# Build context must contain:
# dist/aether-gateway-amd64 (x86_64-unknown-linux-musl cross-compiled binary)
# dist/aether-gateway-arm64 (aarch64-unknown-linux-musl cross-compiled binary)
# dist/frontend/ (npm run build output)
FROM aether-base:latest AS builder
WORKDIR /app
# 复制前端源码并构建(CI 通过 no-cache-filters=builder 确保每次重建)
COPY frontend/ ./frontend/
RUN cd frontend && npm run build
# --- layout stage: create /opt/aether directory structure with symlink ---
# distroless has no shell, so we use busybox to set up the symlink.
FROM busybox:1.37-musl AS layout
# ==================== 运行时镜像 ====================
FROM python:3.13-slim
WORKDIR /app
ARG HUB_RELEASE_REPO=fawney19/Aether
ARG HUB_TAG
ARG TARGETARCH
ARG GITHUB_TOKEN
# 运行时依赖(无 gcc/nodejs/npm,使用 BuildKit 缓存加速)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && apt-get install -y --no-install-recommends \
nginx \
supervisor \
libpq5 \
curl
# 从 base 镜像复制 Python 包
COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packages
# 只复制需要的 Python 可执行文件
COPY --from=builder /usr/local/bin/gunicorn /usr/local/bin/
COPY --from=builder /usr/local/bin/uvicorn /usr/local/bin/
COPY --from=builder /usr/local/bin/alembic /usr/local/bin/
# Hub 预编译二进制(构建时从 GitHub Release 下载)
# GITHUB_TOKEN 可选:未认证 API 限流 60 次/小时,认证后 5000 次/小时
RUN set -eux; \
auth_header=""; \
if [ -n "${GITHUB_TOKEN:-}" ]; then \
auth_header="Authorization: token ${GITHUB_TOKEN}"; \
fi; \
tag="${HUB_TAG:-}"; \
if [ -z "$tag" ]; then \
tag="$(curl -sL ${auth_header:+-H "$auth_header"} "https://api.github.com/repos/${HUB_RELEASE_REPO}/releases" | python3 -c "import json,sys;print(next((r['tag_name'] for r in json.load(sys.stdin) if r.get('tag_name','').startswith('hub-v') and not r.get('draft') and not r.get('prerelease')),''))")"; \
fi; \
if [ -z "$tag" ]; then \
echo "Failed to resolve hub release tag"; \
exit 1; \
fi; \
arch="${TARGETARCH:-}"; \
if [ -z "$arch" ]; then \
arch="$(dpkg --print-architecture)"; \
fi; \
case "$arch" in \
amd64|arm64) ;; \
x86_64) arch="amd64" ;; \
aarch64) arch="arm64" ;; \
*) echo "Unsupported architecture: $arch"; exit 1 ;; \
esac; \
echo "Using Hub release tag: $tag"; \
url="https://github.com/${HUB_RELEASE_REPO}/releases/download/${tag}/aether-hub-linux-${arch}.tar.gz"; \
curl -L --fail -o /tmp/aether-hub.tar.gz "$url"; \
tar xzf /tmp/aether-hub.tar.gz -C /usr/local/bin; \
chmod +x /usr/local/bin/aether-hub; \
rm -f /tmp/aether-hub.tar.gz
# 从 builder 阶段复制前端构建产物
COPY --from=builder /app/frontend/dist /usr/share/nginx/html
RUN chmod -R 755 /usr/share/nginx/html
# 复制后端代码
COPY src/ ./src/
COPY alembic.ini ./
COPY alembic/ ./alembic/
COPY gunicorn_conf.py ./
# Nginx 配置模板
# 策略:白名单后端路由 → 后端代理,其余全部 → 前端 SPA(index.html)
# 智能处理 IP:有外层代理头就透传,没有就用直连 IP
RUN printf '%s\n' \
'map $http_x_real_ip $real_ip {' \
' default $http_x_real_ip;' \
' "" $remote_addr;' \
'}' \
'' \
'map $http_x_forwarded_for $forwarded_for {' \
' default $http_x_forwarded_for;' \
' "" $remote_addr;' \
'}' \
'' \
'server {' \
' listen 80;' \
' server_name _;' \
' root /usr/share/nginx/html;' \
' index index.html;' \
' client_max_body_size 100M;' \
'' \
' # gzip 压缩配置(对 base64 图片等非流式响应有效)' \
' gzip on;' \
' gzip_min_length 256;' \
' gzip_comp_level 5;' \
' gzip_vary on;' \
' gzip_proxied any;' \
' gzip_types application/json text/plain text/css text/javascript application/javascript application/octet-stream;' \
' gzip_disable "msie6";' \
'' \
' # 静态资源:长期缓存' \
' location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {' \
' expires 1y;' \
' add_header Cache-Control "public, no-transform";' \
' try_files $uri =404;' \
' }' \
'' \
' # 安全:阻止访问源码目录' \
' location ~ ^/(src|node_modules)/ {' \
' deny all;' \
' return 404;' \
' }' \
'' \
' # WebSocket 隧道端点(aether-proxy tunnel 模式)' \
' location = /api/internal/proxy-tunnel {' \
' proxy_pass http://127.0.0.1:8085/proxy;' \
' proxy_http_version 1.1;' \
' proxy_set_header Host $host;' \
' proxy_set_header X-Real-IP $real_ip;' \
' proxy_set_header X-Forwarded-For $forwarded_for;' \
' proxy_set_header X-Forwarded-Proto $scheme;' \
' proxy_set_header Upgrade $http_upgrade;' \
' proxy_set_header Connection "upgrade";' \
' # 剥离 CF 头,防止泄露给上游或返回给客户端' \
' proxy_hide_header CF-Connecting-IP;' \
' proxy_hide_header CF-IPCountry;' \
' proxy_hide_header CF-Ray;' \
' proxy_hide_header CF-Visitor;' \
' proxy_hide_header CDN-Loop;' \
' proxy_hide_header True-Client-IP;' \
' proxy_hide_header CF-Worker;' \
' proxy_hide_header CF-EW-Via;' \
' proxy_hide_header CF-Warp-Tag-ID;' \
' proxy_set_header CF-Connecting-IP "";' \
' proxy_set_header CF-IPCountry "";' \
' proxy_set_header CF-Ray "";' \
' proxy_set_header CF-Visitor "";' \
' proxy_set_header CDN-Loop "";' \
' proxy_set_header True-Client-IP "";' \
' proxy_set_header CF-Worker "";' \
' proxy_set_header CF-EW-Via "";' \
' proxy_set_header CF-Warp-Tag-ID "";' \
' proxy_read_timeout 86400s;' \
' proxy_send_timeout 86400s;' \
' }' \
'' \
' # 后端 API 路由(白名单)→ 代理到后端' \
' location ~ ^/(api|v1|v1beta|upload|health)(/|$) {' \
' proxy_pass http://127.0.0.1:PORT_PLACEHOLDER;' \
' proxy_http_version 1.1;' \
' proxy_set_header Host $host;' \
' proxy_set_header X-Real-IP $real_ip;' \
' proxy_set_header X-Forwarded-For $forwarded_for;' \
' proxy_set_header X-Forwarded-Proto $scheme;' \
' proxy_set_header Connection "";' \
' proxy_set_header Accept $http_accept;' \
' proxy_set_header Content-Type $content_type;' \
' proxy_set_header Authorization $http_authorization;' \
' proxy_set_header X-Api-Key $http_x_api_key;' \
' # 剥离 CF 头,防止泄露给上游或返回给客户端' \
' proxy_hide_header CF-Connecting-IP;' \
' proxy_hide_header CF-IPCountry;' \
' proxy_hide_header CF-Ray;' \
' proxy_hide_header CF-Visitor;' \
' proxy_hide_header CDN-Loop;' \
' proxy_hide_header True-Client-IP;' \
' proxy_hide_header CF-Worker;' \
' proxy_hide_header CF-EW-Via;' \
' proxy_hide_header CF-Warp-Tag-ID;' \
' proxy_set_header CF-Connecting-IP "";' \
' proxy_set_header CF-IPCountry "";' \
' proxy_set_header CF-Ray "";' \
' proxy_set_header CF-Visitor "";' \
' proxy_set_header CDN-Loop "";' \
' proxy_set_header True-Client-IP "";' \
' proxy_set_header CF-Worker "";' \
' proxy_set_header CF-EW-Via "";' \
' proxy_set_header CF-Warp-Tag-ID "";' \
' proxy_buffering off;' \
' proxy_cache off;' \
' proxy_request_buffering off;' \
' chunked_transfer_encoding on;' \
' gzip off;' \
' add_header X-Accel-Buffering no;' \
' proxy_connect_timeout 60s;' \
' proxy_send_timeout 3600s;' \
' proxy_read_timeout 3600s;' \
' }' \
'' \
' # API 文档路由 → 代理到后端' \
' location ~ ^/(docs|redoc|openapi\\.json)$ {' \
' proxy_pass http://127.0.0.1:PORT_PLACEHOLDER;' \
' proxy_http_version 1.1;' \
' proxy_set_header Host $host;' \
' proxy_set_header X-Real-IP $real_ip;' \
' proxy_set_header X-Forwarded-For $forwarded_for;' \
' proxy_set_header X-Forwarded-Proto $scheme;' \
' # 剥离 CF 头,防止泄露给上游或返回给客户端' \
' proxy_hide_header CF-Connecting-IP;' \
' proxy_hide_header CF-IPCountry;' \
' proxy_hide_header CF-Ray;' \
' proxy_hide_header CF-Visitor;' \
' proxy_hide_header CDN-Loop;' \
' proxy_hide_header True-Client-IP;' \
' proxy_hide_header CF-Worker;' \
' proxy_hide_header CF-EW-Via;' \
' proxy_hide_header CF-Warp-Tag-ID;' \
' proxy_set_header CF-Connecting-IP "";' \
' proxy_set_header CF-IPCountry "";' \
' proxy_set_header CF-Ray "";' \
' proxy_set_header CF-Visitor "";' \
' proxy_set_header CDN-Loop "";' \
' proxy_set_header True-Client-IP "";' \
' proxy_set_header CF-Worker "";' \
' proxy_set_header CF-EW-Via "";' \
' proxy_set_header CF-Warp-Tag-ID "";' \
' }' \
'' \
' # 所有其他路由 → 前端 SPA(先尝试静态文件,再回退到 index.html)' \
' location / {' \
' try_files $uri $uri/ /index.html;' \
' }' \
'}' > /etc/nginx/sites-available/default.template
# Supervisor 配置
RUN printf '%s\n' \
'[supervisord]' \
'nodaemon=true' \
'logfile=/var/log/supervisor/supervisord.log' \
'pidfile=/var/run/supervisord.pid' \
'' \
'[program:nginx]' \
'command=/bin/bash -c "sed \"s/PORT_PLACEHOLDER/8084/g\" /etc/nginx/sites-available/default.template > /etc/nginx/sites-available/default && /usr/sbin/nginx -g \"daemon off;\""' \
'autostart=true' \
'autorestart=true' \
'stdout_logfile=/var/log/nginx/access.log' \
'stderr_logfile=/var/log/nginx/error.log' \
'' \
'[program:app]' \
'command=/bin/bash -c "MAX_REQUESTS_JITTER=$((${MAX_REQUESTS:-50000}/20)); exec gunicorn src.main:app -c gunicorn_conf.py --preload -w %(ENV_GUNICORN_WORKERS)s -k uvicorn.workers.UvicornWorker --bind 127.0.0.1:8084 --max-requests ${MAX_REQUESTS:-50000} --max-requests-jitter $MAX_REQUESTS_JITTER --access-logfile - --error-logfile - --log-level info"' \
'directory=/app' \
'autostart=true' \
'autorestart=true' \
'stdout_logfile=/dev/stdout' \
'stdout_logfile_maxbytes=0' \
'stderr_logfile=/dev/stderr' \
'stderr_logfile_maxbytes=0' \
'environment=PYTHONUNBUFFERED=1,PYTHONIOENCODING=utf-8,LANG=C.UTF-8,LC_ALL=C.UTF-8,DOCKER_CONTAINER=true' \
'' \
'[program:tunnel-hub]' \
'command=/usr/local/bin/aether-hub --bind 0.0.0.0:8085' \
'autostart=true' \
'autorestart=true' \
'stdout_logfile=/dev/stdout' \
'stdout_logfile_maxbytes=0' \
'stderr_logfile=/dev/stderr' \
'stderr_logfile_maxbytes=0' > /etc/supervisor/conf.d/supervisord.conf
# 创建目录
RUN mkdir -p /var/log/supervisor /app/logs /app/data
# 入口脚本(启动前执行迁移)
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
# 环境变量
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONIOENCODING=utf-8 \
LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
PORT=8084 \
GUNICORN_WORKERS=2 \
MAX_REQUESTS=4000
EXPOSE 80
RUN mkdir -p /opt/aether/releases/image/bin /opt/aether/releases/image/frontend /opt/aether/logs
COPY dist/aether-gateway-${TARGETARCH} /opt/aether/releases/image/bin/aether-gateway
RUN chmod 0755 /opt/aether/releases/image/bin/aether-gateway
COPY dist/frontend/ /opt/aether/releases/image/frontend/
RUN ln -s /opt/aether/releases/image /opt/aether/current
# --- final stage: distroless runtime ---
FROM gcr.io/distroless/static-debian12
COPY --from=layout /opt/aether /opt/aether
WORKDIR /opt/aether
ENV RUST_LOG=aether_gateway=info \
APP_PORT=8084 \
AETHER_UPDATE_STRATEGY=docker \
AETHER_GATEWAY_STATIC_DIR=/opt/aether/current/frontend
EXPOSE 8084
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost/health || exit 1
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
CMD ["/opt/aether/current/bin/aether-gateway", "--healthcheck"]
USER root
ENTRYPOINT ["/opt/aether/current/bin/aether-gateway"]
+128 -284
View File
@@ -1,307 +1,151 @@
# syntax=docker/dockerfile:1
# 运行镜像:从 base 提取产物到精简运行时(国内镜像源版本)
# 构建命令: docker build -f Dockerfile.app.local -t aether-app:latest .
# 用于本地/国内服务器部署
# syntax=docker.m.daocloud.io/docker/dockerfile:1
# Aether 运行镜像:Rust gateway 直接服务 API + 前端静态文件(国内镜像源版本)
# 构建命令: docker build --build-arg AETHER_BUILD_VERSION=v0.7.2 -f Dockerfile.app.local -t aether-app:latest .
FROM aether-base:latest AS builder
ARG RUST_VERSION=1.95.0
ARG NODE_BASE_IMAGE=docker.m.daocloud.io/library/node:22-slim
ARG RUST_BASE_IMAGE=docker.m.daocloud.io/library/rust:${RUST_VERSION}-slim
WORKDIR /app
# ==================== 前端构建 ====================
FROM ${NODE_BASE_IMAGE} AS frontend-builder
ARG AETHER_BUILD_VERSION
ENV AETHER_BUILD_VERSION=${AETHER_BUILD_VERSION} \
AETHER_VERSION=${AETHER_BUILD_VERSION}
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN --mount=type=cache,id=aether-npm-cache,target=/root/.npm,sharing=locked \
npm config set registry https://registry.npmmirror.com && \
npm ci --no-audit --no-fund
COPY frontend/ ./
RUN npm run build
# 复制前端源码并构建
COPY frontend/ ./frontend/
RUN cd frontend && npm run build
# ==================== Rust gateway 构建 ====================
FROM ${RUST_BASE_IMAGE} AS gateway-base
WORKDIR /build
# ==================== 运行时镜像 ====================
FROM python:3.13-slim
# 生产级 release 构建:保留 thin LTO,同时用 lld 缩短最终链接阶段。
ENV CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse \
CARGO_PROFILE_RELEASE_LTO=thin \
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16 \
RUSTFLAGS="-C linker=clang -C link-arg=-fuse-ld=lld"
WORKDIR /app
ARG HUB_RELEASE_REPO=fawney19/Aether
ARG HUB_TAG
ARG TARGETARCH
ARG GITHUB_TOKEN
# GitHub 下载镜像前缀,国内构建时传入可用的镜像加速地址
# 用法: --build-arg GITHUB_MIRROR=https://ghfast.top
# 或: --build-arg GITHUB_MIRROR=https://gh-proxy.com
# 或: --build-arg GITHUB_MIRROR=https://mirror.ghproxy.com
ARG GITHUB_MIRROR
# 运行时依赖(使用清华镜像源 + BuildKit 缓存加速)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list.d/debian.sources && \
apt-get update && apt-get install -y --no-install-recommends \
nginx \
supervisor \
libpq5 \
curl
build-essential \
ca-certificates \
clang \
cmake \
git \
libclang-dev \
libssl-dev \
lld \
pkg-config \
perl
# 从 base 镜像复制 Python 包
COPY --from=builder /usr/local/lib/python3.13/site-packages /usr/local/lib/python3.13/site-packages
RUN --mount=type=cache,id=aether-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,id=aether-cargo-git,target=/usr/local/cargo/git,sharing=locked \
cargo install cargo-chef --locked
# 只复制需要的 Python 可执行文件
COPY --from=builder /usr/local/bin/gunicorn /usr/local/bin/
COPY --from=builder /usr/local/bin/uvicorn /usr/local/bin/
COPY --from=builder /usr/local/bin/alembic /usr/local/bin/
FROM gateway-base AS gateway-planner
COPY Cargo.toml Cargo.lock ./
COPY apps/ ./apps/
COPY crates/ ./crates/
RUN cargo chef prepare --recipe-path recipe.json
# Hub 预编译二进制
# 国内构建: --build-arg GITHUB_MIRROR=https://ghfast.top 即可走镜像下载
# GITHUB_TOKEN 可选:未认证 API 限流 60 次/小时,认证后 5000 次/小时
FROM gateway-base AS gateway-builder
ARG AETHER_BUILD_VERSION
ENV AETHER_BUILD_VERSION=${AETHER_BUILD_VERSION} \
AETHER_VERSION=${AETHER_BUILD_VERSION}
COPY --from=gateway-planner /build/recipe.json ./recipe.json
RUN --mount=type=cache,id=aether-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,id=aether-cargo-git,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,id=aether-cargo-target-local,target=/build/target,sharing=locked \
cargo chef cook --release --locked --package aether-gateway --bin aether-gateway --features jemalloc --recipe-path recipe.json
COPY Cargo.toml Cargo.lock ./
COPY apps/ ./apps/
COPY crates/ ./crates/
RUN --mount=type=cache,id=aether-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,id=aether-cargo-git,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,id=aether-cargo-target-local,target=/build/target,sharing=locked \
set -eux; \
cargo build --release --locked -p aether-gateway --bin aether-gateway --features jemalloc; \
cp target/release/aether-gateway /tmp/aether-gateway
# ==================== 最小运行时打包 ====================
FROM gateway-builder AS runtime-prep
RUN set -eux; \
arch="${TARGETARCH:-}"; \
if [ -z "$arch" ]; then \
arch="$(dpkg --print-architecture)"; \
mkdir -p \
/runtime-root/app/data \
/runtime-root/app/logs \
/runtime-root/etc \
/runtime-root/etc/ssl \
/runtime-root/lib \
/runtime-root/lib64 \
/runtime-root/usr/local/bin; \
cp /tmp/aether-gateway /runtime-root/usr/local/bin/aether-gateway; \
: > /tmp/runtime-libs.txt; \
: > /tmp/runtime-scan-queue.txt; \
printf '%s\n' /tmp/aether-gateway >> /tmp/runtime-scan-queue.txt; \
while [ -s /tmp/runtime-scan-queue.txt ]; do \
current="$(head -n1 /tmp/runtime-scan-queue.txt)"; \
sed -i '1d' /tmp/runtime-scan-queue.txt; \
ldd "$current" | awk '/=>/ { print $3 } $1 ~ /^\// { print $1 }' | while read -r lib; do \
[ -n "$lib" ]; \
if ! grep -Fxq "$lib" /tmp/runtime-libs.txt; then \
printf '%s\n' "$lib" >> /tmp/runtime-libs.txt; \
printf '%s\n' "$lib" >> /tmp/runtime-scan-queue.txt; \
fi; \
done; \
done; \
sort -u /tmp/runtime-libs.txt -o /tmp/runtime-libs.txt; \
while read -r lib; do \
[ -n "$lib" ]; \
dest="/runtime-root$(dirname "$lib")"; \
mkdir -p "$dest"; \
cp -L "$lib" "$dest/"; \
done < /tmp/runtime-libs.txt; \
for lib in \
/lib/x86_64-linux-gnu/libnss_dns.so.2 \
/lib/x86_64-linux-gnu/libnss_files.so.2 \
/lib/x86_64-linux-gnu/libresolv.so.2; do \
if [ -f "$lib" ]; then \
dest="/runtime-root$(dirname "$lib")"; \
mkdir -p "$dest"; \
cp -L "$lib" "$dest/"; \
fi; \
done; \
cp -a /usr/lib/ssl /runtime-root/usr/lib/; \
cp -a /etc/ssl/certs /runtime-root/etc/ssl/; \
if [ -f /etc/ssl/openssl.cnf ]; then \
cp /etc/ssl/openssl.cnf /runtime-root/etc/ssl/openssl.cnf; \
fi; \
case "$arch" in \
amd64|arm64) ;; \
x86_64) arch="amd64" ;; \
aarch64) arch="arm64" ;; \
*) echo "Unsupported architecture: $arch"; exit 1 ;; \
esac; \
auth_header=""; \
if [ -n "${GITHUB_TOKEN:-}" ]; then \
auth_header="Authorization: token ${GITHUB_TOKEN}"; \
fi; \
tag="${HUB_TAG:-}"; \
if [ -z "$tag" ]; then \
tag="$(curl -sL ${auth_header:+-H "$auth_header"} "https://api.github.com/repos/${HUB_RELEASE_REPO}/releases" | python3 -c "import json,sys;print(next((r['tag_name'] for r in json.load(sys.stdin) if r.get('tag_name','').startswith('hub-v') and not r.get('draft') and not r.get('prerelease')),''))")"; \
fi; \
if [ -z "$tag" ]; then \
echo "Failed to resolve hub release tag"; \
exit 1; \
fi; \
echo "Using Hub release tag: $tag"; \
origin_url="https://github.com/${HUB_RELEASE_REPO}/releases/download/${tag}/aether-hub-linux-${arch}.tar.gz"; \
if [ -n "${GITHUB_MIRROR:-}" ]; then \
url="${GITHUB_MIRROR}/https://github.com/${HUB_RELEASE_REPO}/releases/download/${tag}/aether-hub-linux-${arch}.tar.gz"; \
echo "Using mirror: ${GITHUB_MIRROR}"; \
else \
url="$origin_url"; \
fi; \
curl -L --fail -o /tmp/aether-hub.tar.gz "$url"; \
tar xzf /tmp/aether-hub.tar.gz -C /usr/local/bin; \
chmod +x /usr/local/bin/aether-hub; \
rm -f /tmp/aether-hub.tar.gz
if [ -f /etc/nsswitch.conf ]; then \
cp /etc/nsswitch.conf /runtime-root/etc/nsswitch.conf; \
fi
# 从 builder 阶段复制前端构建产物
COPY --from=builder /app/frontend/dist /usr/share/nginx/html
RUN chmod -R 755 /usr/share/nginx/html
# ==================== 运行时镜像 ====================
FROM scratch
# 复制后端代码
COPY src/ ./src/
COPY alembic.ini ./
COPY alembic/ ./alembic/
COPY gunicorn_conf.py ./
# 复制 gateway 二进制
COPY --from=runtime-prep /runtime-root/ /
# Nginx 配置模板
# 策略:白名单后端路由 → 后端代理,其余全部 → 前端 SPA(index.html)
# 智能处理 IP:有外层代理头就透传,没有就用直连 IP
RUN printf '%s\n' \
'map $http_x_real_ip $real_ip {' \
' default $http_x_real_ip;' \
' "" $remote_addr;' \
'}' \
'' \
'map $http_x_forwarded_for $forwarded_for {' \
' default $http_x_forwarded_for;' \
' "" $remote_addr;' \
'}' \
'' \
'server {' \
' listen 80;' \
' server_name _;' \
' root /usr/share/nginx/html;' \
' index index.html;' \
' client_max_body_size 100M;' \
'' \
' # gzip 压缩配置(对 base64 图片等非流式响应有效)' \
' gzip on;' \
' gzip_min_length 256;' \
' gzip_comp_level 5;' \
' gzip_vary on;' \
' gzip_proxied any;' \
' gzip_types application/json text/plain text/css text/javascript application/javascript application/octet-stream;' \
' gzip_disable "msie6";' \
'' \
' # 静态资源:长期缓存' \
' location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {' \
' expires 1y;' \
' add_header Cache-Control "public, no-transform";' \
' try_files $uri =404;' \
' }' \
'' \
' # 安全:阻止访问源码目录' \
' location ~ ^/(src|node_modules)/ {' \
' deny all;' \
' return 404;' \
' }' \
'' \
' # WebSocket 隧道端点(aether-proxy tunnel 模式)' \
' location = /api/internal/proxy-tunnel {' \
' proxy_pass http://127.0.0.1:8085/proxy;' \
' proxy_http_version 1.1;' \
' proxy_set_header Host $host;' \
' proxy_set_header X-Real-IP $real_ip;' \
' proxy_set_header X-Forwarded-For $forwarded_for;' \
' proxy_set_header X-Forwarded-Proto $scheme;' \
' proxy_set_header Upgrade $http_upgrade;' \
' proxy_set_header Connection "upgrade";' \
' # 剥离 CF 头,防止泄露给上游或返回给客户端' \
' proxy_hide_header CF-Connecting-IP;' \
' proxy_hide_header CF-IPCountry;' \
' proxy_hide_header CF-Ray;' \
' proxy_hide_header CF-Visitor;' \
' proxy_hide_header CDN-Loop;' \
' proxy_hide_header True-Client-IP;' \
' proxy_hide_header CF-Worker;' \
' proxy_hide_header CF-EW-Via;' \
' proxy_hide_header CF-Warp-Tag-ID;' \
' proxy_set_header CF-Connecting-IP "";' \
' proxy_set_header CF-IPCountry "";' \
' proxy_set_header CF-Ray "";' \
' proxy_set_header CF-Visitor "";' \
' proxy_set_header CDN-Loop "";' \
' proxy_set_header True-Client-IP "";' \
' proxy_set_header CF-Worker "";' \
' proxy_set_header CF-EW-Via "";' \
' proxy_set_header CF-Warp-Tag-ID "";' \
' proxy_read_timeout 86400s;' \
' proxy_send_timeout 86400s;' \
' }' \
'' \
' # 后端 API 路由(白名单)→ 代理到后端' \
' location ~ ^/(api|v1|v1beta|upload|health)(/|$) {' \
' proxy_pass http://127.0.0.1:PORT_PLACEHOLDER;' \
' proxy_http_version 1.1;' \
' proxy_set_header Host $host;' \
' proxy_set_header X-Real-IP $real_ip;' \
' proxy_set_header X-Forwarded-For $forwarded_for;' \
' proxy_set_header X-Forwarded-Proto $scheme;' \
' proxy_set_header Connection "";' \
' proxy_set_header Accept $http_accept;' \
' proxy_set_header Content-Type $content_type;' \
' proxy_set_header Authorization $http_authorization;' \
' proxy_set_header X-Api-Key $http_x_api_key;' \
' # 剥离 CF 头,防止泄露给上游或返回给客户端' \
' proxy_hide_header CF-Connecting-IP;' \
' proxy_hide_header CF-IPCountry;' \
' proxy_hide_header CF-Ray;' \
' proxy_hide_header CF-Visitor;' \
' proxy_hide_header CDN-Loop;' \
' proxy_hide_header True-Client-IP;' \
' proxy_hide_header CF-Worker;' \
' proxy_hide_header CF-EW-Via;' \
' proxy_hide_header CF-Warp-Tag-ID;' \
' proxy_set_header CF-Connecting-IP "";' \
' proxy_set_header CF-IPCountry "";' \
' proxy_set_header CF-Ray "";' \
' proxy_set_header CF-Visitor "";' \
' proxy_set_header CDN-Loop "";' \
' proxy_set_header True-Client-IP "";' \
' proxy_set_header CF-Worker "";' \
' proxy_set_header CF-EW-Via "";' \
' proxy_set_header CF-Warp-Tag-ID "";' \
' proxy_buffering off;' \
' proxy_cache off;' \
' proxy_request_buffering off;' \
' chunked_transfer_encoding on;' \
' gzip off;' \
' add_header X-Accel-Buffering no;' \
' proxy_connect_timeout 60s;' \
' proxy_send_timeout 3600s;' \
' proxy_read_timeout 3600s;' \
' }' \
'' \
' # API 文档路由 → 代理到后端' \
' location ~ ^/(docs|redoc|openapi\\.json)$ {' \
' proxy_pass http://127.0.0.1:PORT_PLACEHOLDER;' \
' proxy_http_version 1.1;' \
' proxy_set_header Host $host;' \
' proxy_set_header X-Real-IP $real_ip;' \
' proxy_set_header X-Forwarded-For $forwarded_for;' \
' proxy_set_header X-Forwarded-Proto $scheme;' \
' # 剥离 CF 头,防止泄露给上游或返回给客户端' \
' proxy_hide_header CF-Connecting-IP;' \
' proxy_hide_header CF-IPCountry;' \
' proxy_hide_header CF-Ray;' \
' proxy_hide_header CF-Visitor;' \
' proxy_hide_header CDN-Loop;' \
' proxy_hide_header True-Client-IP;' \
' proxy_hide_header CF-Worker;' \
' proxy_hide_header CF-EW-Via;' \
' proxy_hide_header CF-Warp-Tag-ID;' \
' proxy_set_header CF-Connecting-IP "";' \
' proxy_set_header CF-IPCountry "";' \
' proxy_set_header CF-Ray "";' \
' proxy_set_header CF-Visitor "";' \
' proxy_set_header CDN-Loop "";' \
' proxy_set_header True-Client-IP "";' \
' proxy_set_header CF-Worker "";' \
' proxy_set_header CF-EW-Via "";' \
' proxy_set_header CF-Warp-Tag-ID "";' \
' }' \
'' \
' # 所有其他路由 → 前端 SPA(先尝试静态文件,再回退到 index.html)' \
' location / {' \
' try_files $uri $uri/ /index.html;' \
' }' \
'}' > /etc/nginx/sites-available/default.template
# 复制前端构建产物
COPY --from=frontend-builder /app/frontend/dist /srv/frontend
WORKDIR /app
# Supervisor 配置
RUN printf '%s\n' \
'[supervisord]' \
'nodaemon=true' \
'logfile=/var/log/supervisor/supervisord.log' \
'pidfile=/var/run/supervisord.pid' \
'' \
'[program:nginx]' \
'command=/bin/bash -c "sed \"s/PORT_PLACEHOLDER/${PORT:-8084}/g\" /etc/nginx/sites-available/default.template > /etc/nginx/sites-available/default && /usr/sbin/nginx -g \"daemon off;\""' \
'autostart=true' \
'autorestart=true' \
'stdout_logfile=/var/log/nginx/access.log' \
'stderr_logfile=/var/log/nginx/error.log' \
'' \
'[program:app]' \
'command=/bin/bash -c "MAX_REQUESTS_JITTER=$((${MAX_REQUESTS:-50000}/20)); exec gunicorn src.main:app -c gunicorn_conf.py --preload -w %(ENV_GUNICORN_WORKERS)s -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:%(ENV_PORT)s --max-requests ${MAX_REQUESTS:-50000} --max-requests-jitter $MAX_REQUESTS_JITTER --access-logfile - --error-logfile - --log-level info"' \
'directory=/app' \
'autostart=true' \
'autorestart=true' \
'stdout_logfile=/dev/stdout' \
'stdout_logfile_maxbytes=0' \
'stderr_logfile=/dev/stderr' \
'stderr_logfile_maxbytes=0' \
'environment=PYTHONUNBUFFERED=1,PYTHONIOENCODING=utf-8,LANG=C.UTF-8,LC_ALL=C.UTF-8,DOCKER_CONTAINER=true' \
'' \
'[program:tunnel-hub]' \
'command=/usr/local/bin/aether-hub --bind 0.0.0.0:8085' \
'autostart=true' \
'autorestart=true' \
'stdout_logfile=/dev/stdout' \
'stdout_logfile_maxbytes=0' \
'stderr_logfile=/dev/stderr' \
'stderr_logfile_maxbytes=0' > /etc/supervisor/conf.d/supervisord.conf
# 创建目录
RUN mkdir -p /var/log/supervisor /app/logs /app/data
# 入口脚本(启动前执行迁移)
COPY entrypoint.sh /entrypoint.sh
RUN sed -i 's/\r$//' /entrypoint.sh && chmod +x /entrypoint.sh
# 环境变量
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONIOENCODING=utf-8 \
LANG=C.UTF-8 \
ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
PORT=8084 \
GUNICORN_WORKERS=2 \
MAX_REQUESTS=4000
RUST_LOG=aether_gateway=info \
APP_PORT=8084 \
AETHER_UPDATE_STRATEGY=manual \
AETHER_GATEWAY_STATIC_DIR=/srv/frontend
EXPOSE 80
EXPOSE 8084
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost/health || exit 1
CMD ["/usr/local/bin/aether-gateway", "--healthcheck"]
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
ENTRYPOINT ["/usr/local/bin/aether-gateway"]
+150
View File
@@ -0,0 +1,150 @@
# syntax=docker.m.daocloud.io/docker/dockerfile:1
# Aether 本地发布版联调镜像
# 作用:用当前源码构建一个 release-layout 容器,专门测试管理后台在线更新流程。
ARG RUST_VERSION=1.95.0
ARG NODE_BASE_IMAGE=docker.m.daocloud.io/library/node:22-slim
ARG RUST_BASE_IMAGE=docker.m.daocloud.io/library/rust:${RUST_VERSION}-slim
# ==================== 前端构建 ====================
FROM ${NODE_BASE_IMAGE} AS frontend-builder
ARG AETHER_BUILD_VERSION
ENV AETHER_BUILD_VERSION=${AETHER_BUILD_VERSION} \
AETHER_VERSION=${AETHER_BUILD_VERSION}
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN --mount=type=cache,id=aether-npm-cache,target=/root/.npm,sharing=locked \
npm config set registry https://registry.npmmirror.com && \
npm ci --no-audit --no-fund
COPY frontend/ ./
RUN npm run build
# ==================== Rust gateway 构建 ====================
FROM ${RUST_BASE_IMAGE} AS gateway-base
WORKDIR /build
ENV CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse \
CARGO_PROFILE_RELEASE_LTO=thin \
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list.d/debian.sources && \
apt-get update && apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
cmake \
git \
libclang-dev \
libssl-dev \
pkg-config \
perl
RUN --mount=type=cache,id=aether-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,id=aether-cargo-git,target=/usr/local/cargo/git,sharing=locked \
cargo install cargo-chef --locked
FROM gateway-base AS gateway-planner
COPY Cargo.toml Cargo.lock ./
COPY apps/ ./apps/
COPY crates/ ./crates/
RUN cargo chef prepare --recipe-path recipe.json
FROM gateway-base AS gateway-builder
ARG AETHER_BUILD_VERSION
ARG AETHER_BUILD_TYPE=release
ENV AETHER_BUILD_VERSION=${AETHER_BUILD_VERSION} \
AETHER_VERSION=${AETHER_BUILD_VERSION} \
AETHER_BUILD_TYPE=${AETHER_BUILD_TYPE}
COPY --from=gateway-planner /build/recipe.json ./recipe.json
RUN --mount=type=cache,id=aether-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,id=aether-cargo-git,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,id=aether-cargo-target-release-local,target=/build/target,sharing=locked \
cargo chef cook --release --locked --package aether-gateway --bin aether-gateway --features jemalloc --recipe-path recipe.json
COPY Cargo.toml Cargo.lock ./
COPY apps/ ./apps/
COPY crates/ ./crates/
RUN --mount=type=cache,id=aether-cargo-registry,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,id=aether-cargo-git,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,id=aether-cargo-target-release-local,target=/build/target,sharing=locked \
cargo build --release --locked -p aether-gateway --features jemalloc && \
cp target/release/aether-gateway /tmp/aether-gateway
# ==================== 最小运行时打包 ====================
FROM gateway-builder AS runtime-prep
RUN set -eux; \
mkdir -p \
/runtime-root/app/data \
/runtime-root/etc \
/runtime-root/etc/ssl \
/runtime-root/lib \
/runtime-root/lib64 \
/runtime-root/usr/lib \
/runtime-root/opt/aether/logs \
/runtime-root/opt/aether/releases/image/bin \
/runtime-root/opt/aether/releases/image/frontend; \
cp /tmp/aether-gateway /runtime-root/opt/aether/releases/image/bin/aether-gateway; \
ln -s /opt/aether/releases/image /runtime-root/opt/aether/current; \
: > /tmp/runtime-libs.txt; \
: > /tmp/runtime-scan-queue.txt; \
printf '%s\n' /tmp/aether-gateway >> /tmp/runtime-scan-queue.txt; \
while [ -s /tmp/runtime-scan-queue.txt ]; do \
current="$(head -n1 /tmp/runtime-scan-queue.txt)"; \
sed -i '1d' /tmp/runtime-scan-queue.txt; \
ldd "$current" | awk '/=>/ { print $3 } $1 ~ /^\// { print $1 }' | while read -r lib; do \
[ -n "$lib" ]; \
if ! grep -Fxq "$lib" /tmp/runtime-libs.txt; then \
printf '%s\n' "$lib" >> /tmp/runtime-libs.txt; \
printf '%s\n' "$lib" >> /tmp/runtime-scan-queue.txt; \
fi; \
done; \
done; \
sort -u /tmp/runtime-libs.txt -o /tmp/runtime-libs.txt; \
while read -r lib; do \
[ -n "$lib" ]; \
dest="/runtime-root$(dirname "$lib")"; \
mkdir -p "$dest"; \
cp -L "$lib" "$dest/"; \
done < /tmp/runtime-libs.txt; \
for lib in \
/lib/x86_64-linux-gnu/libnss_dns.so.2 \
/lib/x86_64-linux-gnu/libnss_files.so.2 \
/lib/x86_64-linux-gnu/libresolv.so.2; do \
if [ -f "$lib" ]; then \
dest="/runtime-root$(dirname "$lib")"; \
mkdir -p "$dest"; \
cp -L "$lib" "$dest/"; \
fi; \
done; \
cp -a /usr/lib/ssl /runtime-root/usr/lib/; \
cp -a /etc/ssl/certs /runtime-root/etc/ssl/; \
if [ -f /etc/ssl/openssl.cnf ]; then \
cp /etc/ssl/openssl.cnf /runtime-root/etc/ssl/openssl.cnf; \
fi; \
if [ -f /etc/nsswitch.conf ]; then \
cp /etc/nsswitch.conf /runtime-root/etc/nsswitch.conf; \
fi
COPY --from=frontend-builder /app/frontend/dist /runtime-root/opt/aether/releases/image/frontend
# ==================== 运行时镜像 ====================
FROM scratch
COPY --from=runtime-prep /runtime-root/ /
WORKDIR /app
ENV LANG=C.UTF-8 \
LC_ALL=C.UTF-8 \
RUST_LOG=aether_gateway=info \
APP_PORT=8084 \
AETHER_BASE_DIR=/opt/aether \
AETHER_UPDATE_STRATEGY=self \
AETHER_GATEWAY_STATIC_DIR=/opt/aether/current/frontend
EXPOSE 8084
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD ["/opt/aether/current/bin/aether-gateway", "--healthcheck"]
ENTRYPOINT ["/opt/aether/current/bin/aether-gateway"]
-28
View File
@@ -1,28 +0,0 @@
# syntax=docker/dockerfile:1
# 构建镜像:编译环境 + 预编译的依赖
# 用于 GitHub Actions CI 构建(不使用国内镜像源)
# 构建命令: docker build -f Dockerfile.base -t aether-base:latest .
# 只在 pyproject.toml 或 frontend/package*.json 变化时需要重建
FROM python:3.13-slim
WORKDIR /app
# 构建工具(使用 BuildKit 缓存加速)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && apt-get install -y --no-install-recommends \
libpq-dev \
gcc \
nodejs \
npm
# Python 依赖(使用 BuildKit 缓存加速)
COPY pyproject.toml README.md ./
RUN --mount=type=cache,target=/root/.cache/pip \
mkdir -p src && touch src/__init__.py && \
SETUPTOOLS_SCM_PRETEND_VERSION=0.1.0 pip install .
# 前端依赖(只安装,不构建,使用 BuildKit 缓存加速)
COPY frontend/package*.json ./frontend/
RUN --mount=type=cache,target=/root/.npm \
cd frontend && npm ci
-31
View File
@@ -1,31 +0,0 @@
# syntax=docker/dockerfile:1
# 构建镜像:编译环境 + 预编译的依赖(国内镜像源版本)
# 构建命令: docker build -f Dockerfile.base.local -t aether-base:latest .
# 只在 pyproject.toml 或 frontend/package*.json 变化时需要重建
FROM python:3.13-slim
WORKDIR /app
# 构建工具(使用清华镜像源 + BuildKit 缓存加速)
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
sed -i 's/deb.debian.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list.d/debian.sources && \
apt-get update && apt-get install -y --no-install-recommends \
libpq-dev \
gcc \
nodejs \
npm
# pip 镜像源
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
# Python 依赖(使用 BuildKit 缓存加速)
COPY pyproject.toml README.md ./
RUN --mount=type=cache,target=/root/.cache/pip \
mkdir -p src && touch src/__init__.py && \
SETUPTOOLS_SCM_PRETEND_VERSION=0.1.0 pip install .
# 前端依赖(只安装,不构建,使用淘宝镜像源 + BuildKit 缓存加速)
COPY frontend/package*.json ./frontend/
RUN --mount=type=cache,target=/root/.npm \
cd frontend && npm config set registry https://registry.npmmirror.com && npm ci
+567
View File
@@ -0,0 +1,567 @@
SHELL := /bin/bash
DEV_RUST_LOG := info,executor::candidate_loop=debug,stream::execution=debug
ifeq ($(origin RUST_LOG), command line)
DEV_RUST_LOG := $(RUST_LOG)
endif
export DEV_RUST_LOG
.PHONY: dev dev-backend dev-frontend migration backfill
define DEV_BACKEND_SCRIPT
set -euo pipefail
if [ ! -f .env ]; then
echo "=> 未找到 .env,请先执行: cp .env.example .env"
exit 1
fi
set -a
source .env
set +a
dotenv_has_key() {
local key="$$1"
grep -Eq "^[[:space:]]*$${key}=" .env
}
lowercase() {
printf '%s' "$$1" | tr '[:upper:]' '[:lower:]'
}
dev_uses_sqlite_database() {
local driver
local url
driver="$$(lowercase "$${AETHER_DATABASE_DRIVER:-}")"
url="$${AETHER_DATABASE_URL:-$${DATABASE_URL:-}}"
[[ "$${driver}" == "sqlite" || "$${url}" == sqlite:* ]]
}
dev_uses_postgres_database() {
local driver
local url
driver="$$(lowercase "$${AETHER_DATABASE_DRIVER:-}")"
url="$${AETHER_DATABASE_URL:-$${DATABASE_URL:-}}"
if [[ -z "$${driver}" && -z "$${url}" ]]; then
return 0
fi
[[ "$${driver}" == "postgres" || "$${driver}" == "postgresql" || "$${url}" == postgres:* || "$${url}" == postgresql:* ]]
}
dev_uses_redis_runtime() {
local backend
backend="$$(lowercase "$${AETHER_RUNTIME_BACKEND:-}")"
if [[ "$${backend}" == "memory" ]]; then
return 1
fi
if [[ "$${backend}" == "redis" ]]; then
return 0
fi
if dev_uses_sqlite_database; then
return 1
fi
return 0
}
print_dev_infra_hint() {
echo "=> 本地开发依赖未就绪。"
echo "=> 可手动启动 Postgres / Redis:"
echo "=> docker compose up -d postgres redis"
}
check_postgres_ready() {
local host="$$1"
local port="$$2"
if command -v pg_isready >/dev/null 2>&1; then
pg_isready -h "$${host}" -p "$${port}" >/dev/null 2>&1
return $$?
fi
if command -v nc >/dev/null 2>&1; then
nc -z "$${host}" "$${port}" >/dev/null 2>&1
return $$?
fi
return 0
}
check_redis_ready() {
local host="$$1"
local port="$$2"
local password="$$3"
if command -v redis-cli >/dev/null 2>&1; then
REDISCLI_AUTH="$${password}" redis-cli -h "$${host}" -p "$${port}" ping >/dev/null 2>&1
return $$?
fi
if command -v nc >/dev/null 2>&1; then
nc -z "$${host}" "$${port}" >/dev/null 2>&1
return $$?
fi
return 0
}
is_local_host() {
case "$$1" in
localhost|127.0.0.1|::1)
return 0
;;
esac
return 1
}
ensure_dev_infra() {
local postgres_host="$${DB_HOST:-localhost}"
local postgres_port="$${DB_PORT:-5432}"
local redis_host="$${REDIS_HOST:-localhost}"
local redis_port="$${REDIS_PORT:-6379}"
local redis_password="$${REDIS_PASSWORD:-}"
local need_postgres=false
local need_redis=false
local services=()
if dev_uses_postgres_database; then
if ! check_postgres_ready "$${postgres_host}" "$${postgres_port}"; then
if is_local_host "$${postgres_host}"; then
need_postgres=true
services+=(postgres)
else
echo "=> PostgreSQL 不可用: $${postgres_host}:$${postgres_port}"
print_dev_infra_hint
return 1
fi
fi
fi
if dev_uses_redis_runtime; then
if ! check_redis_ready "$${redis_host}" "$${redis_port}" "$${redis_password}"; then
if is_local_host "$${redis_host}"; then
need_redis=true
services+=(redis)
else
echo "=> Redis 不可用: $${redis_host}:$${redis_port}"
print_dev_infra_hint
return 1
fi
fi
fi
if [ "$${#services[@]}" -eq 0 ]; then
return 0
fi
if ! command -v docker >/dev/null 2>&1; then
echo "=> 未找到 docker,无法自动启动本地开发依赖。"
print_dev_infra_hint
return 1
fi
echo "=> 本地开发依赖未就绪,正在启动: docker compose up -d $${services[*]}"
if ! docker compose up -d "$${services[@]}"; then
echo "=> docker compose 启动本地开发依赖失败。"
print_dev_infra_hint
return 1
fi
for _ in {1..100}; do
local ready=true
if [ "$${need_postgres}" = "true" ] && ! check_postgres_ready "$${postgres_host}" "$${postgres_port}"; then
ready=false
fi
if [ "$${need_redis}" = "true" ] && ! check_redis_ready "$${redis_host}" "$${redis_port}" "$${redis_password}"; then
ready=false
fi
if [ "$${ready}" = "true" ]; then
return 0
fi
sleep 0.2
done
if [ "$${need_postgres}" = "true" ] && ! check_postgres_ready "$${postgres_host}" "$${postgres_port}"; then
echo "=> PostgreSQL 不可用: $${postgres_host}:$${postgres_port}"
fi
if [ "$${need_redis}" = "true" ] && ! check_redis_ready "$${redis_host}" "$${redis_port}" "$${redis_password}"; then
echo "=> Redis 不可用: $${redis_host}:$${redis_port}"
fi
print_dev_infra_hint
return 1
}
print_startup_failure_hint() {
local log_file="$$1"
if [ -n "$${log_file}" ] && [ -f "$${log_file}" ]; then
if grep -Eq "database schema is behind" "$${log_file}"; then
echo "=> 检测到数据库 schema 落后,请执行: make migration"
return
fi
if grep -Eq "database backfills are behind" "$${log_file}"; then
echo "=> 检测到待执行 backfills,请执行: make backfill"
return
fi
fi
echo "=> 未识别到明确的修复动作,请根据上面的日志继续排查。"
}
wait_for_startup() {
local pid="$$1"
local timeout_seconds="$$2"
local service_name="$$3"
shift 3
STARTUP_WAIT_EARLY_EXIT=false
local attempts=$$((timeout_seconds * 10))
if [ "$${attempts}" -lt 1 ]; then
attempts=1
fi
for ((i = 0; i < attempts; i++)); do
if "$$@" >/dev/null 2>&1; then
return 0
fi
if ! kill -0 "$${pid}" >/dev/null 2>&1; then
STARTUP_WAIT_EARLY_EXIT=true
echo "=> $${service_name} 启动进程已提前退出,请检查上面的日志。"
print_startup_failure_hint "$${GATEWAY_LOG_FILE}"
return 1
fi
sleep 0.1
done
if "$$@" >/dev/null 2>&1; then
return 0
fi
if ! kill -0 "$${pid}" >/dev/null 2>&1; then
STARTUP_WAIT_EARLY_EXIT=true
echo "=> $${service_name} 启动进程已提前退出,请检查上面的日志。"
print_startup_failure_hint "$${GATEWAY_LOG_FILE}"
return 1
fi
echo "=> $${service_name} 在 $${timeout_seconds}s 内未通过启动检查。"
echo "=> 如果这是冷编译或存在并发 cargo 构建,可调大启动超时后重试。"
return 1
}
create_gateway_log_file() {
local tmp_root="$${TMPDIR:-/tmp}"
tmp_root="$${tmp_root%/}"
GATEWAY_LOG_DIR="$$(mktemp -d "$${tmp_root}/aether-dev-startup.XXXXXX")"
GATEWAY_LOG_FILE="$${GATEWAY_LOG_DIR}/gateway.log"
: > "$${GATEWAY_LOG_FILE}"
}
cleanup() {
local status="$${1:-0}"
trap - INT TERM EXIT
if [ -n "$${GATEWAY_PID:-}" ]; then
echo ""
echo "=> 停止 aether-gateway..."
kill "$${GATEWAY_PID}" >/dev/null 2>&1 || true
wait "$${GATEWAY_PID}" >/dev/null 2>&1 || true
fi
if [ -n "$${GATEWAY_LOG_FILE:-}" ] && [ -f "$${GATEWAY_LOG_FILE}" ]; then
rm -f "$${GATEWAY_LOG_FILE}"
fi
if [ -n "$${GATEWAY_LOG_DIR:-}" ] && [ -d "$${GATEWAY_LOG_DIR}" ]; then
rmdir "$${GATEWAY_LOG_DIR}" >/dev/null 2>&1 || true
fi
exit "$${status}"
}
trap 'cleanup 130' INT
trap 'cleanup 143' TERM
trap 'cleanup $$?' EXIT
export APP_PORT="$${APP_PORT:-8084}"
export RUST_LOG="$${DEV_RUST_LOG}"
RUST_SERVICE_STARTUP_TIMEOUT_SECONDS="$${RUST_SERVICE_STARTUP_TIMEOUT_SECONDS:-180}"
GATEWAY_STARTUP_TIMEOUT_SECONDS="$${GATEWAY_STARTUP_TIMEOUT_SECONDS:-$${RUST_SERVICE_STARTUP_TIMEOUT_SECONDS}}"
export AETHER_GATEWAY_VIDEO_TASK_TRUTH_SOURCE_MODE="$${AETHER_GATEWAY_VIDEO_TASK_TRUTH_SOURCE_MODE:-rust-authoritative}"
if dev_uses_postgres_database; then
export DATABASE_URL="postgresql://$${DB_USER:-postgres}:$${DB_PASSWORD:-}@$${DB_HOST:-localhost}:$${DB_PORT:-5432}/$${DB_NAME:-aether}"
if ! dotenv_has_key "AETHER_GATEWAY_DATA_POSTGRES_URL"; then
export AETHER_GATEWAY_DATA_POSTGRES_URL="$${DATABASE_URL}"
fi
fi
if dev_uses_redis_runtime; then
export REDIS_URL="redis://:$${REDIS_PASSWORD:-}@$${REDIS_HOST:-localhost}:$${REDIS_PORT:-6379}/0"
if ! dotenv_has_key "AETHER_GATEWAY_DATA_REDIS_URL"; then
export AETHER_GATEWAY_DATA_REDIS_URL="$${REDIS_URL}"
fi
else
unset REDIS_URL
unset AETHER_GATEWAY_DATA_REDIS_URL
fi
if ! dotenv_has_key "AETHER_GATEWAY_DATA_ENCRYPTION_KEY"; then
export AETHER_GATEWAY_DATA_ENCRYPTION_KEY="$${ENCRYPTION_KEY:-}"
fi
export DB_POOL_SIZE="$${DB_POOL_SIZE:-5}"
export DB_MAX_OVERFLOW="$${DB_MAX_OVERFLOW:-5}"
export HTTP_MAX_CONNECTIONS="$${HTTP_MAX_CONNECTIONS:-20}"
export HTTP_KEEPALIVE_CONNECTIONS="$${HTTP_KEEPALIVE_CONNECTIONS:-5}"
if ! command -v cargo >/dev/null 2>&1; then
echo "=> 未找到 cargo,无法启动 aether-gateway。请先安装 Rust toolchain。"
exit 1
fi
if ! command -v curl >/dev/null 2>&1; then
echo "=> 未找到 curl,无法检查 aether-gateway 健康状态。请先安装 curl。"
exit 1
fi
if [ -z "$${RUSTC_WRAPPER:-}" ] && command -v sccache >/dev/null 2>&1; then
export RUSTC_WRAPPER="$$(command -v sccache)"
echo "=> 启用 Rust 编译缓存: $${RUSTC_WRAPPER}"
fi
if ! ensure_dev_infra; then
exit 1
fi
GATEWAY_PID=""
GATEWAY_LOG_DIR=""
GATEWAY_LOG_FILE=""
STARTUP_WAIT_EARLY_EXIT=false
create_gateway_log_file
echo "=> 启动 aether-gateway (Rust frontdoor: 0.0.0.0:$${APP_PORT})..."
echo "=> 日志过滤: $${RUST_LOG}"
echo "=> 执行命令: cargo run -p aether-gateway -- --app-port $${APP_PORT}"
cargo run -p aether-gateway -- --app-port "$${APP_PORT}" > >(
tee -a "$${GATEWAY_LOG_FILE}"
) 2>&1 &
GATEWAY_PID=$$!
if ! wait_for_startup "$${GATEWAY_PID}" "$${GATEWAY_STARTUP_TIMEOUT_SECONDS}" "aether-gateway" curl -sf "http://127.0.0.1:$${APP_PORT}/_gateway/health"; then
if [ "$${STARTUP_WAIT_EARLY_EXIT}" = "true" ]; then
GATEWAY_PID=""
fi
exit 1
fi
if wait "$${GATEWAY_PID}"; then
gateway_exit_code=0
else
gateway_exit_code=$$?
fi
GATEWAY_PID=""
if [ "$${gateway_exit_code}" -ne 130 ] && [ "$${gateway_exit_code}" -ne 143 ]; then
echo "=> aether-gateway 运行失败并已退出,请检查上面的日志。"
print_startup_failure_hint "$${GATEWAY_LOG_FILE}"
fi
exit "$${gateway_exit_code}"
endef
export DEV_BACKEND_SCRIPT
define DEV_SCRIPT
set -euo pipefail
backend_pid=""
frontend_pid=""
cleanup() {
local status="$${1:-0}"
trap - INT TERM EXIT
if [ -n "$${backend_pid}" ] || [ -n "$${frontend_pid}" ]; then
echo ""
echo "=> 停止本地开发服务..."
if [ -n "$${backend_pid}" ]; then
kill "$${backend_pid}" >/dev/null 2>&1 || true
wait "$${backend_pid}" >/dev/null 2>&1 || true
fi
if [ -n "$${frontend_pid}" ]; then
kill "$${frontend_pid}" >/dev/null 2>&1 || true
wait "$${frontend_pid}" >/dev/null 2>&1 || true
fi
fi
exit "$${status}"
}
wait_for_backend_ready() {
while :; do
if curl -sf "http://127.0.0.1:$${APP_PORT}/_gateway/health" >/dev/null 2>&1; then
return 0
fi
if ! kill -0 "$${backend_pid}" >/dev/null 2>&1; then
if wait "$${backend_pid}"; then
status=0
else
status=$$?
fi
if [ "$${status}" -ne 0 ]; then
echo "=> 后端进程已退出 (status $${status})"
else
echo "=> 后端进程已退出"
fi
backend_pid=""
cleanup "$${status}"
fi
sleep 0.2
done
}
trap 'cleanup 130' INT
trap 'cleanup 143' TERM
trap 'cleanup $$?' EXIT
if [ -f .env ]; then
set -a
source .env
set +a
fi
export APP_PORT="$${APP_PORT:-8084}"
echo "=> 启动后端: RUST_LOG=$${DEV_RUST_LOG} cargo run -p aether-gateway -- --app-port $${APP_PORT:-8084}"
/bin/bash -euo pipefail -c "$$DEV_BACKEND_SCRIPT" &
backend_pid=$$!
echo "=> 等待后端健康检查: http://127.0.0.1:$${APP_PORT}/_gateway/health"
wait_for_backend_ready
echo "=> 启动前端: cd frontend && npm run dev"
( cd frontend && exec npm run dev ) &
frontend_pid=$$!
while :; do
if ! kill -0 "$${backend_pid}" >/dev/null 2>&1; then
if wait "$${backend_pid}"; then
status=0
else
status=$$?
fi
if [ "$${status}" -ne 0 ]; then
echo "=> 后端进程已退出 (status $${status})"
else
echo "=> 后端进程已退出"
fi
backend_pid=""
cleanup "$${status}"
fi
if ! kill -0 "$${frontend_pid}" >/dev/null 2>&1; then
if wait "$${frontend_pid}"; then
status=0
else
status=$$?
fi
if [ "$${status}" -ne 0 ]; then
echo "=> 前端进程已退出 (status $${status})"
else
echo "=> 前端进程已退出"
fi
frontend_pid=""
cleanup "$${status}"
fi
sleep 1
done
endef
export DEV_SCRIPT
define DB_TASK_SCRIPT
set -euo pipefail
if [ -z "$${DB_TASK_FLAG:-}" ] || [ -z "$${DB_TASK_LABEL:-}" ]; then
echo "=> 内部错误: DB_TASK_FLAG / DB_TASK_LABEL 未设置"
exit 1
fi
if [ ! -f .env ]; then
echo "=> 未找到 .env,请先执行: cp .env.example .env"
exit 1
fi
set -a
source .env
set +a
dotenv_has_key() {
local key="$$1"
grep -Eq "^[[:space:]]*$${key}=" .env
}
lowercase() {
printf '%s' "$$1" | tr '[:upper:]' '[:lower:]'
}
uses_postgres_database() {
local driver
local url
driver="$$(lowercase "$${AETHER_DATABASE_DRIVER:-}")"
url="$${AETHER_DATABASE_URL:-$${DATABASE_URL:-}}"
if [[ -z "$${driver}" && -z "$${url}" ]]; then
return 0
fi
[[ "$${driver}" == "postgres" || "$${driver}" == "postgresql" || "$${url}" == postgres:* || "$${url}" == postgresql:* ]]
}
if uses_postgres_database; then
export DATABASE_URL="postgresql://$${DB_USER:-postgres}:$${DB_PASSWORD:-}@$${DB_HOST:-localhost}:$${DB_PORT:-5432}/$${DB_NAME:-aether}"
if ! dotenv_has_key "AETHER_GATEWAY_DATA_POSTGRES_URL"; then
export AETHER_GATEWAY_DATA_POSTGRES_URL="$${DATABASE_URL}"
fi
fi
if ! dotenv_has_key "AETHER_GATEWAY_DATA_ENCRYPTION_KEY"; then
export AETHER_GATEWAY_DATA_ENCRYPTION_KEY="$${ENCRYPTION_KEY:-}"
fi
if ! command -v cargo >/dev/null 2>&1; then
echo "=> 未找到 cargo,无法执行 $${DB_TASK_LABEL}。请先安装 Rust toolchain。"
exit 1
fi
echo "=> 执行 $${DB_TASK_LABEL}: cargo run -p aether-gateway -- $${DB_TASK_FLAG}"
exec cargo run -p aether-gateway -- "$${DB_TASK_FLAG}"
endef
export DB_TASK_SCRIPT
dev:
@$(SHELL) -euo pipefail -c "$$DEV_SCRIPT"
dev-backend:
@$(SHELL) -euo pipefail -c "$$DEV_BACKEND_SCRIPT"
dev-frontend:
@cd frontend && npm run dev
migration:
@DB_TASK_FLAG=--migrate DB_TASK_LABEL="数据库迁移" $(SHELL) -euo pipefail -c "$$DB_TASK_SCRIPT"
backfill:
@DB_TASK_FLAG=--apply-backfills DB_TASK_LABEL="数据库 backfill" $(SHELL) -euo pipefail -c "$$DB_TASK_SCRIPT"
+99 -100
View File
@@ -11,6 +11,7 @@
<p align="center">
<a href="#简介">简介</a> •
<a href="#部署">部署</a> •
<a href="#api-文档">API 文档</a> •
<a href="#环境变量">环境变量</a> •
<a href="#qa">Q&A</a>
</p>
@@ -43,126 +44,124 @@ cd Aether
# 2. 配置环境变量
cp .env.example .env
python generate_keys.py # 生成密钥, 并将生成的密钥填入 .env
# 生成 JWT_SECRET_KEY / ENCRYPTION_KEY, 并填入 .env
./generate_keys.sh
# 编辑 .env 设置 ADMIN_PASSWORD
# 3. 部署 / 更新(自动执行数据库迁移)
# 3. 首次部署 / 更新 (从以下部署形态任选其一)
# Postgres + Redis (适用于企业或多人使用)
docker compose pull && docker compose up -d
# 4. 升级前备份 (可选)
docker compose exec postgres pg_dump -U postgres aether | gzip > backup_$(date +%Y%m%d_%H%M%S).sql.gz
# Single Node (适用于个人用户或朋友分享)
docker compose -f docker-compose.single-node.yml pull && docker compose -f docker-compose.single-node.yml up -d
```
### Docker Compose(本地构建镜像)
### 一键更新
Docker Compose 部署后,可在部署目录直接执行:
```bash
# 1. 克隆代码
git clone https://github.com/fawney19/Aether.git
cd Aether
./update.sh
```
# 2. 配置环境变量
cp .env.example .env
python generate_keys.py # 生成密钥, 并将生成的密钥填入 .env
`update.sh` 会拉取最新 `app` 镜像并重建 `app` 容器,Docker named volumes、`./data` 和 `./logs` 不会被删除。Single Node 部署也可显式指定:
# 3. 部署 / 更新(自动构建、启动、迁移)
git pull
```bash
./update.sh --mode single-node
```
仓库自带的 Docker Compose 默认把应用日志输出到容器 `stdout/stderr`,直接用 `docker compose logs -f app` 查看,并由 Docker 轮转日志,避免正式发布镜像切换到非 root 用户后再被宿主机挂载日志目录的权限问题拖垮启动。如果你确实需要文件日志,需要在 compose 里把 `AETHER_LOG_DESTINATION` 改成 `file|both`,并额外挂载一个容器用户可写的目录到 `/opt/aether/logs`。
管理后台右上角“版本信息”会检测新版本。Docker Compose 部署只提示版本,实际更新继续执行 `./update.sh`;systemd / launchd / 二进制部署才使用后台自更新,流程是下载对应平台的 GitHub Release 包、强制校验 `SHA256SUMS`、解压到 `/opt/aether/releases/<version>`,再切换 `/opt/aether/current` 并退出进程,交给 systemd / launchd 拉起新版本。
源码或本地构建版本不会启用后台在线更新,请继续使用源码更新流程。Docker Compose 用户如果希望“容器重建后也保持镜像层面的新版本”,仍建议定期运行 `./update.sh` 拉取并重建 app 镜像。服务器访问 GitHub 需要代理时,可设置 `AETHER_UPDATE_PROXY_URL`,也兼容 `UPDATE_PROXY_URL`、`HTTPS_PROXY`、`ALL_PROXY`、`HTTP_PROXY` 以及 `NO_PROXY`。共享出口触发 GitHub API 限流时,可设置只读 `AETHER_UPDATE_GITHUB_TOKEN`,也兼容 `GITHUB_TOKEN` / `GH_TOKEN`。下载总超时默认 600 秒,连续无响应/无数据默认 30 秒,可通过 `AETHER_UPDATE_DOWNLOAD_TIMEOUT_SECS` 和 `AETHER_UPDATE_DOWNLOAD_IDLE_TIMEOUT_SECS` 调整。
标准 Docker Compose 使用 Docker named volumes 存放 Postgres/Redis/MySQL 数据;Single Node 使用部署目录下的 `./data` 存放 SQLite 数据。
如果是本地源码构建镜像的部署,继续使用:
```bash
./deploy.sh
```
### 本地开发
如果要在本机联调“管理后台在线更新”本身,可启动仓库内置的 release-layout 测试环境:
```bash
# 启动依赖
docker compose -f docker-compose.build.yml up -d postgres redis
# 后端
uv sync
./dev.sh
# 前端
cd frontend && npm install && npm run dev
docker compose -f docker-compose.release-local.yml up -d --build
```
## Aether Proxy (可选)
这套环境会用当前源码构建一个本地测试镜像,但编译为 `release` 类型,并默认伪装成 `v0.7.0`,这样后台会按正式发布版逻辑开放“立即更新”。默认监听 `http://127.0.0.1:18085`,数据目录使用 `./data-release-local`;日志默认走 `docker logs`,不会影响你正在跑的源码构建容器。
Aether Proxy 是配套的正向代理节点,部署在海外 VPS 上,为墙内的 Aether 实例中转 API 流量。或者部署在其他服务器为指定的提供商、账号、Key使用不同的节点访问。支持 TUI 向导一键配置、systemd 服务管理、TLS 加密、DNS 缓存及连接池调优。
如果这套容器在 `prepare-update` 时访问 GitHub 失败,而你本机是通过代理出网,请在 `.env` 里把 `AETHER_UPDATE_PROXY_URL` 写成宿主机地址,例如 `http://host.docker.internal:7890`;容器内的 `127.0.0.1` 指向容器自身,不是宿主机。
如果想重置这套联调环境(包括 `/opt/aether/current` 和已下载的历史版本),执行:
```bash
docker compose -f docker-compose.release-local.yml down -v
```
可选变量:
- `AETHER_RELEASE_LOCAL_VERSION`:本地联调镜像对外声明的当前版本,默认 `v0.7.0`
- `AETHER_RELEASE_LOCAL_PORT`:本地联调端口,默认 `18085`
- `LOCAL_RELEASE_APP_IMAGE`:本地联调镜像名,默认 `aether-app:release-local`
### 一键安装(默认 Single Node:Linux systemd / macOS launchd + SQLite)
```bash
git clone https://github.com/fawney19/Aether.git
cd Aether
curl -fsSL https://raw.githubusercontent.com/fawney19/Aether/main/install.sh | sudo bash
```
## 本地开发
依赖 Docker、Rust toolchain、Node.js 和 make。
```bash
make dev
```
`make dev` 会同时启动后端 `aether-gateway` 和前端 `frontend` 的 Vite dev server。需要单独启动时可使用 `make dev-backend` 或 `make dev-frontend`。
Postgres / Redis 本地依赖未就绪时,`make dev` 会自动执行 `docker compose up -d postgres redis`。
## Aether Tunnel (可选)
Aether Tunnel 是配套的正向代理节点,部署在海外 VPS 上,为墙内的 Aether 实例中转 API 流量。
- Docker Compose 部署或下载预编译二进制直接运行
- 通过 `aether-proxy setup` 完成交互式配置,自动注册为系统服务
- 详细文档见 [aether-proxy/README.md](aether-proxy/README.md)
- 提供 macOS/Linux 与 Windows 一键脚本,自动下载最新 `tunnel-v*` 制品并向现有 `aether-tunnel.toml` 追加 `[[servers]]`
- 通过 `aether-tunnel setup` 完成交互式配置,自动注册为系统服务
- 详细文档见 [apps/aether-tunnel/README.md](apps/aether-tunnel/README.md)
## API 文档
- Embeddings: [OpenAI compatible `POST /v1/embeddings`](docs/api/embeddings.md)
- Rerank: [OpenAI/Jina compatible `POST /v1/rerank`](docs/api/rerank.md)
- Responses WebSocket mode: [protocol and Aether behavior](docs/WebSocket-Mode.md)
- WebSocket probes: [Codex](docs/operations/codex-responses-websocket-probe.md) · [OpenAI Responses](docs/operations/openai-responses-websocket-probe.md)
## 环境变量
### 必需配置
| 变量 | 说明 |
|------|------|
| `DB_PASSWORD` | PostgreSQL 数据库密码 |
| `REDIS_PASSWORD` | Redis 密码 |
| `JWT_SECRET_KEY` | JWT 签名密钥(使用 `generate_keys.py` 生成) |
| `ENCRYPTION_KEY` | API Key 加密密钥(更换后需重新配置 Provider Key) |
| `ADMIN_EMAIL` | 初始管理员邮箱 |
| `ADMIN_USERNAME` | 初始管理员用户名 |
| `ADMIN_PASSWORD` | 初始管理员密码 |
### 可选配置
| 变量 | 默认值 | 说明 |
|------|--------|------|
| `APP_PORT` | 8084 | 应用端口 |
| `API_KEY_PREFIX` | sk | API Key 前缀 |
| `LOG_LEVEL` | INFO | 日志级别 (DEBUG/INFO/WARNING/ERROR) |
| `GUNICORN_WORKERS` | 2 | Gunicorn 工作进程数 |
| `DB_PORT` | 5432 | PostgreSQL 端口 |
| `REDIS_PORT` | 6379 | Redis 端口 |
## Q&A
### Q: 如何开启/关闭请求体记录?
管理员在 **系统设置** 中配置日志记录的详细程度:
| 级别 | 记录内容 |
|------|----------|
| Base | 基本请求信息 |
| Headers | Base + 请求头 |
| Full | Headers + 请求体 |
### Q: 更新出问题如何回滚?
**有备份的情况(推荐):**
```bash
# 1. 停止应用
docker compose stop app
# 2. 恢复数据库(先清空再导入)
docker compose exec -T postgres psql -U postgres -c "DROP DATABASE aether; CREATE DATABASE aether;"
gunzip < backup_xxx.sql.gz | docker compose exec -T postgres psql -U postgres -d aether
# 3. 拉取旧版本镜像并重启
# 方式一:使用具体版本 tag(如果有发布版本号)
# 将 docker-compose.yml 中 image 从 ghcr.io/fawney19/aether:latest 改为指定版本
# 方式二:使用之前记录的镜像 digest
# 将 image 改为 ghcr.io/fawney19/aether@sha256:xxxxx
docker compose up -d app
```
> 可以在升级前通过 `docker inspect ghcr.io/fawney19/aether:latest --format '{{index .RepoDigests 0}}'` 记录当前镜像 digest,方便回滚时使用。
**没有备份的情况:**
```bash
# 1. 用当前容器回退数据库迁移(回退 1 步,按需调整数字)
docker compose exec app alembic downgrade -1
# 2. 查看回退后的版本确认正确
docker compose exec app alembic current
# 3. 切回旧镜像并重启(同上方式修改 docker-compose.yml 中的 image)
docker compose up -d app
```
> 注意:没有备份的回滚依赖 alembic downgrade,如果迁移涉及不可逆的数据变更(如删除列),可能无法完全恢复数据。因此强烈建议升级前备份。
- `APP_PORT`:`aether-gateway` 唯一监听端口,固定绑定 `0.0.0.0:${APP_PORT}`
- `DATABASE_URL`:数据库连接串;SQLite 例如 `sqlite:///opt/aether/data/aether.db`,Postgres 例如 `postgresql://postgres:aether@postgres:5432/aether`
- `AETHER_GATEWAY_DATA_POSTGRES_MIN_CONNECTIONS` / `AETHER_GATEWAY_DATA_POSTGRES_MAX_CONNECTIONS`:数据库连接池手动覆盖值;未配置时 SQLite 固定 `1/1`,Postgres/MySQL 按每核 `4` 条自动推导,总池范围为 `32-100`。该预算按进程计算,多实例部署应按数据库连接上限显式分配
- `AETHER_GATEWAY_MAX_IN_FLIGHT_REQUESTS`:单实例请求并发上限;未配置时按 CPU 自动推导(基础范围 `512-65536`),低文件描述符预算时会进一步下调
- `AETHER_GATEWAY_REQUEST_BODY_BUFFER_BUDGET_MB`:单实例同时读取和解压请求体的加权内存预算,默认 `256MB`
- `AETHER_GATEWAY_REQUEST_BODY_READ_TIMEOUT_MS`:请求体完整读取超时,默认 `120000ms`
- `AETHER_MAX_REQUEST_BODY_MB`:可选的单请求解压后请求体上限;未配置或设为 `0` 时不限制
- `AETHER_MAX_INTERNAL_BUFFERED_BODY_MB`:可选的 heartbeat、管理探测等内部整包响应体上限;未配置或设为 `0` 时不限制
- `AETHER_TUNNEL_NODE_STATUS_QUEUE_CAPACITY`:隧道节点状态上报队列容量,默认 `1024`;满载时拒绝新事件,避免控制面故障导致无界内存增长
- `AETHER_GATEWAY_SECURITY_CACHE_TTL_MS`:IP 黑白名单本地缓存时间,默认 `1000ms`,写操作会主动失效相关缓存
- `AETHER_MAX_REDACTED_SYNC_RESPONSE_BODY_MB`:可选的 PII 恢复同步响应缓冲上限;未配置或设为 `0` 时不限制
- `REDIS_URL`:Redis 连接串;仅 Postgres + Redis 的 Docker Compose 部署需要配置
- `AETHER_RUNTIME_BACKEND=memory|redis`:运行时缓存/协调后端。SQLite 默认用 `memory`,不会连接 Redis;多节点部署和需要跨 gateway 重启恢复 OpenAI Responses continuation history 的部署必须使用共享 Redis
- `AETHER_GATEWAY_AUTO_PREPARE_DATABASE`:常规启动前自动执行挂起的 schema migration 和 backfill;仓库自带的 `docker-compose.yml` 默认开启
- `JWT_SECRET_KEY` / `ENCRYPTION_KEY`:认证和敏感数据加密所需密钥
- `API_KEY_PREFIX`:用户和管理员新建 API Key 时使用的前缀,默认 `sk`
- `ADMIN_USERNAME` / `ADMIN_PASSWORD` / `ADMIN_EMAIL`:首次启动时自举首个本地管理员;`install.sh` 会提示输入管理员密码
- `CORS_ORIGINS` / `CORS_ALLOW_CREDENTIALS`:前端跨域来源控制;如果要跨域带登录 Cookie,`CORS_ORIGINS` 不能写 `*`
- `RUST_LOG`:Rust 日志过滤,例如 `aether_gateway=info`、`aether_gateway=debug,sqlx=warn`
- Docker Compose 的 `DB_PASSWORD` / `REDIS_PASSWORD` 默认使用 `aether`
---
@@ -180,4 +179,4 @@ docker compose up -d app
## Star History
[![Star History Chart](https://api.star-history.com/svg?repos=fawney19/Aether&type=Date)](https://star-history.com/#fawney19/Aether&Date)
[![Star History Chart](https://api.star-history.com/svg?repos=fawney19/Aether&type=date&legend=top-left)](https://www.star-history.com/?repos=fawney19%2FAether&type=date&legend=top-left)
-3
View File
@@ -1,3 +0,0 @@
target/
.git/
.DS_Store
-2012
View File
File diff suppressed because it is too large Load Diff
-27
View File
@@ -1,27 +0,0 @@
[package]
name = "aether-hub"
version = "0.2.0"
edition = "2021"
description = "Tunnel Hub for Aether - frame router between workers and proxies"
[dependencies]
tokio = { version = "1", features = ["full"] }
axum = { version = "0.8", features = ["ws"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
clap = { version = "4", features = ["derive", "env"] }
dashmap = "6"
parking_lot = "0.12"
flate2 = "1"
futures-util = "0.3"
bytes = "1"
async-stream = "0.3"
http-body-util = "0.1"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
[profile.release]
lto = true
strip = true
codegen-units = 1
-34
View File
@@ -1,34 +0,0 @@
# syntax=docker/dockerfile:1
FROM rust:1.85-slim AS builder
WORKDIR /build/aether-hub
# 可选:配置国内 Cargo 镜像源(本地构建时传 --build-arg CARGO_MIRROR=1)
ARG CARGO_MIRROR
RUN if [ -n "$CARGO_MIRROR" ]; then \
printf '[source.crates-io]\nreplace-with = "tuna"\n\n[source.tuna]\nregistry = "sparse+https://mirrors.tuna.tsinghua.edu.cn/crates.io-index/"\n' \
> /usr/local/cargo/config.toml; \
fi
# 先构建依赖层,最大化后续代码变更时的缓存命中
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && printf 'fn main() {}\n' > src/main.rs
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,target=/build/aether-hub/target,sharing=locked \
cargo build --release --locked
RUN rm -rf src
COPY src ./src
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,target=/build/aether-hub/target,sharing=locked \
cargo build --release --locked && \
cp target/release/aether-hub /tmp/aether-hub
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && \
rm -rf /var/lib/apt/lists/*
COPY --from=builder /tmp/aether-hub /usr/local/bin/aether-hub
EXPOSE 8085
ENTRYPOINT ["/usr/local/bin/aether-hub"]
CMD ["--bind", "0.0.0.0:8085"]
-38
View File
@@ -1,38 +0,0 @@
# aether-hub
`aether-hub` 是 Tunnel Hub 服务,负责在 proxy 与 worker 之间路由帧。
已集成在Docker镜像中, 无需单独部署。
## 部署端指定 Hub 版本并构建
```bash
cd /path/to/Aether
./deploy.sh --hub-tag hub-v0.1.0
```
不指定 `--hub-tag` 时,`./deploy.sh` 会自动解析最新 `hub-v*` release,并在构建 app 镜像时从 GitHub Release 下载对应架构的 Hub 二进制。
## build.sh 模式说明
- 默认是 `binary` 模式(`cross` 构建二进制)。
- `--upload <hub-vX.Y.Z>` 会把构建产物上传到 GitHub Release。
- 加 `--image` 后进入镜像模式(`docker buildx`,可选)。
常用参数:
- `--tag <tag>`: 镜像 tag
- `--image-name <name>`: 镜像名(默认 `ghcr.io/fawney19/aether-hub`)
- `--platforms <list>`: 例如 `linux/amd64,linux/arm64`
- `--push`: 推送镜像
- `--load`: 加载到本地 Docker(单平台)
- `--latest`: 额外打 `latest` tag
## 运行时参数
- `TUNNEL_HUB_WORKER_IDLE_TIMEOUT`:worker 心跳空闲超时,默认 `60` 秒
- `TUNNEL_HUB_OUTBOUND_QUEUE_CAPACITY`:单连接出站队列容量,默认 `128`;队列打满时会把连接视为拥塞并主动关闭,避免 Hub 内存无限增长
## 与部署脚本关系
- `./deploy.sh`: 本地构建部署(会本地构建 app/base,并在构建 app 时从 GitHub Release 下载 Hub,可用 `--hub-tag` 固定版本)。
-272
View File
@@ -1,272 +0,0 @@
#!/bin/bash
# aether-hub 构建脚本
#
# 支持两种模式:
# 1) binary 模式(默认): 构建多架构二进制并可上传 GitHub Release
# 2) image 模式: 构建并推送/加载 Docker 镜像(推荐生产发布用)
#
# 示例:
# # binary 模式(兼容旧行为)
# ./build.sh
# ./build.sh amd64
# ./build.sh --upload hub-v0.1.0
#
# # image 模式(多架构推送)
# ./build.sh --image --tag v0.2.5 --push --latest
# ./build.sh --image --tag sha-abc123 --image-name ghcr.io/fawney19/aether-hub --push
# ./build.sh --image --tag local-test --platforms linux/amd64 --load
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
DIST_DIR="$SCRIPT_DIR/dist"
# -------------------------------
# Defaults
# -------------------------------
MODE="binary" # binary | image
# binary mode options
UPLOAD=false
UPLOAD_TAG=""
BINARY_TARGETS=""
# image mode options
IMAGE_NAME="${IMAGE_NAME:-ghcr.io/fawney19/aether-hub}"
IMAGE_TAG=""
IMAGE_PLATFORMS="linux/amd64,linux/arm64"
IMAGE_PUSH=false
IMAGE_LOAD=false
IMAGE_LATEST=false
usage() {
cat <<'EOF'
用法:
./build.sh [binary-args]
./build.sh --image [image-args]
binary 模式(默认):
amd64|arm64 仅构建指定架构(可重复)
--upload <hub-vX.Y.Z> 上传到 GitHub Release(需要 gh CLI)
image 模式:
--image 启用镜像模式
--tag <tag> 镜像 tag(默认自动从 git describe 推导)
--image-name <name> 镜像名(默认 ghcr.io/fawney19/aether-hub)
--platforms <list> 平台列表,逗号分隔(默认 linux/amd64,linux/arm64)
--push 推送镜像到仓库
--load 加载到本地 Docker(仅单平台)
--latest 额外打 latest tag
通用:
-h, --help 显示帮助
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--image)
MODE="image"
shift
;;
--tag)
IMAGE_TAG="${2:-}"
shift 2
;;
--image-name)
IMAGE_NAME="${2:-}"
shift 2
;;
--platforms)
IMAGE_PLATFORMS="${2:-}"
shift 2
;;
--push)
IMAGE_PUSH=true
shift
;;
--load)
IMAGE_LOAD=true
shift
;;
--latest)
IMAGE_LATEST=true
shift
;;
--upload)
UPLOAD=true
UPLOAD_TAG="${2:-}"
shift 2
;;
amd64|arm64)
BINARY_TARGETS="$BINARY_TARGETS $1"
shift
;;
-h|--help)
usage
exit 0
;;
*)
echo "❌ 未知参数: $1"
usage
exit 1
;;
esac
done
build_binary() {
if [ -z "$BINARY_TARGETS" ]; then
BINARY_TARGETS="amd64 arm64"
fi
if ! command -v cross >/dev/null 2>&1; then
echo "❌ 需要安装 cross: cargo install cross --git https://github.com/cross-rs/cross"
exit 1
fi
mkdir -p "$DIST_DIR"
echo "🔨 开始构建 aether-hub 二进制..."
echo " 目标平台: $BINARY_TARGETS"
echo ""
ARTIFACTS=""
for arch in $BINARY_TARGETS; do
case "$arch" in
amd64) target="x86_64-unknown-linux-gnu" ;;
arm64) target="aarch64-unknown-linux-gnu" ;;
*) echo "❌ 未知架构: $arch"; exit 1 ;;
esac
echo ">>> 构建 $arch ($target)..."
cd "$SCRIPT_DIR"
cross build --release --target "$target" --locked
BIN="target/$target/release/aether-hub"
if [ ! -f "$BIN" ]; then
echo "❌ 未找到二进制文件: $BIN"
exit 1
fi
ARCHIVE="$DIST_DIR/aether-hub-linux-$arch.tar.gz"
tar czf "$ARCHIVE" -C "target/$target/release" aether-hub
ARTIFACTS="$ARTIFACTS $ARCHIVE"
SIZE=$(du -h "$ARCHIVE" | cut -f1)
echo "✅ $arch 构建完成: $ARCHIVE ($SIZE)"
echo ""
done
cd "$DIST_DIR"
shasum -a 256 aether-hub-*.tar.gz > SHA256SUMS.txt
echo "📋 SHA256 校验和:"
cat SHA256SUMS.txt
echo ""
if [ "$UPLOAD" = true ]; then
if [ -z "$UPLOAD_TAG" ]; then
echo "❌ --upload 需要指定 tag,例如: ./build.sh --upload hub-v0.1.0"
exit 1
fi
if ! command -v gh >/dev/null 2>&1; then
echo "❌ 需要安装 GitHub CLI: brew install gh"
exit 1
fi
echo "📦 上传到 GitHub Release: $UPLOAD_TAG"
cd "$PROJECT_DIR"
if ! git rev-parse "$UPLOAD_TAG" >/dev/null 2>&1; then
git tag "$UPLOAD_TAG"
git push origin "$UPLOAD_TAG"
fi
gh release create "$UPLOAD_TAG" \
--title "aether-hub ${UPLOAD_TAG#hub-}" \
--generate-notes \
$ARTIFACTS \
"$DIST_DIR/SHA256SUMS.txt"
echo "✅ 上传完成!"
fi
echo "🎉 binary 模式完成!"
}
build_image() {
if ! command -v docker >/dev/null 2>&1; then
echo "❌ 未找到 docker,请先安装 Docker"
exit 1
fi
if ! docker buildx version >/dev/null 2>&1; then
echo "❌ 未找到 docker buildx,请先启用 buildx"
exit 1
fi
if [ "$IMAGE_PUSH" = true ] && [ "$IMAGE_LOAD" = true ]; then
echo "❌ --push 与 --load 不能同时使用"
exit 1
fi
if [ "$IMAGE_PUSH" = false ] && [ "$IMAGE_LOAD" = false ]; then
# image 模式默认走 push,符合发布场景
IMAGE_PUSH=true
fi
if [ -z "$IMAGE_TAG" ]; then
IMAGE_TAG=$(git -C "$PROJECT_DIR" describe --tags --always 2>/dev/null | sed 's/^v//')
if [ -z "$IMAGE_TAG" ]; then
IMAGE_TAG=$(date +%Y%m%d%H%M%S)
fi
fi
if [ "$IMAGE_LOAD" = true ] && [[ "$IMAGE_PLATFORMS" == *,* ]]; then
echo "❌ --load 仅支持单平台,请用 --platforms linux/amd64(或 arm64)"
exit 1
fi
local ref="${IMAGE_NAME}:${IMAGE_TAG}"
local cmd=(docker buildx build
--platform "$IMAGE_PLATFORMS"
-f "$SCRIPT_DIR/Dockerfile"
-t "$ref"
)
if [ "$IMAGE_LATEST" = true ]; then
cmd+=(-t "${IMAGE_NAME}:latest")
fi
if [ "$IMAGE_PUSH" = true ]; then
cmd+=(--push)
else
cmd+=(--load)
fi
cmd+=("$SCRIPT_DIR")
echo "🔨 开始构建 aether-hub 镜像..."
echo " image: $ref"
echo " platforms: $IMAGE_PLATFORMS"
echo " mode: $([ "$IMAGE_PUSH" = true ] && echo push || echo load)"
echo ""
"${cmd[@]}"
if [ "$IMAGE_PUSH" = true ]; then
echo "✅ 镜像已推送: $ref"
if [ "$IMAGE_LATEST" = true ]; then
echo "✅ 镜像已推送: ${IMAGE_NAME}:latest"
fi
else
echo "✅ 镜像已加载到本地: $ref"
fi
echo "🎉 image 模式完成!"
}
if [ "$MODE" = "image" ]; then
build_image
else
build_binary
fi
-85
View File
@@ -1,85 +0,0 @@
use reqwest::Client;
#[derive(Clone)]
pub struct ControlPlaneClient {
client: Option<Client>,
base_url: String,
}
impl ControlPlaneClient {
pub fn new(base_url: String) -> Self {
let client = Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()
.ok();
Self { client, base_url }
}
pub fn disabled() -> Self {
Self {
client: None,
base_url: String::new(),
}
}
pub async fn heartbeat_ack(&self, payload: &[u8]) -> Result<Vec<u8>, String> {
let Some(client) = &self.client else {
return Ok(b"{}".to_vec());
};
let url = format!(
"{}/api/internal/hub/heartbeat",
self.base_url.trim_end_matches('/')
);
let response = client
.post(&url)
.header("content-type", "application/json")
.body(payload.to_vec())
.send()
.await
.map_err(|e| format!("heartbeat callback request failed: {e}"))?;
if !response.status().is_success() {
return Err(format!(
"heartbeat callback failed with status {}",
response.status()
));
}
response
.bytes()
.await
.map(|bytes| bytes.to_vec())
.map_err(|e| format!("heartbeat callback body read failed: {e}"))
}
pub async fn push_node_status(
&self,
node_id: &str,
connected: bool,
conn_count: usize,
) -> Result<(), String> {
let Some(client) = &self.client else {
return Ok(());
};
let url = format!(
"{}/api/internal/hub/node-status",
self.base_url.trim_end_matches('/')
);
let response = client
.post(&url)
.json(&serde_json::json!({
"node_id": node_id,
"connected": connected,
"conn_count": conn_count,
}))
.send()
.await
.map_err(|e| format!("node-status callback request failed: {e}"))?;
if response.status().is_success() {
Ok(())
} else {
Err(format!(
"node-status callback failed with status {}",
response.status()
))
}
}
}
-878
View File
@@ -1,878 +0,0 @@
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
use axum::extract::ws::Message;
use bytes::Bytes;
use dashmap::DashMap;
use parking_lot::{Mutex, RwLock};
use tokio::sync::mpsc;
use tokio::sync::mpsc::error::TrySendError;
use tokio::sync::{watch, Notify};
use tracing::{debug, info, warn};
use crate::control_plane::ControlPlaneClient;
use crate::protocol;
const MAX_REQUEST_BODY_FRAME_SIZE: usize = 32 * 1024;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SendStatus {
Queued,
Closed,
Congested,
}
#[derive(Debug, Clone, Copy)]
pub struct ConnConfig {
pub ping_interval: Duration,
pub idle_timeout: Duration,
pub outbound_queue_capacity: usize,
}
pub struct BoundedOutbound {
tx: mpsc::Sender<Message>,
close_tx: watch::Sender<bool>,
closing: AtomicBool,
}
impl BoundedOutbound {
pub fn new(tx: mpsc::Sender<Message>, close_tx: watch::Sender<bool>) -> Self {
Self {
tx,
close_tx,
closing: AtomicBool::new(false),
}
}
pub fn send(&self, msg: Message) -> SendStatus {
if self.is_closing() {
return SendStatus::Closed;
}
match self.tx.try_send(msg) {
Ok(()) => SendStatus::Queued,
Err(TrySendError::Closed(_)) => {
self.mark_closing();
SendStatus::Closed
}
Err(TrySendError::Full(_)) => {
self.mark_closing();
SendStatus::Congested
}
}
}
pub fn is_closing(&self) -> bool {
self.closing.load(Ordering::Acquire)
}
pub fn mark_closing(&self) -> bool {
if self.closing.swap(true, Ordering::AcqRel) {
return false;
}
let _ = self.close_tx.send(true);
true
}
}
pub struct ProxyConn {
pub id: u64,
pub node_id: String,
pub node_name: String,
pub outbound: BoundedOutbound,
next_stream_id: AtomicU32,
pub stream_count: AtomicUsize,
pub max_streams: usize,
}
impl ProxyConn {
pub fn new(
id: u64,
node_id: String,
node_name: String,
tx: mpsc::Sender<Message>,
close_tx: watch::Sender<bool>,
max_streams: usize,
) -> Self {
Self {
id,
node_id,
node_name,
outbound: BoundedOutbound::new(tx, close_tx),
next_stream_id: AtomicU32::new(2),
stream_count: AtomicUsize::new(0),
max_streams,
}
}
pub fn alloc_stream_id(&self) -> Option<u32> {
let mut current = self.stream_count.load(Ordering::Relaxed);
loop {
if current >= self.max_streams || !self.is_available() {
return None;
}
match self.stream_count.compare_exchange_weak(
current,
current + 1,
Ordering::AcqRel,
Ordering::Relaxed,
) {
Ok(_) => break,
Err(observed) => current = observed,
}
}
let sid = loop {
let current_sid = self.next_stream_id.load(Ordering::Relaxed);
let next_sid = if current_sid >= 0xFFFF_FFFE {
2
} else {
current_sid + 2
};
if self
.next_stream_id
.compare_exchange_weak(current_sid, next_sid, Ordering::AcqRel, Ordering::Relaxed)
.is_ok()
{
break current_sid;
}
};
Some(sid)
}
pub fn release_stream(&self) {
let mut current = self.stream_count.load(Ordering::Relaxed);
while current > 0 {
match self.stream_count.compare_exchange_weak(
current,
current - 1,
Ordering::AcqRel,
Ordering::Relaxed,
) {
Ok(_) => return,
Err(observed) => current = observed,
}
}
}
pub fn is_available(&self) -> bool {
!self.outbound.is_closing()
}
pub fn request_close(&self) {
self.outbound.mark_closing();
}
pub fn send(&self, msg: Message) -> SendStatus {
let was_closing = self.outbound.is_closing();
let status = self.outbound.send(msg);
if status == SendStatus::Congested && !was_closing {
warn!(
conn_id = self.id,
node_id = %self.node_id,
node_name = %self.node_name,
queued_streams = self.stream_count.load(Ordering::Relaxed),
"proxy outbound queue full, closing congested connection"
);
}
status
}
}
#[derive(Debug, Clone)]
pub struct LocalResponseHead {
pub status: u16,
pub headers: Vec<(String, String)>,
}
#[derive(Debug)]
pub enum LocalBodyEvent {
Chunk(Bytes),
End,
Error(String),
}
#[derive(Debug, Default)]
struct LocalWaitState {
response: Option<LocalResponseHead>,
error: Option<String>,
}
pub struct LocalStream {
pub id: u64,
proxy_conn_id: u64,
proxy_stream_id: u32,
wait_state: Mutex<LocalWaitState>,
headers_notify: Notify,
body_tx: mpsc::Sender<LocalBodyEvent>,
body_rx: Mutex<Option<mpsc::Receiver<LocalBodyEvent>>>,
terminal: AtomicBool,
}
impl LocalStream {
fn new(id: u64, proxy_conn_id: u64, proxy_stream_id: u32) -> Self {
let (body_tx, body_rx) = mpsc::channel(128);
Self {
id,
proxy_conn_id,
proxy_stream_id,
wait_state: Mutex::new(LocalWaitState::default()),
headers_notify: Notify::new(),
body_tx,
body_rx: Mutex::new(Some(body_rx)),
terminal: AtomicBool::new(false),
}
}
pub async fn wait_headers(&self, timeout: Duration) -> Result<LocalResponseHead, String> {
tokio::time::timeout(timeout, async {
loop {
let outcome = {
let state = self.wait_state.lock();
if let Some(response) = &state.response {
return Ok(response.clone());
}
state.error.clone()
};
if let Some(error) = outcome {
return Err(error);
}
self.headers_notify.notified().await;
}
})
.await
.map_err(|_| "timed out waiting for response headers".to_string())?
}
pub fn take_body_receiver(&self) -> Option<mpsc::Receiver<LocalBodyEvent>> {
self.body_rx.lock().take()
}
fn set_response_headers(&self, meta: protocol::ResponseMeta) {
let mut notify = false;
{
let mut state = self.wait_state.lock();
if state.response.is_none() && state.error.is_none() {
state.response = Some(LocalResponseHead {
status: meta.status,
headers: meta.headers,
});
notify = true;
}
}
if notify {
self.headers_notify.notify_waiters();
}
}
fn push_body_chunk(&self, payload: Bytes) -> bool {
if self.terminal.load(Ordering::Acquire) {
return false;
}
self.body_tx
.try_send(LocalBodyEvent::Chunk(payload))
.is_ok()
}
fn finish(&self) {
if self.terminal.swap(true, Ordering::AcqRel) {
return;
}
let mut notify = false;
{
let mut state = self.wait_state.lock();
if state.response.is_none() && state.error.is_none() {
state.error = Some("stream ended before response headers".to_string());
notify = true;
}
}
if notify {
self.headers_notify.notify_waiters();
}
let _ = self.body_tx.try_send(LocalBodyEvent::End);
}
fn fail(&self, error: impl Into<String>) {
if self.terminal.swap(true, Ordering::AcqRel) {
return;
}
let error = error.into();
let mut notify = false;
{
let mut state = self.wait_state.lock();
if state.response.is_none() && state.error.is_none() {
state.error = Some(error.clone());
notify = true;
}
}
if notify {
self.headers_notify.notify_waiters();
}
let _ = self.body_tx.try_send(LocalBodyEvent::Error(error));
}
}
pub struct HubRouter {
proxy_conns: RwLock<HashMap<String, Vec<Arc<ProxyConn>>>>,
proxy_conns_by_id: DashMap<u64, Arc<ProxyConn>>,
local_streams: DashMap<u64, Arc<LocalStream>>,
proxy_to_local: DashMap<(u64, u32), u64>,
next_conn_id: AtomicU64,
next_local_stream_id: AtomicU64,
control_plane: ControlPlaneClient,
}
impl HubRouter {
pub fn new(control_plane: ControlPlaneClient) -> Arc<Self> {
Arc::new(Self {
proxy_conns: RwLock::new(HashMap::new()),
proxy_conns_by_id: DashMap::new(),
local_streams: DashMap::new(),
proxy_to_local: DashMap::new(),
next_conn_id: AtomicU64::new(1),
next_local_stream_id: AtomicU64::new(1),
control_plane,
})
}
pub fn alloc_conn_id(&self) -> u64 {
self.next_conn_id.fetch_add(1, Ordering::Relaxed)
}
pub fn register_proxy(&self, conn: Arc<ProxyConn>) {
let node_id = conn.node_id.clone();
let node_name = conn.node_name.clone();
let conn_id = conn.id;
self.proxy_conns_by_id.insert(conn_id, conn.clone());
let pool_size = {
let mut map = self.proxy_conns.write();
map.entry(node_id.clone()).or_default().push(conn);
map.get(&node_id).map(|v| v.len()).unwrap_or(0)
};
info!(
node_id = %node_id,
node_name = %node_name,
conn_id = conn_id,
pool_size = pool_size,
"proxy connected"
);
self.notify_node_status(node_id, true, pool_size);
}
pub fn unregister_proxy(&self, conn_id: u64, node_id: &str) {
self.proxy_conns_by_id.remove(&conn_id);
let pool_size = {
let mut map = self.proxy_conns.write();
if let Some(conns) = map.get_mut(node_id) {
conns.retain(|c| c.id != conn_id);
if conns.is_empty() {
map.remove(node_id);
}
}
map.get(node_id).map(|v| v.len()).unwrap_or(0)
};
info!(
node_id = %node_id,
conn_id = conn_id,
remaining = pool_size,
"proxy disconnected"
);
self.cancel_streams_for_proxy(conn_id);
self.notify_node_status(node_id.to_string(), pool_size > 0, pool_size);
}
fn notify_node_status(&self, node_id: String, connected: bool, conn_count: usize) {
let control_plane = self.control_plane.clone();
tokio::spawn(async move {
if let Err(error) = control_plane
.push_node_status(&node_id, connected, conn_count)
.await
{
warn!(
node_id = %node_id,
connected = connected,
conn_count = conn_count,
error = %error,
"failed to push node status to app control plane"
);
}
});
}
fn get_proxy_conn(&self, node_id: &str) -> Option<Arc<ProxyConn>> {
let map = self.proxy_conns.read();
let conns = map.get(node_id)?;
conns
.iter()
.filter(|c| c.is_available())
.min_by_key(|c| c.stream_count.load(Ordering::Relaxed))
.cloned()
}
pub fn open_local_stream(
&self,
node_id: &str,
meta: &protocol::RequestMeta,
) -> Result<Arc<LocalStream>, String> {
let proxy_conn = self
.get_proxy_conn(node_id)
.ok_or_else(|| format!("no proxy connection for node {node_id}"))?;
let proxy_stream_id = proxy_conn
.alloc_stream_id()
.ok_or_else(|| format!("stream limit reached for node {node_id}"))?;
// Encode frames before registering the stream so that encoding failures
// (practically impossible but theoretically possible) don't leak a stream
// slot or orphan map entries.
let meta_json = match serde_json::to_vec(meta) {
Ok(json) => json,
Err(e) => {
proxy_conn.release_stream();
return Err(format!("failed to encode request metadata: {e}"));
}
};
let (meta_payload, meta_flags) = match protocol::compress_payload(&meta_json) {
Ok(result) => result,
Err(e) => {
proxy_conn.release_stream();
return Err(format!("failed to compress request metadata: {e}"));
}
};
let header_frame = protocol::encode_frame(
proxy_stream_id,
protocol::REQUEST_HEADERS,
meta_flags,
&meta_payload,
);
// Frames encoded successfully -- now register the stream.
let local_stream_id = self.next_local_stream_id.fetch_add(1, Ordering::Relaxed);
let local_stream = Arc::new(LocalStream::new(
local_stream_id,
proxy_conn.id,
proxy_stream_id,
));
self.local_streams
.insert(local_stream_id, local_stream.clone());
self.proxy_to_local
.insert((proxy_conn.id, proxy_stream_id), local_stream_id);
match proxy_conn.send(Message::Binary(header_frame.into())) {
SendStatus::Queued => Ok(local_stream),
SendStatus::Closed | SendStatus::Congested => {
self.cleanup_local_stream(local_stream_id);
proxy_conn.release_stream();
Err("proxy connection congested".to_string())
}
}
}
pub fn push_local_request_body(
&self,
local_stream_id: u64,
payload: Bytes,
end_stream: bool,
) -> Result<(), String> {
let stream = self
.local_streams
.get(&local_stream_id)
.map(|entry| entry.value().clone())
.ok_or_else(|| "local stream not found".to_string())?;
let proxy_conn = self
.proxy_conns_by_id
.get(&stream.proxy_conn_id)
.map(|entry| entry.value().clone())
.ok_or_else(|| "proxy connection unavailable".to_string())?;
let total_chunks = payload.len().div_ceil(MAX_REQUEST_BODY_FRAME_SIZE);
if total_chunks == 0 {
if end_stream {
self.send_request_body_frame(&proxy_conn, stream.proxy_stream_id, &[], true)?;
}
} else {
for (index, chunk) in payload.chunks(MAX_REQUEST_BODY_FRAME_SIZE).enumerate() {
let is_last_chunk = index + 1 == total_chunks;
self.send_request_body_frame(
&proxy_conn,
stream.proxy_stream_id,
chunk,
end_stream && is_last_chunk,
)?;
}
}
Ok(())
}
fn send_request_body_frame(
&self,
proxy_conn: &Arc<ProxyConn>,
proxy_stream_id: u32,
payload: &[u8],
end_stream: bool,
) -> Result<(), String> {
let (body_payload, body_flags) = protocol::compress_payload(payload)
.map_err(|e| format!("failed to compress request body: {e}"))?;
let body_frame = protocol::encode_frame(
proxy_stream_id,
protocol::REQUEST_BODY,
body_flags
| if end_stream {
protocol::FLAG_END_STREAM
} else {
0
},
&body_payload,
);
match proxy_conn.send(Message::Binary(body_frame.into())) {
SendStatus::Queued => Ok(()),
SendStatus::Closed | SendStatus::Congested => {
Err("proxy connection congested".to_string())
}
}
}
pub fn cancel_local_stream(&self, local_stream_id: u64, reason: &str) {
let Some((_, stream)) = self.local_streams.remove(&local_stream_id) else {
return;
};
self.proxy_to_local
.remove(&(stream.proxy_conn_id, stream.proxy_stream_id));
if let Some(pc) = self.proxy_conns_by_id.get(&stream.proxy_conn_id) {
pc.release_stream();
let frame = protocol::encode_stream_error(stream.proxy_stream_id, reason);
let _ = pc.send(Message::Binary(frame.into()));
}
stream.fail(reason.to_string());
}
fn cleanup_local_stream(&self, local_stream_id: u64) {
let Some((_, stream)) = self.local_streams.remove(&local_stream_id) else {
return;
};
self.proxy_to_local
.remove(&(stream.proxy_conn_id, stream.proxy_stream_id));
}
pub async fn handle_proxy_frame(&self, proxy_conn_id: u64, data: &mut [u8]) {
let header = match protocol::FrameHeader::parse(data) {
Some(h) => h,
None => return,
};
let expected_len = protocol::HEADER_SIZE + header.payload_len as usize;
if data.len() < expected_len {
return;
}
match header.msg_type {
protocol::RESPONSE_HEADERS => {
self.route_response_headers(proxy_conn_id, header, data);
}
protocol::RESPONSE_BODY => {
self.route_response_body(proxy_conn_id, header, data);
}
protocol::STREAM_END => {
self.finish_proxy_stream(proxy_conn_id, header.stream_id);
}
protocol::STREAM_ERROR => {
let message = protocol::decode_payload(data, &header)
.ok()
.and_then(|payload| String::from_utf8(payload).ok())
.unwrap_or_else(|| "stream error".to_string());
self.fail_proxy_stream(proxy_conn_id, header.stream_id, message);
}
protocol::HEARTBEAT_DATA => {
self.handle_heartbeat(proxy_conn_id, header.stream_id, data, &header)
.await;
}
protocol::PING => {
let payload = protocol::frame_payload_by_header(data, &header).unwrap_or(&[]);
let pong = protocol::encode_pong(payload);
if let Some(pc) = self.proxy_conns_by_id.get(&proxy_conn_id) {
let _ = pc.send(Message::Binary(pong.into()));
}
}
protocol::PONG => {}
protocol::GOAWAY => {
warn!(
proxy_conn_id = proxy_conn_id,
"received GOAWAY from proxy connection"
);
}
_ => {
debug!(
msg_type = header.msg_type,
proxy_conn_id = proxy_conn_id,
"unexpected frame type from proxy"
);
}
}
}
fn route_response_headers(
&self,
proxy_conn_id: u64,
header: protocol::FrameHeader,
data: &[u8],
) {
let Some(local_id) = self.lookup_local_stream(proxy_conn_id, header.stream_id) else {
return;
};
let Ok(payload) = protocol::decode_payload(data, &header) else {
self.fail_proxy_stream(
proxy_conn_id,
header.stream_id,
"failed to decode response headers",
);
return;
};
let Ok(meta) = serde_json::from_slice::<protocol::ResponseMeta>(&payload) else {
self.fail_proxy_stream(
proxy_conn_id,
header.stream_id,
"invalid response headers payload",
);
return;
};
if let Some(entry) = self.local_streams.get(&local_id) {
entry.value().set_response_headers(meta);
}
}
fn route_response_body(&self, proxy_conn_id: u64, header: protocol::FrameHeader, data: &[u8]) {
let Some(local_id) = self.lookup_local_stream(proxy_conn_id, header.stream_id) else {
return;
};
let Ok(payload) = protocol::decode_payload(data, &header) else {
self.fail_proxy_stream(
proxy_conn_id,
header.stream_id,
"failed to decode response body",
);
return;
};
let stream = match self.local_streams.get(&local_id) {
Some(entry) => entry.value().clone(),
None => return,
};
if !stream.push_body_chunk(Bytes::from(payload)) {
self.cancel_local_stream(local_id, "local relay response congested");
}
}
fn handle_stream_cleanup(
&self,
proxy_conn_id: u64,
proxy_stream_id: u32,
) -> Option<Arc<LocalStream>> {
let local_id = self
.proxy_to_local
.remove(&(proxy_conn_id, proxy_stream_id))
.map(|(_, local_id)| local_id)?;
let stream = self
.local_streams
.remove(&local_id)
.map(|(_, stream)| stream)?;
if let Some(pc) = self.proxy_conns_by_id.get(&proxy_conn_id) {
pc.release_stream();
}
Some(stream)
}
fn finish_proxy_stream(&self, proxy_conn_id: u64, proxy_stream_id: u32) {
if let Some(stream) = self.handle_stream_cleanup(proxy_conn_id, proxy_stream_id) {
stream.finish();
}
}
fn fail_proxy_stream(
&self,
proxy_conn_id: u64,
proxy_stream_id: u32,
error: impl Into<String>,
) {
if let Some(stream) = self.handle_stream_cleanup(proxy_conn_id, proxy_stream_id) {
stream.fail(error.into());
}
}
fn lookup_local_stream(&self, proxy_conn_id: u64, proxy_stream_id: u32) -> Option<u64> {
self.proxy_to_local
.get(&(proxy_conn_id, proxy_stream_id))
.map(|entry| *entry.value())
}
async fn handle_heartbeat(
&self,
proxy_conn_id: u64,
stream_id: u32,
data: &[u8],
header: &protocol::FrameHeader,
) {
let payload = match protocol::decode_payload(data, header) {
Ok(payload) => payload,
Err(error) => {
warn!(proxy_conn_id = proxy_conn_id, error = %error, "failed to decode heartbeat payload");
return;
}
};
let ack_payload = match self.control_plane.heartbeat_ack(&payload).await {
Ok(payload) => payload,
Err(error) => {
warn!(proxy_conn_id = proxy_conn_id, error = %error, "control-plane heartbeat callback failed");
b"{}".to_vec()
}
};
if let Some(pc) = self.proxy_conns_by_id.get(&proxy_conn_id) {
let frame = protocol::encode_frame(stream_id, protocol::HEARTBEAT_ACK, 0, &ack_payload);
let _ = pc.send(Message::Binary(frame.into()));
}
}
fn cancel_streams_for_proxy(&self, proxy_conn_id: u64) {
let mut cancelled = 0usize;
self.proxy_to_local.retain(|key, local_id| {
if key.0 != proxy_conn_id {
return true;
}
if let Some((_, stream)) = self.local_streams.remove(local_id) {
stream.fail("proxy disconnected".to_string());
}
cancelled += 1;
false
});
if cancelled > 0 {
warn!(
proxy_conn_id = proxy_conn_id,
streams_cancelled = cancelled,
"cancelled in-flight streams due to proxy disconnect"
);
}
}
pub fn stats(&self) -> HubStats {
let proxy_conns = self.proxy_conns.read();
let total_proxy = proxy_conns.values().map(|v| v.len()).sum();
let nodes = proxy_conns.len();
drop(proxy_conns);
HubStats {
proxy_connections: total_proxy,
nodes,
active_streams: self.local_streams.len(),
}
}
}
#[derive(serde::Serialize)]
pub struct HubStats {
pub proxy_connections: usize,
pub nodes: usize,
pub active_streams: usize,
}
#[cfg(test)]
mod tests {
use super::*;
fn build_meta() -> protocol::RequestMeta {
protocol::RequestMeta {
method: "GET".to_string(),
url: "https://example.com".to_string(),
headers: HashMap::new(),
timeout: 30,
}
}
#[tokio::test]
async fn cancel_local_stream_notifies_proxy() {
let hub = HubRouter::new(ControlPlaneClient::disabled());
let (proxy_tx, mut proxy_rx) = mpsc::channel(8);
let (proxy_close_tx, _) = watch::channel(false);
let proxy = Arc::new(ProxyConn::new(
100,
"node-1".to_string(),
"Node 1".to_string(),
proxy_tx,
proxy_close_tx,
16,
));
hub.register_proxy(proxy);
let stream = hub
.open_local_stream("node-1", &build_meta())
.expect("open local stream");
let _ = proxy_rx.try_recv().expect("headers frame");
hub.push_local_request_body(stream.id, Bytes::new(), true)
.expect("finish empty body");
let _ = proxy_rx.try_recv().expect("body frame");
hub.cancel_local_stream(stream.id, "client dropped");
let cancelled = proxy_rx.try_recv().expect("cancel frame");
let cancelled_data = match cancelled {
Message::Binary(data) => data.to_vec(),
other => panic!("unexpected message: {other:?}"),
};
let header = protocol::FrameHeader::parse(&cancelled_data).expect("cancel frame header");
assert_eq!(header.msg_type, protocol::STREAM_ERROR);
}
#[tokio::test]
async fn push_local_request_body_splits_large_payload_and_marks_end() {
let hub = HubRouter::new(ControlPlaneClient::disabled());
let (proxy_tx, mut proxy_rx) = mpsc::channel(8);
let (proxy_close_tx, _) = watch::channel(false);
let proxy = Arc::new(ProxyConn::new(
200,
"node-2".to_string(),
"Node 2".to_string(),
proxy_tx,
proxy_close_tx,
16,
));
hub.register_proxy(proxy);
let stream = hub
.open_local_stream("node-2", &build_meta())
.expect("open local stream");
let _ = proxy_rx.try_recv().expect("headers frame");
let payload = Bytes::from(vec![b'x'; MAX_REQUEST_BODY_FRAME_SIZE + 17]);
hub.push_local_request_body(stream.id, payload, true)
.expect("push request body");
let first = match proxy_rx.try_recv().expect("first body frame") {
Message::Binary(data) => data.to_vec(),
other => panic!("unexpected message: {other:?}"),
};
let first_header = protocol::FrameHeader::parse(&first).expect("first body header");
assert_eq!(first_header.msg_type, protocol::REQUEST_BODY);
assert_eq!(first_header.flags & protocol::FLAG_END_STREAM, 0);
let second = match proxy_rx.try_recv().expect("second body frame") {
Message::Binary(data) => data.to_vec(),
other => panic!("unexpected message: {other:?}"),
};
let second_header = protocol::FrameHeader::parse(&second).expect("second body header");
assert_eq!(second_header.msg_type, protocol::REQUEST_BODY);
assert_ne!(second_header.flags & protocol::FLAG_END_STREAM, 0);
}
}
-262
View File
@@ -1,262 +0,0 @@
use std::io;
use std::net::SocketAddr;
use std::time::Duration;
use async_stream::stream;
use axum::body::{Body, Bytes};
use axum::extract::{ConnectInfo, Path, Request, State};
use axum::http::{HeaderMap, HeaderName, HeaderValue, Response, StatusCode};
use axum::response::IntoResponse;
use bytes::BytesMut;
use futures_util::StreamExt;
use tracing::warn;
use crate::hub::{LocalBodyEvent, LocalStream};
use crate::protocol;
use crate::AppState;
pub const TUNNEL_ERROR_HEADER: &str = "x-aether-tunnel-error";
const MAX_RELAY_META_LEN: usize = 256 * 1024;
struct StreamGuard {
hub: std::sync::Arc<crate::hub::HubRouter>,
stream_id: u64,
finished: bool,
}
impl Drop for StreamGuard {
fn drop(&mut self) {
if !self.finished {
self.hub
.cancel_local_stream(self.stream_id, "local relay client dropped");
}
}
}
pub async fn relay_request(
Path(node_id): Path<String>,
State(state): State<AppState>,
ConnectInfo(addr): ConnectInfo<SocketAddr>,
request: Request,
) -> impl IntoResponse {
if !addr.ip().is_loopback() {
return tunnel_error_response(
StatusCode::FORBIDDEN,
"forbidden",
"local relay only accepts loopback requests",
);
}
let mut body_stream = request.into_body().into_data_stream();
let mut envelope_buf = BytesMut::new();
let mut meta: Option<protocol::RequestMeta> = None;
let mut stream: Option<std::sync::Arc<LocalStream>> = None;
while let Some(chunk_result) = body_stream.next().await {
let chunk = match chunk_result {
Ok(chunk) => chunk,
Err(error) => {
if let Some(active_stream) = &stream {
state
.hub
.cancel_local_stream(active_stream.id, "failed to read relay request body");
}
warn!(error = %error, "failed to read local relay request body");
return tunnel_error_response(
StatusCode::BAD_GATEWAY,
"relay",
"failed to read relay request body",
);
}
};
if stream.is_none() {
envelope_buf.extend_from_slice(&chunk);
let Some((parsed_meta, body_offset)) = (match try_decode_envelope_meta(&envelope_buf) {
Ok(result) => result,
Err(error) => {
return tunnel_error_response(StatusCode::BAD_REQUEST, "bad_request", &error);
}
}) else {
continue;
};
let opened_stream = match state.hub.open_local_stream(&node_id, &parsed_meta) {
Ok(stream) => stream,
Err(error) => {
return tunnel_error_response(
StatusCode::SERVICE_UNAVAILABLE,
"connect",
&error,
);
}
};
if envelope_buf.len() > body_offset {
let first_body_chunk = Bytes::copy_from_slice(&envelope_buf[body_offset..]);
if let Err(error) =
state
.hub
.push_local_request_body(opened_stream.id, first_body_chunk, false)
{
state.hub.cancel_local_stream(opened_stream.id, &error);
return tunnel_error_response(
StatusCode::SERVICE_UNAVAILABLE,
"connect",
&error,
);
}
}
envelope_buf.clear();
meta = Some(parsed_meta);
stream = Some(opened_stream);
continue;
}
let Some(active_stream) = &stream else {
continue;
};
if let Err(error) = state
.hub
.push_local_request_body(active_stream.id, chunk, false)
{
state.hub.cancel_local_stream(active_stream.id, &error);
return tunnel_error_response(StatusCode::SERVICE_UNAVAILABLE, "connect", &error);
}
}
let (meta, stream) = match (meta, stream) {
(Some(meta), Some(stream)) => (meta, stream),
_ => {
return tunnel_error_response(
StatusCode::BAD_REQUEST,
"bad_request",
"relay envelope metadata truncated",
);
}
};
if let Err(error) = state
.hub
.push_local_request_body(stream.id, Bytes::new(), true)
{
state.hub.cancel_local_stream(stream.id, &error);
return tunnel_error_response(StatusCode::SERVICE_UNAVAILABLE, "connect", &error);
}
let request_guard = StreamGuard {
hub: state.hub.clone(),
stream_id: stream.id,
finished: false,
};
let wait_timeout = Duration::from_secs(meta.timeout.clamp(5, 300));
let response_head = match stream.wait_headers(wait_timeout).await {
Ok(response) => response,
Err(error) => {
state.hub.cancel_local_stream(stream.id, &error);
return tunnel_error_response(StatusCode::GATEWAY_TIMEOUT, "timeout", &error);
}
};
let Some(mut body_rx) = stream.take_body_receiver() else {
state
.hub
.cancel_local_stream(stream.id, "missing relay response body receiver");
return tunnel_error_response(
StatusCode::BAD_GATEWAY,
"relay",
"missing relay response body receiver",
);
};
let hub = state.hub.clone();
let stream_id = stream.id;
let body_stream = stream! {
let mut guard = request_guard;
guard.hub = hub;
guard.stream_id = stream_id;
while let Some(event) = body_rx.recv().await {
match event {
LocalBodyEvent::Chunk(chunk) => yield Ok::<Bytes, io::Error>(chunk),
LocalBodyEvent::End => {
guard.finished = true;
break;
}
LocalBodyEvent::Error(error) => {
guard.finished = true;
yield Err(io::Error::other(error));
break;
}
}
}
guard.finished = true;
};
let mut builder = Response::builder().status(response_head.status);
if let Some(headers) = builder.headers_mut() {
append_headers(headers, &response_head.headers);
}
match builder.body(Body::from_stream(body_stream)) {
Ok(response) => response,
Err(error) => {
warn!(error = %error, "failed to build relay response");
tunnel_error_response(
StatusCode::BAD_GATEWAY,
"relay",
"failed to build relay response",
)
}
}
}
fn try_decode_envelope_meta(
buffer: &BytesMut,
) -> Result<Option<(protocol::RequestMeta, usize)>, String> {
if buffer.len() < 4 {
return Ok(None);
}
let meta_len = u32::from_be_bytes([buffer[0], buffer[1], buffer[2], buffer[3]]) as usize;
if meta_len > MAX_RELAY_META_LEN {
return Err("relay metadata too large".to_string());
}
let meta_end = 4usize
.checked_add(meta_len)
.ok_or_else(|| "relay envelope length overflow".to_string())?;
if buffer.len() < meta_end {
return Ok(None);
}
let meta = serde_json::from_slice::<protocol::RequestMeta>(&buffer[4..meta_end])
.map_err(|e| format!("invalid relay metadata: {e}"))?;
Ok(Some((meta, meta_end)))
}
fn append_headers(target: &mut HeaderMap, headers: &[(String, String)]) {
for (name, value) in headers {
let Ok(name) = HeaderName::from_bytes(name.as_bytes()) else {
continue;
};
let Ok(value) = HeaderValue::from_str(value) else {
continue;
};
target.append(name, value);
}
}
fn tunnel_error_response(status: StatusCode, kind: &str, message: &str) -> Response<Body> {
let mut builder = Response::builder().status(status);
if let Some(headers) = builder.headers_mut() {
headers.insert(
HeaderName::from_static(TUNNEL_ERROR_HEADER),
HeaderValue::from_str(kind).unwrap_or_else(|_| HeaderValue::from_static("relay")),
);
headers.insert(
axum::http::header::CONTENT_TYPE,
HeaderValue::from_static("text/plain; charset=utf-8"),
);
}
builder
.body(Body::from(message.to_string()))
.unwrap_or_else(|_| Response::new(Body::from("relay error")))
}
-167
View File
@@ -1,167 +0,0 @@
mod control_plane;
mod hub;
mod local_relay;
mod protocol;
mod proxy_conn;
use std::net::SocketAddr;
use std::time::Duration;
use axum::extract::ws::WebSocketUpgrade;
use axum::extract::State;
use axum::response::{IntoResponse, Json};
use axum::routing::{get, post};
use axum::Router;
use clap::Parser;
use tracing::{info, warn};
use crate::control_plane::ControlPlaneClient;
use crate::hub::{ConnConfig, HubRouter};
use crate::local_relay::relay_request;
#[derive(Parser, Debug)]
#[command(name = "aether-hub", about = "Tunnel Hub for Aether")]
struct Args {
/// Bind address
#[arg(long, default_value = "0.0.0.0:8085", env = "TUNNEL_HUB_BIND")]
bind: String,
/// Proxy-side idle timeout in seconds (0 to disable)
#[arg(long, default_value_t = 0, env = "TUNNEL_HUB_PROXY_IDLE_TIMEOUT")]
proxy_idle_timeout: u64,
/// Ping interval in seconds (for both sides)
#[arg(long, default_value_t = 15, env = "TUNNEL_HUB_PING_INTERVAL")]
ping_interval: u64,
/// Max concurrent streams per proxy connection
#[arg(long, default_value_t = 2048, env = "TUNNEL_HUB_MAX_STREAMS")]
max_streams: usize,
/// Per-connection outbound queue capacity before treating the socket as congested
#[arg(
long,
default_value_t = 128,
env = "TUNNEL_HUB_OUTBOUND_QUEUE_CAPACITY"
)]
outbound_queue_capacity: usize,
/// Local Aether app base URL for control-plane callbacks
#[arg(
long,
default_value = "http://127.0.0.1:8084",
env = "TUNNEL_HUB_APP_BASE_URL"
)]
app_base_url: String,
}
#[derive(Clone)]
pub struct AppState {
pub hub: std::sync::Arc<HubRouter>,
pub proxy_conn_cfg: ConnConfig,
pub max_streams: usize,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Initialize tracing
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "aether_hub=info".into()),
)
.init();
let args = Args::parse();
let hub = HubRouter::new(ControlPlaneClient::new(args.app_base_url));
let outbound_queue_capacity = args.outbound_queue_capacity.clamp(8, 4096);
let ping_interval = Duration::from_secs(args.ping_interval);
let state = AppState {
hub,
proxy_conn_cfg: ConnConfig {
ping_interval,
idle_timeout: Duration::from_secs(args.proxy_idle_timeout),
outbound_queue_capacity,
},
max_streams: args.max_streams,
};
let app = Router::new()
.route("/health", get(health))
.route("/stats", get(stats))
.route("/proxy", get(ws_proxy))
.route("/local/relay/{node_id}", post(relay_request))
.with_state(state);
let listener = tokio::net::TcpListener::bind(&args.bind).await?;
info!(bind = %args.bind, "aether-hub started");
axum::serve(
listener,
app.into_make_service_with_connect_info::<SocketAddr>(),
)
.await?;
Ok(())
}
// ---------------------------------------------------------------------------
// HTTP endpoints
// ---------------------------------------------------------------------------
async fn health() -> impl IntoResponse {
Json(serde_json::json!({"status": "ok"}))
}
async fn stats(State(state): State<AppState>) -> impl IntoResponse {
Json(state.hub.stats())
}
// ---------------------------------------------------------------------------
// WebSocket endpoints
// ---------------------------------------------------------------------------
async fn ws_proxy(
ws: WebSocketUpgrade,
State(state): State<AppState>,
headers: axum::http::HeaderMap,
) -> impl IntoResponse {
let node_id = headers
.get("x-node-id")
.and_then(|v| v.to_str().ok())
.unwrap_or("")
.trim()
.to_string();
let node_name = headers
.get("x-node-name")
.and_then(|v| v.to_str().ok())
.unwrap_or(&node_id)
.trim()
.to_string();
let max_streams: usize = headers
.get("x-tunnel-max-streams")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse().ok())
.unwrap_or(state.max_streams)
.clamp(64, 2048);
if node_id.is_empty() {
warn!("proxy connection rejected: missing X-Node-ID header");
return axum::http::StatusCode::BAD_REQUEST.into_response();
}
ws.max_frame_size(64 * 1024 * 1024)
.on_upgrade(move |socket| {
proxy_conn::handle_proxy_connection(
socket,
state.hub,
node_id,
node_name,
max_streams,
state.proxy_conn_cfg,
)
})
.into_response()
}
-176
View File
@@ -1,176 +0,0 @@
/// Tunnel binary frame protocol
///
/// Frame format (10-byte header + payload):
/// | stream_id (4B) | msg_type (1B) | flags (1B) | payload_len (4B) | payload (NB) |
use std::io::Read;
use flate2::read::GzDecoder;
use flate2::write::GzEncoder;
use flate2::Compression;
pub const HEADER_SIZE: usize = 10;
// Message types
pub const REQUEST_HEADERS: u8 = 0x01;
pub const REQUEST_BODY: u8 = 0x02;
pub const RESPONSE_HEADERS: u8 = 0x03;
pub const RESPONSE_BODY: u8 = 0x04;
pub const STREAM_END: u8 = 0x05;
pub const STREAM_ERROR: u8 = 0x06;
pub const PING: u8 = 0x10;
pub const PONG: u8 = 0x11;
pub const GOAWAY: u8 = 0x12;
pub const HEARTBEAT_DATA: u8 = 0x13;
pub const HEARTBEAT_ACK: u8 = 0x14;
// Flags
pub const FLAG_END_STREAM: u8 = 0x01;
pub const FLAG_GZIP_COMPRESSED: u8 = 0x02;
#[derive(Debug, Clone, Copy)]
pub struct FrameHeader {
pub stream_id: u32,
pub msg_type: u8,
pub flags: u8,
pub payload_len: u32,
}
impl FrameHeader {
/// Parse frame header from raw bytes (must be >= HEADER_SIZE)
#[inline]
pub fn parse(data: &[u8]) -> Option<Self> {
if data.len() < HEADER_SIZE {
return None;
}
Some(Self {
stream_id: u32::from_be_bytes([data[0], data[1], data[2], data[3]]),
msg_type: data[4],
flags: data[5],
payload_len: u32::from_be_bytes([data[6], data[7], data[8], data[9]]),
})
}
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct RequestMeta {
pub method: String,
pub url: String,
pub headers: std::collections::HashMap<String, String>,
#[serde(default = "default_timeout", deserialize_with = "deserialize_timeout")]
pub timeout: u64,
}
fn default_timeout() -> u64 {
60
}
fn deserialize_timeout<'de, D>(deserializer: D) -> Result<u64, D::Error>
where
D: serde::Deserializer<'de>,
{
#[derive(serde::Deserialize)]
#[serde(untagged)]
enum TimeoutValue {
Int(u64),
Float(f64),
}
match <TimeoutValue as serde::Deserialize>::deserialize(deserializer)? {
TimeoutValue::Int(v) => Ok(v),
TimeoutValue::Float(v) => {
if !v.is_finite() || v < 0.0 {
return Err(serde::de::Error::custom(
"timeout must be a non-negative finite number",
));
}
if v.fract() != 0.0 {
return Err(serde::de::Error::custom("timeout must be integer seconds"));
}
if v > (u64::MAX as f64) {
return Err(serde::de::Error::custom("timeout is too large"));
}
Ok(v as u64)
}
}
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct ResponseMeta {
pub status: u16,
pub headers: Vec<(String, String)>,
}
pub fn encode_frame(stream_id: u32, msg_type: u8, flags: u8, payload: &[u8]) -> Vec<u8> {
let mut buf = Vec::with_capacity(HEADER_SIZE + payload.len());
buf.extend_from_slice(&stream_id.to_be_bytes());
buf.push(msg_type);
buf.push(flags);
buf.extend_from_slice(&(payload.len() as u32).to_be_bytes());
buf.extend_from_slice(payload);
buf
}
/// Encode a STREAM_ERROR frame for a given stream_id with an error message
pub fn encode_stream_error(stream_id: u32, msg: &str) -> Vec<u8> {
encode_frame(stream_id, STREAM_ERROR, 0, msg.as_bytes())
}
/// Encode a PING frame (stream_id=0)
pub fn encode_ping() -> Vec<u8> {
encode_frame(0, PING, 0, &[])
}
/// Encode a PONG frame (stream_id=0, echo payload)
pub fn encode_pong(payload: &[u8]) -> Vec<u8> {
encode_frame(0, PONG, 0, payload)
}
/// Encode a GOAWAY frame (stream_id=0)
pub fn encode_goaway() -> Vec<u8> {
encode_frame(0, GOAWAY, 0, &[])
}
#[inline]
pub fn frame_payload_by_header<'a>(data: &'a [u8], header: &FrameHeader) -> Option<&'a [u8]> {
let payload_len = header.payload_len as usize;
let end = HEADER_SIZE.checked_add(payload_len)?;
if data.len() < end {
return None;
}
Some(&data[HEADER_SIZE..end])
}
pub fn decode_payload(data: &[u8], header: &FrameHeader) -> Result<Vec<u8>, String> {
let payload = frame_payload_by_header(data, header)
.ok_or_else(|| "incomplete frame payload".to_string())?;
if header.flags & FLAG_GZIP_COMPRESSED != 0 {
let mut decoder = GzDecoder::new(payload);
let mut decoded = Vec::new();
decoder
.read_to_end(&mut decoded)
.map_err(|e| format!("failed to decompress payload: {e}"))?;
Ok(decoded)
} else {
Ok(payload.to_vec())
}
}
pub fn compress_payload(payload: &[u8]) -> Result<(Vec<u8>, u8), std::io::Error> {
maybe_recompress_payload(payload, true)
}
fn maybe_recompress_payload(
payload: &[u8],
prefer_gzip: bool,
) -> Result<(Vec<u8>, u8), std::io::Error> {
if !prefer_gzip {
return Ok((payload.to_vec(), 0));
}
let mut encoder = GzEncoder::new(Vec::new(), Compression::default());
std::io::Write::write_all(&mut encoder, payload)?;
let compressed = encoder.finish()?;
if compressed.len() < payload.len() {
Ok((compressed, FLAG_GZIP_COMPRESSED))
} else {
Ok((payload.to_vec(), 0))
}
}
-156
View File
@@ -1,156 +0,0 @@
/// Proxy-side WebSocket connection handler
///
/// Handles the lifecycle of a single aether-proxy connection:
/// accept -> authenticate (headers) -> read loop -> cleanup
use std::sync::Arc;
use std::time::Duration;
use axum::extract::ws::{Message, WebSocket};
use futures_util::{SinkExt, StreamExt};
use tokio::sync::{mpsc, watch};
use tracing::{debug, info, warn};
use crate::hub::{ConnConfig, HubRouter, ProxyConn, SendStatus};
use crate::protocol;
/// Maximum single frame size: 64 MB
const MAX_FRAME_SIZE: usize = 64 * 1024 * 1024;
pub async fn handle_proxy_connection(
ws: WebSocket,
hub: Arc<HubRouter>,
node_id: String,
node_name: String,
max_streams: usize,
cfg: ConnConfig,
) {
let conn_id = hub.alloc_conn_id();
let (mut ws_tx, ws_rx) = ws.split();
let (tx, mut rx) = mpsc::channel::<Message>(cfg.outbound_queue_capacity);
let (close_tx, mut close_rx) = watch::channel(false);
let conn = Arc::new(ProxyConn::new(
conn_id,
node_id.clone(),
node_name.clone(),
tx,
close_tx,
max_streams,
));
hub.register_proxy(conn.clone());
let writer = tokio::spawn(async move {
loop {
tokio::select! {
msg = rx.recv() => match msg {
Some(msg) => {
if ws_tx.send(msg).await.is_err() {
break;
}
}
None => break,
},
changed = close_rx.changed() => {
if changed.is_err() || *close_rx.borrow() {
break;
}
}
}
}
let _ = ws_tx.close().await;
});
let ping_conn = conn.clone();
let ping_interval = cfg.ping_interval;
let ping_task = tokio::spawn(async move {
loop {
tokio::time::sleep(ping_interval).await;
let ping = protocol::encode_ping();
if !matches!(
ping_conn.send(Message::Binary(ping.into())),
SendStatus::Queued
) {
break;
}
}
});
let reader_hub = hub.clone();
let reader_conn = conn.clone();
let reader = tokio::spawn(async move {
run_proxy_reader(ws_rx, reader_hub, reader_conn, cfg.idle_timeout).await;
});
let _ = reader.await;
ping_task.abort();
conn.request_close();
hub.unregister_proxy(conn_id, &node_id);
drop(conn);
tokio::time::sleep(Duration::from_millis(100)).await;
writer.abort();
let _ = writer.await;
}
async fn run_proxy_reader(
mut ws_rx: futures_util::stream::SplitStream<WebSocket>,
hub: Arc<HubRouter>,
conn: Arc<ProxyConn>,
idle_timeout: Duration,
) {
let idle_enabled = !idle_timeout.is_zero();
let mut oversized_count = 0u32;
loop {
let msg = if idle_enabled {
tokio::select! {
msg = ws_rx.next() => msg,
_ = tokio::time::sleep(idle_timeout) => {
warn!(conn_id = conn.id, node_id = %conn.node_id, "proxy idle timeout");
let _ = conn.send(Message::Binary(protocol::encode_goaway().into()));
conn.request_close();
break;
}
}
} else {
ws_rx.next().await
};
match msg {
Some(Ok(Message::Binary(data))) => {
let mut data = data.to_vec();
if data.len() > MAX_FRAME_SIZE {
oversized_count += 1;
warn!(
conn_id = conn.id,
size = data.len(),
"oversized frame from proxy"
);
if oversized_count >= 5 {
warn!(conn_id = conn.id, "too many oversized frames, closing");
conn.request_close();
break;
}
continue;
}
oversized_count = 0;
if data.len() < protocol::HEADER_SIZE {
debug!(conn_id = conn.id, "frame too small, skipping");
continue;
}
hub.handle_proxy_frame(conn.id, &mut data).await;
}
Some(Ok(Message::Close(_))) | None => {
info!(conn_id = conn.id, node_id = %conn.node_id, "proxy WebSocket closed");
break;
}
Some(Err(e)) => {
warn!(conn_id = conn.id, error = %e, "proxy WebSocket error");
break;
}
_ => {}
}
}
}
-8
View File
@@ -1,8 +0,0 @@
# Aether server URL
AETHER_PROXY_AETHER_URL=https://aether.example.com
# Management Token (ae_xxx, must belong to an ADMIN user)
AETHER_PROXY_MANAGEMENT_TOKEN=ae_xxxxx
# Node identification
AETHER_PROXY_NODE_NAME=proxy-01
-3403
View File
File diff suppressed because it is too large Load Diff
-44
View File
@@ -1,44 +0,0 @@
[package]
name = "aether-proxy"
version = "0.2.5"
edition = "2021"
description = "Tunnel proxy for Aether"
[dependencies]
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream", "http2"] }
hyper = { version = "1", features = ["client", "http1", "http2"] }
hyper-util = { version = "0.1", features = ["client", "client-legacy", "http1", "http2", "tokio"] }
http-body-util = "0.1"
tokio-tungstenite = { version = "0.24", features = ["rustls-tls-webpki-roots"] }
tokio-rustls = "0.26"
futures-util = "0.3"
base64 = "0.22"
clap = { version = "4", features = ["derive", "env"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
thiserror = "2"
bytes = "1"
sha2 = "0.10"
hex = "0.4"
anyhow = "1"
arc-swap = "1"
toml = "0.8"
rustls = { version = "0.23", features = ["ring"] }
ratatui = "0.30"
crossterm = "0.28"
url = "2"
sysinfo = "0.32"
libc = "0.2"
flate2 = "1"
tar = "0.4"
socket2 = { version = "0.5", features = ["all"] }
tower-service = "0.3"
webpki-roots = "0.26"
[profile.release]
lto = true
strip = true
codegen-units = 1
-10
View File
@@ -1,10 +0,0 @@
FROM debian:bookworm-slim
ARG TARGETARCH
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*
COPY build/linux-${TARGETARCH}/aether-proxy /usr/local/bin/aether-proxy
ENTRYPOINT ["aether-proxy"]
-154
View File
@@ -1,154 +0,0 @@
# aether-proxy
Aether Tunnel 代理节点,部署在海外 VPS 上,通过 WebSocket 隧道为 Aether 实例中转 API 流量。
Tunnel 模式下代理节点**无需对外监听端口**,仅需出站连接到 Aether 服务器。
## 安装
### Docker Compose 部署
```bash
cp .env.example .env
# 编辑 .env 填入 AETHER_PROXY_AETHER_URL 和 AETHER_PROXY_MANAGEMENT_TOKEN
docker compose up -d
```
### 下载预编译二进制
<!-- DOWNLOAD_TABLE_START -->
| Platform | Download |
|----------|----------|
| Linux x86_64 | [aether-proxy-linux-amd64.tar.gz](https://github.com/fawney19/Aether/releases/download/proxy-v0.2.4/aether-proxy-linux-amd64.tar.gz) |
| Linux ARM64 | [aether-proxy-linux-arm64.tar.gz](https://github.com/fawney19/Aether/releases/download/proxy-v0.2.4/aether-proxy-linux-arm64.tar.gz) |
| macOS x86_64 | [aether-proxy-macos-amd64.tar.gz](https://github.com/fawney19/Aether/releases/download/proxy-v0.2.4/aether-proxy-macos-amd64.tar.gz) |
| macOS ARM64 | [aether-proxy-macos-arm64.tar.gz](https://github.com/fawney19/Aether/releases/download/proxy-v0.2.4/aether-proxy-macos-arm64.tar.gz) |
| Windows x86_64 | [aether-proxy-windows-amd64.zip](https://github.com/fawney19/Aether/releases/download/proxy-v0.2.4/aether-proxy-windows-amd64.zip) |
<!-- DOWNLOAD_TABLE_END -->
## 快速开始
```bash
# 1. 首次安装配置(TUI 向导,勾选 Install Service 随系统启动服务)
sudo ./aether-proxy setup
# 2. 日常管理 (勾选 Install Service 作为系统服务的情况下)
aether-proxy status # 看状态
aether-proxy logs # 看日志
sudo aether-proxy start # 启动服务
sudo aether-proxy stop # 停止服务
sudo aether-proxy restart # 重启服务
# 3. 重新配置(改完自动重启服务)
sudo aether-proxy setup
# 4. 彻底卸载
sudo aether-proxy uninstall
```
完成向导后, 配置自动保存到 `aether-proxy.toml`,如果启用了 Install Service,将自动注册并启动 systemd 服务。
### 直接运行
如果不需要安装为系统服务,可以直接运行。缺少必填参数时会自动进入 setup 向导:
```bash
./aether-proxy
```
## 配置
配置按以下优先级加载(高优先级覆盖低优先级):
1. CLI 参数
2. 环境变量(`AETHER_PROXY_*`)
3. 配置文件(`aether-proxy.toml`,或通过 `AETHER_PROXY_CONFIG` 指定路径)
### 参数一览
#### 基础配置
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--aether-url` | `AETHER_PROXY_AETHER_URL` | **必填** | Aether 服务器地址 |
| `--management-token` | `AETHER_PROXY_MANAGEMENT_TOKEN` | **必填** | 管理员 Token(`ae_xxx` 格式) |
| `--public-ip` | `AETHER_PROXY_PUBLIC_IP` | 自动检测 | 公网 IP |
| `--node-name` | `AETHER_PROXY_NODE_NAME` | `proxy-01` | 节点名称标识 |
| `--node-region` | `AETHER_PROXY_NODE_REGION` | 自动检测 | 地区标识 |
| `--heartbeat-interval` | `AETHER_PROXY_HEARTBEAT_INTERVAL` | `30` | 心跳间隔(秒) |
| `--allowed-ports` | `AETHER_PROXY_ALLOWED_PORTS` | `80,443,8080,8443` | 允许代理的目标端口 |
#### Tunnel 连接
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--tunnel-connections` | `AETHER_PROXY_TUNNEL_CONNECTIONS` | `3` | 到 Aether 的连接池大小 |
| `--tunnel-max-streams` | `AETHER_PROXY_TUNNEL_MAX_STREAMS` | 自动(硬件估算) | 单连接最大并发 stream 数 |
| `--tunnel-connect-timeout-secs` | `AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT_SECS` | `15` | TCP + TLS 握手超时(秒) |
| `--tunnel-tcp-keepalive-secs` | `AETHER_PROXY_TUNNEL_TCP_KEEPALIVE_SECS` | `30` | TCP keepalive 初始延迟(秒) |
| `--tunnel-tcp-nodelay` | `AETHER_PROXY_TUNNEL_TCP_NODELAY` | `true` | 禁用 Nagle 算法 |
| `--tunnel-ping-interval-secs` | `AETHER_PROXY_TUNNEL_PING_INTERVAL_SECS` | `15` | WebSocket Ping 频率(秒) |
| `--tunnel-stale-timeout-secs` | `AETHER_PROXY_TUNNEL_STALE_TIMEOUT_SECS` | `45` | 无数据断连阈值(秒) |
| `--tunnel-reconnect-base-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS` | `500` | 指数退避基础延迟(毫秒) |
| `--tunnel-reconnect-max-ms` | `AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS` | `30000` | 指数退避上限(毫秒) |
#### 上游 HTTP 请求
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--upstream-connect-timeout-secs` | `AETHER_PROXY_UPSTREAM_CONNECT_TIMEOUT_SECS` | `30` | 上游建连超时(秒) |
| `--upstream-pool-max-idle-per-host` | `AETHER_PROXY_UPSTREAM_POOL_MAX_IDLE_PER_HOST` | `64` | 每 Host 最大空闲连接数 |
| `--upstream-pool-idle-timeout-secs` | `AETHER_PROXY_UPSTREAM_POOL_IDLE_TIMEOUT_SECS` | `300` | 连接池空闲超时(秒) |
| `--upstream-tcp-keepalive-secs` | `AETHER_PROXY_UPSTREAM_TCP_KEEPALIVE_SECS` | `60` | TCP keepalive(秒,0 关闭) |
| `--upstream-tcp-nodelay` | `AETHER_PROXY_UPSTREAM_TCP_NODELAY` | `true` | 启用 TCP_NODELAY |
#### Aether API 客户端
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--aether-request-timeout-secs` | `AETHER_PROXY_AETHER_REQUEST_TIMEOUT_SECS` | `10` | 请求总超时(秒) |
| `--aether-connect-timeout-secs` | `AETHER_PROXY_AETHER_CONNECT_TIMEOUT_SECS` | `10` | 建连超时(秒) |
| `--aether-retry-max-attempts` | `AETHER_PROXY_AETHER_RETRY_MAX_ATTEMPTS` | `3` | 最大重试次数 |
#### DNS 与安全
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--dns-cache-ttl-secs` | `AETHER_PROXY_DNS_CACHE_TTL_SECS` | `60` | DNS 缓存 TTL(秒) |
| `--dns-cache-capacity` | `AETHER_PROXY_DNS_CACHE_CAPACITY` | `1024` | DNS 缓存容量(条目数) |
#### 日志
| 参数 | 环境变量 | 默认值 | 说明 |
|------|----------|--------|------|
| `--log-level` | `AETHER_PROXY_LOG_LEVEL` | `info` | 日志级别 |
| `--log-json` | `AETHER_PROXY_LOG_JSON` | `false` | JSON 格式日志 |
### 多服务器配置
在 `aether-proxy.toml` 中使用 `[[servers]]` 配置多个 Aether 服务器:
```toml
[[servers]]
aether_url = "https://aether-1.example.com"
management_token = "ae_xxx"
node_name = "jp-proxy-01"
[[servers]]
aether_url = "https://aether-2.example.com"
management_token = "ae_yyy"
node_name = "jp-proxy-02"
```
## 发布新版本
推送 `proxy-v*` 格式的 tag,GitHub Actions 会自动:
- 编译所有平台二进制并发布到 Releases
- 构建 Docker 镜像并推送到 GHCR 和 Docker Hub
- 更新 README 中的下载链接表格
```bash
git tag proxy-v0.2.0
git push origin proxy-v0.2.0
```
-14
View File
@@ -1,14 +0,0 @@
services:
aether-proxy:
image: ghcr.io/fawney19/aether-proxy:latest
container_name: aether-proxy
restart: unless-stopped
env_file:
- .env
environment:
AETHER_PROXY_LOG_JSON: "true"
logging:
driver: json-file
options:
max-size: "50m"
max-file: "3"
-360
View File
@@ -1,360 +0,0 @@
//! Application lifecycle: initialization, task orchestration, and shutdown.
use std::sync::atomic::AtomicU64;
use std::sync::{Arc, RwLock};
use std::time::Duration;
use arc_swap::ArcSwap;
use tokio::signal;
use tokio::sync::{watch, Mutex};
use tracing::{error, info, warn};
use crate::config::{Config, ServerEntry};
use crate::net;
use crate::registration::client::AetherClient;
use crate::runtime::{self, DynamicConfig};
use crate::state::{AppState, ProxyMetrics, ServerContext};
use crate::upstream_client;
use crate::{hardware, target_filter, tunnel};
/// Run the full application lifecycle after config has been parsed.
pub async fn run(mut config: Config, servers: Vec<ServerEntry>) -> anyhow::Result<()> {
config.validate()?;
init_tracing(&config);
info!(
version = env!("CARGO_PKG_VERSION"),
node_name = %config.node_name,
server_count = servers.len(),
"aether-proxy starting (tunnel mode)"
);
// Resolve public IP (best-effort for region info)
let public_ip = match &config.public_ip {
Some(ip) => ip.clone(),
None => net::detect_public_ip()
.await
.unwrap_or_else(|_| "0.0.0.0".to_string()),
};
// Auto-detect region if not configured
if config.node_region.is_none() {
if let Some(region) = net::detect_region(&public_ip).await {
config.node_region = Some(region);
}
}
// Collect hardware info (once at startup, sent during registration)
let hw_info = hardware::collect();
// Auto-detect tunnel_max_streams from hardware if not explicitly set
if config.tunnel_max_streams.is_none() {
let auto = (hw_info.estimated_max_concurrency / 10).clamp(64, 1024) as u32;
config.tunnel_max_streams = Some(auto);
info!(
tunnel_max_streams = auto,
"auto-detected tunnel_max_streams from hardware"
);
}
info!(
max_concurrency = hw_info.estimated_max_concurrency,
"hardware info collected"
);
let dns_cache = Arc::new(target_filter::DnsCache::new(
Duration::from_secs(config.dns_cache_ttl_secs),
config.dns_cache_capacity,
));
// Build Hyper client for tunnel upstream requests (shared).
// DNS still flows through validated addresses from DnsCache, while the
// custom connector exposes per-request connect/TLS timing when available.
let upstream_client = upstream_client::build_upstream_client(&config, Arc::clone(&dns_cache));
// Register with each Aether server and build per-server contexts.
// Wrapped in Arc<Mutex> so retry_failed_registrations can append later.
let server_contexts: Arc<Mutex<Vec<Arc<ServerContext>>>> = Arc::new(Mutex::new(Vec::new()));
let mut failed_entries: Vec<(String, ServerEntry)> = Vec::new();
for (i, entry) in servers.iter().enumerate() {
let label = if servers.len() == 1 {
"server".to_string()
} else {
format!("server-{}", i)
};
let node_name = entry
.node_name
.clone()
.unwrap_or_else(|| config.node_name.clone());
let client = Arc::new(AetherClient::new(
&config,
&entry.aether_url,
&entry.management_token,
));
match client
.register(&config, &node_name, &public_ip, Some(&hw_info))
.await
{
Ok(node_id) => {
info!(server = %label, node_id = %node_id, url = %entry.aether_url, node_name = %node_name, "registered");
// Initialize dynamic config with per-server node_name (not global),
// so that the heartbeat and reconnect use the correct name.
let mut dynamic = DynamicConfig::from_config(&config);
dynamic.node_name = node_name.clone();
server_contexts.lock().await.push(Arc::new(ServerContext {
server_label: label,
aether_url: entry.aether_url.clone(),
management_token: entry.management_token.clone(),
node_name,
node_id: Arc::new(RwLock::new(node_id)),
aether_client: client,
dynamic: Arc::new(ArcSwap::from_pointee(dynamic)),
active_connections: Arc::new(AtomicU64::new(0)),
metrics: Arc::new(ProxyMetrics::new()),
}));
}
Err(e) => {
warn!(
server = %label,
url = %entry.aether_url,
error = %e,
"registration failed, will retry in background"
);
failed_entries.push((label, entry.clone()));
}
}
}
{
let ctx_count = server_contexts.lock().await.len();
if ctx_count == 0 && failed_entries.is_empty() {
anyhow::bail!("no servers configured");
}
if ctx_count == 0 {
anyhow::bail!(
"no servers registered successfully (all {} failed)",
failed_entries.len()
);
}
}
// Build shared application state
let tunnel_tls_config = Arc::new(crate::tunnel::client::build_tls_config());
let state = Arc::new(AppState {
config: Arc::new(config),
dns_cache,
upstream_client,
tunnel_tls_config,
});
// Shutdown signal channel
let (shutdown_tx, shutdown_rx) = watch::channel(false);
info!(
active_servers = server_contexts.lock().await.len(),
"running in tunnel mode"
);
// Spawn tunnel connections per server (pool_size connections each)
let pool_size = state.config.tunnel_connections.max(1) as usize;
let mut tunnel_handles = Vec::new();
for server in server_contexts.lock().await.iter() {
for conn_idx in 0..pool_size {
let s = Arc::clone(&state);
let srv = Arc::clone(server);
let rx = shutdown_rx.clone();
tunnel_handles.push(tokio::spawn(async move {
tunnel::run(&s, &srv, conn_idx, rx).await;
}));
}
}
// Spawn background retry for failed server registrations
if !failed_entries.is_empty() {
let retry_state = Arc::clone(&state);
let retry_contexts = Arc::clone(&server_contexts);
let retry_public_ip = public_ip.clone();
let retry_hw_info = hw_info.clone();
let retry_shutdown = shutdown_rx.clone();
let retry_pool_size = pool_size;
tokio::spawn(async move {
retry_failed_registrations(
retry_state,
retry_contexts,
failed_entries,
retry_public_ip,
retry_hw_info,
retry_pool_size,
retry_shutdown,
)
.await;
});
}
// Wait for shutdown signal
wait_for_shutdown().await;
info!("shutdown signal received, cleaning up...");
let _ = shutdown_tx.send(true);
// Graceful unregister from all servers (including retry-registered ones)
for server in server_contexts.lock().await.iter() {
let node_id = server.node_id.read().unwrap().clone();
if let Err(e) = server.aether_client.unregister(&node_id).await {
error!(
server = %server.server_label,
error = %e,
"unregister failed during shutdown"
);
}
}
// Wait for all tunnel tasks
for h in tunnel_handles {
let _ = h.await;
}
info!("aether-proxy stopped");
Ok(())
}
/// Retry interval for failed server registrations (5 minutes).
const REGISTRATION_RETRY_INTERVAL: Duration = Duration::from_secs(300);
/// Max registration retry attempts before giving up.
const REGISTRATION_RETRY_MAX: u32 = 12;
/// Background task that retries registration for servers that failed at startup.
async fn retry_failed_registrations(
state: Arc<AppState>,
server_contexts: Arc<Mutex<Vec<Arc<ServerContext>>>>,
failed: Vec<(String, ServerEntry)>,
public_ip: String,
hw_info: crate::hardware::HardwareInfo,
pool_size: usize,
mut shutdown: watch::Receiver<bool>,
) {
for (label, entry) in &failed {
let node_name = entry
.node_name
.clone()
.unwrap_or_else(|| state.config.node_name.clone());
let client = Arc::new(AetherClient::new(
&state.config,
&entry.aether_url,
&entry.management_token,
));
let mut attempt = 0u32;
let node_id = loop {
attempt += 1;
tokio::select! {
_ = tokio::time::sleep(REGISTRATION_RETRY_INTERVAL) => {}
_ = shutdown.changed() => {
info!(server = %label, "shutdown during registration retry");
return;
}
}
match client
.register(&state.config, &node_name, &public_ip, Some(&hw_info))
.await
{
Ok(id) => {
info!(server = %label, node_id = %id, attempt, "registration retry succeeded");
break id;
}
Err(e) => {
warn!(
server = %label,
attempt,
max = REGISTRATION_RETRY_MAX,
error = %e,
"registration retry failed"
);
if attempt >= REGISTRATION_RETRY_MAX {
error!(server = %label, "giving up registration after {} attempts", attempt);
return;
}
}
}
};
// Build server context and spawn tunnels
let mut dynamic = DynamicConfig::from_config(&state.config);
dynamic.node_name = node_name.clone();
let server = Arc::new(ServerContext {
server_label: label.clone(),
aether_url: entry.aether_url.clone(),
management_token: entry.management_token.clone(),
node_name,
node_id: Arc::new(RwLock::new(node_id)),
aether_client: client,
dynamic: Arc::new(ArcSwap::from_pointee(dynamic)),
active_connections: Arc::new(AtomicU64::new(0)),
metrics: Arc::new(ProxyMetrics::new()),
});
// Add to shared list so shutdown can unregister this server
server_contexts.lock().await.push(Arc::clone(&server));
for conn_idx in 0..pool_size {
let s = Arc::clone(&state);
let srv = Arc::clone(&server);
let rx = shutdown.clone();
tokio::spawn(async move {
tunnel::run(&s, &srv, conn_idx, rx).await;
});
}
}
}
fn init_tracing(config: &Config) {
use tracing_subscriber::prelude::*;
use tracing_subscriber::{reload, EnvFilter};
let filter = EnvFilter::try_new(&config.log_level).unwrap_or_else(|_| EnvFilter::new("info"));
let (filter_layer, reload_handle) = reload::Layer::new(filter);
runtime::set_log_reloader(Box::new(move |level: &str| {
if let Ok(new_filter) = EnvFilter::try_new(level) {
let _ = reload_handle.modify(|f| *f = new_filter);
}
}));
if config.log_json {
tracing_subscriber::registry()
.with(filter_layer)
.with(tracing_subscriber::fmt::layer().json())
.init();
} else {
tracing_subscriber::registry()
.with(filter_layer)
.with(tracing_subscriber::fmt::layer())
.init();
}
}
async fn wait_for_shutdown() {
let ctrl_c = async {
signal::ctrl_c()
.await
.expect("failed to install Ctrl+C handler");
};
#[cfg(unix)]
let terminate = async {
signal::unix::signal(signal::unix::SignalKind::terminate())
.expect("failed to install SIGTERM handler")
.recv()
.await;
};
#[cfg(not(unix))]
let terminate = std::future::pending::<()>();
tokio::select! {
_ = ctrl_c => {},
_ = terminate => {},
}
}
-656
View File
@@ -1,656 +0,0 @@
use std::path::Path;
use clap::Parser;
use serde::{Deserialize, Serialize};
/// Fields that existed in 0.1.x but were removed in 0.2.0.
const LEGACY_ONLY_KEYS: &[&str] = &[
"hmac_key",
"listen_port",
"timestamp_tolerance",
"connect_timeout_secs",
"tls_handshake_timeout_secs",
"enable_tls",
"tls_cert",
"tls_key",
];
/// Fields renamed from 0.1.x `delegate_*` to 0.2.0 `upstream_*`.
const DELEGATE_TO_UPSTREAM: &[(&str, &str)] = &[
(
"delegate_connect_timeout_secs",
"upstream_connect_timeout_secs",
),
(
"delegate_pool_max_idle_per_host",
"upstream_pool_max_idle_per_host",
),
(
"delegate_pool_idle_timeout_secs",
"upstream_pool_idle_timeout_secs",
),
("delegate_tcp_keepalive_secs", "upstream_tcp_keepalive_secs"),
("delegate_tcp_nodelay", "upstream_tcp_nodelay"),
];
/// Aether tunnel proxy.
///
/// Deployed on overseas VPS to relay API traffic for Aether instances
/// behind the GFW. Connects to Aether via WebSocket tunnel, registers
/// with Aether, and relays upstream requests.
#[derive(Parser, Debug, Clone)]
#[command(version, about)]
pub struct Config {
/// Aether server URL (e.g. https://aether.example.com)
#[arg(long, env = "AETHER_PROXY_AETHER_URL")]
pub aether_url: String,
/// Management Token for Aether admin API (ae_xxx)
#[arg(long, env = "AETHER_PROXY_MANAGEMENT_TOKEN")]
pub management_token: String,
/// Public IP address of this node (auto-detected if omitted)
#[arg(long, env = "AETHER_PROXY_PUBLIC_IP")]
pub public_ip: Option<String>,
/// Human-readable node name
#[arg(long, env = "AETHER_PROXY_NODE_NAME", default_value = "proxy-01")]
pub node_name: String,
/// Region label (e.g. ap-northeast-1)
#[arg(long, env = "AETHER_PROXY_NODE_REGION")]
pub node_region: Option<String>,
/// Heartbeat interval in seconds
#[arg(long, env = "AETHER_PROXY_HEARTBEAT_INTERVAL", default_value_t = 30)]
pub heartbeat_interval: u64,
/// Allowed destination ports (default: 80,443,8080,8443)
#[arg(
long,
env = "AETHER_PROXY_ALLOWED_PORTS",
value_delimiter = ',',
default_values_t = vec![80, 443, 8080, 8443]
)]
pub allowed_ports: Vec<u16>,
/// Aether API request timeout in seconds
#[arg(
long,
env = "AETHER_PROXY_AETHER_REQUEST_TIMEOUT",
default_value_t = 10
)]
pub aether_request_timeout_secs: u64,
/// Aether API connect timeout in seconds
#[arg(
long,
env = "AETHER_PROXY_AETHER_CONNECT_TIMEOUT",
default_value_t = 10
)]
pub aether_connect_timeout_secs: u64,
/// Aether API max idle connections per host
#[arg(
long,
env = "AETHER_PROXY_AETHER_POOL_MAX_IDLE_PER_HOST",
default_value_t = 8
)]
pub aether_pool_max_idle_per_host: usize,
/// Aether API idle timeout in seconds
#[arg(
long,
env = "AETHER_PROXY_AETHER_POOL_IDLE_TIMEOUT",
default_value_t = 90
)]
pub aether_pool_idle_timeout_secs: u64,
/// Aether API TCP keepalive in seconds (0 disables)
#[arg(long, env = "AETHER_PROXY_AETHER_TCP_KEEPALIVE", default_value_t = 60)]
pub aether_tcp_keepalive_secs: u64,
/// Aether API TCP_NODELAY
#[arg(long, env = "AETHER_PROXY_AETHER_TCP_NODELAY", default_value_t = true)]
pub aether_tcp_nodelay: bool,
/// Enable HTTP/2 when talking to Aether API
#[arg(long, env = "AETHER_PROXY_AETHER_HTTP2", default_value_t = true)]
pub aether_http2: bool,
/// Aether API retry attempts (including initial)
#[arg(
long,
env = "AETHER_PROXY_AETHER_RETRY_MAX_ATTEMPTS",
default_value_t = 3
)]
pub aether_retry_max_attempts: u32,
/// Aether API retry base delay in milliseconds
#[arg(
long,
env = "AETHER_PROXY_AETHER_RETRY_BASE_DELAY_MS",
default_value_t = 200
)]
pub aether_retry_base_delay_ms: u64,
/// Aether API retry max delay in milliseconds
#[arg(
long,
env = "AETHER_PROXY_AETHER_RETRY_MAX_DELAY_MS",
default_value_t = 2000
)]
pub aether_retry_max_delay_ms: u64,
/// Maximum concurrent TCP connections (defaults to hardware estimate)
#[arg(long, env = "AETHER_PROXY_MAX_CONCURRENT_CONNECTIONS")]
pub max_concurrent_connections: Option<u64>,
/// DNS cache TTL in seconds
#[arg(long, env = "AETHER_PROXY_DNS_CACHE_TTL", default_value_t = 60)]
pub dns_cache_ttl_secs: u64,
/// DNS cache capacity (entries)
#[arg(long, env = "AETHER_PROXY_DNS_CACHE_CAPACITY", default_value_t = 1024)]
pub dns_cache_capacity: usize,
/// Upstream HTTP client connect timeout in seconds
#[arg(
long,
env = "AETHER_PROXY_UPSTREAM_CONNECT_TIMEOUT",
default_value_t = 30
)]
pub upstream_connect_timeout_secs: u64,
/// Upstream HTTP client max idle connections per host
#[arg(
long,
env = "AETHER_PROXY_UPSTREAM_POOL_MAX_IDLE_PER_HOST",
default_value_t = 64
)]
pub upstream_pool_max_idle_per_host: usize,
/// Upstream HTTP client idle timeout in seconds
#[arg(
long,
env = "AETHER_PROXY_UPSTREAM_POOL_IDLE_TIMEOUT",
default_value_t = 300
)]
pub upstream_pool_idle_timeout_secs: u64,
/// Upstream TCP keepalive in seconds (0 disables)
#[arg(
long,
env = "AETHER_PROXY_UPSTREAM_TCP_KEEPALIVE",
default_value_t = 60
)]
pub upstream_tcp_keepalive_secs: u64,
/// Upstream TCP_NODELAY
#[arg(
long,
env = "AETHER_PROXY_UPSTREAM_TCP_NODELAY",
default_value_t = true
)]
pub upstream_tcp_nodelay: bool,
/// Log level (trace, debug, info, warn, error)
#[arg(long, env = "AETHER_PROXY_LOG_LEVEL", default_value = "info")]
pub log_level: String,
/// Output logs as JSON
#[arg(long, env = "AETHER_PROXY_LOG_JSON", default_value_t = false)]
pub log_json: bool,
/// Tunnel reconnect base delay in milliseconds (used by exponential backoff)
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS",
default_value_t = 500
)]
pub tunnel_reconnect_base_ms: u64,
/// Tunnel reconnect max delay in milliseconds (cap for exponential backoff)
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS",
default_value_t = 30000
)]
pub tunnel_reconnect_max_ms: u64,
/// WebSocket tunnel ping interval in seconds
#[arg(long, env = "AETHER_PROXY_TUNNEL_PING_INTERVAL", default_value_t = 15)]
pub tunnel_ping_interval_secs: u64,
/// Maximum concurrent streams over tunnel (auto-detected from hardware if omitted)
#[arg(long, env = "AETHER_PROXY_TUNNEL_MAX_STREAMS")]
pub tunnel_max_streams: Option<u32>,
/// WebSocket tunnel TCP connect timeout in seconds
#[arg(
long,
env = "AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT",
default_value_t = 15
)]
pub tunnel_connect_timeout_secs: u64,
/// WebSocket tunnel TCP keepalive in seconds (0 disables)
#[arg(long, env = "AETHER_PROXY_TUNNEL_TCP_KEEPALIVE", default_value_t = 30)]
pub tunnel_tcp_keepalive_secs: u64,
/// WebSocket tunnel TCP_NODELAY
#[arg(long, env = "AETHER_PROXY_TUNNEL_TCP_NODELAY", default_value_t = true)]
pub tunnel_tcp_nodelay: bool,
/// Tunnel connection staleness timeout in seconds (triggers reconnect if no data received)
#[arg(long, env = "AETHER_PROXY_TUNNEL_STALE_TIMEOUT", default_value_t = 45)]
pub tunnel_stale_timeout_secs: u64,
/// Number of parallel WebSocket tunnel connections per server (connection pool)
#[arg(long, env = "AETHER_PROXY_TUNNEL_CONNECTIONS", default_value_t = 3)]
pub tunnel_connections: u32,
}
impl Config {
/// Validate configuration values are within sane ranges.
/// Called after parsing to catch misconfigurations early.
pub fn validate(&self) -> anyhow::Result<()> {
if self.heartbeat_interval == 0 {
anyhow::bail!("heartbeat_interval must be > 0");
}
if self.heartbeat_interval > 3600 {
anyhow::bail!("heartbeat_interval must be <= 3600");
}
if self.allowed_ports.is_empty() {
anyhow::bail!("allowed_ports must not be empty");
}
for &port in &self.allowed_ports {
if port == 0 {
anyhow::bail!("allowed_ports: port 0 is not valid");
}
}
if self.tunnel_connect_timeout_secs == 0 {
anyhow::bail!("tunnel_connect_timeout_secs must be > 0");
}
if self.tunnel_ping_interval_secs == 0 {
anyhow::bail!("tunnel_ping_interval_secs must be > 0");
}
if self.tunnel_stale_timeout_secs <= self.tunnel_ping_interval_secs {
anyhow::bail!(
"tunnel_stale_timeout_secs ({}) must be > tunnel_ping_interval_secs ({})",
self.tunnel_stale_timeout_secs,
self.tunnel_ping_interval_secs
);
}
if self.tunnel_connections == 0 {
anyhow::bail!("tunnel_connections must be > 0");
}
if self.aether_retry_max_attempts == 0 {
anyhow::bail!("aether_retry_max_attempts must be >= 1");
}
if self.upstream_connect_timeout_secs == 0 {
anyhow::bail!("upstream_connect_timeout_secs must be > 0");
}
Ok(())
}
}
/// Per-server connection config (used in multi-server TOML `[[servers]]`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServerEntry {
pub aether_url: String,
pub management_token: String,
/// Per-server node name override. Falls back to the global `node_name`.
pub node_name: Option<String>,
}
// ---------------------------------------------------------------------------
// TOML config file support
// ---------------------------------------------------------------------------
/// Serializable config for TOML file persistence.
/// All fields are optional -- only populated values are written.
#[derive(Debug, Default, Serialize, Deserialize)]
pub struct ConfigFile {
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub management_token: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub public_ip: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub node_name: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub node_region: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub heartbeat_interval: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub allowed_ports: Option<Vec<u16>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_request_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_connect_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_pool_max_idle_per_host: Option<usize>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_pool_idle_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_tcp_keepalive_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_tcp_nodelay: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_http2: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_retry_max_attempts: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_retry_base_delay_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub aether_retry_max_delay_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub max_concurrent_connections: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub dns_cache_ttl_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub dns_cache_capacity: Option<usize>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_connect_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_pool_max_idle_per_host: Option<usize>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_pool_idle_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_tcp_keepalive_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub upstream_tcp_nodelay: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub log_level: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub log_json: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_reconnect_base_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_reconnect_max_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_ping_interval_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_max_streams: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_connect_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_tcp_keepalive_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_tcp_nodelay: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_stale_timeout_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tunnel_connections: Option<u32>,
/// Multi-server config: each entry connects to a separate Aether instance.
/// When present, top-level aether_url/management_token are ignored for
/// tunnel connections (but still injected as env for clap compatibility).
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub servers: Vec<ServerEntry>,
}
impl ConfigFile {
/// Load from a TOML file.
pub fn load(path: &Path) -> anyhow::Result<Self> {
let content = std::fs::read_to_string(path)?;
Ok(toml::from_str(&content)?)
}
/// Save to a TOML file.
pub fn save(&self, path: &Path) -> anyhow::Result<()> {
let content = toml::to_string_pretty(self)?;
std::fs::write(path, content)?;
Ok(())
}
/// Detect and migrate a 0.1.x config file to 0.2.0 format in-place.
///
/// Returns `true` if migration was performed, `false` if already current.
/// The original file is backed up as `<name>.v1.bak` before rewriting.
pub fn migrate_legacy(path: &Path) -> anyhow::Result<bool> {
let content = match std::fs::read_to_string(path) {
Ok(c) => c,
Err(_) => return Ok(false),
};
let mut table: toml::map::Map<String, toml::Value> = toml::from_str(&content)?;
// Detect legacy format: presence of any 0.1.x-only key.
let is_legacy = LEGACY_ONLY_KEYS.iter().any(|k| table.contains_key(*k))
|| DELEGATE_TO_UPSTREAM
.iter()
.any(|(old, _)| table.contains_key(*old));
if !is_legacy {
return Ok(false);
}
// 1. Rename delegate_* -> upstream_* (carry over user-customized values)
for &(old, new) in DELEGATE_TO_UPSTREAM {
if let Some(val) = table.remove(old) {
table.entry(new.to_string()).or_insert(val);
}
}
// 2. Build [[servers]] from top-level aether_url + management_token + node_name
if !table.contains_key("servers") {
let aether_url = table.get("aether_url").and_then(|v| v.as_str());
let management_token = table.get("management_token").and_then(|v| v.as_str());
if let (Some(url), Some(token)) = (aether_url, management_token) {
let mut entry = toml::map::Map::new();
entry.insert("aether_url".into(), toml::Value::String(url.to_string()));
entry.insert(
"management_token".into(),
toml::Value::String(token.to_string()),
);
if let Some(name) = table.get("node_name").and_then(|v| v.as_str()) {
entry.insert("node_name".into(), toml::Value::String(name.to_string()));
}
table.insert(
"servers".into(),
toml::Value::Array(vec![toml::Value::Table(entry)]),
);
}
}
// 3. Remove top-level fields that are now in [[servers]] or obsolete
table.remove("aether_url");
table.remove("management_token");
table.remove("node_name");
for &key in LEGACY_ONLY_KEYS {
table.remove(key);
}
// 4. Backup original file (abort migration if backup fails)
let backup_path = path.with_extension("v1.bak");
std::fs::copy(path, &backup_path).map_err(|e| {
anyhow::anyhow!(
"failed to backup config before migration: {} -> {}: {}",
path.display(),
backup_path.display(),
e
)
})?;
// 5. Write migrated config
let new_content = toml::to_string_pretty(&table)?;
std::fs::write(path, &new_content)?;
eprintln!(" Config migrated from 0.1.x to 0.2.0 format.");
eprintln!(" Backup saved: {}", backup_path.display());
Ok(true)
}
/// Resolve the effective server list.
///
/// If `[[servers]]` is present, use it. Otherwise fall back to the
/// top-level `aether_url` + `management_token` as a single server.
pub fn effective_servers(&self) -> Vec<ServerEntry> {
if !self.servers.is_empty() {
return self.servers.clone();
}
match (&self.aether_url, &self.management_token) {
(Some(url), Some(token)) => vec![ServerEntry {
aether_url: url.clone(),
management_token: token.clone(),
node_name: None,
}],
_ => vec![],
}
}
/// Inject values as environment variables so clap picks them up.
///
/// Only sets variables that are **not** already present in the
/// environment, preserving the precedence: CLI > env > config file.
pub fn inject_env(&self) {
self.inject_env_inner(false);
}
/// Inject values as environment variables, **overriding** any existing
/// values. Used after setup to ensure the freshly-saved config takes
/// effect before re-parsing.
pub fn inject_env_override(&self) {
self.inject_env_inner(true);
}
fn inject_env_inner(&self, force: bool) {
macro_rules! set {
($env:expr, $val:expr) => {
if let Some(ref v) = $val {
if force || std::env::var($env).is_err() {
std::env::set_var($env, v.to_string());
}
}
};
}
// When top-level fields are absent, fall back to the first [[servers]]
// entry so that clap's required `aether_url` / `management_token` are
// satisfied even with the new config format.
let first_server = self.servers.first();
let aether_url = self
.aether_url
.as_deref()
.or(first_server.map(|s| s.aether_url.as_str()));
let management_token = self
.management_token
.as_deref()
.or(first_server.map(|s| s.management_token.as_str()));
let node_name = self
.node_name
.as_deref()
.or(first_server.and_then(|s| s.node_name.as_deref()));
set!("AETHER_PROXY_AETHER_URL", aether_url);
set!("AETHER_PROXY_MANAGEMENT_TOKEN", management_token);
set!("AETHER_PROXY_PUBLIC_IP", self.public_ip);
set!("AETHER_PROXY_NODE_NAME", node_name);
set!("AETHER_PROXY_NODE_REGION", self.node_region);
set!("AETHER_PROXY_HEARTBEAT_INTERVAL", self.heartbeat_interval);
set!(
"AETHER_PROXY_AETHER_REQUEST_TIMEOUT",
self.aether_request_timeout_secs
);
set!(
"AETHER_PROXY_AETHER_CONNECT_TIMEOUT",
self.aether_connect_timeout_secs
);
set!(
"AETHER_PROXY_AETHER_POOL_MAX_IDLE_PER_HOST",
self.aether_pool_max_idle_per_host
);
set!(
"AETHER_PROXY_AETHER_POOL_IDLE_TIMEOUT",
self.aether_pool_idle_timeout_secs
);
set!(
"AETHER_PROXY_AETHER_TCP_KEEPALIVE",
self.aether_tcp_keepalive_secs
);
set!("AETHER_PROXY_AETHER_TCP_NODELAY", self.aether_tcp_nodelay);
set!("AETHER_PROXY_AETHER_HTTP2", self.aether_http2);
set!(
"AETHER_PROXY_AETHER_RETRY_MAX_ATTEMPTS",
self.aether_retry_max_attempts
);
set!(
"AETHER_PROXY_AETHER_RETRY_BASE_DELAY_MS",
self.aether_retry_base_delay_ms
);
set!(
"AETHER_PROXY_AETHER_RETRY_MAX_DELAY_MS",
self.aether_retry_max_delay_ms
);
set!(
"AETHER_PROXY_MAX_CONCURRENT_CONNECTIONS",
self.max_concurrent_connections
);
set!("AETHER_PROXY_DNS_CACHE_TTL", self.dns_cache_ttl_secs);
set!("AETHER_PROXY_DNS_CACHE_CAPACITY", self.dns_cache_capacity);
set!(
"AETHER_PROXY_UPSTREAM_CONNECT_TIMEOUT",
self.upstream_connect_timeout_secs
);
set!(
"AETHER_PROXY_UPSTREAM_POOL_MAX_IDLE_PER_HOST",
self.upstream_pool_max_idle_per_host
);
set!(
"AETHER_PROXY_UPSTREAM_POOL_IDLE_TIMEOUT",
self.upstream_pool_idle_timeout_secs
);
set!(
"AETHER_PROXY_UPSTREAM_TCP_KEEPALIVE",
self.upstream_tcp_keepalive_secs
);
set!(
"AETHER_PROXY_UPSTREAM_TCP_NODELAY",
self.upstream_tcp_nodelay
);
set!("AETHER_PROXY_LOG_LEVEL", self.log_level);
set!("AETHER_PROXY_LOG_JSON", self.log_json);
set!(
"AETHER_PROXY_TUNNEL_RECONNECT_BASE_MS",
self.tunnel_reconnect_base_ms
);
set!(
"AETHER_PROXY_TUNNEL_RECONNECT_MAX_MS",
self.tunnel_reconnect_max_ms
);
set!(
"AETHER_PROXY_TUNNEL_PING_INTERVAL",
self.tunnel_ping_interval_secs
);
set!("AETHER_PROXY_TUNNEL_MAX_STREAMS", self.tunnel_max_streams);
set!(
"AETHER_PROXY_TUNNEL_CONNECT_TIMEOUT",
self.tunnel_connect_timeout_secs
);
set!(
"AETHER_PROXY_TUNNEL_TCP_KEEPALIVE",
self.tunnel_tcp_keepalive_secs
);
set!("AETHER_PROXY_TUNNEL_TCP_NODELAY", self.tunnel_tcp_nodelay);
set!(
"AETHER_PROXY_TUNNEL_STALE_TIMEOUT",
self.tunnel_stale_timeout_secs
);
set!("AETHER_PROXY_TUNNEL_CONNECTIONS", self.tunnel_connections);
// allowed_ports needs special handling (comma-separated)
if let Some(ref ports) = self.allowed_ports {
if force || std::env::var("AETHER_PROXY_ALLOWED_PORTS").is_err() {
let s: String = ports
.iter()
.map(|p| p.to_string())
.collect::<Vec<_>>()
.join(",");
std::env::set_var("AETHER_PROXY_ALLOWED_PORTS", s);
}
}
}
}
-79
View File
@@ -1,79 +0,0 @@
use serde::Serialize;
use sysinfo::System;
use tracing::info;
/// Hardware information collected at startup.
///
/// The struct is `Serialize`-able so it can be sent directly as the
/// `hardware_info` JSON bag in the registration request. New fields
/// can be added without database schema migrations.
#[derive(Debug, Clone, Serialize)]
pub struct HardwareInfo {
pub cpu_cores: u32,
pub total_memory_mb: u64,
pub os_info: String,
pub fd_limit: u64,
#[serde(skip)]
pub estimated_max_concurrency: u64,
}
/// Collect hardware information and estimate max concurrency.
///
/// Should be called once at startup -- hardware does not change at runtime.
pub fn collect() -> HardwareInfo {
let sys = System::new_all();
let cpu_cores = sys.cpus().len() as u32;
let total_memory_mb = sys.total_memory() / (1024 * 1024);
let os_info = format!(
"{} {}",
System::name().unwrap_or_else(|| "Unknown".into()),
System::os_version().unwrap_or_default(),
)
.trim()
.to_string();
// Estimate max concurrent connections:
// - Each tokio async task uses ~8-16 KB stack + heap buffers
// - OS file descriptor limit is often the real bottleneck
// - Conservative formula: min(fd_limit - 100, ram_mb * 40, cpu_cores * 2000)
let fd_limit = get_fd_limit();
let by_fd = fd_limit.saturating_sub(100);
let by_ram = total_memory_mb.saturating_mul(40);
let by_cpu = (cpu_cores as u64).saturating_mul(2000);
let estimated_max_concurrency = by_fd.min(by_ram).min(by_cpu);
info!(
cpu_cores,
total_memory_mb,
os_info = %os_info,
fd_limit,
estimated_max_concurrency,
"hardware info collected"
);
HardwareInfo {
cpu_cores,
total_memory_mb,
os_info,
fd_limit,
estimated_max_concurrency,
}
}
/// Read the soft file-descriptor limit (RLIMIT_NOFILE).
fn get_fd_limit() -> u64 {
#[cfg(unix)]
{
let mut rlim = libc::rlimit {
rlim_cur: 0,
rlim_max: 0,
};
let ret = unsafe { libc::getrlimit(libc::RLIMIT_NOFILE, &mut rlim) };
if ret == 0 {
return rlim.rlim_cur;
}
}
// Fallback for non-unix or error
1024
}
-168
View File
@@ -1,168 +0,0 @@
mod app;
mod config;
mod hardware;
mod net;
mod registration;
mod runtime;
mod setup;
mod state;
mod target_filter;
mod tunnel;
mod upstream_client;
use std::path::PathBuf;
use clap::{CommandFactory, FromArgMatches, Parser};
use config::Config;
/// Default config file name.
const DEFAULT_CONFIG: &str = "aether-proxy.toml";
/// Build the full clap command: Config args + discoverable subcommands.
///
/// `subcommand_negates_reqs` lets subcommands bypass the required Config
/// flags so that e.g. `aether-proxy setup` doesn't demand `--aether-url`.
fn build_command() -> clap::Command {
Config::command()
.subcommand(
clap::Command::new("setup")
.about("Interactive setup wizard (TUI)")
.arg(
clap::Arg::new("config_path")
.help("Path to config file")
.default_value(DEFAULT_CONFIG),
),
)
.subcommand(clap::Command::new("start").about("Start the systemd service"))
.subcommand(clap::Command::new("status").about("Show service status"))
.subcommand(clap::Command::new("logs").about("Tail service logs"))
.subcommand(clap::Command::new("restart").about("Restart the systemd service"))
.subcommand(clap::Command::new("stop").about("Stop the systemd service"))
.subcommand(clap::Command::new("uninstall").about("Uninstall the systemd service"))
.subcommand(
clap::Command::new("upgrade")
.about("Self-upgrade from GitHub releases")
.arg(clap::Arg::new("version").help("Target version (e.g. 0.2.0)")),
)
.subcommand_negates_reqs(true)
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
rustls::crypto::ring::default_provider()
.install_default()
.map_err(|_| anyhow::anyhow!("Failed to install rustls CryptoProvider"))?;
// Load config file as env-var defaults (before clap parsing)
let config_file_path =
std::env::var("AETHER_PROXY_CONFIG").unwrap_or_else(|_| DEFAULT_CONFIG.to_string());
let config_path = std::path::Path::new(&config_file_path);
if config_path.exists() {
// Migrate legacy 0.1.x config to 0.2.0 format if needed
if let Err(e) = config::ConfigFile::migrate_legacy(config_path) {
eprintln!(" WARNING: config migration failed: {}", e);
}
if let Ok(file_cfg) = config::ConfigFile::load(config_path) {
file_cfg.inject_env();
}
}
// Parse CLI (subcommands + config args in one pass)
match build_command().try_get_matches() {
Ok(matches) => match matches.subcommand() {
Some(("setup", sub_m)) => {
let path = sub_m
.get_one::<String>("config_path")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from(DEFAULT_CONFIG));
handle_setup_result(setup::run(path)?).await
}
Some(("start", _)) => setup::service::cmd_start(),
Some(("status", _)) => setup::service::cmd_status(),
Some(("logs", _)) => setup::service::cmd_logs(),
Some(("restart", _)) => setup::service::cmd_restart(),
Some(("stop", _)) => setup::service::cmd_stop(),
Some(("uninstall", _)) => setup::service::cmd_uninstall(),
Some(("upgrade", sub_m)) => {
let version = sub_m.get_one::<String>("version").cloned();
setup::upgrade::cmd_upgrade(version).await
}
Some(_) => unreachable!(),
None => {
// No subcommand — run the proxy with parsed config.
let config = Config::from_arg_matches(&matches)?;
run_proxy(config).await
}
},
Err(e) => {
if e.kind() == clap::error::ErrorKind::MissingRequiredArgument {
eprintln!("Missing required config, launching setup wizard...\n");
handle_setup_result(setup::run(PathBuf::from(&config_file_path))?).await
} else {
e.exit();
}
}
}
}
/// Decide what to do after the setup wizard completes.
async fn handle_setup_result(outcome: setup::SetupOutcome) -> anyhow::Result<()> {
match outcome {
setup::SetupOutcome::ServiceInstalled => Ok(()),
setup::SetupOutcome::ReadyToRun(config_path) => {
// Reload config from the file that setup just wrote, overriding
// any stale env vars from a previous config.
match config::ConfigFile::load(&config_path) {
Ok(file_cfg) => file_cfg.inject_env_override(),
Err(e) => anyhow::bail!("failed to reload config after setup: {}", e),
}
// Parse from env-only (argv may still contain "setup" etc.)
let config = Config::try_parse_from(["aether-proxy"])
.map_err(|e| anyhow::anyhow!("config invalid after setup: {}", e))?;
eprintln!(" Starting proxy...\n");
run_proxy(config).await
}
setup::SetupOutcome::Cancelled => {
eprintln!(" Setup cancelled.");
Ok(())
}
}
}
/// Start the proxy server, checking for systemd conflicts first.
async fn run_proxy(config: Config) -> anyhow::Result<()> {
// Warn if systemd service is already running (would cause port conflict).
// Skip this check when we ARE the systemd service (INVOCATION_ID is set by systemd).
if std::env::var_os("INVOCATION_ID").is_none() && setup::service::is_service_active() {
eprintln!("Warning: systemd service is already running.");
eprintln!("Use `./aether-proxy stop` to stop it first, or manage via subcommands:");
eprintln!(" ./aether-proxy status / logs / restart / stop");
std::process::exit(1);
}
// Resolve server list: prefer [[servers]] from TOML, fall back to CLI/env single server.
let config_path =
std::env::var("AETHER_PROXY_CONFIG").unwrap_or_else(|_| DEFAULT_CONFIG.to_string());
let servers = if std::path::Path::new(&config_path).exists() {
config::ConfigFile::load(std::path::Path::new(&config_path))
.ok()
.map(|f| f.effective_servers())
.filter(|s| !s.is_empty())
.unwrap_or_else(|| {
vec![config::ServerEntry {
aether_url: config.aether_url.clone(),
management_token: config.management_token.clone(),
node_name: None,
}]
})
} else {
vec![config::ServerEntry {
aether_url: config.aether_url.clone(),
management_token: config.management_token.clone(),
node_name: None,
}]
};
app::run(config, servers).await
}
-257
View File
@@ -1,257 +0,0 @@
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use reqwest::{Client, StatusCode};
use serde::{Deserialize, Serialize};
use tokio::time::sleep;
use tracing::{debug, error, info};
use crate::config::Config;
use crate::hardware::HardwareInfo;
#[derive(Debug, Serialize)]
struct RegisterRequest {
name: String,
ip: String,
port: u16,
#[serde(skip_serializing_if = "Option::is_none")]
region: Option<String>,
heartbeat_interval: u64,
#[serde(skip_serializing_if = "Option::is_none")]
hardware_info: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
estimated_max_concurrency: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
proxy_metadata: Option<serde_json::Value>,
tunnel_mode: bool,
}
#[derive(Debug, Deserialize)]
pub struct RegisterResponse {
pub node_id: String,
}
/// Remote configuration pushed by the Aether management backend.
#[derive(Debug, Clone, Deserialize)]
pub struct RemoteConfig {
pub node_name: Option<String>,
pub allowed_ports: Option<Vec<u16>>,
pub log_level: Option<String>,
pub heartbeat_interval: Option<u64>,
}
#[derive(Debug, Serialize)]
struct UnregisterRequest {
node_id: String,
}
/// Aether API client for proxy node lifecycle management.
pub struct AetherClient {
http: Client,
base_url: String,
token: String,
retry_max_attempts: u32,
retry_base_delay: Duration,
retry_max_delay: Duration,
}
impl AetherClient {
pub fn new(config: &Config, aether_url: &str, management_token: &str) -> Self {
let mut builder = Client::builder()
.timeout(Duration::from_secs(config.aether_request_timeout_secs))
.connect_timeout(Duration::from_secs(config.aether_connect_timeout_secs))
.pool_max_idle_per_host(config.aether_pool_max_idle_per_host)
.pool_idle_timeout(Duration::from_secs(config.aether_pool_idle_timeout_secs))
.tcp_nodelay(config.aether_tcp_nodelay);
if config.aether_tcp_keepalive_secs > 0 {
builder =
builder.tcp_keepalive(Some(Duration::from_secs(config.aether_tcp_keepalive_secs)));
} else {
builder = builder.tcp_keepalive(None);
}
if config.aether_http2 {
builder = builder.http2_adaptive_window(true);
}
let http = builder.build().expect("failed to create HTTP client");
let retry_base_delay = Duration::from_millis(config.aether_retry_base_delay_ms);
let retry_max_delay =
Duration::from_millis(config.aether_retry_max_delay_ms).max(retry_base_delay);
Self {
http,
base_url: aether_url.trim_end_matches('/').to_string(),
token: management_token.to_string(),
retry_max_attempts: config.aether_retry_max_attempts.max(1),
retry_base_delay,
retry_max_delay,
}
}
/// Register this node with Aether (idempotent upsert by ip:port).
///
/// Returns the stable node_id assigned by Aether.
pub async fn register(
&self,
config: &Config,
node_name: &str,
public_ip: &str,
hw: Option<&HardwareInfo>,
) -> anyhow::Result<String> {
let url = format!("{}/api/admin/proxy-nodes/register", self.base_url);
let body = RegisterRequest {
name: node_name.to_string(),
ip: public_ip.to_string(),
port: 0,
region: config.node_region.clone(),
heartbeat_interval: config.heartbeat_interval,
hardware_info: hw.and_then(|h| serde_json::to_value(h).ok()),
estimated_max_concurrency: hw.map(|h| h.estimated_max_concurrency),
proxy_metadata: Some(serde_json::json!({
"version": env!("CARGO_PKG_VERSION"),
})),
tunnel_mode: true,
};
info!(
url = %url,
name = %body.name,
ip = %body.ip,
"registering with Aether"
);
let resp = self
.send_with_retry(
|| {
self.http
.post(&url)
.header("Authorization", format!("Bearer {}", self.token))
.json(&body)
},
"register",
)
.await?;
let status = resp.status();
if !status.is_success() {
let text = resp.text().await.unwrap_or_default();
anyhow::bail!("register failed (HTTP {}): {}", status, text);
}
let data: RegisterResponse = resp.json().await?;
info!(node_id = %data.node_id, "registered successfully");
Ok(data.node_id)
}
/// Unregister this node from Aether (graceful shutdown).
pub async fn unregister(&self, node_id: &str) -> anyhow::Result<()> {
let url = format!("{}/api/admin/proxy-nodes/unregister", self.base_url);
let body = UnregisterRequest {
node_id: node_id.to_string(),
};
info!(node_id = %node_id, "unregistering from Aether");
let resp = self
.send_with_retry(
|| {
self.http
.post(&url)
.header("Authorization", format!("Bearer {}", self.token))
.json(&body)
},
"unregister",
)
.await;
match resp {
Ok(r) if r.status().is_success() => {
info!(node_id = %node_id, "unregistered successfully");
Ok(())
}
Ok(r) => {
let text = r.text().await.unwrap_or_default();
error!(body = %text, "unregister failed");
anyhow::bail!("unregister failed: {}", text);
}
Err(e) => {
// Best-effort during shutdown
error!(error = %e, "unregister request failed");
anyhow::bail!("unregister request failed: {}", e);
}
}
}
async fn send_with_retry<F>(
&self,
mut make_req: F,
label: &str,
) -> Result<reqwest::Response, reqwest::Error>
where
F: FnMut() -> reqwest::RequestBuilder,
{
let mut attempt: u32 = 0;
let mut delay = self.retry_base_delay;
loop {
attempt = attempt.saturating_add(1);
let resp = make_req().send().await;
match resp {
Ok(resp) => {
if should_retry_status(resp.status()) && attempt < self.retry_max_attempts {
let sleep_for = jitter_delay(delay);
debug!(
attempt,
status = %resp.status(),
sleep_ms = sleep_for.as_millis(),
label,
"Aether request retrying"
);
sleep(sleep_for).await;
let next_delay = delay.checked_mul(2).unwrap_or(self.retry_max_delay);
delay = std::cmp::min(next_delay, self.retry_max_delay);
continue;
}
return Ok(resp);
}
Err(e) => {
if attempt < self.retry_max_attempts {
let sleep_for = jitter_delay(delay);
debug!(
attempt,
error = %e,
sleep_ms = sleep_for.as_millis(),
label,
"Aether request retrying"
);
sleep(sleep_for).await;
let next_delay = delay.checked_mul(2).unwrap_or(self.retry_max_delay);
delay = std::cmp::min(next_delay, self.retry_max_delay);
continue;
}
return Err(e);
}
}
}
}
}
fn should_retry_status(status: StatusCode) -> bool {
status.is_server_error()
|| status == StatusCode::TOO_MANY_REQUESTS
|| status == StatusCode::REQUEST_TIMEOUT
}
fn jitter_delay(base: Duration) -> Duration {
if base.is_zero() {
return base;
}
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.subsec_nanos() as u64)
.unwrap_or(0);
let jitter_ms = nanos % 100;
base + Duration::from_millis(jitter_ms)
}
-256
View File
@@ -1,256 +0,0 @@
//! Systemd service installation for aether-proxy.
//!
//! Called from the setup TUI when the user enables "Install Service".
//! The unit file points to the binary and config at their current
//! absolute paths -- no files are copied.
use std::path::Path;
use std::process::Command;
const UNIT_PATH: &str = "/etc/systemd/system/aether-proxy.service";
const SERVICE_NAME: &str = "aether-proxy";
/// Whether systemd service installation is possible (systemd present + root).
pub fn is_available() -> bool {
is_systemd_available() && is_root()
}
/// Install aether-proxy as a systemd service. Must be run as root.
pub fn install_service(config_path: &Path) -> anyhow::Result<()> {
if !is_systemd_available() {
anyhow::bail!("systemd not available");
}
if !is_root() {
anyhow::bail!("root required, use: sudo ./aether-proxy setup");
}
let exe_path = std::env::current_exe()?.canonicalize()?;
let exe_str = exe_path
.to_str()
.ok_or_else(|| anyhow::anyhow!("binary path contains invalid UTF-8"))?;
let config_abs = std::fs::canonicalize(config_path)?;
let config_str = config_abs
.to_str()
.ok_or_else(|| anyhow::anyhow!("config path contains invalid UTF-8"))?;
let working_dir = config_abs
.parent()
.unwrap_or_else(|| Path::new("/"))
.to_str()
.unwrap_or("/");
// Stop existing service if running (ignore errors)
if Path::new(UNIT_PATH).exists() {
eprintln!(" Stopping existing service...");
let _ = Command::new("systemctl")
.args(["stop", SERVICE_NAME])
.status();
}
// Write unit file
eprintln!(" Generating systemd unit file...");
eprintln!(" Binary: {}", exe_str);
eprintln!(" Config: {}", config_str);
eprintln!(" WorkDir: {}", working_dir);
let unit_content = format!(
"[Unit]\n\
Description=Aether Proxy\n\
After=network.target\n\
\n\
[Service]\n\
Type=simple\n\
WorkingDirectory={working_dir}\n\
Environment=AETHER_PROXY_CONFIG={config_str}\n\
ExecStart={exe_str}\n\
Restart=on-failure\n\
RestartSec=5\n\
LimitNOFILE=65535\n\
UMask=0077\n\
\n\
[Install]\n\
WantedBy=multi-user.target\n",
);
std::fs::write(UNIT_PATH, &unit_content)?;
// Reload and enable
eprintln!(" Enabling and starting service...");
run_cmd("systemctl", &["daemon-reload"])?;
run_cmd("systemctl", &["enable", "--now", SERVICE_NAME])?;
// Verify
eprintln!();
let output = Command::new("systemctl")
.args(["is-active", SERVICE_NAME])
.output()?;
let state = String::from_utf8_lossy(&output.stdout).trim().to_string();
if state == "active" {
eprintln!(" Service started successfully!");
} else {
eprintln!(" Service state: {} (check logs)", state);
}
eprintln!();
eprintln!(" Commands:");
eprintln!(" ./aether-proxy status # service status");
eprintln!(" ./aether-proxy logs # tail logs");
eprintln!(" sudo ./aether-proxy restart # restart");
eprintln!(" sudo ./aether-proxy stop # stop");
eprintln!(" sudo ./aether-proxy uninstall # remove service");
eprintln!();
Ok(())
}
fn is_systemd_available() -> bool {
Command::new("systemctl")
.arg("--version")
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
}
pub(crate) fn is_root() -> bool {
#[cfg(unix)]
{
unsafe { libc::geteuid() == 0 }
}
#[cfg(not(unix))]
{
false
}
}
/// Whether a systemd unit file is currently installed.
pub fn is_installed() -> bool {
Path::new(UNIT_PATH).exists()
}
/// Remove the systemd service (called from setup TUI when Install Service is toggled off).
pub fn uninstall_service() -> anyhow::Result<()> {
if !Path::new(UNIT_PATH).exists() {
return Ok(());
}
eprintln!(" Stopping and removing existing service...");
let _ = Command::new("systemctl")
.args(["disable", "--now", SERVICE_NAME])
.status();
std::fs::remove_file(UNIT_PATH)?;
eprintln!(" Removed {}", UNIT_PATH);
run_cmd("systemctl", &["daemon-reload"])?;
eprintln!(" Service uninstalled.");
eprintln!();
Ok(())
}
/// Check if the systemd service is currently active.
pub fn is_service_active() -> bool {
std::path::Path::new(UNIT_PATH).exists()
&& Command::new("systemctl")
.args(["is-active", "--quiet", SERVICE_NAME])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
}
// ── CLI subcommands (systemd wrappers) ──────────────────────────────────────
fn ensure_service_installed() -> anyhow::Result<()> {
if !std::path::Path::new(UNIT_PATH).exists() {
anyhow::bail!("service not installed, run `sudo ./aether-proxy setup` first");
}
Ok(())
}
fn ensure_root_and_service() -> anyhow::Result<()> {
ensure_service_installed()?;
if !is_root() {
anyhow::bail!("root required, use: sudo ./aether-proxy <command>");
}
Ok(())
}
/// `aether-proxy status` -- show service status.
pub fn cmd_status() -> anyhow::Result<()> {
ensure_service_installed()?;
let status = Command::new("systemctl")
.args(["status", SERVICE_NAME])
.status()?;
// systemctl status returns non-zero when inactive; that's fine
std::process::exit(status.code().unwrap_or(1));
}
/// `aether-proxy logs` -- tail service logs.
pub fn cmd_logs() -> anyhow::Result<()> {
ensure_service_installed()?;
let status = Command::new("journalctl")
.args(["-u", SERVICE_NAME, "-f", "--no-pager", "-n", "100"])
.status()?;
std::process::exit(status.code().unwrap_or(1));
}
/// `aether-proxy start` -- start the service.
pub fn cmd_start() -> anyhow::Result<()> {
ensure_root_and_service()?;
run_cmd("systemctl", &["start", SERVICE_NAME])?;
eprintln!(" Service started.");
Ok(())
}
/// `aether-proxy restart` -- restart the service.
pub fn cmd_restart() -> anyhow::Result<()> {
ensure_root_and_service()?;
run_cmd("systemctl", &["restart", SERVICE_NAME])?;
eprintln!(" Service restarted.");
Ok(())
}
/// `aether-proxy stop` -- stop the service.
pub fn cmd_stop() -> anyhow::Result<()> {
ensure_root_and_service()?;
run_cmd("systemctl", &["stop", SERVICE_NAME])?;
eprintln!(" Service stopped.");
Ok(())
}
/// `aether-proxy uninstall` -- disable and remove the systemd service.
pub fn cmd_uninstall() -> anyhow::Result<()> {
ensure_root_and_service()?;
eprintln!(" Stopping and disabling service...");
let _ = Command::new("systemctl")
.args(["disable", "--now", SERVICE_NAME])
.status();
if std::path::Path::new(UNIT_PATH).exists() {
std::fs::remove_file(UNIT_PATH)?;
eprintln!(" Removed {}", UNIT_PATH);
}
run_cmd("systemctl", &["daemon-reload"])?;
eprintln!(" Service uninstalled.");
eprintln!();
eprintln!(" Config file and TLS certs are preserved. Remove manually if needed.");
Ok(())
}
pub(crate) fn run_cmd(program: &str, args: &[&str]) -> anyhow::Result<()> {
let display = format!("{} {}", program, args.join(" "));
eprintln!(" > {}", display);
let status = Command::new(program).args(args).status()?;
if !status.success() {
anyhow::bail!("command failed: {}", display);
}
Ok(())
}
-868
View File
@@ -1,868 +0,0 @@
//! Interactive TUI for configuring aether-proxy.
//!
//! Launched via `aether-proxy setup [path]`. Presents a full-screen form
//! backed by ratatui where the user can navigate fields, edit values, and
//! save to a TOML config file. Supports multi-server configuration via
//! a tabbed interface.
use std::io;
use std::path::PathBuf;
use std::time::{Duration, Instant};
use crossterm::event::{self, Event, KeyCode, KeyEvent, KeyEventKind, KeyModifiers};
use crossterm::execute;
use crossterm::terminal::{self, EnterAlternateScreen, LeaveAlternateScreen};
use ratatui::backend::CrosstermBackend;
use ratatui::layout::{Constraint, Layout, Rect};
use ratatui::style::{Color, Modifier, Style};
use ratatui::text::{Line, Span};
use ratatui::widgets::{Block, Borders, Paragraph};
use ratatui::Frame;
use ratatui::Terminal;
use crate::config::{ConfigFile, ServerEntry};
/// Outcome of the setup wizard, returned to the caller.
pub enum SetupOutcome {
/// Config saved; systemd service installed and started.
ServiceInstalled,
/// Config saved; no service -- caller should start the proxy directly.
ReadyToRun(PathBuf),
/// User quit without saving.
Cancelled,
}
/// Column width reserved for the field label (chars).
const LABEL_WIDTH: usize = 22;
// -- Field types --------------------------------------------------------------
#[derive(Clone, Copy, PartialEq)]
enum FieldKind {
Text,
Secret,
Bool,
LogLevel,
}
struct Field {
label: &'static str,
key: &'static str,
value: String,
kind: FieldKind,
required: bool,
help: &'static str,
}
// -- Server tab ---------------------------------------------------------------
/// A single server tab's editable fields.
struct ServerTab {
fields: Vec<Field>,
}
impl ServerTab {
fn new() -> Self {
Self {
fields: vec![
Field {
label: "Aether URL",
key: "aether_url",
value: String::new(),
kind: FieldKind::Text,
required: true,
help: "Aether URL (e.g. https://aether.example.com)",
},
Field {
label: "Management Token",
key: "management_token",
value: String::new(),
kind: FieldKind::Secret,
required: true,
help: "Aether Management Token (ae_xxx)",
},
Field {
label: "Node Name",
key: "node_name",
value: "proxy-01".into(),
kind: FieldKind::Text,
required: true,
help: "Node name for identification in Aether dashboard",
},
],
}
}
fn from_entry(entry: &ServerEntry) -> Self {
let mut tab = Self::new();
tab.fields[0].value = entry.aether_url.clone();
tab.fields[1].value = entry.management_token.clone();
if let Some(ref name) = entry.node_name {
tab.fields[2].value = name.clone();
}
tab
}
}
// -- App state ----------------------------------------------------------------
#[derive(PartialEq)]
enum Mode {
Normal,
Editing,
}
struct App {
server_tabs: Vec<ServerTab>,
active_tab: usize,
global_fields: Vec<Field>,
selected: usize,
mode: Mode,
edit_buffer: String,
edit_cursor: usize,
config_path: PathBuf,
modified: bool,
message: Option<(String, Instant, bool)>,
scroll_offset: usize,
saved_once: bool,
pending_quit: bool,
confirm_delete: bool,
}
impl App {
fn new(config_path: PathBuf) -> Self {
Self {
server_tabs: vec![ServerTab::new()],
active_tab: 0,
global_fields: vec![
Field {
label: "Log Level",
key: "log_level",
value: "info".into(),
kind: FieldKind::LogLevel,
required: true,
help: "Log level -- Enter to cycle: trace / debug / info / warn / error",
},
Field {
label: "Log JSON",
key: "log_json",
value: "false".into(),
kind: FieldKind::Bool,
required: true,
help: "Output logs as JSON -- Enter to toggle",
},
Field {
label: "Install Service",
key: "install_service",
value: if super::service::is_available() {
"true"
} else {
"false"
}
.into(),
kind: FieldKind::Bool,
required: true,
help: "Install as systemd service (requires root) -- Enter to toggle",
},
],
selected: 0,
mode: Mode::Normal,
edit_buffer: String::new(),
edit_cursor: 0,
config_path,
modified: false,
message: None,
scroll_offset: 0,
saved_once: false,
pending_quit: false,
confirm_delete: false,
}
}
// -- Field accessors (unified index across server + global) ---------------
fn server_field_count(&self) -> usize {
self.server_tabs[self.active_tab].fields.len()
}
fn total_field_count(&self) -> usize {
self.server_field_count() + self.global_fields.len()
}
fn selected_field(&self) -> &Field {
let sc = self.server_field_count();
if self.selected < sc {
&self.server_tabs[self.active_tab].fields[self.selected]
} else {
&self.global_fields[self.selected - sc]
}
}
fn selected_field_mut(&mut self) -> &mut Field {
let sc = self.server_field_count();
if self.selected < sc {
&mut self.server_tabs[self.active_tab].fields[self.selected]
} else {
&mut self.global_fields[self.selected - sc]
}
}
fn clamp_selection(&mut self) {
let max = self.total_field_count();
if self.selected >= max {
self.selected = max.saturating_sub(1);
}
self.scroll_offset = 0;
self.confirm_delete = false;
}
// -- Config <-> fields -----------------------------------------------------
fn load_from_file(&mut self) {
if let Ok(cfg) = ConfigFile::load(&self.config_path) {
self.apply_config(&cfg);
}
}
fn apply_config(&mut self, cfg: &ConfigFile) {
// Global fields
for field in &mut self.global_fields {
let val: Option<String> = match field.key {
"log_level" => cfg.log_level.clone(),
"log_json" => cfg.log_json.map(|v| v.to_string()),
_ => None,
};
if let Some(v) = val {
field.value = v;
}
}
// Server tabs
let servers = cfg.effective_servers();
if servers.is_empty() {
let mut tab = ServerTab::new();
// Single-server fallback: use top-level node_name
if let Some(ref name) = cfg.node_name {
tab.fields[2].value = name.clone();
}
self.server_tabs = vec![tab];
} else {
self.server_tabs = servers.iter().map(ServerTab::from_entry).collect();
// For single-server mode, node_name might be in top-level only
if self.server_tabs.len() == 1 && self.server_tabs[0].fields[2].value.is_empty() {
if let Some(ref name) = cfg.node_name {
self.server_tabs[0].fields[2].value = name.clone();
}
}
}
self.active_tab = 0;
self.selected = 0;
self.scroll_offset = 0;
}
fn to_config(&self) -> ConfigFile {
let get_global = |key: &str| -> Option<String> {
self.global_fields
.iter()
.find(|f| f.key == key)
.map(|f| f.value.clone())
.filter(|v| !v.is_empty())
};
let get_tab = |tab: &ServerTab, key: &str| -> Option<String> {
tab.fields
.iter()
.find(|f| f.key == key)
.map(|f| f.value.clone())
.filter(|v| !v.is_empty())
};
let mut cfg = ConfigFile {
log_level: get_global("log_level"),
log_json: get_global("log_json").and_then(|v| v.parse().ok()),
..ConfigFile::default()
};
// Always write [[servers]] format; old top-level fields are read-only compat
cfg.servers = self
.server_tabs
.iter()
.map(|tab| ServerEntry {
aether_url: get_tab(tab, "aether_url").unwrap_or_default(),
management_token: get_tab(tab, "management_token").unwrap_or_default(),
node_name: get_tab(tab, "node_name"),
})
.collect();
cfg
}
fn save(&mut self) -> anyhow::Result<()> {
let cfg = self.to_config();
cfg.save(&self.config_path)?;
// Restrict config file permissions to owner-only (contains management token).
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ =
std::fs::set_permissions(&self.config_path, std::fs::Permissions::from_mode(0o600));
}
self.modified = false;
self.saved_once = true;
self.message = Some((
format!("saved to {}", self.config_path.display()),
Instant::now(),
false,
));
Ok(())
}
// -- Scrolling ---------------------------------------------------------------
fn ensure_visible(&mut self, visible_rows: usize) {
if visible_rows == 0 {
return;
}
// Account for separator line between server and global fields
let display_row = if self.selected >= self.server_field_count() {
self.selected + 1
} else {
self.selected
};
if display_row < self.scroll_offset {
self.scroll_offset = display_row;
} else if display_row >= self.scroll_offset + visible_rows {
self.scroll_offset = display_row - visible_rows + 1;
}
}
// -- Key handling -------------------------------------------------------------
/// Returns `true` when the app should exit.
fn handle_key(&mut self, key: KeyEvent) -> bool {
// Expire old messages (but keep quit-confirmation messages alive)
if let Some((_, when, _)) = &self.message {
if !self.pending_quit && !self.confirm_delete && when.elapsed() > Duration::from_secs(4)
{
self.message = None;
}
}
match self.mode {
Mode::Normal => self.handle_normal(key),
Mode::Editing => {
self.handle_edit(key);
false
}
}
}
fn handle_normal(&mut self, key: KeyEvent) -> bool {
// -- Quit handling (with unsaved-changes confirmation) -----------------
let is_quit_key = matches!(key.code, KeyCode::Char('q') | KeyCode::Esc);
if is_quit_key {
if !self.modified || self.pending_quit {
return true;
}
self.pending_quit = true;
self.confirm_delete = false;
self.message = Some((
"unsaved changes! q again to discard, ^S to save".into(),
Instant::now(),
true,
));
return false;
}
// Any other key cancels pending quit / pending delete
if self.pending_quit {
self.pending_quit = false;
self.message = None;
}
if self.confirm_delete && !matches!(key.code, KeyCode::Delete | KeyCode::Char('x')) {
self.confirm_delete = false;
self.message = None;
}
match key.code {
KeyCode::Char('s')
if key.modifiers.contains(KeyModifiers::CONTROL)
|| key.modifiers.contains(KeyModifiers::SUPER) =>
{
if let Err(e) = self.save() {
self.message = Some((format!("error: {}", e), Instant::now(), true));
}
}
KeyCode::Up | KeyCode::Char('k') => {
self.selected = self.selected.saturating_sub(1);
}
KeyCode::Down | KeyCode::Char('j') => {
if self.selected + 1 < self.total_field_count() {
self.selected += 1;
}
}
KeyCode::Home => self.selected = 0,
KeyCode::End => self.selected = self.total_field_count() - 1,
KeyCode::Enter | KeyCode::Char(' ') => {
let kind = self.selected_field().kind;
let key_str = self.selected_field().key;
let value = self.selected_field().value.clone();
match kind {
FieldKind::Bool => {
let toggled = if value == "true" { "false" } else { "true" };
if key_str == "install_service"
&& toggled == "true"
&& !super::service::is_available()
{
self.message = Some((
"requires root with systemd, use: sudo aether-proxy setup".into(),
Instant::now(),
true,
));
} else {
self.selected_field_mut().value = toggled.into();
self.modified = true;
}
}
FieldKind::LogLevel => {
const LEVELS: &[&str] = &["trace", "debug", "info", "warn", "error"];
let idx = LEVELS.iter().position(|l| *l == value).unwrap_or(2);
self.selected_field_mut().value = LEVELS[(idx + 1) % LEVELS.len()].into();
self.modified = true;
}
_ => {
self.edit_buffer = value;
self.edit_cursor = self.edit_buffer.chars().count();
self.mode = Mode::Editing;
}
}
}
// -- Tab navigation --
KeyCode::Tab => {
if self.server_tabs.len() > 1 {
self.active_tab = (self.active_tab + 1) % self.server_tabs.len();
self.clamp_selection();
}
}
KeyCode::BackTab => {
if self.server_tabs.len() > 1 {
self.active_tab = if self.active_tab == 0 {
self.server_tabs.len() - 1
} else {
self.active_tab - 1
};
self.clamp_selection();
}
}
KeyCode::Char(c @ '1'..='9') if !key.modifiers.contains(KeyModifiers::CONTROL) => {
let idx = (c as usize) - ('1' as usize);
if idx < self.server_tabs.len() && idx != self.active_tab {
self.active_tab = idx;
self.clamp_selection();
}
}
// -- Add / remove server --
KeyCode::Char('+') | KeyCode::Char('a') => {
self.server_tabs.push(ServerTab::new());
self.active_tab = self.server_tabs.len() - 1;
self.selected = 0;
self.scroll_offset = 0;
self.modified = true;
self.message = Some((
format!("added server {}", self.server_tabs.len()),
Instant::now(),
false,
));
}
KeyCode::Delete | KeyCode::Char('x') => {
if self.server_tabs.len() <= 1 {
self.message =
Some(("cannot remove the last server".into(), Instant::now(), true));
} else if self.confirm_delete {
let removed = self.active_tab + 1;
self.server_tabs.remove(self.active_tab);
self.active_tab = self.active_tab.min(self.server_tabs.len() - 1);
self.clamp_selection();
self.modified = true;
self.message =
Some((format!("server {} removed", removed), Instant::now(), false));
} else {
self.confirm_delete = true;
self.message = Some((
"press Delete/x again to remove this server".into(),
Instant::now(),
true,
));
}
}
_ => {}
}
false
}
fn handle_edit(&mut self, key: KeyEvent) {
match key.code {
KeyCode::Esc => {
self.mode = Mode::Normal;
}
KeyCode::Enter => {
if self.validate_edit() {
self.selected_field_mut().value = self.edit_buffer.clone();
self.modified = true;
self.mode = Mode::Normal;
} else {
self.message = Some(("invalid format".into(), Instant::now(), true));
}
}
KeyCode::Backspace => {
if self.edit_cursor > 0 {
self.edit_cursor -= 1;
let byte = self.char_byte_pos(self.edit_cursor);
self.edit_buffer.remove(byte);
}
}
KeyCode::Delete => {
if self.edit_cursor < self.edit_buffer.chars().count() {
let byte = self.char_byte_pos(self.edit_cursor);
self.edit_buffer.remove(byte);
}
}
KeyCode::Left => {
self.edit_cursor = self.edit_cursor.saturating_sub(1);
}
KeyCode::Right => {
let len = self.edit_buffer.chars().count();
if self.edit_cursor < len {
self.edit_cursor += 1;
}
}
KeyCode::Home => self.edit_cursor = 0,
KeyCode::End => self.edit_cursor = self.edit_buffer.chars().count(),
KeyCode::Char(c) => {
let byte = self.char_byte_pos(self.edit_cursor);
self.edit_buffer.insert(byte, c);
self.edit_cursor += 1;
}
_ => {}
}
}
fn validate_edit(&self) -> bool {
true
}
/// Byte offset of the char at `char_idx`.
fn char_byte_pos(&self, char_idx: usize) -> usize {
self.edit_buffer
.char_indices()
.nth(char_idx)
.map(|(i, _)| i)
.unwrap_or(self.edit_buffer.len())
}
}
// -- Rendering ----------------------------------------------------------------
fn ui(f: &mut Frame, app: &mut App) {
let area = f.area();
let title = if app.modified {
" Aether Proxy Setup [*] "
} else {
" Aether Proxy Setup "
};
let outer = Block::default()
.borders(Borders::ALL)
.title(title)
.title_alignment(ratatui::layout::Alignment::Center)
.border_style(Style::default().fg(Color::Cyan));
let inner = outer.inner(area);
f.render_widget(outer, area);
// Split: fields | tab bar | footer
let chunks = Layout::vertical([
Constraint::Min(1),
Constraint::Length(1),
Constraint::Length(4),
])
.split(inner);
render_fields(f, app, chunks[0]);
render_tab_bar(f, app, chunks[1]);
render_footer(f, app, chunks[2]);
}
fn render_fields(f: &mut Frame, app: &mut App, area: Rect) {
let visible = area.height as usize;
app.ensure_visible(visible);
let server_count = app.server_field_count();
let mut lines: Vec<Line> = Vec::new();
// display_row tracks the actual row index (including separator)
let mut display_row: usize = 0;
// Server fields
for i in 0..server_count {
if display_row >= app.scroll_offset && display_row < app.scroll_offset + visible {
lines.push(build_field_line(app, i, display_row));
}
display_row += 1;
}
// Separator line
if display_row >= app.scroll_offset && display_row < app.scroll_offset + visible {
lines.push(Line::from(Span::styled(
" ----------------------------------------",
Style::default().fg(Color::DarkGray),
)));
}
display_row += 1;
// Global fields
for i in 0..app.global_fields.len() {
let field_idx = server_count + i;
if display_row >= app.scroll_offset && display_row < app.scroll_offset + visible {
lines.push(build_field_line(app, field_idx, display_row));
}
display_row += 1;
}
let paragraph = Paragraph::new(lines);
f.render_widget(paragraph, area);
// Cursor position while editing
if app.mode == Mode::Editing {
let sel_display_row = if app.selected >= server_count {
app.selected + 1
} else {
app.selected
};
let row_in_view = sel_display_row.saturating_sub(app.scroll_offset);
let prefix: u16 = 3 + LABEL_WIDTH as u16 + 2;
let cx = area.x + prefix + app.edit_cursor as u16;
let cy = area.y + row_in_view as u16;
if cx < area.x + area.width && cy < area.y + area.height {
f.set_cursor_position((cx, cy));
}
}
}
fn build_field_line(app: &App, field_idx: usize, _display_row: usize) -> Line<'static> {
let sc = app.server_field_count();
let field = if field_idx < sc {
&app.server_tabs[app.active_tab].fields[field_idx]
} else {
&app.global_fields[field_idx - sc]
};
let selected = field_idx == app.selected;
let indicator = if selected { " > " } else { " " };
let label_style = if selected {
Style::default()
.fg(Color::Cyan)
.add_modifier(Modifier::BOLD)
} else {
Style::default().fg(Color::DarkGray)
};
let padded_label = format!("{:<width$}", field.label, width = LABEL_WIDTH);
let (value_text, value_style) = if app.mode == Mode::Editing && selected {
(app.edit_buffer.clone(), Style::default().fg(Color::Yellow))
} else {
field_display(field)
};
Line::from(vec![
Span::styled(indicator.to_string(), label_style),
Span::styled(padded_label, label_style),
Span::raw(" "),
Span::styled(value_text, value_style),
])
}
/// Returns (display_text, style) for a field in normal mode.
fn field_display(field: &Field) -> (String, Style) {
if field.value.is_empty() {
let text = if field.required {
"(required)".into()
} else {
"-".into()
};
let color = if field.required {
Color::Red
} else {
Color::DarkGray
};
return (text, Style::default().fg(color));
}
match field.kind {
FieldKind::Secret => (
"*".repeat(field.value.len().min(20)),
Style::default().fg(Color::White),
),
FieldKind::Bool => {
if field.value == "true" {
("[x] on".into(), Style::default().fg(Color::Green))
} else {
("[ ] off".into(), Style::default().fg(Color::DarkGray))
}
}
FieldKind::LogLevel => {
let color = match field.value.as_str() {
"trace" => Color::Magenta,
"debug" => Color::Blue,
"info" => Color::Green,
"warn" => Color::Yellow,
"error" => Color::Red,
_ => Color::White,
};
(field.value.clone(), Style::default().fg(color))
}
_ => (field.value.clone(), Style::default().fg(Color::White)),
}
}
fn render_tab_bar(f: &mut Frame, app: &App, area: Rect) {
let mut spans: Vec<Span> = Vec::new();
spans.push(Span::raw(" "));
for (i, tab) in app.server_tabs.iter().enumerate() {
let num = i + 1;
let name = tab
.fields
.iter()
.find(|f| f.key == "node_name")
.filter(|f| !f.value.is_empty())
.map(|f| f.value.clone())
.unwrap_or_else(|| format!("Server {}", num));
let label = format!(" {} {} ", num, name);
if i == app.active_tab {
spans.push(Span::styled(
label,
Style::default()
.fg(Color::Black)
.bg(Color::Cyan)
.add_modifier(Modifier::BOLD),
));
} else {
spans.push(Span::styled(label, Style::default().fg(Color::DarkGray)));
}
spans.push(Span::raw(" "));
}
spans.push(Span::styled(" + Add ", Style::default().fg(Color::Green)));
f.render_widget(Paragraph::new(Line::from(spans)), area);
}
fn render_footer(f: &mut Frame, app: &App, area: Rect) {
let help = app.selected_field().help;
let keybindings = if app.mode == Mode::Editing {
"Enter confirm Esc cancel"
} else if app.server_tabs.len() > 1 {
"j/k select Enter edit Tab switch + add x remove ^S save q quit"
} else {
"j/k select Enter edit + add server ^S save q quit"
};
let mut status_spans: Vec<Span> = vec![Span::styled(
format!(" {}", keybindings),
Style::default().fg(Color::DarkGray),
)];
if let Some((msg, _, is_err)) = &app.message {
let color = if *is_err { Color::Red } else { Color::Green };
status_spans.push(Span::raw(" "));
status_spans.push(Span::styled(msg.clone(), Style::default().fg(color)));
}
let footer_text = vec![
Line::raw(""),
Line::from(Span::styled(
format!(" {}", help),
Style::default().fg(Color::DarkGray),
)),
Line::from(status_spans),
];
let footer = Paragraph::new(footer_text).block(
Block::default()
.borders(Borders::TOP)
.border_style(Style::default().fg(Color::DarkGray)),
);
f.render_widget(footer, area);
}
// -- Entry point --------------------------------------------------------------
pub fn run(config_path: PathBuf) -> anyhow::Result<SetupOutcome> {
terminal::enable_raw_mode()?;
let mut stdout = io::stdout();
execute!(stdout, EnterAlternateScreen)?;
let backend = CrosstermBackend::new(stdout);
let mut terminal = Terminal::new(backend)?;
let mut app = App::new(config_path.clone());
app.load_from_file();
let result = event_loop(&mut terminal, &mut app);
terminal::disable_raw_mode()?;
execute!(terminal.backend_mut(), LeaveAlternateScreen)?;
terminal.show_cursor()?;
result?;
// -- Post-TUI: decide outcome ---------------------------------------------
if !app.saved_once {
return Ok(SetupOutcome::Cancelled);
}
eprintln!();
eprintln!(" Config saved to {}", config_path.display());
eprintln!();
let wants_service = app
.global_fields
.iter()
.find(|f| f.key == "install_service")
.map(|f| f.value == "true")
.unwrap_or(false);
if wants_service {
match super::service::install_service(&config_path) {
Ok(()) => return Ok(SetupOutcome::ServiceInstalled),
Err(e) => {
eprintln!(" Service install failed: {}", e);
eprintln!(" Starting proxy directly instead.\n");
}
}
} else if super::service::is_installed() {
if let Err(e) = super::service::uninstall_service() {
eprintln!(" Service uninstall failed: {}", e);
eprintln!();
}
}
Ok(SetupOutcome::ReadyToRun(config_path))
}
fn event_loop(
terminal: &mut Terminal<CrosstermBackend<io::Stdout>>,
app: &mut App,
) -> anyhow::Result<()> {
loop {
terminal.draw(|f| ui(f, app))?;
if event::poll(Duration::from_millis(200))? {
if let Event::Key(key) = event::read()? {
if key.kind == KeyEventKind::Press && app.handle_key(key) {
break;
}
}
}
}
Ok(())
}
-77
View File
@@ -1,77 +0,0 @@
//! Shared application state passed to all subsystems.
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, RwLock};
use std::time::Duration;
use crate::config::Config;
use crate::registration::client::AetherClient;
use crate::runtime::SharedDynamicConfig;
use crate::target_filter::DnsCache;
use crate::upstream_client::UpstreamClient;
/// Central application state shared across all servers/tunnels.
pub struct AppState {
pub config: Arc<Config>,
/// DNS cache for upstream target resolution (shared).
pub dns_cache: Arc<DnsCache>,
/// Hyper client for tunnel upstream requests with validated DNS and connection timing.
pub upstream_client: UpstreamClient,
/// Shared TLS config for tunnel WebSocket connections (avoids re-parsing root CAs on each reconnect).
pub tunnel_tls_config: Arc<rustls::ClientConfig>,
}
/// Per-server state: one instance per Aether server connection.
pub struct ServerContext {
/// Human-readable label for logging (e.g. "server-0").
pub server_label: String,
/// Aether server URL for this connection.
pub aether_url: String,
/// Management token for this server.
pub management_token: String,
/// Resolved node name at registration time (per-server override or global fallback).
/// After startup, the active node_name is read from `dynamic` (may be updated remotely).
#[allow(dead_code)]
pub node_name: String,
/// Node ID assigned by this Aether server.
pub node_id: Arc<RwLock<String>>,
/// API client for this server.
pub aether_client: Arc<AetherClient>,
/// Dynamic config from this server's heartbeat ACKs.
pub dynamic: SharedDynamicConfig,
/// Per-server active connection count.
pub active_connections: Arc<AtomicU64>,
/// Per-server request/latency metrics.
pub metrics: Arc<ProxyMetrics>,
}
/// Aggregate metrics for reporting to Aether.
pub struct ProxyMetrics {
pub total_requests: AtomicU64,
/// Cumulative connection-establishment latency in nanoseconds
/// (DNS + TCP/TLS + TTFB, excludes response body streaming).
pub total_latency_ns: AtomicU64,
pub failed_requests: AtomicU64,
pub dns_failures: AtomicU64,
pub stream_errors: AtomicU64,
}
impl ProxyMetrics {
pub fn new() -> Self {
Self {
total_requests: AtomicU64::new(0),
total_latency_ns: AtomicU64::new(0),
failed_requests: AtomicU64::new(0),
dns_failures: AtomicU64::new(0),
stream_errors: AtomicU64::new(0),
}
}
/// Record a completed request with its connection-establishment latency
/// (DNS + TCP/TLS + TTFB, excludes response body streaming).
pub fn record_request(&self, connect_elapsed: Duration) {
let nanos = u64::try_from(connect_elapsed.as_nanos()).unwrap_or(u64::MAX);
self.total_requests.fetch_add(1, Ordering::Release);
self.total_latency_ns.fetch_add(nanos, Ordering::Release);
}
}
-238
View File
@@ -1,238 +0,0 @@
//! WebSocket tunnel client: connect, authenticate, and run the tunnel.
use std::sync::Arc;
use std::time::Duration;
use tokio::net::TcpStream;
use tokio::sync::watch;
use tokio_tungstenite::tungstenite::client::IntoClientRequest;
use tokio_tungstenite::tungstenite::http;
use tokio_tungstenite::tungstenite::protocol::WebSocketConfig;
use tracing::{debug, info, warn};
use crate::state::{AppState, ServerContext};
use super::{dispatcher, heartbeat, writer};
/// Outcome of a tunnel session.
pub enum TunnelOutcome {
/// Graceful shutdown requested by the local process.
Shutdown,
/// Remote side disconnected or connection lost — should reconnect.
Disconnected,
}
/// Connect to Aether's WebSocket tunnel endpoint and run until disconnected.
///
/// `conn_idx` identifies which connection in the pool this is (0-based).
/// Only connection 0 sends heartbeats to avoid resetting shared metrics.
pub async fn connect_and_run(
state: &Arc<AppState>,
server: &Arc<ServerContext>,
conn_idx: usize,
shutdown: &mut watch::Receiver<bool>,
) -> Result<TunnelOutcome, anyhow::Error> {
let ws_url = build_tunnel_url(server);
info!(url = %ws_url, conn = conn_idx, "connecting tunnel");
// Build WebSocket request with auth headers
let mut request = ws_url.clone().into_client_request()?;
let headers = request.headers_mut();
headers.insert(
"Authorization",
http::HeaderValue::from_str(&format!("Bearer {}", server.management_token))?,
);
let node_id = server.node_id.read().unwrap().clone();
headers.insert("X-Node-Id", http::HeaderValue::from_str(&node_id)?);
// Use dynamic node_name (may be updated by remote config) instead of
// the static server.node_name, so that remote name changes take effect
// on the next reconnect.
let dynamic_node_name = server.dynamic.load().node_name.clone();
headers.insert(
"X-Node-Name",
http::HeaderValue::from_str(&dynamic_node_name)?,
);
// Advertise per-connection max concurrent streams so the backend can
// respect the proxy's capacity limit (backward-compatible: old backends
// ignore this header).
let max_streams = state.config.tunnel_max_streams.unwrap_or(128);
headers.insert("X-Tunnel-Max-Streams", http::HeaderValue::from(max_streams));
// Parse host:port from URL
let uri: http::Uri = ws_url.parse()?;
let host = uri
.host()
.ok_or_else(|| anyhow::anyhow!("missing host in tunnel URL"))?;
let is_tls = uri.scheme_str() == Some("wss");
let port = uri.port_u16().unwrap_or(if is_tls { 443 } else { 80 });
// TCP connect with timeout
let connect_timeout = Duration::from_secs(state.config.tunnel_connect_timeout_secs);
let tcp_stream = tokio::time::timeout(connect_timeout, TcpStream::connect((host, port)))
.await
.map_err(|_| {
anyhow::anyhow!(
"tunnel TCP connect timeout ({}s)",
connect_timeout.as_secs()
)
})??;
// Configure TCP parameters via socket2
configure_tcp_socket(&tcp_stream, state);
// WebSocket upgrade (with TLS if wss://)
let connector = if is_tls {
Some(tokio_tungstenite::Connector::Rustls(Arc::clone(
&state.tunnel_tls_config,
)))
} else {
None
};
// Match Python-side _MAX_FRAME_SIZE (64 MiB) to prevent tungstenite's
// default 16 MiB limit from rejecting large AI API payloads (multi-image
// base64 requests can exceed 16 MiB).
let ws_config = WebSocketConfig {
max_frame_size: Some(64 << 20),
max_message_size: Some(64 << 20),
..Default::default()
};
let handshake_timeout = Duration::from_secs(state.config.tunnel_connect_timeout_secs);
let (ws_stream, _response) = tokio::time::timeout(
handshake_timeout,
tokio_tungstenite::client_async_tls_with_config(
request,
tcp_stream,
Some(ws_config),
connector,
),
)
.await
.map_err(|_| {
anyhow::anyhow!(
"tunnel WebSocket handshake timeout ({}s)",
handshake_timeout.as_secs()
)
})??;
info!(
conn = conn_idx,
tcp_keepalive_secs = state.config.tunnel_tcp_keepalive_secs,
tcp_nodelay = state.config.tunnel_tcp_nodelay,
connect_timeout_secs = state.config.tunnel_connect_timeout_secs,
stale_timeout_secs = state.config.tunnel_stale_timeout_secs,
"tunnel connected"
);
// NOTE: reconnect_attempts reset is handled by the caller (mod.rs)
// based on how long the connection stayed alive.
// Split into read/write halves
let (ws_sink, ws_read) = futures_util::StreamExt::split(ws_stream);
// Spawn writer task (with WebSocket ping keepalive)
let ping_interval = Duration::from_secs(state.config.tunnel_ping_interval_secs);
let (frame_tx, mut writer_handle) = writer::spawn_writer(ws_sink, ping_interval);
// Spawn heartbeat task (only for primary connection to avoid
// resetting shared atomic metrics via swap(0))
let hb_handle = if conn_idx == 0 {
heartbeat::spawn(
Arc::clone(&state.config),
Arc::clone(server),
frame_tx.clone(),
shutdown.clone(),
)
} else {
heartbeat::spawn_noop()
};
// Run dispatcher (blocks until disconnect or shutdown).
// Also watch for writer exit — if the write half dies (e.g. the peer
// closed the connection) but the read half stays open, dispatcher would
// block forever on `ws_stream.next()`. Monitoring `writer_handle`
// ensures we detect this and trigger a reconnect promptly.
let state_clone = Arc::clone(state);
let server_clone = Arc::clone(server);
let outcome = tokio::select! {
result = dispatcher::run(state_clone, server_clone, ws_read, frame_tx.clone(), hb_handle) => {
match result {
Ok(()) => TunnelOutcome::Disconnected,
Err(e) => return Err(e),
}
}
writer_result = &mut writer_handle => {
match writer_result {
Ok(()) => warn!("writer task exited normally, triggering reconnect"),
Err(e) => {
if e.is_panic() {
tracing::error!(error = %e, "writer task panicked, triggering reconnect");
} else {
warn!(error = %e, "writer task cancelled, triggering reconnect");
}
}
}
TunnelOutcome::Disconnected
}
_ = shutdown.changed() => {
debug!("shutdown during tunnel dispatch");
TunnelOutcome::Shutdown
}
};
// Drop our sender; the writer will exit once all stream handler clones
// are also dropped (i.e. after they finish their in-flight work).
drop(frame_tx);
// Wait for the writer task to finish with a generous timeout — the
// dispatcher already waits up to 30s for stream handlers, so 35s here
// covers that plus a small margin.
// Skip if the writer already exited (the select branch that fired).
if !writer_handle.is_finished() {
let _ = tokio::time::timeout(Duration::from_secs(35), writer_handle).await;
}
info!("tunnel disconnected");
Ok(outcome)
}
/// Configure TCP keepalive and NODELAY on an established socket.
fn configure_tcp_socket(stream: &TcpStream, state: &Arc<AppState>) {
let sock_ref = socket2::SockRef::from(stream);
if state.config.tunnel_tcp_keepalive_secs > 0 {
let keepalive = socket2::TcpKeepalive::new()
.with_time(Duration::from_secs(state.config.tunnel_tcp_keepalive_secs))
.with_interval(Duration::from_secs(5));
#[cfg(not(target_os = "windows"))]
let keepalive = keepalive.with_retries(3);
if let Err(e) = sock_ref.set_tcp_keepalive(&keepalive) {
warn!(error = %e, "failed to set TCP keepalive on tunnel socket");
}
}
if state.config.tunnel_tcp_nodelay {
if let Err(e) = sock_ref.set_nodelay(true) {
warn!(error = %e, "failed to set TCP_NODELAY on tunnel socket");
}
}
}
/// Build rustls ClientConfig with system root certificates.
pub fn build_tls_config() -> rustls::ClientConfig {
let root_store =
rustls::RootCertStore::from_iter(webpki_roots::TLS_SERVER_ROOTS.iter().cloned());
rustls::ClientConfig::builder()
.with_root_certificates(root_store)
.with_no_client_auth()
}
fn build_tunnel_url(server: &ServerContext) -> String {
let base = server.aether_url.trim_end_matches('/');
let ws_base = if base.starts_with("https://") {
base.replacen("https://", "wss://", 1)
} else if base.starts_with("http://") {
base.replacen("http://", "ws://", 1)
} else {
format!("wss://{}", base)
};
format!("{}/api/internal/proxy-tunnel", ws_base)
}
-249
View File
@@ -1,249 +0,0 @@
//! Frame dispatcher: reads incoming WebSocket frames and routes them.
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use bytes::Bytes;
use futures_util::StreamExt;
use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use tokio_tungstenite::tungstenite::Message;
use tracing::{debug, error, info, warn};
use crate::state::{AppState, ServerContext};
use super::heartbeat::HeartbeatHandle;
use super::protocol::{decompress_if_gzip, Frame, MsgType, RequestMeta};
use super::stream_handler;
use super::writer::FrameSender;
/// Run the dispatcher loop, reading from the WebSocket stream.
pub async fn run<S>(
state: Arc<AppState>,
server: Arc<ServerContext>,
mut ws_stream: S,
frame_tx: FrameSender,
heartbeat: HeartbeatHandle,
) -> Result<(), anyhow::Error>
where
S: StreamExt<Item = Result<Message, tokio_tungstenite::tungstenite::Error>>
+ Unpin
+ Send
+ 'static,
{
// Active streams: stream_id -> body sender
let mut streams: HashMap<u32, mpsc::Sender<Frame>> = HashMap::new();
// Track spawned stream handlers so we can wait for them on shutdown
let mut handler_handles: Vec<JoinHandle<()>> = Vec::new();
let max_streams = state.config.tunnel_max_streams.unwrap_or(128) as usize;
let mut frames_since_cleanup: u32 = 0;
let stale_timeout = Duration::from_secs(state.config.tunnel_stale_timeout_secs);
// Track last time we received any data to detect stale connections
let mut last_data_at = tokio::time::Instant::now();
let read_err = loop {
let msg_result = tokio::select! {
msg = ws_stream.next() => {
match msg {
Some(r) => r,
None => break None,
}
}
_ = tokio::time::sleep_until(last_data_at + stale_timeout) => {
warn!(
stale_secs = stale_timeout.as_secs(),
"tunnel connection stale, no data received"
);
break None;
}
};
let msg = match msg_result {
Ok(m) => m,
Err(e) => {
error!(error = %e, "WebSocket read error");
break Some(e);
}
};
// Any successfully received message proves the connection is alive
last_data_at = tokio::time::Instant::now();
let data = match msg {
Message::Binary(data) => Bytes::from(data),
Message::Ping(_) => continue,
Message::Pong(_) => continue,
Message::Close(_) => {
info!("received WebSocket close");
break None;
}
_ => continue,
};
let frame = match Frame::decode(data) {
Ok(f) => f,
Err(e) => {
warn!(error = %e, "failed to decode frame");
continue;
}
};
match frame.msg_type {
MsgType::RequestHeaders => {
// Decompress if the frame is gzip-compressed, then parse metadata
let payload = match decompress_if_gzip(&frame) {
Ok(p) => p,
Err(e) => {
warn!(stream_id = frame.stream_id, error = %e, "frame decompress failed");
continue;
}
};
let meta: RequestMeta = match serde_json::from_slice(&payload) {
Ok(m) => m,
Err(e) => {
warn!(stream_id = frame.stream_id, error = %e, "invalid request metadata");
// Use try_send to avoid blocking the read loop
if frame_tx
.try_send(Frame::new(
frame.stream_id,
MsgType::StreamError,
0,
Bytes::from(format!("invalid request metadata: {e}")),
))
.is_err()
{
warn!(
stream_id = frame.stream_id,
"writer channel full, StreamError dropped"
);
}
continue;
}
};
if streams.len() >= max_streams {
warn!(
stream_id = frame.stream_id,
"max concurrent streams reached"
);
if frame_tx
.try_send(Frame::new(
frame.stream_id,
MsgType::StreamError,
0,
Bytes::from("max concurrent streams reached"),
))
.is_err()
{
warn!(
stream_id = frame.stream_id,
"writer channel full, StreamError dropped"
);
}
continue;
}
// Create body channel and spawn handler
let (body_tx, body_rx) = mpsc::channel::<Frame>(64);
streams.insert(frame.stream_id, body_tx);
let state_clone = Arc::clone(&state);
let server_clone = Arc::clone(&server);
let tx_clone = frame_tx.clone();
let sid = frame.stream_id;
let handle = tokio::spawn(async move {
stream_handler::handle_stream(
state_clone,
server_clone,
sid,
meta,
body_rx,
tx_clone,
)
.await;
});
handler_handles.push(handle);
debug!(stream_id = frame.stream_id, "new stream started");
}
MsgType::RequestBody => {
if let Some(tx) = streams.get(&frame.stream_id) {
let is_end = frame.is_end_stream();
let sid = frame.stream_id;
let _ = tx.send(frame).await;
if is_end {
streams.remove(&sid);
}
}
}
MsgType::StreamEnd | MsgType::StreamError => {
// Client-side cancellation or end
if let Some(tx) = streams.remove(&frame.stream_id) {
let _ = tx.send(frame).await;
}
}
MsgType::Ping => {
// Use try_send to avoid blocking the read loop when writer is congested
if frame_tx
.try_send(Frame::control(MsgType::Pong, frame.payload))
.is_err()
{
warn!("writer channel full, Pong dropped");
}
}
MsgType::HeartbeatAck => {
heartbeat.on_ack(frame.payload).await;
}
MsgType::GoAway => {
info!("received GOAWAY");
break None;
}
_ => {
debug!(msg_type = ?frame.msg_type, "ignoring unexpected frame type");
}
}
// Periodically clean up finished handles to avoid unbounded growth.
// Trigger every 64 frames OR when the count exceeds max_streams.
frames_since_cleanup += 1;
if frames_since_cleanup >= 64 || handler_handles.len() > max_streams {
handler_handles.retain(|h| !h.is_finished());
frames_since_cleanup = 0;
}
};
// Drop body senders so stream handlers waiting on body_rx will unblock
streams.clear();
// Wait for active stream handlers to finish so their frame_tx clones
// are dropped before the writer closes the sink.
drain_handlers(handler_handles).await;
match read_err {
Some(e) => Err(e.into()),
None => Ok(()),
}
}
/// Wait for all active stream handlers to finish (with a timeout).
async fn drain_handlers(handles: Vec<JoinHandle<()>>) {
if handles.is_empty() {
return;
}
let count = handles.len();
debug!(count, "waiting for active stream handlers to finish");
let _ = tokio::time::timeout(Duration::from_secs(30), async {
for h in handles {
let _ = h.await;
}
})
.await;
}
-340
View File
@@ -1,340 +0,0 @@
//! Tunnel heartbeat: sends metrics over the tunnel, processes ACKs.
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use std::time::Duration;
use std::time::SystemTime;
use std::time::UNIX_EPOCH;
use bytes::Bytes;
use tokio::sync::watch;
use tracing::{debug, info, warn};
use crate::config::Config;
use crate::registration::client::RemoteConfig;
use crate::runtime;
use crate::state::ServerContext;
use super::protocol::{Frame, MsgType};
use super::writer::FrameSender;
const CURRENT_VERSION: &str = env!("CARGO_PKG_VERSION");
static UPGRADE_IN_PROGRESS: AtomicBool = AtomicBool::new(false);
static NON_ROOT_UPGRADE_WARNED: AtomicBool = AtomicBool::new(false);
enum AckDecision {
Accept {
heartbeat_id: Option<u64>,
upgrade_to: Option<String>,
},
Ignore,
}
/// Handle for the dispatcher to forward HeartbeatAck frames.
#[derive(Clone)]
pub struct HeartbeatHandle {
ack_tx: tokio::sync::mpsc::Sender<Bytes>,
}
impl HeartbeatHandle {
pub async fn on_ack(&self, payload: Bytes) {
let _ = self.ack_tx.send(payload).await;
}
}
/// Create a no-op heartbeat handle that silently discards ACKs.
/// Used for non-primary tunnel connections (conn_idx > 0) to avoid
/// resetting shared atomic metrics via `swap(0)`.
pub fn spawn_noop() -> HeartbeatHandle {
let (ack_tx, _) = tokio::sync::mpsc::channel::<Bytes>(1);
// receiver is immediately dropped; on_ack() calls will silently fail
HeartbeatHandle { ack_tx }
}
#[derive(Debug, Clone, Copy, Default)]
struct HeartbeatSnapshot {
requests: u64,
latency_ns: u64,
failed: u64,
dns_failures: u64,
stream_errors: u64,
}
/// Spawn the heartbeat task. Returns a handle for forwarding ACKs.
pub fn spawn(
_config: Arc<Config>,
server: Arc<ServerContext>,
frame_tx: FrameSender,
mut shutdown: watch::Receiver<bool>,
) -> HeartbeatHandle {
let (ack_tx, mut ack_rx) = tokio::sync::mpsc::channel::<Bytes>(4);
tokio::spawn(async move {
// Read initial interval from dynamic config (may be updated by remote config).
let initial_interval = Duration::from_secs(server.dynamic.load().heartbeat_interval);
let mut current_interval = initial_interval;
// At most one in-flight heartbeat snapshot is tracked at a time.
// Snapshot is only cleared after receiving an ACK, which avoids losing
// interval counters when ACK/frame delivery is temporarily unstable.
let mut pending: Option<(u64, HeartbeatSnapshot)> = None;
let mut next_heartbeat_id: u64 = 1;
let heartbeat_session_id = format!(
"{}-{}",
std::process::id(),
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_nanos()
);
// Skip first immediate tick by sleeping first.
tokio::time::sleep(current_interval).await;
loop {
tokio::select! {
_ = tokio::time::sleep(current_interval) => {
let (heartbeat_id, snapshot) = if let Some((id, snap)) = pending {
(id, snap)
} else {
let snap = collect_snapshot(&server);
let id = next_heartbeat_id;
next_heartbeat_id = next_heartbeat_id.wrapping_add(1);
if next_heartbeat_id == 0 {
next_heartbeat_id = 1;
}
pending = Some((id, snap));
(id, snap)
};
let payload = build_heartbeat_payload(
&server,
&heartbeat_session_id,
heartbeat_id,
snapshot
);
let frame = Frame::control(MsgType::HeartbeatData, payload);
if frame_tx.send(frame).await.is_err() {
if let Some((_, snap)) = pending.take() {
restore_snapshot(&server, snap);
}
break; // Writer closed
}
debug!("sent heartbeat data");
// Re-read interval from dynamic config (remote config may have
// updated it since the last heartbeat).
let new_interval = Duration::from_secs(
server.dynamic.load().heartbeat_interval
);
if new_interval != current_interval {
debug!(
old_secs = current_interval.as_secs(),
new_secs = new_interval.as_secs(),
"heartbeat interval updated from dynamic config"
);
current_interval = new_interval;
}
}
Some(ack_payload) = ack_rx.recv() => {
match handle_ack(&server, &ack_payload) {
AckDecision::Accept {
heartbeat_id: ack_id,
upgrade_to,
} => {
if let Some((pending_id, _)) = pending {
match ack_id {
Some(id) if id == pending_id => {
pending = None;
}
None => {
// Backward-compatible with servers that don't echo
// heartbeat_id in ACK payload yet.
pending = None;
}
_ => {}
}
}
maybe_trigger_upgrade(upgrade_to);
}
AckDecision::Ignore => {}
}
}
_ = shutdown.changed() => {
debug!("heartbeat task shutting down");
if let Some((_, snap)) = pending.take() {
restore_snapshot(&server, snap);
}
break;
}
}
}
});
HeartbeatHandle { ack_tx }
}
fn collect_snapshot(server: &ServerContext) -> HeartbeatSnapshot {
HeartbeatSnapshot {
requests: server.metrics.total_requests.swap(0, Ordering::AcqRel),
latency_ns: server.metrics.total_latency_ns.swap(0, Ordering::AcqRel),
failed: server.metrics.failed_requests.swap(0, Ordering::AcqRel),
dns_failures: server.metrics.dns_failures.swap(0, Ordering::AcqRel),
stream_errors: server.metrics.stream_errors.swap(0, Ordering::AcqRel),
}
}
fn restore_snapshot(server: &ServerContext, snap: HeartbeatSnapshot) {
if snap.requests > 0 {
server
.metrics
.total_requests
.fetch_add(snap.requests, Ordering::Release);
}
if snap.latency_ns > 0 {
server
.metrics
.total_latency_ns
.fetch_add(snap.latency_ns, Ordering::Release);
}
if snap.failed > 0 {
server
.metrics
.failed_requests
.fetch_add(snap.failed, Ordering::Release);
}
if snap.dns_failures > 0 {
server
.metrics
.dns_failures
.fetch_add(snap.dns_failures, Ordering::Release);
}
if snap.stream_errors > 0 {
server
.metrics
.stream_errors
.fetch_add(snap.stream_errors, Ordering::Release);
}
}
fn build_heartbeat_payload(
server: &ServerContext,
heartbeat_session_id: &str,
heartbeat_id: u64,
snapshot: HeartbeatSnapshot,
) -> Bytes {
let node_id = server.node_id.read().unwrap().clone();
let avg_latency_ms = if snapshot.requests > 0 {
Some(snapshot.latency_ns as f64 / snapshot.requests as f64 / 1_000_000.0)
} else {
None
};
let payload = serde_json::json!({
"node_id": node_id,
"heartbeat_session_id": heartbeat_session_id,
"heartbeat_id": heartbeat_id,
"active_connections": server.active_connections.load(Ordering::Acquire),
"total_requests": snapshot.requests,
"avg_latency_ms": avg_latency_ms,
"failed_requests": snapshot.failed,
"dns_failures": snapshot.dns_failures,
"stream_errors": snapshot.stream_errors,
"proxy_metadata": {
"version": CURRENT_VERSION,
},
});
Bytes::from(serde_json::to_vec(&payload).unwrap_or_default())
}
fn handle_ack(server: &ServerContext, payload: &[u8]) -> AckDecision {
if payload.is_empty() {
return AckDecision::Accept {
heartbeat_id: None,
upgrade_to: None,
};
}
#[derive(serde::Deserialize)]
struct AckPayload {
#[serde(default)]
remote_config: Option<RemoteConfig>,
#[serde(default)]
config_version: u64,
#[serde(default)]
heartbeat_id: Option<u64>,
#[serde(default)]
upgrade_to: Option<String>,
}
match serde_json::from_slice::<AckPayload>(payload) {
Ok(ack) => {
if let Some(ref rc) = ack.remote_config {
runtime::apply_remote_config(&server.dynamic, rc, ack.config_version);
}
AckDecision::Accept {
heartbeat_id: ack.heartbeat_id,
upgrade_to: ack.upgrade_to.and_then(normalize_upgrade_target),
}
}
Err(e) => {
warn!(error = %e, "failed to parse heartbeat ACK");
AckDecision::Ignore
}
}
}
fn normalize_upgrade_target(raw: String) -> Option<String> {
let trimmed = raw.trim();
if trimmed.is_empty() {
return None;
}
let normalized = trimmed.strip_prefix("proxy-v").unwrap_or(trimmed);
if normalized == CURRENT_VERSION {
return None;
}
Some(normalized.to_string())
}
fn maybe_trigger_upgrade(version: Option<String>) {
let Some(target_version) = version else {
return;
};
if !crate::setup::service::is_root() {
if NON_ROOT_UPGRADE_WARNED
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
.is_ok()
{
warn!(
target_version = %target_version,
"remote upgrade skipped: root privileges are required"
);
}
return;
}
if UPGRADE_IN_PROGRESS
.compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
.is_err()
{
debug!(target_version = %target_version, "upgrade already in progress, ignoring");
return;
}
tokio::spawn(async move {
info!(target_version = %target_version, "received remote upgrade instruction");
match crate::setup::upgrade::perform_upgrade(&target_version).await {
Ok(()) => {
info!(target_version = %target_version, "remote upgrade finished");
}
Err(e) => {
warn!(
target_version = %target_version,
error = %e,
"remote upgrade failed"
);
UPGRADE_IN_PROGRESS.store(false, Ordering::Release);
}
}
});
}
-236
View File
@@ -1,236 +0,0 @@
pub mod client;
pub mod dispatcher;
pub mod heartbeat;
pub mod protocol;
pub mod stream_handler;
pub mod writer;
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use tokio::sync::watch;
use tracing::{error, info};
use crate::state::{AppState, ServerContext};
/// If a tunnel stays connected at least this long, treat the next disconnect
/// as a non-failure and reset reconnect backoff.
const STABLE_SESSION_RESET_AFTER: Duration = Duration::from_secs(30);
/// Startup staggering step per secondary connection, used to avoid
/// simultaneous bursts when a pool of tunnels starts together.
const STARTUP_STAGGER_STEP_MS: u64 = 150;
/// Upper bound for startup staggering.
const MAX_STARTUP_STAGGER_MS: u64 = 1_500;
/// Keep a tiny floor for repeated reconnects; first retry is still immediate.
const MIN_RECONNECT_DELAY_MS: u64 = 50;
/// Even under sustained failures, keep probing frequently so recovery is fast
/// once cross-border network quality improves.
const RECONNECT_PROBE_MAX_DELAY_MS: u64 = 3_000;
/// Run the tunnel mode main loop (connect, dispatch, reconnect).
///
/// `conn_idx` identifies which connection in the pool this is (0-based).
/// Only connection 0 sends heartbeats to avoid resetting shared metrics.
pub async fn run(
state: &Arc<AppState>,
server: &Arc<ServerContext>,
conn_idx: usize,
mut shutdown: watch::Receiver<bool>,
) {
info!(server = %server.server_label, conn = conn_idx, "starting tunnel");
let reconnect_salt = compute_connection_salt(server, conn_idx);
let startup_delay = compute_startup_stagger(conn_idx, reconnect_salt);
if !startup_delay.is_zero() {
info!(
server = %server.server_label,
conn = conn_idx,
delay_ms = startup_delay.as_millis(),
"startup stagger before first connect"
);
tokio::select! {
_ = tokio::time::sleep(startup_delay) => {}
_ = shutdown.changed() => {
info!(server = %server.server_label, conn = conn_idx, "shutdown requested during startup stagger");
return;
}
}
}
let mut consecutive_failures: u32 = 0;
loop {
let started_at = Instant::now();
match client::connect_and_run(state, server, conn_idx, &mut shutdown).await {
Ok(client::TunnelOutcome::Shutdown) => {
info!(server = %server.server_label, conn = conn_idx, "tunnel shut down gracefully");
return;
}
Ok(client::TunnelOutcome::Disconnected) => {
info!(server = %server.server_label, conn = conn_idx, "tunnel disconnected, reconnecting");
}
Err(e) => {
error!(server = %server.server_label, conn = conn_idx, error = %e, "tunnel connection error, reconnecting");
}
}
if *shutdown.borrow() {
info!(server = %server.server_label, conn = conn_idx, "shutdown requested, not reconnecting");
return;
}
// Reset backoff after a stable session to keep recovery snappy when
// failures are only occasional.
let connected_for = started_at.elapsed();
if connected_for >= STABLE_SESSION_RESET_AFTER {
consecutive_failures = 0;
} else {
consecutive_failures = consecutive_failures.saturating_add(1);
}
let reconnect_delay = compute_reconnect_delay(
state.config.tunnel_reconnect_base_ms,
state.config.tunnel_reconnect_max_ms,
consecutive_failures,
reconnect_salt,
);
info!(
server = %server.server_label,
conn = conn_idx,
failures = consecutive_failures,
delay_ms = reconnect_delay.as_millis(),
"waiting before reconnect"
);
tokio::select! {
_ = tokio::time::sleep(reconnect_delay) => {}
_ = shutdown.changed() => {
info!(server = %server.server_label, conn = conn_idx, "shutdown requested during reconnect wait");
return;
}
}
}
}
fn compute_connection_salt(server: &ServerContext, conn_idx: usize) -> u64 {
// FNV-1a style hash over server label + connection index.
let mut h: u64 = 0xcbf29ce484222325;
for &b in server.server_label.as_bytes() {
h ^= b as u64;
h = h.wrapping_mul(0x100000001b3);
}
h ^= conn_idx as u64;
mix_u64(h)
}
fn compute_startup_stagger(conn_idx: usize, salt: u64) -> Duration {
if conn_idx == 0 {
return Duration::ZERO;
}
let base = (conn_idx as u64).saturating_mul(STARTUP_STAGGER_STEP_MS);
let jitter = mix_u64(salt) % 301; // 0..=300ms
Duration::from_millis((base + jitter).min(MAX_STARTUP_STAGGER_MS))
}
fn compute_reconnect_delay(
base_ms: u64,
max_ms: u64,
consecutive_failures: u32,
salt: u64,
) -> Duration {
// First retry should be immediate to maximize recovery speed on transient
// blips (the user's primary expectation in poor networks).
if consecutive_failures <= 1 {
return Duration::ZERO;
}
// Keep a sane minimum for repeated failures.
let base_ms = base_ms.max(MIN_RECONNECT_DELAY_MS);
let max_ms = max_ms.max(base_ms);
let cap_ms = compute_reconnect_cap_ms(base_ms, max_ms, consecutive_failures)
.min(RECONNECT_PROBE_MAX_DELAY_MS.max(base_ms));
// Equal-jitter: randomize in [cap/2, cap], preventing synchronized reconnect
// storms while keeping reconnect latency bounded.
if cap_ms <= 1 {
return Duration::from_millis(cap_ms);
}
let half = cap_ms / 2;
let span = cap_ms - half;
let now_nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.subsec_nanos() as u64)
.unwrap_or(0);
let mixed = mix_u64(now_nanos ^ salt);
let jitter = if span == 0 { 0 } else { mixed % (span + 1) };
Duration::from_millis(half + jitter)
}
fn compute_reconnect_cap_ms(base_ms: u64, max_ms: u64, consecutive_failures: u32) -> u64 {
if consecutive_failures <= 1 {
return base_ms.min(max_ms);
}
let shift = (consecutive_failures - 1).min(31);
let factor = 1u64 << shift;
base_ms.saturating_mul(factor).min(max_ms)
}
fn mix_u64(mut x: u64) -> u64 {
// SplitMix64 finalizer - cheap bit mixing for pseudo-random jitter.
x ^= x >> 30;
x = x.wrapping_mul(0xbf58476d1ce4e5b9);
x ^= x >> 27;
x = x.wrapping_mul(0x94d049bb133111eb);
x ^ (x >> 31)
}
#[cfg(test)]
mod tests {
use std::time::Duration;
use super::{
compute_reconnect_cap_ms, compute_reconnect_delay, compute_startup_stagger,
MAX_STARTUP_STAGGER_MS, RECONNECT_PROBE_MAX_DELAY_MS, STARTUP_STAGGER_STEP_MS,
};
#[test]
fn reconnect_cap_grows_exponentially_and_caps() {
let base = 500;
let max = 30_000;
assert_eq!(compute_reconnect_cap_ms(base, max, 0), 500);
assert_eq!(compute_reconnect_cap_ms(base, max, 1), 500);
assert_eq!(compute_reconnect_cap_ms(base, max, 2), 1_000);
assert_eq!(compute_reconnect_cap_ms(base, max, 3), 2_000);
assert_eq!(compute_reconnect_cap_ms(base, max, 4), 4_000);
assert_eq!(compute_reconnect_cap_ms(base, max, 5), 8_000);
assert_eq!(compute_reconnect_cap_ms(base, max, 6), 16_000);
assert_eq!(compute_reconnect_cap_ms(base, max, 7), 30_000);
assert_eq!(compute_reconnect_cap_ms(base, max, 20), 30_000);
}
#[test]
fn startup_stagger_is_zero_for_primary_and_bounded_for_secondary() {
assert_eq!(compute_startup_stagger(0, 42), Duration::ZERO);
let d1 = compute_startup_stagger(1, 42);
let d2 = compute_startup_stagger(2, 42);
assert!(d1 >= Duration::from_millis(STARTUP_STAGGER_STEP_MS));
assert!(d1 <= Duration::from_millis(MAX_STARTUP_STAGGER_MS));
assert!(d2 >= Duration::from_millis(STARTUP_STAGGER_STEP_MS * 2));
assert!(d2 <= Duration::from_millis(MAX_STARTUP_STAGGER_MS));
}
#[test]
fn reconnect_delay_is_immediate_on_first_failure() {
assert_eq!(compute_reconnect_delay(700, 45_000, 1, 123), Duration::ZERO);
}
#[test]
fn reconnect_delay_stays_within_probe_ceiling_after_many_failures() {
let d = compute_reconnect_delay(500, 45_000, 100, 12345);
assert!(d <= Duration::from_millis(RECONNECT_PROBE_MAX_DELAY_MS));
}
}
-260
View File
@@ -1,260 +0,0 @@
//! Binary frame protocol for WebSocket tunnel multiplexing.
//!
//! Frame layout (10-byte header + variable payload):
//! ```text
//! | stream_id (4B) | msg_type (1B) | flags (1B) | payload_len (4B) | payload (NB) |
//! ```
use bytes::{Buf, BufMut, Bytes, BytesMut};
pub const HEADER_SIZE: usize = 10;
/// Frame flags.
pub mod flags {
pub const END_STREAM: u8 = 0x01;
pub const GZIP_COMPRESSED: u8 = 0x02;
}
/// Message types for the tunnel protocol.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum MsgType {
RequestHeaders = 0x01,
RequestBody = 0x02,
ResponseHeaders = 0x03,
ResponseBody = 0x04,
StreamEnd = 0x05,
StreamError = 0x06,
Ping = 0x10,
Pong = 0x11,
GoAway = 0x12,
HeartbeatData = 0x13,
HeartbeatAck = 0x14,
}
impl MsgType {
pub fn from_u8(v: u8) -> Option<Self> {
match v {
0x01 => Some(Self::RequestHeaders),
0x02 => Some(Self::RequestBody),
0x03 => Some(Self::ResponseHeaders),
0x04 => Some(Self::ResponseBody),
0x05 => Some(Self::StreamEnd),
0x06 => Some(Self::StreamError),
0x10 => Some(Self::Ping),
0x11 => Some(Self::Pong),
0x12 => Some(Self::GoAway),
0x13 => Some(Self::HeartbeatData),
0x14 => Some(Self::HeartbeatAck),
_ => None,
}
}
}
/// A single multiplexed frame.
#[derive(Debug, Clone)]
pub struct Frame {
pub stream_id: u32,
pub msg_type: MsgType,
pub flags: u8,
pub payload: Bytes,
}
impl Frame {
pub fn new(stream_id: u32, msg_type: MsgType, flags: u8, payload: impl Into<Bytes>) -> Self {
Self {
stream_id,
msg_type,
flags,
payload: payload.into(),
}
}
/// Control frame (stream_id = 0).
pub fn control(msg_type: MsgType, payload: impl Into<Bytes>) -> Self {
Self::new(0, msg_type, 0, payload)
}
pub fn is_end_stream(&self) -> bool {
self.flags & flags::END_STREAM != 0
}
pub fn is_gzip(&self) -> bool {
self.flags & flags::GZIP_COMPRESSED != 0
}
/// Encode into a binary buffer.
pub fn encode(&self) -> Bytes {
let mut buf = BytesMut::with_capacity(HEADER_SIZE + self.payload.len());
buf.put_u32(self.stream_id);
buf.put_u8(self.msg_type as u8);
buf.put_u8(self.flags);
buf.put_u32(self.payload.len() as u32);
buf.put(self.payload.clone());
buf.freeze()
}
/// Decode from a binary buffer.
pub fn decode(mut data: Bytes) -> Result<Self, ProtocolError> {
if data.len() < HEADER_SIZE {
return Err(ProtocolError::TooShort {
expected: HEADER_SIZE,
actual: data.len(),
});
}
let stream_id = data.get_u32();
let msg_type_raw = data.get_u8();
let frame_flags = data.get_u8();
let payload_len = data.get_u32() as usize;
if data.remaining() < payload_len {
return Err(ProtocolError::Incomplete {
expected: HEADER_SIZE + payload_len,
actual: HEADER_SIZE + data.remaining(),
});
}
let msg_type =
MsgType::from_u8(msg_type_raw).ok_or(ProtocolError::UnknownMsgType(msg_type_raw))?;
let payload = data.split_to(payload_len);
Ok(Self {
stream_id,
msg_type,
flags: frame_flags,
payload,
})
}
}
/// Protocol errors.
#[derive(Debug, thiserror::Error)]
pub enum ProtocolError {
#[error("frame too short: expected {expected} bytes, got {actual}")]
TooShort { expected: usize, actual: usize },
#[error("frame incomplete: expected {expected} bytes, got {actual}")]
Incomplete { expected: usize, actual: usize },
#[error("unknown message type: 0x{0:02x}")]
UnknownMsgType(u8),
}
/// JSON payload for REQUEST_HEADERS frames.
#[derive(Debug, serde::Deserialize)]
pub struct RequestMeta {
pub method: String,
pub url: String,
pub headers: std::collections::HashMap<String, String>,
#[serde(default = "default_timeout", deserialize_with = "deserialize_timeout")]
pub timeout: u64,
}
fn default_timeout() -> u64 {
60
}
fn deserialize_timeout<'de, D>(deserializer: D) -> Result<u64, D::Error>
where
D: serde::Deserializer<'de>,
{
#[derive(serde::Deserialize)]
#[serde(untagged)]
enum TimeoutValue {
Int(u64),
Float(f64),
}
match <TimeoutValue as serde::Deserialize>::deserialize(deserializer)? {
TimeoutValue::Int(v) => Ok(v),
TimeoutValue::Float(v) => {
if !v.is_finite() || v < 0.0 {
return Err(serde::de::Error::custom(
"timeout must be a non-negative finite number",
));
}
if v.fract() != 0.0 {
return Err(serde::de::Error::custom("timeout must be integer seconds"));
}
if v > (u64::MAX as f64) {
return Err(serde::de::Error::custom("timeout is too large"));
}
Ok(v as u64)
}
}
}
/// JSON payload for RESPONSE_HEADERS frames.
#[derive(Debug, serde::Serialize)]
pub struct ResponseMeta {
pub status: u16,
/// Header list preserving duplicates (e.g. multiple Set-Cookie).
pub headers: Vec<(String, String)>,
}
// ---------------------------------------------------------------------------
// Tunnel frame compression helpers
// ---------------------------------------------------------------------------
/// Minimum payload size to attempt gzip compression (bytes).
const COMPRESS_MIN_SIZE: usize = 512;
/// If the frame has the GZIP_COMPRESSED flag, decompress the payload; otherwise
/// return a clone of the raw payload bytes.
pub fn decompress_if_gzip(frame: &Frame) -> Result<Bytes, std::io::Error> {
if frame.is_gzip() {
decompress_gzip(&frame.payload)
} else {
Ok(frame.payload.clone())
}
}
/// Gzip-compress `data` if it is large enough and compression actually shrinks
/// the payload. Returns `(payload, extra_flags)` where `extra_flags` contains
/// `GZIP_COMPRESSED` when compression was applied.
pub fn compress_payload(data: Bytes) -> (Bytes, u8) {
if data.len() >= COMPRESS_MIN_SIZE {
if let Ok(compressed) = compress_gzip(&data) {
if compressed.len() < data.len() {
return (compressed, flags::GZIP_COMPRESSED);
}
}
}
(data, 0)
}
fn decompress_gzip(data: &[u8]) -> Result<Bytes, std::io::Error> {
use flate2::read::GzDecoder;
use std::io::Read;
let mut decoder = GzDecoder::new(data);
let mut buf = Vec::new();
decoder.read_to_end(&mut buf)?;
Ok(Bytes::from(buf))
}
fn compress_gzip(data: &[u8]) -> Result<Bytes, std::io::Error> {
use flate2::write::GzEncoder;
use flate2::Compression;
use std::io::Write;
let mut encoder = GzEncoder::new(Vec::new(), Compression::fast());
encoder.write_all(data)?;
let compressed = encoder.finish()?;
Ok(Bytes::from(compressed))
}
#[cfg(test)]
mod tests {
use super::RequestMeta;
#[test]
fn request_meta_accepts_integer_timeout() {
let raw = br#"{"method":"GET","url":"https://example.com","headers":{},"timeout":15}"#;
let meta: RequestMeta = serde_json::from_slice(raw).expect("parse request meta");
assert_eq!(meta.timeout, 15);
}
#[test]
fn request_meta_accepts_integer_like_float_timeout() {
let raw = br#"{"method":"GET","url":"https://example.com","headers":{},"timeout":15.0}"#;
let meta: RequestMeta = serde_json::from_slice(raw).expect("parse request meta");
assert_eq!(meta.timeout, 15);
}
}
-506
View File
@@ -1,506 +0,0 @@
//! Per-stream request handler.
//!
//! Receives request frames, executes the upstream HTTP request,
//! and sends response frames back through the writer channel.
use std::io;
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::Ordering;
use std::sync::Arc;
use std::time::{Duration, Instant};
use bytes::Bytes;
use futures_util::stream;
use futures_util::StreamExt;
use http_body_util::BodyExt;
use hyper::body::Frame as BodyFrame;
use tokio::sync::mpsc;
use tracing::{debug, warn};
use crate::state::{AppState, ServerContext};
use crate::target_filter;
use crate::upstream_client;
use super::protocol::{
compress_payload, decompress_if_gzip, flags, Frame as TunnelFrame, MsgType, RequestMeta,
ResponseMeta,
};
use super::writer::FrameSender;
/// Maximum response body chunk size per frame (32 KB).
const MAX_CHUNK_SIZE: usize = 32 * 1024;
/// Timeout for sending a single frame to the writer channel.
/// If the writer is congested (TCP backpressure), we abandon the stream
/// rather than blocking indefinitely and exhausting the stream pool.
const FRAME_SEND_TIMEOUT: Duration = Duration::from_secs(30);
/// Minimum allowed upstream request timeout (seconds).
const MIN_TIMEOUT_SECS: u64 = 5;
/// Maximum allowed upstream request timeout (seconds).
const MAX_TIMEOUT_SECS: u64 = 300;
/// Headers that must not be forwarded to upstream (hop-by-hop or security-sensitive).
///
/// `host` and `content-length` are managed by the HTTP client (reqwest/hyper):
/// - `host` → translated to `:authority` pseudo-header in HTTP/2; forwarding
/// the original `host` alongside `:authority` triggers PROTOCOL_ERROR on
/// strict H2 implementations (e.g. Google APIs).
/// - `content-length` → recalculated by hyper from the actual body; a stale
/// value from the tunnel (body may have been re-compressed) causes H2
/// PROTOCOL_ERROR when it mismatches the real frame length.
const BLOCKED_HEADERS: &[&str] = &[
"connection",
"content-length",
"host",
"keep-alive",
"proxy-authenticate",
"proxy-authorization",
"proxy-connection",
"te",
"trailer",
"transfer-encoding",
"upgrade",
];
/// Handle a single stream: receive body, execute upstream, send response.
pub async fn handle_stream(
state: Arc<AppState>,
server: Arc<ServerContext>,
stream_id: u32,
meta: RequestMeta,
body_rx: mpsc::Receiver<TunnelFrame>,
frame_tx: FrameSender,
) {
server.active_connections.fetch_add(1, Ordering::Release);
let connect_elapsed =
handle_stream_inner(&state, &server, stream_id, meta, body_rx, &frame_tx).await;
server.active_connections.fetch_sub(1, Ordering::Release);
if let Some(d) = connect_elapsed {
server.metrics.record_request(d);
}
}
/// Send a frame to the writer with a timeout. Returns false if send failed.
async fn send_frame(tx: &FrameSender, frame: TunnelFrame) -> bool {
match tokio::time::timeout(FRAME_SEND_TIMEOUT, tx.send(frame)).await {
Ok(Ok(())) => true,
Ok(Err(_)) => {
// Channel closed (writer exited)
false
}
Err(_) => {
// Timeout — writer is congested
warn!("frame send timeout (writer congested), abandoning stream");
false
}
}
}
/// Returns the connection-establishment duration (DNS + TCP/TLS + TTFB) if the
/// upstream request succeeded, or `None` if the request never reached the
/// response-headers stage.
async fn handle_stream_inner(
state: &AppState,
server: &ServerContext,
stream_id: u32,
meta: RequestMeta,
body_rx: mpsc::Receiver<TunnelFrame>,
frame_tx: &FrameSender,
) -> Option<Duration> {
// Validate target
let target_url = match url::Url::parse(&meta.url) {
Ok(u) => u,
Err(e) => {
send_error(frame_tx, stream_id, &format!("invalid URL: {e}")).await;
return None;
}
};
// Only allow http/https schemes (block file://, data://, etc.)
match target_url.scheme() {
"http" | "https" => {}
other => {
send_error(
frame_tx,
stream_id,
&format!("unsupported URL scheme: {other}"),
)
.await;
return None;
}
}
let host = match target_url.host_str() {
Some(h) => h.to_string(),
None => {
send_error(frame_tx, stream_id, "missing host in URL").await;
return None;
}
};
let port = target_url.port_or_known_default().unwrap_or(443);
// DNS + target validation (populates dns_cache for SafeDnsResolver)
let connect_start = Instant::now();
{
let allowed_ports = Arc::clone(&server.dynamic.load().allowed_ports);
if let Err(e) =
target_filter::validate_target(&host, port, &allowed_ports, &state.dns_cache).await
{
server.metrics.dns_failures.fetch_add(1, Ordering::Release);
send_error(frame_tx, stream_id, &format!("target blocked: {e}")).await;
return None;
}
}
let dns_ms = connect_start.elapsed().as_millis() as u64;
// Execute upstream request
let client = &state.upstream_client;
let timeout = Duration::from_secs(meta.timeout.clamp(MIN_TIMEOUT_SECS, MAX_TIMEOUT_SECS));
let request_body_size = Arc::new(AtomicUsize::new(0));
let request_body = build_streaming_request_body(body_rx, Arc::clone(&request_body_size));
let method: hyper::Method = meta.method.parse().unwrap_or(hyper::Method::GET);
let mut request = match hyper::Request::builder()
.method(method)
.uri(meta.url.as_str())
.body(request_body)
{
Ok(request) => request,
Err(e) => {
send_error(
frame_tx,
stream_id,
&format!("invalid upstream request: {e}"),
)
.await;
return None;
}
};
let headers = request.headers_mut();
for (k, v) in &meta.headers {
let k_lower = k.to_ascii_lowercase();
if BLOCKED_HEADERS.contains(&k_lower.as_str()) {
continue;
}
if let (Ok(name), Ok(value)) = (
hyper::header::HeaderName::from_bytes(k.as_bytes()),
hyper::header::HeaderValue::from_str(v),
) {
headers.insert(name, value);
}
}
let mut captured_connection = upstream_client::capture_connection(&mut request);
let connection_start = Instant::now();
let connection_capture = tokio::spawn(async move {
let connected = captured_connection.wait_for_connection_metadata().await;
connected
.as_ref()
.map(|_| connection_start.elapsed().as_millis() as u64)
});
let upstream_start = Instant::now();
let response = match tokio::time::timeout(timeout, client.request(request)).await {
Ok(Ok(response)) => response,
Ok(Err(e)) => {
connection_capture.abort();
server
.metrics
.failed_requests
.fetch_add(1, Ordering::Release);
let msg = if e.is_connect() {
format!("upstream connect error: {e}")
} else {
format!("upstream error: {e}")
};
send_error(frame_tx, stream_id, &msg).await;
return None;
}
Err(_) => {
connection_capture.abort();
server
.metrics
.failed_requests
.fetch_add(1, Ordering::Release);
send_error(frame_tx, stream_id, "upstream timeout").await;
return None;
}
};
// Capture connection-establishment duration (DNS + TCP/TLS + TTFB)
// before proceeding to stream the response body.
let connect_elapsed = connect_start.elapsed();
// Send RESPONSE_HEADERS
let status = response.status().as_u16();
let ttfb_ms = upstream_start.elapsed().as_millis() as u64;
// Short timeout: on connection reuse hyper may never fire the connect
// callback, so avoid blocking indefinitely.
let connection_acquire_ms =
match tokio::time::timeout(Duration::from_millis(100), connection_capture).await {
Ok(Ok(ms)) => ms,
Ok(Err(_)) => None, // JoinError (task panicked / cancelled)
Err(_) => None, // timeout -- task is detached but lightweight
};
let request_timing =
upstream_client::resolve_request_timing(&response, connection_acquire_ms, ttfb_ms);
let mut resp_headers: Vec<(String, String)> = Vec::with_capacity(response.headers().len() + 1);
for (k, v) in response.headers() {
if let Ok(vs) = v.to_str() {
resp_headers.push((k.as_str().to_string(), vs.to_string()));
}
}
let timing = serde_json::json!({
"dns_ms": dns_ms,
"connection_acquire_ms": request_timing.connection_acquire_ms,
"connection_reused": request_timing.connection_reused,
"connect_ms": request_timing.connect_ms,
"tls_ms": request_timing.tls_ms,
"ttfb_ms": ttfb_ms,
"upstream_ms": ttfb_ms,
"response_wait_ms": request_timing.response_wait_ms,
"upstream_processing_ms": request_timing.response_wait_ms,
"timing_source": "instrumented_connector",
"total_ms": connect_elapsed.as_millis() as u64,
"body_size": request_body_size.load(Ordering::Relaxed),
"mode": "tunnel",
});
resp_headers.push(("x-proxy-timing".to_string(), timing.to_string()));
let resp_meta = ResponseMeta {
status,
headers: resp_headers,
};
let meta_json: Bytes = serde_json::to_vec(&resp_meta).unwrap_or_default().into();
let (meta_payload, meta_flags) = compress_payload(meta_json);
if !send_frame(
frame_tx,
TunnelFrame::new(
stream_id,
MsgType::ResponseHeaders,
meta_flags,
meta_payload,
),
)
.await
{
return Some(connect_elapsed);
}
// Stream response body — relay upstream bytes through the tunnel.
// Apply tunnel-level frame compression for chunks that benefit from it
// (e.g. uncompressed SSE text). Already-compressed data (gzip/br from
// upstream Content-Encoding) won't shrink further and will be sent as-is
// thanks to the size check in compress_payload().
let mut stream = response.into_body().into_data_stream();
while let Some(chunk_result) = stream.next().await {
match chunk_result {
Ok(chunk) => {
if chunk.len() <= MAX_CHUNK_SIZE {
let (payload, extra_flags) = compress_payload(chunk);
if !send_frame(
frame_tx,
TunnelFrame::new(stream_id, MsgType::ResponseBody, extra_flags, payload),
)
.await
{
return Some(connect_elapsed);
}
} else {
// Split oversized chunks, compress each slice
let mut offset = 0;
while offset < chunk.len() {
let end = (offset + MAX_CHUNK_SIZE).min(chunk.len());
let slice = chunk.slice(offset..end);
let (payload, extra_flags) = compress_payload(slice);
if !send_frame(
frame_tx,
TunnelFrame::new(
stream_id,
MsgType::ResponseBody,
extra_flags,
payload,
),
)
.await
{
return Some(connect_elapsed);
}
offset = end;
}
}
}
Err(e) => {
server.metrics.stream_errors.fetch_add(1, Ordering::Release);
warn!(stream_id, error = %e, "upstream body read error");
send_error(frame_tx, stream_id, &format!("body read error: {e}")).await;
return Some(connect_elapsed);
}
}
}
// Send STREAM_END
let _ = send_frame(
frame_tx,
TunnelFrame::new(
stream_id,
MsgType::StreamEnd,
flags::END_STREAM,
Bytes::new(),
),
)
.await;
debug!(stream_id, status, "stream completed");
Some(connect_elapsed)
}
async fn send_error(tx: &FrameSender, stream_id: u32, msg: &str) {
// Error frames use best-effort delivery — don't block if writer is congested
let _ = send_frame(
tx,
TunnelFrame::new(
stream_id,
MsgType::StreamError,
0,
Bytes::from(msg.to_string()),
),
)
.await;
}
fn build_streaming_request_body(
body_rx: mpsc::Receiver<TunnelFrame>,
body_size: Arc<AtomicUsize>,
) -> upstream_client::UpstreamRequestBody {
let body_stream = stream::unfold(
(body_rx, body_size, false),
|(mut body_rx, body_size, finished)| async move {
if finished {
return None;
}
loop {
let frame = match body_rx.recv().await {
Some(frame) => frame,
None => return None,
};
match frame.msg_type {
MsgType::RequestBody => {
let end_stream = frame.is_end_stream();
let payload = match decompress_if_gzip(&frame) {
Ok(payload) => payload,
Err(error) => {
let err =
io::Error::other(format!("gzip decompress failed: {error}"));
return Some((Err(err), (body_rx, body_size, true)));
}
};
if payload.is_empty() {
if end_stream {
return None;
}
continue;
}
body_size.fetch_add(payload.len(), Ordering::Relaxed);
return Some((
Ok(BodyFrame::data(payload)),
(body_rx, body_size, end_stream),
));
}
MsgType::StreamError => {
let message = String::from_utf8(frame.payload.to_vec())
.unwrap_or_else(|_| "client cancelled request body".to_string());
return Some((Err(io::Error::other(message)), (body_rx, body_size, true)));
}
MsgType::StreamEnd => return None,
_ => continue,
}
}
},
);
upstream_client::stream_request_body(body_stream)
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn streaming_request_body_yields_chunks_and_tracks_size() {
let (tx, rx) = mpsc::channel(4);
let body_size = Arc::new(AtomicUsize::new(0));
let mut body = build_streaming_request_body(rx, Arc::clone(&body_size));
tx.send(TunnelFrame::new(
1,
MsgType::RequestBody,
0,
Bytes::from_static(b"abc"),
))
.await
.expect("send first chunk");
tx.send(TunnelFrame::new(
1,
MsgType::RequestBody,
flags::END_STREAM,
Bytes::from_static(b"def"),
))
.await
.expect("send final chunk");
drop(tx);
let first = body
.frame()
.await
.expect("first frame")
.expect("first frame ok")
.into_data()
.expect("first data frame");
let second = body
.frame()
.await
.expect("second frame")
.expect("second frame ok")
.into_data()
.expect("second data frame");
assert_eq!(first, Bytes::from_static(b"abc"));
assert_eq!(second, Bytes::from_static(b"def"));
assert!(body.frame().await.is_none());
assert_eq!(body_size.load(Ordering::Relaxed), 6);
}
#[tokio::test]
async fn streaming_request_body_surfaces_client_cancel_as_error() {
let (tx, rx) = mpsc::channel(4);
let body_size = Arc::new(AtomicUsize::new(0));
let mut body = build_streaming_request_body(rx, Arc::clone(&body_size));
tx.send(TunnelFrame::new(
1,
MsgType::StreamError,
0,
Bytes::from_static(b"client cancelled"),
))
.await
.expect("send cancel frame");
drop(tx);
let err = body
.frame()
.await
.expect("error frame present")
.expect_err("body should surface cancellation error");
assert!(err.to_string().contains("client cancelled"));
assert!(body.frame().await.is_none());
assert_eq!(body_size.load(Ordering::Relaxed), 0);
}
}
-63
View File
@@ -1,63 +0,0 @@
//! Dedicated WebSocket writer task.
//!
//! All frame writes go through an mpsc channel to a single writer task,
//! avoiding contention on the WebSocket sink. The writer also sends
//! periodic WebSocket Ping frames to keep the connection alive through
//! intermediary proxies (Nginx, Cloudflare, etc.).
use std::time::Duration;
use futures_util::SinkExt;
use tokio::sync::mpsc;
use tokio::task::JoinHandle;
use tokio_tungstenite::tungstenite::Message;
use tracing::{debug, error, trace};
use super::protocol::Frame;
/// Sender half — cloned by stream handlers and heartbeat.
pub type FrameSender = mpsc::Sender<Frame>;
/// Spawn the writer task. Returns the sender and a JoinHandle for cleanup.
///
/// `ping_interval` controls WebSocket-level Ping frequency (typically 15s).
/// This keeps the connection alive through intermediary proxies/load-balancers.
pub fn spawn_writer<S>(mut sink: S, ping_interval: Duration) -> (FrameSender, JoinHandle<()>)
where
S: SinkExt<Message, Error = tokio_tungstenite::tungstenite::Error> + Unpin + Send + 'static,
{
let (tx, mut rx) = mpsc::channel::<Frame>(256);
let handle = tokio::spawn(async move {
let mut ping_ticker = tokio::time::interval(ping_interval);
ping_ticker.tick().await; // skip first immediate tick
loop {
tokio::select! {
frame = rx.recv() => {
match frame {
Some(frame) => {
let data = frame.encode();
if let Err(e) = sink.send(Message::Binary(data.into())).await {
error!(error = %e, "failed to write frame to WebSocket");
break;
}
}
None => break, // all senders dropped
}
}
_ = ping_ticker.tick() => {
if let Err(e) = sink.send(Message::Ping(vec![])).await {
error!(error = %e, "failed to send WebSocket ping");
break;
}
trace!("sent WebSocket ping");
}
}
}
debug!("writer task exiting");
let _ = sink.close().await;
});
(tx, handle)
}
-444
View File
@@ -1,444 +0,0 @@
use std::future::Future;
use std::io;
use std::net::IpAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
use std::time::Duration;
use bytes::Bytes;
use futures_util::Stream;
use http_body_util::combinators::UnsyncBoxBody;
use http_body_util::{BodyExt, StreamBody};
use hyper::body::Frame;
use hyper::rt;
use hyper::Response;
use hyper::Uri;
pub use hyper_util::client::legacy::connect::capture_connection;
use hyper_util::client::legacy::connect::dns::Name;
use hyper_util::client::legacy::connect::{Connected, Connection, HttpConnector};
use hyper_util::client::legacy::Client;
use hyper_util::rt::{TokioExecutor, TokioIo, TokioTimer};
use rustls::pki_types::ServerName;
use rustls::ClientConfig;
use tokio::net::TcpStream;
use tokio_rustls::TlsConnector;
use tower_service::Service;
use crate::config::Config;
use crate::target_filter::{self, DnsCache};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
type PlainStream = TokioIo<TcpStream>;
type TlsStream = TokioIo<tokio_rustls::client::TlsStream<TcpStream>>;
pub type UpstreamRequestBody = UnsyncBoxBody<Bytes, io::Error>;
pub type UpstreamClient = Client<InstrumentedConnector, UpstreamRequestBody>;
pub fn stream_request_body<S>(stream: S) -> UpstreamRequestBody
where
S: Stream<Item = Result<Frame<Bytes>, io::Error>> + Send + 'static,
{
StreamBody::new(stream).boxed_unsync()
}
#[derive(Clone, Copy, Debug, Default)]
pub struct ConnectTiming {
pub connect_ms: u64,
pub tls_ms: u64,
}
#[derive(Clone, Copy, Debug, Default)]
pub struct RequestTiming {
pub connection_acquire_ms: u64,
pub connect_ms: u64,
pub tls_ms: u64,
pub response_wait_ms: u64,
pub connection_reused: bool,
}
#[derive(Clone)]
pub struct ValidatedResolver {
dns_cache: Arc<DnsCache>,
}
impl ValidatedResolver {
pub fn new(dns_cache: Arc<DnsCache>) -> Self {
Self { dns_cache }
}
}
pub struct ValidatedAddrs {
inner: std::vec::IntoIter<std::net::SocketAddr>,
}
impl Iterator for ValidatedAddrs {
type Item = std::net::SocketAddr;
fn next(&mut self) -> Option<Self::Item> {
self.inner.next()
}
}
impl Service<Name> for ValidatedResolver {
type Response = ValidatedAddrs;
type Error = io::Error;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, name: Name) -> Self::Future {
let dns_cache = Arc::clone(&self.dns_cache);
let host = name.as_str().to_string();
Box::pin(async move {
if let Some(addrs) = dns_cache.get_by_host(&host).await {
return Ok(ValidatedAddrs {
inner: (*addrs).clone().into_iter(),
});
}
let resolved = target_filter::resolve_public_addrs(&host, 0, dns_cache.as_ref())
.await
.map_err(|err| io::Error::other(err.to_string()))?;
Ok(ValidatedAddrs {
inner: resolved.into_iter(),
})
})
}
}
#[derive(Clone)]
pub struct InstrumentedConnector {
http: HttpConnector<ValidatedResolver>,
tls_config: Arc<ClientConfig>,
}
impl Service<Uri> for InstrumentedConnector {
type Response = TimedConn;
type Error = BoxError;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
self.http.poll_ready(cx).map_err(Into::into)
}
fn call(&mut self, dst: Uri) -> Self::Future {
let scheme = dst.scheme_str().map(|value| value.to_ascii_lowercase());
let tls_config = Arc::clone(&self.tls_config);
let connecting = self.http.call(dst.clone());
let connect_start = std::time::Instant::now();
Box::pin(async move {
match scheme.as_deref() {
Some("http") => {
let tcp = connecting.await.map_err(|err| Box::new(err) as BoxError)?;
let connect_ms = connect_start.elapsed().as_millis() as u64;
Ok(TimedConn::new(
MaybeHttpsStream::Http(tcp),
ConnectTiming {
connect_ms,
tls_ms: 0,
},
))
}
Some("https") => {
let server_name = resolve_server_name(&dst)?;
let tcp = connecting.await.map_err(|err| Box::new(err) as BoxError)?;
let connect_ms = connect_start.elapsed().as_millis() as u64;
let tls_start = std::time::Instant::now();
let tls_stream = TlsConnector::from(tls_config)
.connect(server_name, tcp.into_inner())
.await
.map_err(io::Error::other)?;
let tls_ms = tls_start.elapsed().as_millis() as u64;
Ok(TimedConn::new(
MaybeHttpsStream::Https(TokioIo::new(tls_stream)),
ConnectTiming { connect_ms, tls_ms },
))
}
Some(other) => Err(io::Error::other(format!("unsupported scheme {other}")).into()),
None => Err(io::Error::other("missing scheme").into()),
}
})
}
}
pub fn build_upstream_client(config: &Config, dns_cache: Arc<DnsCache>) -> UpstreamClient {
let mut http = HttpConnector::new_with_resolver(ValidatedResolver::new(dns_cache));
http.enforce_http(false);
http.set_connect_timeout(Some(Duration::from_secs(
config.upstream_connect_timeout_secs,
)));
http.set_nodelay(config.upstream_tcp_nodelay);
if config.upstream_tcp_keepalive_secs > 0 {
http.set_keepalive(Some(Duration::from_secs(
config.upstream_tcp_keepalive_secs,
)));
} else {
http.set_keepalive(None);
}
let connector = InstrumentedConnector {
http,
tls_config: build_tls_config(),
};
let mut builder = Client::builder(TokioExecutor::new());
builder.pool_max_idle_per_host(config.upstream_pool_max_idle_per_host);
builder.pool_idle_timeout(Duration::from_secs(config.upstream_pool_idle_timeout_secs));
builder.pool_timer(TokioTimer::new());
builder.build(connector)
}
pub fn resolve_request_timing<B>(
response: &Response<B>,
connection_acquire_ms: Option<u64>,
ttfb_ms: u64,
) -> RequestTiming {
let raw = response
.extensions()
.get::<ConnectTiming>()
.copied()
.unwrap_or_default();
let raw_connection_ms = raw.connect_ms.saturating_add(raw.tls_ms);
let measured_acquire_ms = connection_acquire_ms.unwrap_or(raw_connection_ms.min(ttfb_ms));
let likely_reused = measured_acquire_ms <= 5 && raw_connection_ms > 0;
let connector_matches_request = raw_connection_ms <= measured_acquire_ms.saturating_add(25);
let (connect_ms, tls_ms) = if likely_reused || !connector_matches_request {
(0, 0)
} else {
(raw.connect_ms, raw.tls_ms)
};
RequestTiming {
connection_acquire_ms: measured_acquire_ms,
connect_ms,
tls_ms,
response_wait_ms: ttfb_ms.saturating_sub(measured_acquire_ms),
connection_reused: likely_reused,
}
}
fn build_tls_config() -> Arc<ClientConfig> {
let root_store =
rustls::RootCertStore::from_iter(webpki_roots::TLS_SERVER_ROOTS.iter().cloned());
let mut config = ClientConfig::builder()
.with_root_certificates(root_store)
.with_no_client_auth();
config.alpn_protocols = vec![b"h2".to_vec(), b"http/1.1".to_vec()];
Arc::new(config)
}
fn resolve_server_name(uri: &Uri) -> Result<ServerName<'static>, BoxError> {
let host = uri.host().ok_or_else(|| io::Error::other("missing host"))?;
let host = host.trim_start_matches('[').trim_end_matches(']');
if let Ok(ip) = host.parse::<IpAddr>() {
return Ok(ServerName::from(ip));
}
Ok(ServerName::try_from(host.to_string())?)
}
pub struct TimedConn {
inner: MaybeHttpsStream,
timing: ConnectTiming,
}
impl TimedConn {
fn new(inner: MaybeHttpsStream, timing: ConnectTiming) -> Self {
Self { inner, timing }
}
}
impl Connection for TimedConn {
fn connected(&self) -> Connected {
self.inner.connected().extra(self.timing)
}
}
impl rt::Read for TimedConn {
fn poll_read(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: rt::ReadBufCursor<'_>,
) -> Poll<Result<(), io::Error>> {
Pin::new(&mut self.inner).poll_read(cx, buf)
}
}
impl rt::Write for TimedConn {
fn poll_write(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &[u8],
) -> Poll<Result<usize, io::Error>> {
Pin::new(&mut self.inner).poll_write(cx, buf)
}
fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), io::Error>> {
Pin::new(&mut self.inner).poll_flush(cx)
}
fn poll_shutdown(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Result<(), io::Error>> {
Pin::new(&mut self.inner).poll_shutdown(cx)
}
fn is_write_vectored(&self) -> bool {
self.inner.is_write_vectored()
}
fn poll_write_vectored(
mut self: Pin<&mut Self>,
cx: &mut Context<'_>,
bufs: &[std::io::IoSlice<'_>],
) -> Poll<Result<usize, io::Error>> {
Pin::new(&mut self.inner).poll_write_vectored(cx, bufs)
}
}
pub enum MaybeHttpsStream {
Http(PlainStream),
Https(TlsStream),
}
impl Connection for MaybeHttpsStream {
fn connected(&self) -> Connected {
match self {
Self::Http(stream) => stream.connected(),
Self::Https(stream) => {
let (tcp, tls) = stream.inner().get_ref();
if tls.alpn_protocol() == Some(b"h2") {
tcp.connected().negotiated_h2()
} else {
tcp.connected()
}
}
}
}
}
impl rt::Read for MaybeHttpsStream {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: rt::ReadBufCursor<'_>,
) -> Poll<Result<(), io::Error>> {
match Pin::get_mut(self) {
Self::Http(stream) => Pin::new(stream).poll_read(cx, buf),
Self::Https(stream) => Pin::new(stream).poll_read(cx, buf),
}
}
}
impl rt::Write for MaybeHttpsStream {
fn poll_write(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &[u8],
) -> Poll<Result<usize, io::Error>> {
match Pin::get_mut(self) {
Self::Http(stream) => Pin::new(stream).poll_write(cx, buf),
Self::Https(stream) => Pin::new(stream).poll_write(cx, buf),
}
}
fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), io::Error>> {
match Pin::get_mut(self) {
Self::Http(stream) => Pin::new(stream).poll_flush(cx),
Self::Https(stream) => Pin::new(stream).poll_flush(cx),
}
}
fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Result<(), io::Error>> {
match Pin::get_mut(self) {
Self::Http(stream) => Pin::new(stream).poll_shutdown(cx),
Self::Https(stream) => Pin::new(stream).poll_shutdown(cx),
}
}
fn is_write_vectored(&self) -> bool {
match self {
Self::Http(stream) => stream.is_write_vectored(),
Self::Https(stream) => stream.is_write_vectored(),
}
}
fn poll_write_vectored(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
bufs: &[std::io::IoSlice<'_>],
) -> Poll<Result<usize, io::Error>> {
match Pin::get_mut(self) {
Self::Http(stream) => Pin::new(stream).poll_write_vectored(cx, bufs),
Self::Https(stream) => Pin::new(stream).poll_write_vectored(cx, bufs),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use hyper::Response;
#[test]
fn fresh_connection_uses_connector_breakdown() {
let mut response = Response::new(());
response.extensions_mut().insert(ConnectTiming {
connect_ms: 80,
tls_ms: 40,
});
let timing = resolve_request_timing(&response, Some(125), 600);
assert_eq!(timing.connection_acquire_ms, 125);
assert_eq!(timing.connect_ms, 80);
assert_eq!(timing.tls_ms, 40);
assert_eq!(timing.response_wait_ms, 475);
assert!(!timing.connection_reused);
}
#[test]
fn reused_connection_zeroes_stale_connect_timings() {
let mut response = Response::new(());
response.extensions_mut().insert(ConnectTiming {
connect_ms: 70,
tls_ms: 30,
});
let timing = resolve_request_timing(&response, Some(0), 310);
assert_eq!(timing.connection_acquire_ms, 0);
assert_eq!(timing.connect_ms, 0);
assert_eq!(timing.tls_ms, 0);
assert_eq!(timing.response_wait_ms, 310);
assert!(timing.connection_reused);
}
#[test]
fn falls_back_to_connector_timings_when_capture_missing() {
let mut response = Response::new(());
response.extensions_mut().insert(ConnectTiming {
connect_ms: 55,
tls_ms: 25,
});
let timing = resolve_request_timing(&response, None, 400);
assert_eq!(timing.connection_acquire_ms, 80);
assert_eq!(timing.connect_ms, 55);
assert_eq!(timing.tls_ms, 25);
assert_eq!(timing.response_wait_ms, 320);
assert!(!timing.connection_reused);
}
}
-51
View File
@@ -1,51 +0,0 @@
# Alembic 配置文件
# 用于数据库版本化迁移
[alembic]
# 迁移脚本存放目录
script_location = alembic
# 模板文件
file_template = %%(year)d%%(month).2d%%(day).2d_%%(hour).2d%%(minute).2d_%%(rev)s_%%(slug)s
# 时区(用于生成迁移文件的时间戳)
timezone = UTC
# 数据库连接 URL(会被 env.py 从环境变量覆盖)
# Docker 环境中会从 DATABASE_URL 环境变量读取
sqlalchemy.url = postgresql://postgres:${DB_PASSWORD}@localhost:5432/aether
# 日志配置
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
-121
View File
@@ -1,121 +0,0 @@
"""
Alembic 环境配置
用于数据库迁移的运行时环境设置
"""
import os
import sys
from logging.config import fileConfig
from pathlib import Path
from sqlalchemy import engine_from_config, pool, text
from alembic import context
# 添加项目根目录到 Python 路径
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
# 加载 .env 文件(本地开发时需要)
try:
from dotenv import load_dotenv
env_file = Path(__file__).parent.parent / ".env"
if env_file.exists():
load_dotenv(env_file)
except ImportError:
pass
# 导入所有数据库模型(确保 Alembic 能检测到所有表)
from src.models.database import Base
# Alembic Config 对象
config = context.config
# 从环境变量获取数据库 URL
# 优先使用 DATABASE_URL,否则从 DB_PASSWORD 自动构建(与 docker compose 保持一致)
database_url = os.getenv("DATABASE_URL")
if not database_url:
db_password = os.getenv("DB_PASSWORD", "")
db_host = os.getenv("DB_HOST", "localhost")
db_port = os.getenv("DB_PORT", "5432")
db_name = os.getenv("DB_NAME", "aether")
db_user = os.getenv("DB_USER", "postgres")
database_url = f"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"
config.set_main_option("sqlalchemy.url", database_url)
# 配置日志
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# 目标元数据(包含所有表定义)
target_metadata = Base.metadata
# PostgreSQL 全局迁移锁,避免多进程并发执行 Alembic 导致竞态(重复加列/索引等)
# 使用会话级 advisory lock(pg_advisory_lock),在迁移完成后手动释放。
# ID 由 crc32("aether-alembic-migration") 拼接生成,仅需全局唯一即可。
MIGRATION_ADVISORY_LOCK_ID = 582694137405821
def run_migrations_offline() -> None:
"""
离线模式运行迁移
在离线模式下,不需要连接数据库,
只生成 SQL 脚本
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
compare_type=True, # 比较列类型变更
compare_server_default=True, # 比较默认值变更
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online() -> None:
"""
在线模式运行迁移
在线模式下,直接连接数据库执行迁移
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
try:
# 使用会话级 advisory lock(非事务级),避免干扰 Alembic 的事务管理。
# pg_advisory_lock 在会话结束时自动释放,不受 COMMIT/ROLLBACK 影响。
if connection.dialect.name == "postgresql":
connection.execute(
text("SELECT pg_advisory_lock(:lock_id)"),
{"lock_id": MIGRATION_ADVISORY_LOCK_ID},
)
connection.commit()
context.configure(
connection=connection,
target_metadata=target_metadata,
compare_type=True,
compare_server_default=True,
transaction_per_migration=True, # 每个迁移文件独立事务,完成即提交
)
with context.begin_transaction():
context.run_migrations()
except Exception:
raise
# 根据模式选择运行方式
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
-26
View File
@@ -1,26 +0,0 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
"""应用迁移:升级到新版本"""
${upgrades if upgrades else "pass"}
def downgrade() -> None:
"""回滚迁移:降级到旧版本"""
${downgrades if downgrades else "pass"}
-775
View File
@@ -1,775 +0,0 @@
"""Baseline migration - all tables consolidated
Revision ID: 20251210_baseline
Revises:
Create Date: 2024-12-10
This is the consolidated baseline migration that creates all tables from scratch.
Includes all schema changes up to circuit breaker v2.
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers
revision = "20251210_baseline"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
# Create ENUM types (with IF NOT EXISTS for idempotency)
op.execute("DO $$ BEGIN CREATE TYPE userrole AS ENUM ('admin', 'user'); EXCEPTION WHEN duplicate_object THEN NULL; END $$")
op.execute(
"DO $$ BEGIN CREATE TYPE providerbillingtype AS ENUM ('monthly_quota', 'pay_as_you_go', 'free_tier'); EXCEPTION WHEN duplicate_object THEN NULL; END $$"
)
# ==================== users ====================
op.create_table(
"users",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("email", sa.String(255), unique=True, index=True, nullable=False),
sa.Column("username", sa.String(100), unique=True, index=True, nullable=False),
sa.Column("password_hash", sa.String(255), nullable=False),
sa.Column(
"role",
postgresql.ENUM("admin", "user", name="userrole", create_type=False),
nullable=False,
server_default="user",
),
sa.Column("allowed_providers", sa.JSON, nullable=True),
sa.Column("allowed_endpoints", sa.JSON, nullable=True),
sa.Column("allowed_models", sa.JSON, nullable=True),
sa.Column("model_capability_settings", sa.JSON, nullable=True),
sa.Column("quota_usd", sa.Float, nullable=True),
sa.Column("used_usd", sa.Float, server_default="0.0"),
sa.Column("total_usd", sa.Float, server_default="0.0"),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("is_deleted", sa.Boolean, server_default="false", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True),
)
# ==================== providers ====================
op.create_table(
"providers",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("name", sa.String(100), unique=True, index=True, nullable=False),
sa.Column("display_name", sa.String(100), nullable=False),
sa.Column("description", sa.Text, nullable=True),
sa.Column("website", sa.String(500), nullable=True),
sa.Column(
"billing_type",
postgresql.ENUM(
"monthly_quota", "pay_as_you_go", "free_tier", name="providerbillingtype", create_type=False
),
nullable=False,
server_default="pay_as_you_go",
),
sa.Column("monthly_quota_usd", sa.Float, nullable=True),
sa.Column("monthly_used_usd", sa.Float, server_default="0.0"),
sa.Column("quota_reset_day", sa.Integer, server_default="30"),
sa.Column("quota_last_reset_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("quota_expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("rpm_limit", sa.Integer, nullable=True),
sa.Column("rpm_used", sa.Integer, server_default="0"),
sa.Column("rpm_reset_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("provider_priority", sa.Integer, server_default="100"),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("rate_limit", sa.Integer, nullable=True),
sa.Column("concurrent_limit", sa.Integer, nullable=True),
sa.Column("config", sa.JSON, nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== global_models ====================
op.create_table(
"global_models",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("name", sa.String(100), unique=True, index=True, nullable=False),
sa.Column("display_name", sa.String(100), nullable=False),
sa.Column("description", sa.Text, nullable=True),
sa.Column("icon_url", sa.String(500), nullable=True),
sa.Column("official_url", sa.String(500), nullable=True),
sa.Column("default_price_per_request", sa.Float, nullable=True),
sa.Column("default_tiered_pricing", sa.JSON, nullable=False),
sa.Column("default_supports_vision", sa.Boolean, server_default="false", nullable=True),
sa.Column("default_supports_function_calling", sa.Boolean, server_default="false", nullable=True),
sa.Column("default_supports_streaming", sa.Boolean, server_default="true", nullable=True),
sa.Column("default_supports_extended_thinking", sa.Boolean, server_default="false", nullable=True),
sa.Column("default_supports_image_generation", sa.Boolean, server_default="false", nullable=True),
sa.Column("supported_capabilities", sa.JSON, nullable=True),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("usage_count", sa.Integer, server_default="0", nullable=False, index=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== api_keys ====================
op.create_table(
"api_keys",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"user_id", sa.String(36), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column("key_hash", sa.String(64), unique=True, index=True, nullable=False),
sa.Column("key_encrypted", sa.Text, nullable=True),
sa.Column("name", sa.String(100), nullable=True),
sa.Column("total_requests", sa.Integer, server_default="0"),
sa.Column("total_cost_usd", sa.Float, server_default="0.0"),
sa.Column("balance_used_usd", sa.Float, server_default="0.0"),
sa.Column("current_balance_usd", sa.Float, nullable=True),
sa.Column("is_standalone", sa.Boolean, server_default="false", nullable=False),
sa.Column("allowed_providers", sa.JSON, nullable=True),
sa.Column("allowed_endpoints", sa.JSON, nullable=True),
sa.Column("allowed_api_formats", sa.JSON, nullable=True),
sa.Column("allowed_models", sa.JSON, nullable=True),
sa.Column("rate_limit", sa.Integer, server_default="100"),
sa.Column("concurrent_limit", sa.Integer, server_default="5", nullable=True),
sa.Column("force_capabilities", sa.JSON, nullable=True),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("last_used_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("auto_delete_on_expiry", sa.Boolean, server_default="false", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== provider_endpoints ====================
op.create_table(
"provider_endpoints",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"provider_id",
sa.String(36),
sa.ForeignKey("providers.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("api_format", sa.String(50), nullable=False),
sa.Column("base_url", sa.String(500), nullable=False),
sa.Column("headers", sa.JSON, nullable=True),
sa.Column("timeout", sa.Integer, server_default="300"),
sa.Column("max_retries", sa.Integer, server_default="3"),
sa.Column("max_concurrent", sa.Integer, nullable=True),
sa.Column("rate_limit", sa.Integer, nullable=True),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("custom_path", sa.String(200), nullable=True),
sa.Column("config", sa.JSON, nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("provider_id", "api_format", name="uq_provider_api_format"),
)
op.create_index(
"idx_endpoint_format_active", "provider_endpoints", ["api_format", "is_active"]
)
# ==================== models ====================
op.create_table(
"models",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"provider_id", sa.String(36), sa.ForeignKey("providers.id"), nullable=False
),
sa.Column(
"global_model_id",
sa.String(36),
sa.ForeignKey("global_models.id"),
nullable=False,
index=True,
),
sa.Column("provider_model_name", sa.String(200), nullable=False),
sa.Column("price_per_request", sa.Float, nullable=True),
sa.Column("tiered_pricing", sa.JSON, nullable=True),
sa.Column("supports_vision", sa.Boolean, nullable=True),
sa.Column("supports_function_calling", sa.Boolean, nullable=True),
sa.Column("supports_streaming", sa.Boolean, nullable=True),
sa.Column("supports_extended_thinking", sa.Boolean, nullable=True),
sa.Column("supports_image_generation", sa.Boolean, nullable=True),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("is_available", sa.Boolean, server_default="true"),
sa.Column("config", sa.JSON, nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("provider_id", "provider_model_name", name="uq_provider_model"),
)
# ==================== model_mappings ====================
op.create_table(
"model_mappings",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("source_model", sa.String(200), nullable=False, index=True),
sa.Column(
"target_global_model_id",
sa.String(36),
sa.ForeignKey("global_models.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column(
"provider_id", sa.String(36), sa.ForeignKey("providers.id"), nullable=True, index=True
),
sa.Column("mapping_type", sa.String(20), nullable=False, server_default="alias", index=True),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("source_model", "provider_id", name="uq_model_mapping_source_provider"),
)
# ==================== provider_api_keys ====================
op.create_table(
"provider_api_keys",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"endpoint_id",
sa.String(36),
sa.ForeignKey("provider_endpoints.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("api_key", sa.String(500), nullable=False),
sa.Column("name", sa.String(100), nullable=False),
sa.Column("note", sa.String(500), nullable=True),
sa.Column("rate_multiplier", sa.Float, server_default="1.0", nullable=False),
sa.Column("internal_priority", sa.Integer, server_default="50"),
sa.Column("global_priority", sa.Integer, nullable=True),
sa.Column("max_concurrent", sa.Integer, nullable=True),
sa.Column("rate_limit", sa.Integer, nullable=True),
sa.Column("daily_limit", sa.Integer, nullable=True),
sa.Column("monthly_limit", sa.Integer, nullable=True),
sa.Column("allowed_models", sa.JSON, nullable=True),
sa.Column("capabilities", sa.JSON, nullable=True),
sa.Column("learned_max_concurrent", sa.Integer, nullable=True),
sa.Column("concurrent_429_count", sa.Integer, server_default="0", nullable=False),
sa.Column("rpm_429_count", sa.Integer, server_default="0", nullable=False),
sa.Column("last_429_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_429_type", sa.String(50), nullable=True),
sa.Column("last_concurrent_peak", sa.Integer, nullable=True),
sa.Column("adjustment_history", sa.JSON, nullable=True),
# Sliding window fields (replaces high_utilization_start)
sa.Column("utilization_samples", sa.JSON, nullable=True),
sa.Column("last_probe_increase_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("health_score", sa.Float, server_default="1.0"),
sa.Column("consecutive_failures", sa.Integer, server_default="0"),
sa.Column("last_failure_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("cache_ttl_minutes", sa.Integer, server_default="5", nullable=False),
sa.Column("max_probe_interval_minutes", sa.Integer, server_default="32", nullable=False),
sa.Column("circuit_breaker_open", sa.Boolean, server_default="false", nullable=False),
sa.Column("circuit_breaker_open_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("next_probe_at", sa.DateTime(timezone=True), nullable=True),
# Circuit breaker v2 fields
sa.Column("request_results_window", sa.JSON, nullable=True),
sa.Column("half_open_until", sa.DateTime(timezone=True), nullable=True),
sa.Column("half_open_successes", sa.Integer, server_default="0", nullable=True),
sa.Column("half_open_failures", sa.Integer, server_default="0", nullable=True),
sa.Column("request_count", sa.Integer, server_default="0"),
sa.Column("success_count", sa.Integer, server_default="0"),
sa.Column("error_count", sa.Integer, server_default="0"),
sa.Column("total_response_time_ms", sa.Integer, server_default="0"),
sa.Column("last_used_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error_msg", sa.Text, nullable=True),
sa.Column("is_active", sa.Boolean, server_default="true", nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== usage ====================
op.create_table(
"usage",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"user_id",
sa.String(36),
sa.ForeignKey("users.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column(
"api_key_id",
sa.String(36),
sa.ForeignKey("api_keys.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column("request_id", sa.String(100), unique=True, index=True, nullable=False),
sa.Column("provider", sa.String(100), nullable=False),
sa.Column("model", sa.String(100), nullable=False),
sa.Column("target_model", sa.String(100), nullable=True),
sa.Column(
"provider_id",
sa.String(36),
sa.ForeignKey("providers.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column(
"provider_endpoint_id",
sa.String(36),
sa.ForeignKey("provider_endpoints.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column(
"provider_api_key_id",
sa.String(36),
sa.ForeignKey("provider_api_keys.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column("input_tokens", sa.Integer, server_default="0"),
sa.Column("output_tokens", sa.Integer, server_default="0"),
sa.Column("total_tokens", sa.Integer, server_default="0"),
sa.Column("cache_creation_input_tokens", sa.Integer, server_default="0"),
sa.Column("cache_read_input_tokens", sa.Integer, server_default="0"),
sa.Column("input_cost_usd", sa.Float, server_default="0.0"),
sa.Column("output_cost_usd", sa.Float, server_default="0.0"),
sa.Column("cache_cost_usd", sa.Float, server_default="0.0"),
sa.Column("cache_creation_cost_usd", sa.Float, server_default="0.0"),
sa.Column("cache_read_cost_usd", sa.Float, server_default="0.0"),
sa.Column("request_cost_usd", sa.Float, server_default="0.0"),
sa.Column("total_cost_usd", sa.Float, server_default="0.0"),
sa.Column("actual_input_cost_usd", sa.Float, server_default="0.0"),
sa.Column("actual_output_cost_usd", sa.Float, server_default="0.0"),
sa.Column("actual_cache_creation_cost_usd", sa.Float, server_default="0.0"),
sa.Column("actual_cache_read_cost_usd", sa.Float, server_default="0.0"),
sa.Column("actual_request_cost_usd", sa.Float, server_default="0.0"),
sa.Column("actual_total_cost_usd", sa.Float, server_default="0.0"),
sa.Column("rate_multiplier", sa.Float, server_default="1.0"),
sa.Column("input_price_per_1m", sa.Float, nullable=True),
sa.Column("output_price_per_1m", sa.Float, nullable=True),
sa.Column("cache_creation_price_per_1m", sa.Float, nullable=True),
sa.Column("cache_read_price_per_1m", sa.Float, nullable=True),
sa.Column("price_per_request", sa.Float, nullable=True),
sa.Column("request_type", sa.String(50), nullable=True),
sa.Column("api_format", sa.String(50), nullable=True),
sa.Column("is_stream", sa.Boolean, server_default="false"),
sa.Column("status_code", sa.Integer, nullable=True),
sa.Column("error_message", sa.Text, nullable=True),
sa.Column("response_time_ms", sa.Integer, nullable=True),
sa.Column("status", sa.String(20), server_default="completed", nullable=False, index=True),
sa.Column("request_headers", sa.JSON, nullable=True),
sa.Column("request_body", sa.JSON, nullable=True),
sa.Column("provider_request_headers", sa.JSON, nullable=True),
sa.Column("response_headers", sa.JSON, nullable=True),
sa.Column("response_body", sa.JSON, nullable=True),
sa.Column("request_body_compressed", sa.LargeBinary, nullable=True),
sa.Column("response_body_compressed", sa.LargeBinary, nullable=True),
sa.Column("request_metadata", sa.JSON, nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
index=True,
),
)
# usage 表复合索引(优化常见查询)
op.create_index("idx_usage_user_created", "usage", ["user_id", "created_at"])
op.create_index("idx_usage_apikey_created", "usage", ["api_key_id", "created_at"])
op.create_index("idx_usage_provider_model_created", "usage", ["provider", "model", "created_at"])
# ==================== user_quotas ====================
op.create_table(
"user_quotas",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"user_id", sa.String(36), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column("quota_type", sa.String(50), nullable=False),
sa.Column("quota_usd", sa.Float, nullable=False),
sa.Column("period_start", sa.DateTime(timezone=True), nullable=False),
sa.Column("period_end", sa.DateTime(timezone=True), nullable=False),
sa.Column("used_usd", sa.Float, server_default="0.0"),
sa.Column("is_active", sa.Boolean, server_default="true"),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== system_configs ====================
op.create_table(
"system_configs",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("key", sa.String(100), unique=True, nullable=False),
sa.Column("value", sa.JSON, nullable=False),
sa.Column("description", sa.Text, nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== user_preferences ====================
op.create_table(
"user_preferences",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"user_id",
sa.String(36),
sa.ForeignKey("users.id", ondelete="CASCADE"),
unique=True,
nullable=False,
),
sa.Column("avatar_url", sa.String(500), nullable=True),
sa.Column("bio", sa.Text, nullable=True),
sa.Column(
"default_provider_id", sa.String(36), sa.ForeignKey("providers.id"), nullable=True
),
sa.Column("theme", sa.String(20), server_default="light"),
sa.Column("language", sa.String(10), server_default="zh-CN"),
sa.Column("timezone", sa.String(50), server_default="Asia/Shanghai"),
sa.Column("email_notifications", sa.Boolean, server_default="true"),
sa.Column("usage_alerts", sa.Boolean, server_default="true"),
sa.Column("announcement_notifications", sa.Boolean, server_default="true"),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== announcements ====================
op.create_table(
"announcements",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("title", sa.String(200), nullable=False),
sa.Column("content", sa.Text, nullable=False),
sa.Column("type", sa.String(20), server_default="info"),
sa.Column("priority", sa.Integer, server_default="0"),
sa.Column(
"author_id",
sa.String(36),
sa.ForeignKey("users.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column("is_active", sa.Boolean, server_default="true", index=True),
sa.Column("is_pinned", sa.Boolean, server_default="false"),
sa.Column("start_time", sa.DateTime(timezone=True), nullable=True),
sa.Column("end_time", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
index=True,
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== announcement_reads ====================
op.create_table(
"announcement_reads",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"user_id", sa.String(36), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column(
"announcement_id", sa.String(36), sa.ForeignKey("announcements.id"), nullable=False
),
sa.Column(
"read_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "announcement_id", name="uq_user_announcement"),
)
# ==================== audit_logs ====================
op.create_table(
"audit_logs",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column("event_type", sa.String(50), nullable=False, index=True),
sa.Column(
"user_id",
sa.String(36),
sa.ForeignKey("users.id", ondelete="SET NULL"),
nullable=True,
index=True,
),
sa.Column("api_key_id", sa.String(36), nullable=True),
sa.Column("description", sa.Text, nullable=False),
sa.Column("ip_address", sa.String(45), nullable=True),
sa.Column("user_agent", sa.String(500), nullable=True),
sa.Column("request_id", sa.String(100), nullable=True, index=True),
sa.Column("event_metadata", sa.JSON, nullable=True),
sa.Column("status_code", sa.Integer, nullable=True),
sa.Column("error_message", sa.Text, nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
index=True,
),
)
# ==================== request_candidates ====================
op.create_table(
"request_candidates",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("request_id", sa.String(100), nullable=False, index=True),
sa.Column(
"user_id", sa.String(36), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=True
),
sa.Column(
"api_key_id",
sa.String(36),
sa.ForeignKey("api_keys.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column("candidate_index", sa.Integer, nullable=False),
sa.Column("retry_index", sa.Integer, nullable=False, server_default="0"),
sa.Column(
"provider_id",
sa.String(36),
sa.ForeignKey("providers.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column(
"endpoint_id",
sa.String(36),
sa.ForeignKey("provider_endpoints.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column(
"key_id",
sa.String(36),
sa.ForeignKey("provider_api_keys.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column("status", sa.String(20), nullable=False),
sa.Column("skip_reason", sa.Text, nullable=True),
sa.Column("is_cached", sa.Boolean, server_default="false"),
sa.Column("status_code", sa.Integer, nullable=True),
sa.Column("error_type", sa.String(50), nullable=True),
sa.Column("error_message", sa.Text, nullable=True),
sa.Column("latency_ms", sa.Integer, nullable=True),
sa.Column("concurrent_requests", sa.Integer, nullable=True),
sa.Column("extra_data", sa.JSON, nullable=True),
sa.Column("required_capabilities", sa.JSON, nullable=True),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.UniqueConstraint(
"request_id", "candidate_index", "retry_index", name="uq_request_candidate_with_retry"
),
)
op.create_index("idx_request_candidates_request_id", "request_candidates", ["request_id"])
op.create_index("idx_request_candidates_status", "request_candidates", ["status"])
op.create_index("idx_request_candidates_provider_id", "request_candidates", ["provider_id"])
# ==================== stats_daily ====================
op.create_table(
"stats_daily",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("date", sa.DateTime(timezone=True), nullable=False, unique=True, index=True),
sa.Column("total_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("success_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("error_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("input_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("output_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("cache_creation_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("cache_read_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("total_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("actual_total_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("input_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("output_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("cache_creation_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("cache_read_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("avg_response_time_ms", sa.Float, server_default="0.0", nullable=False),
sa.Column("fallback_count", sa.Integer, server_default="0", nullable=False),
sa.Column("unique_models", sa.Integer, server_default="0", nullable=False),
sa.Column("unique_providers", sa.Integer, server_default="0", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== stats_summary ====================
op.create_table(
"stats_summary",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("cutoff_date", sa.DateTime(timezone=True), nullable=False),
sa.Column("all_time_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("all_time_success_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("all_time_error_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("all_time_input_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("all_time_output_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column(
"all_time_cache_creation_tokens", sa.BigInteger, server_default="0", nullable=False
),
sa.Column("all_time_cache_read_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("all_time_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("all_time_actual_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column("total_users", sa.Integer, server_default="0", nullable=False),
sa.Column("active_users", sa.Integer, server_default="0", nullable=False),
sa.Column("total_api_keys", sa.Integer, server_default="0", nullable=False),
sa.Column("active_api_keys", sa.Integer, server_default="0", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
# ==================== stats_user_daily ====================
op.create_table(
"stats_user_daily",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column(
"user_id", sa.String(36), sa.ForeignKey("users.id", ondelete="CASCADE"), nullable=False
),
sa.Column("date", sa.DateTime(timezone=True), nullable=False, index=True),
sa.Column("total_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("success_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("error_requests", sa.Integer, server_default="0", nullable=False),
sa.Column("input_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("output_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("cache_creation_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("cache_read_tokens", sa.BigInteger, server_default="0", nullable=False),
sa.Column("total_cost", sa.Float, server_default="0.0", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("user_id", "date", name="uq_stats_user_daily"),
)
op.create_index("idx_stats_user_daily_user_date", "stats_user_daily", ["user_id", "date"])
# ==================== api_key_provider_mappings ====================
op.create_table(
"api_key_provider_mappings",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"api_key_id",
sa.String(36),
sa.ForeignKey("api_keys.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column(
"provider_id",
sa.String(36),
sa.ForeignKey("providers.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("priority_adjustment", sa.Integer, server_default="0"),
sa.Column("weight_multiplier", sa.Float, server_default="1.0"),
sa.Column("is_enabled", sa.Boolean, server_default="true", nullable=False),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.UniqueConstraint("api_key_id", "provider_id", name="uq_apikey_provider"),
)
op.create_index(
"idx_apikey_provider_enabled", "api_key_provider_mappings", ["api_key_id", "is_enabled"]
)
# ==================== provider_usage_tracking ====================
op.create_table(
"provider_usage_tracking",
sa.Column("id", sa.String(36), primary_key=True, index=True),
sa.Column(
"provider_id",
sa.String(36),
sa.ForeignKey("providers.id", ondelete="CASCADE"),
nullable=False,
index=True,
),
sa.Column("window_start", sa.DateTime(timezone=True), nullable=False, index=True),
sa.Column("window_end", sa.DateTime(timezone=True), nullable=False),
sa.Column("total_requests", sa.Integer, server_default="0"),
sa.Column("successful_requests", sa.Integer, server_default="0"),
sa.Column("failed_requests", sa.Integer, server_default="0"),
sa.Column("avg_response_time_ms", sa.Float, server_default="0.0"),
sa.Column("total_response_time_ms", sa.Float, server_default="0.0"),
sa.Column("total_cost_usd", sa.Float, server_default="0.0"),
sa.Column(
"created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
sa.Column(
"updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False
),
)
op.create_index(
"idx_provider_window", "provider_usage_tracking", ["provider_id", "window_start"]
)
op.create_index("idx_window_time", "provider_usage_tracking", ["window_start", "window_end"])
def downgrade() -> None:
# Drop tables in reverse order (respecting foreign key dependencies)
op.drop_table("provider_usage_tracking")
op.drop_table("api_key_provider_mappings")
op.drop_table("stats_user_daily")
op.drop_table("stats_summary")
op.drop_table("stats_daily")
op.drop_table("request_candidates")
op.drop_table("audit_logs")
op.drop_table("announcement_reads")
op.drop_table("announcements")
op.drop_table("user_preferences")
op.drop_table("system_configs")
op.drop_table("user_quotas")
op.drop_table("usage")
op.drop_table("provider_api_keys")
op.drop_table("model_mappings")
op.drop_table("models")
op.drop_table("provider_endpoints")
op.drop_table("api_keys")
op.drop_table("global_models")
op.drop_table("providers")
op.drop_table("users")
# Drop ENUM types
op.execute("DROP TYPE IF EXISTS providerbillingtype")
op.execute("DROP TYPE IF EXISTS userrole")
@@ -1,315 +0,0 @@
"""remove_model_mappings_add_aliases
合并迁移:
1. 添加 provider_model_aliases 字段到 models 表
2. 迁移 model_mappings 数据到 provider_model_aliases
3. 删除 model_mappings 表
4. 添加索引优化别名解析性能
Revision ID: e9b3d63f0cbf
Revises: 20251210_baseline
Create Date: 2025-12-14 13:00:22.828183+00:00
"""
import json
from datetime import datetime, timezone
import sqlalchemy as sa
from alembic import op
from sqlalchemy.orm import Session
# revision identifiers, used by Alembic.
revision = 'e9b3d63f0cbf'
down_revision = '20251210_baseline'
branch_labels = None
depends_on = None
def column_exists(bind, table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
result = bind.execute(
sa.text(
"""
SELECT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = :table_name AND column_name = :column_name
)
"""
),
{"table_name": table_name, "column_name": column_name},
)
return result.scalar()
def table_exists(bind, table_name: str) -> bool:
"""检查表是否存在"""
result = bind.execute(
sa.text(
"""
SELECT EXISTS (
SELECT 1 FROM information_schema.tables
WHERE table_name = :table_name
)
"""
),
{"table_name": table_name},
)
return result.scalar()
def index_exists(bind, index_name: str) -> bool:
"""检查索引是否存在"""
result = bind.execute(
sa.text(
"""
SELECT EXISTS (
SELECT 1 FROM pg_indexes
WHERE indexname = :index_name
)
"""
),
{"index_name": index_name},
)
return result.scalar()
def upgrade() -> None:
"""添加 provider_model_aliases 字段,迁移数据,删除 model_mappings 表"""
bind = op.get_bind()
# 1. 添加 provider_model_aliases 字段(如果不存在)
if not column_exists(bind, "models", "provider_model_aliases"):
op.add_column(
'models',
sa.Column('provider_model_aliases', sa.JSON(), nullable=True)
)
# 2. 迁移 model_mappings 数据(如果表存在)
session = Session(bind=bind)
model_mappings_table = sa.table(
"model_mappings",
sa.column("source_model", sa.String),
sa.column("target_global_model_id", sa.String),
sa.column("provider_id", sa.String),
sa.column("mapping_type", sa.String),
sa.column("is_active", sa.Boolean),
)
models_table = sa.table(
"models",
sa.column("id", sa.String),
sa.column("provider_id", sa.String),
sa.column("global_model_id", sa.String),
sa.column("provider_model_aliases", sa.JSON),
sa.column("updated_at", sa.DateTime(timezone=True)),
)
def normalize_alias_list(value) -> list[dict]:
"""将 DB 返回的 JSON 值规范化为 list[{'name': str, 'priority': int}]"""
if value is None:
return []
if isinstance(value, str):
try:
value = json.loads(value) if value else []
except Exception:
return []
if not isinstance(value, list):
return []
normalized: list[dict] = []
for item in value:
if not isinstance(item, dict):
continue
raw_name = item.get("name")
if not isinstance(raw_name, str):
continue
name = raw_name.strip()
if not name:
continue
raw_priority = item.get("priority", 1)
try:
priority = int(raw_priority)
except Exception:
priority = 1
if priority < 1:
priority = 1
normalized.append({"name": name, "priority": priority})
return normalized
# 查询所有活跃的 provider 级别 alias(只迁移 is_active=True 且 mapping_type='alias' 的)
# 全局别名/映射不迁移(新架构不再支持 source_model -> GlobalModel.name 的解析)
# 仅当 model_mappings 表存在时执行迁移
if table_exists(bind, "model_mappings"):
mappings = session.execute(
sa.select(
model_mappings_table.c.source_model,
model_mappings_table.c.target_global_model_id,
model_mappings_table.c.provider_id,
)
.where(
model_mappings_table.c.is_active.is_(True),
model_mappings_table.c.provider_id.isnot(None),
model_mappings_table.c.mapping_type == "alias",
)
.order_by(model_mappings_table.c.provider_id, model_mappings_table.c.source_model)
).all()
# 按 (provider_id, target_global_model_id) 分组,收集别名
alias_groups: dict = {}
for source_model, target_global_model_id, provider_id in mappings:
if not isinstance(source_model, str):
continue
source_model = source_model.strip()
if not source_model:
continue
if not isinstance(provider_id, str) or not provider_id:
continue
if not isinstance(target_global_model_id, str) or not target_global_model_id:
continue
key = (provider_id, target_global_model_id)
if key not in alias_groups:
alias_groups[key] = []
priority = len(alias_groups[key]) + 1
alias_groups[key].append({"name": source_model, "priority": priority})
# 更新对应的 models 记录
for (provider_id, global_model_id), aliases in alias_groups.items():
model_row = session.execute(
sa.select(models_table.c.id, models_table.c.provider_model_aliases)
.where(
models_table.c.provider_id == provider_id,
models_table.c.global_model_id == global_model_id,
)
.limit(1)
).first()
if model_row:
model_id = model_row[0]
existing_aliases = normalize_alias_list(model_row[1])
existing_names = {a["name"] for a in existing_aliases}
merged_aliases = list(existing_aliases)
for alias in aliases:
name = alias.get("name")
if not isinstance(name, str):
continue
name = name.strip()
if not name or name in existing_names:
continue
merged_aliases.append(
{
"name": name,
"priority": len(merged_aliases) + 1,
}
)
existing_names.add(name)
session.execute(
models_table.update()
.where(models_table.c.id == model_id)
.values(
provider_model_aliases=merged_aliases if merged_aliases else None,
updated_at=datetime.now(timezone.utc),
)
)
session.commit()
# 3. 删除 model_mappings 表
op.drop_table('model_mappings')
# 4. 添加索引优化别名解析性能
# provider_model_name 索引(支持精确匹配,如果不存在)
if not index_exists(bind, "idx_model_provider_model_name"):
op.create_index(
"idx_model_provider_model_name",
"models",
["provider_model_name"],
unique=False,
postgresql_where=sa.text("is_active = true"),
)
# provider_model_aliases GIN 索引(支持 JSONB 查询,仅 PostgreSQL)
if bind.dialect.name == "postgresql":
# 将 json 列转为 jsonb(jsonb 性能更好且支持 GIN 索引)
# 使用 IF NOT EXISTS 风格的检查来避免重复转换
op.execute(
"""
DO $$
BEGIN
IF EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = 'models'
AND column_name = 'provider_model_aliases'
AND data_type = 'json'
) THEN
ALTER TABLE models
ALTER COLUMN provider_model_aliases TYPE jsonb
USING provider_model_aliases::jsonb;
END IF;
END $$;
"""
)
# 创建 GIN 索引
op.execute(
"""
CREATE INDEX IF NOT EXISTS idx_model_provider_model_aliases_gin
ON models USING gin(provider_model_aliases jsonb_path_ops)
WHERE is_active = true
"""
)
def downgrade() -> None:
"""恢复 model_mappings 表,移除 provider_model_aliases 字段和索引"""
bind = op.get_bind()
# 1. 删除索引
op.drop_index("idx_model_provider_model_name", table_name="models")
if bind.dialect.name == "postgresql":
op.execute("DROP INDEX IF EXISTS idx_model_provider_model_aliases_gin")
# 将 jsonb 列还原为 json
op.execute(
"""
ALTER TABLE models
ALTER COLUMN provider_model_aliases TYPE json
USING provider_model_aliases::json
"""
)
# 2. 恢复 model_mappings 表
op.create_table(
'model_mappings',
sa.Column('id', sa.String(36), primary_key=True),
sa.Column('source_model', sa.String(200), nullable=False),
sa.Column(
'target_global_model_id',
sa.String(36),
sa.ForeignKey('global_models.id', ondelete='CASCADE'),
nullable=False,
),
sa.Column('provider_id', sa.String(36), sa.ForeignKey('providers.id'), nullable=True),
sa.Column('mapping_type', sa.String(20), nullable=False, server_default='alias'),
sa.Column('is_active', sa.Boolean(), nullable=False, server_default='true'),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.UniqueConstraint('source_model', 'provider_id', name='uq_model_mapping_source_provider'),
)
op.create_index('ix_model_mappings_source_model', 'model_mappings', ['source_model'])
op.create_index('ix_model_mappings_target_global_model_id', 'model_mappings', ['target_global_model_id'])
op.create_index('ix_model_mappings_provider_id', 'model_mappings', ['provider_id'])
op.create_index('ix_model_mappings_mapping_type', 'model_mappings', ['mapping_type'])
# 3. 移除 provider_model_aliases 字段
op.drop_column('models', 'provider_model_aliases')
@@ -1,47 +0,0 @@
"""add first_byte_time_ms to usage table
Revision ID: 180e63a9c83a
Revises: e9b3d63f0cbf
Create Date: 2025-12-15 17:07:44.631032+00:00
"""
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = '180e63a9c83a'
down_revision = 'e9b3d63f0cbf'
branch_labels = None
depends_on = None
def column_exists(bind, table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
result = bind.execute(
sa.text(
"""
SELECT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = :table_name AND column_name = :column_name
)
"""
),
{"table_name": table_name, "column_name": column_name},
)
return result.scalar()
def upgrade() -> None:
"""应用迁移:升级到新版本"""
bind = op.get_bind()
# 添加首字时间字段到 usage 表(如果不存在)
if not column_exists(bind, "usage", "first_byte_time_ms"):
op.add_column('usage', sa.Column('first_byte_time_ms', sa.Integer(), nullable=True))
def downgrade() -> None:
"""回滚迁移:降级到旧版本"""
# 删除首字时间字段
op.drop_column('usage', 'first_byte_time_ms')
@@ -1,110 +0,0 @@
"""refactor global_model to use config json field
Revision ID: 1cc6942cf06f
Revises: 180e63a9c83a
Create Date: 2025-12-16 03:11:32.480976+00:00
"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision = '1cc6942cf06f'
down_revision = '180e63a9c83a'
branch_labels = None
depends_on = None
def column_exists(bind, table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
result = bind.execute(
sa.text(
"""
SELECT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = :table_name AND column_name = :column_name
)
"""
),
{"table_name": table_name, "column_name": column_name},
)
return result.scalar()
def upgrade() -> None:
"""应用迁移:升级到新版本
1. 添加 config 列
2. 把旧数据迁移到 config
3. 删除旧列
"""
bind = op.get_bind()
# 检查是否已经迁移过(config 列存在且旧列不存在)
has_config = column_exists(bind, "global_models", "config")
has_old_columns = column_exists(bind, "global_models", "default_supports_streaming")
if has_config and not has_old_columns:
# 已完成迁移,跳过
return
# 1. 添加 config 列(使用 JSONB 类型,支持索引和更高效的查询)
if not has_config:
op.add_column('global_models', sa.Column('config', postgresql.JSONB(), nullable=True))
# 2. 迁移数据:把旧字段合并到 config JSON(仅当旧列存在时)
if has_old_columns:
op.execute("""
UPDATE global_models
SET config = jsonb_strip_nulls(jsonb_build_object(
'streaming', COALESCE(default_supports_streaming, true),
'vision', CASE WHEN COALESCE(default_supports_vision, false) THEN true ELSE NULL END,
'function_calling', CASE WHEN COALESCE(default_supports_function_calling, false) THEN true ELSE NULL END,
'extended_thinking', CASE WHEN COALESCE(default_supports_extended_thinking, false) THEN true ELSE NULL END,
'image_generation', CASE WHEN COALESCE(default_supports_image_generation, false) THEN true ELSE NULL END,
'description', description,
'icon_url', icon_url,
'official_url', official_url
))
""")
# 3. 删除旧列
op.drop_column('global_models', 'default_supports_streaming')
op.drop_column('global_models', 'default_supports_vision')
op.drop_column('global_models', 'default_supports_function_calling')
op.drop_column('global_models', 'default_supports_extended_thinking')
op.drop_column('global_models', 'default_supports_image_generation')
op.drop_column('global_models', 'description')
op.drop_column('global_models', 'icon_url')
op.drop_column('global_models', 'official_url')
def downgrade() -> None:
"""回滚迁移:降级到旧版本"""
# 1. 添加旧列
op.add_column('global_models', sa.Column('icon_url', sa.VARCHAR(length=500), nullable=True))
op.add_column('global_models', sa.Column('official_url', sa.VARCHAR(length=500), nullable=True))
op.add_column('global_models', sa.Column('description', sa.TEXT(), nullable=True))
op.add_column('global_models', sa.Column('default_supports_streaming', sa.BOOLEAN(), nullable=True))
op.add_column('global_models', sa.Column('default_supports_vision', sa.BOOLEAN(), nullable=True))
op.add_column('global_models', sa.Column('default_supports_function_calling', sa.BOOLEAN(), nullable=True))
op.add_column('global_models', sa.Column('default_supports_extended_thinking', sa.BOOLEAN(), nullable=True))
op.add_column('global_models', sa.Column('default_supports_image_generation', sa.BOOLEAN(), nullable=True))
# 2. 从 config 恢复数据
op.execute("""
UPDATE global_models
SET
default_supports_streaming = COALESCE((config->>'streaming')::boolean, true),
default_supports_vision = COALESCE((config->>'vision')::boolean, false),
default_supports_function_calling = COALESCE((config->>'function_calling')::boolean, false),
default_supports_extended_thinking = COALESCE((config->>'extended_thinking')::boolean, false),
default_supports_image_generation = COALESCE((config->>'image_generation')::boolean, false),
description = config->>'description',
icon_url = config->>'icon_url',
official_url = config->>'official_url'
""")
# 3. 删除 config 列
op.drop_column('global_models', 'config')
@@ -1,57 +0,0 @@
"""add proxy field to provider_endpoints
Revision ID: f30f9936f6a2
Revises: 1cc6942cf06f
Create Date: 2025-12-18 06:31:58.451112+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = 'f30f9936f6a2'
down_revision = '1cc6942cf06f'
branch_labels = None
depends_on = None
def column_exists(table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col['name'] for col in inspector.get_columns(table_name)]
return column_name in columns
def get_column_type(table_name: str, column_name: str) -> str:
"""获取列的类型"""
bind = op.get_bind()
inspector = inspect(bind)
for col in inspector.get_columns(table_name):
if col['name'] == column_name:
return str(col['type']).upper()
return ''
def upgrade() -> None:
"""添加 proxy 字段到 provider_endpoints 表"""
if not column_exists('provider_endpoints', 'proxy'):
# 字段不存在,直接添加 JSONB 类型
op.add_column('provider_endpoints', sa.Column('proxy', JSONB(), nullable=True))
else:
# 字段已存在,检查是否需要转换类型
col_type = get_column_type('provider_endpoints', 'proxy')
if 'JSONB' not in col_type:
# 如果是 JSON 类型,转换为 JSONB
op.execute(
'ALTER TABLE provider_endpoints '
'ALTER COLUMN proxy TYPE JSONB USING proxy::jsonb'
)
def downgrade() -> None:
"""移除 proxy 字段"""
if column_exists('provider_endpoints', 'proxy'):
op.drop_column('provider_endpoints', 'proxy')
@@ -1,86 +0,0 @@
"""add stats_daily_model table and rename provider_model_aliases
Revision ID: a1b2c3d4e5f6
Revises: f30f9936f6a2
Create Date: 2025-12-20 12:00:00.000000+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = 'a1b2c3d4e5f6'
down_revision = 'f30f9936f6a2'
branch_labels = None
depends_on = None
def table_exists(table_name: str) -> bool:
"""检查表是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col['name'] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
"""创建 stats_daily_model 表,重命名 provider_model_aliases 为 provider_model_mappings"""
# 1. 创建 stats_daily_model 表
if not table_exists('stats_daily_model'):
op.create_table(
'stats_daily_model',
sa.Column('id', sa.String(36), primary_key=True),
sa.Column('date', sa.DateTime(timezone=True), nullable=False),
sa.Column('model', sa.String(100), nullable=False),
sa.Column('total_requests', sa.Integer(), nullable=False, default=0),
sa.Column('input_tokens', sa.BigInteger(), nullable=False, default=0),
sa.Column('output_tokens', sa.BigInteger(), nullable=False, default=0),
sa.Column('cache_creation_tokens', sa.BigInteger(), nullable=False, default=0),
sa.Column('cache_read_tokens', sa.BigInteger(), nullable=False, default=0),
sa.Column('total_cost', sa.Float(), nullable=False, default=0.0),
sa.Column('avg_response_time_ms', sa.Float(), nullable=False, default=0.0),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now(), onupdate=sa.func.now()),
sa.UniqueConstraint('date', 'model', name='uq_stats_daily_model'),
)
# 创建索引
op.create_index('idx_stats_daily_model_date', 'stats_daily_model', ['date'])
op.create_index('idx_stats_daily_model_date_model', 'stats_daily_model', ['date', 'model'])
# 2. 重命名 models 表的 provider_model_aliases 为 provider_model_mappings
if column_exists('models', 'provider_model_aliases') and not column_exists('models', 'provider_model_mappings'):
op.alter_column('models', 'provider_model_aliases', new_column_name='provider_model_mappings')
def index_exists(table_name: str, index_name: str) -> bool:
"""检查索引是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
indexes = [idx['name'] for idx in inspector.get_indexes(table_name)]
return index_name in indexes
def downgrade() -> None:
"""删除 stats_daily_model 表,恢复 provider_model_aliases 列名"""
# 恢复列名
if column_exists('models', 'provider_model_mappings') and not column_exists('models', 'provider_model_aliases'):
op.alter_column('models', 'provider_model_mappings', new_column_name='provider_model_aliases')
# 删除表
if table_exists('stats_daily_model'):
if index_exists('stats_daily_model', 'idx_stats_daily_model_date_model'):
op.drop_index('idx_stats_daily_model_date_model', table_name='stats_daily_model')
if index_exists('stats_daily_model', 'idx_stats_daily_model_date'):
op.drop_index('idx_stats_daily_model_date', table_name='stats_daily_model')
op.drop_table('stats_daily_model')
@@ -1,65 +0,0 @@
"""add usage table composite indexes for query optimization
Revision ID: b2c3d4e5f6g7
Revises: a1b2c3d4e5f6
Create Date: 2025-12-20 15:00:00.000000+00:00
"""
from alembic import op
from sqlalchemy import text
# revision identifiers, used by Alembic.
revision = 'b2c3d4e5f6g7'
down_revision = 'a1b2c3d4e5f6'
branch_labels = None
depends_on = None
def upgrade() -> None:
"""为 usage 表添加复合索引以优化常见查询
注意:这些索引已经在 baseline 迁移中创建。
此迁移仅用于从旧版本升级的场景,新安装会跳过。
"""
conn = op.get_bind()
# 检查 usage 表是否存在
result = conn.execute(text(
"SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'usage')"
))
if not result.scalar():
# 表不存在,跳过
return
# 定义需要创建的索引
indexes = [
("idx_usage_user_created", "ON usage (user_id, created_at)"),
("idx_usage_apikey_created", "ON usage (api_key_id, created_at)"),
("idx_usage_provider_model_created", "ON usage (provider, model, created_at)"),
]
# 分别检查并创建每个索引
for index_name, index_def in indexes:
result = conn.execute(text(
f"SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE indexname = '{index_name}')"
))
if result.scalar():
continue # 索引已存在,跳过
conn.execute(text(f"CREATE INDEX {index_name} {index_def}"))
def downgrade() -> None:
"""删除复合索引"""
conn = op.get_bind()
# 使用 IF EXISTS 避免索引不存在时报错
conn.execute(text(
"DROP INDEX IF EXISTS idx_usage_provider_model_created"
))
conn.execute(text(
"DROP INDEX IF EXISTS idx_usage_apikey_created"
))
conn.execute(text(
"DROP INDEX IF EXISTS idx_usage_user_created"
))
@@ -1,161 +0,0 @@
"""add ldap authentication support
Revision ID: c3d4e5f6g7h8
Revises: b2c3d4e5f6g7
Create Date: 2026-01-01 14:00:00.000000+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import text
# revision identifiers, used by Alembic.
revision = 'c3d4e5f6g7h8'
down_revision = 'b2c3d4e5f6g7'
branch_labels = None
depends_on = None
def _type_exists(conn, type_name: str) -> bool:
"""检查 PostgreSQL 类型是否存在"""
result = conn.execute(
text("SELECT 1 FROM pg_type WHERE typname = :name"),
{"name": type_name}
)
return result.scalar() is not None
def _column_exists(conn, table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
result = conn.execute(
text("""
SELECT 1 FROM information_schema.columns
WHERE table_name = :table AND column_name = :column
"""),
{"table": table_name, "column": column_name}
)
return result.scalar() is not None
def _index_exists(conn, index_name: str) -> bool:
"""检查索引是否存在"""
result = conn.execute(
text("SELECT 1 FROM pg_indexes WHERE indexname = :name"),
{"name": index_name}
)
return result.scalar() is not None
def _table_exists(conn, table_name: str) -> bool:
"""检查表是否存在"""
result = conn.execute(
text("""
SELECT 1 FROM information_schema.tables
WHERE table_name = :name AND table_schema = 'public'
"""),
{"name": table_name}
)
return result.scalar() is not None
def upgrade() -> None:
"""添加 LDAP 认证支持
1. 创建 authsource 枚举类型
2. 在 users 表添加 auth_source 字段和 LDAP 标识字段
3. 创建 ldap_configs 表
"""
conn = op.get_bind()
# 1. 创建 authsource 枚举类型(幂等)
if not _type_exists(conn, 'authsource'):
conn.execute(text("CREATE TYPE authsource AS ENUM ('local', 'ldap')"))
# 2. 在 users 表添加字段(幂等)
if not _column_exists(conn, 'users', 'auth_source'):
op.add_column('users', sa.Column(
'auth_source',
sa.Enum('local', 'ldap', name='authsource', create_type=False),
nullable=False,
server_default='local'
))
if not _column_exists(conn, 'users', 'ldap_dn'):
op.add_column('users', sa.Column('ldap_dn', sa.String(length=512), nullable=True))
if not _column_exists(conn, 'users', 'ldap_username'):
op.add_column('users', sa.Column('ldap_username', sa.String(length=255), nullable=True))
# 创建索引(幂等)
if not _index_exists(conn, 'ix_users_ldap_dn'):
op.create_index('ix_users_ldap_dn', 'users', ['ldap_dn'])
if not _index_exists(conn, 'ix_users_ldap_username'):
op.create_index('ix_users_ldap_username', 'users', ['ldap_username'])
# 3. 创建 ldap_configs 表(幂等)
if not _table_exists(conn, 'ldap_configs'):
op.create_table(
'ldap_configs',
sa.Column('id', sa.Integer(), autoincrement=True, nullable=False),
sa.Column('server_url', sa.String(length=255), nullable=False),
sa.Column('bind_dn', sa.String(length=255), nullable=False),
sa.Column('bind_password_encrypted', sa.Text(), nullable=True),
sa.Column('base_dn', sa.String(length=255), nullable=False),
sa.Column('user_search_filter', sa.String(length=500), nullable=False, server_default='(uid={username})'),
sa.Column('username_attr', sa.String(length=50), nullable=False, server_default='uid'),
sa.Column('email_attr', sa.String(length=50), nullable=False, server_default='mail'),
sa.Column('display_name_attr', sa.String(length=50), nullable=False, server_default='cn'),
sa.Column('is_enabled', sa.Boolean(), nullable=False, server_default='false'),
sa.Column('is_exclusive', sa.Boolean(), nullable=False, server_default='false'),
sa.Column('use_starttls', sa.Boolean(), nullable=False, server_default='false'),
sa.Column('connect_timeout', sa.Integer(), nullable=False, server_default='10'),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False, server_default=sa.text('now()')),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False, server_default=sa.text('now()')),
sa.PrimaryKeyConstraint('id')
)
def downgrade() -> None:
"""回滚 LDAP 认证支持
警告:回滚前请确保:
1. 已备份数据库
2. 没有 LDAP 用户需要保留
"""
conn = op.get_bind()
# 检查是否存在 LDAP 用户,防止数据丢失
if _column_exists(conn, 'users', 'auth_source'):
result = conn.execute(text("SELECT COUNT(*) FROM users WHERE auth_source = 'ldap'"))
ldap_user_count = result.scalar()
if ldap_user_count and ldap_user_count > 0:
raise RuntimeError(
f"无法回滚:存在 {ldap_user_count} 个 LDAP 用户。"
f"请先删除或转换这些用户,或使用 --force 参数强制回滚(将丢失数据)。"
)
# 1. 删除 ldap_configs 表(幂等)
if _table_exists(conn, 'ldap_configs'):
op.drop_table('ldap_configs')
# 2. 删除 users 表的 LDAP 相关字段(幂等)
if _index_exists(conn, 'ix_users_ldap_username'):
op.drop_index('ix_users_ldap_username', table_name='users')
if _index_exists(conn, 'ix_users_ldap_dn'):
op.drop_index('ix_users_ldap_dn', table_name='users')
if _column_exists(conn, 'users', 'ldap_username'):
op.drop_column('users', 'ldap_username')
if _column_exists(conn, 'users', 'ldap_dn'):
op.drop_column('users', 'ldap_dn')
if _column_exists(conn, 'users', 'auth_source'):
op.drop_column('users', 'auth_source')
# 3. 删除 authsource 枚举类型(幂等)
# 注意:不使用 CASCADE,因为此时所有依赖应该已被删除
if _type_exists(conn, 'authsource'):
conn.execute(text("DROP TYPE authsource"))
@@ -1,131 +0,0 @@
"""add_management_tokens_table
Revision ID: ad55f1d008b7
Revises: c3d4e5f6g7h8
Create Date: 2026-01-06 15:24:10.660394+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = 'ad55f1d008b7'
down_revision = 'c3d4e5f6g7h8'
branch_labels = None
depends_on = None
def table_exists(table_name: str) -> bool:
"""检查表是否存在"""
conn = op.get_bind()
inspector = inspect(conn)
return table_name in inspector.get_table_names()
def index_exists(table_name: str, index_name: str) -> bool:
"""检查索引是否存在"""
conn = op.get_bind()
inspector = inspect(conn)
try:
indexes = inspector.get_indexes(table_name)
return any(idx["name"] == index_name for idx in indexes)
except Exception:
return False
def constraint_exists(table_name: str, constraint_name: str) -> bool:
"""检查约束是否存在"""
conn = op.get_bind()
inspector = inspect(conn)
try:
constraints = inspector.get_unique_constraints(table_name)
if any(c["name"] == constraint_name for c in constraints):
return True
# 也检查 check 约束
check_constraints = inspector.get_check_constraints(table_name)
if any(c["name"] == constraint_name for c in check_constraints):
return True
return False
except Exception:
return False
def upgrade() -> None:
"""应用迁移:创建 management_tokens 表"""
# 幂等性检查
if table_exists("management_tokens"):
# 表已存在,检查是否需要添加约束
if not constraint_exists("management_tokens", "uq_management_tokens_user_name"):
op.create_unique_constraint(
"uq_management_tokens_user_name",
"management_tokens",
["user_id", "name"],
)
# 添加 IP 白名单非空检查约束
if not constraint_exists("management_tokens", "check_allowed_ips_not_empty"):
op.create_check_constraint(
"check_allowed_ips_not_empty",
"management_tokens",
"allowed_ips IS NULL OR allowed_ips::text = 'null' OR json_array_length(allowed_ips) > 0",
)
return
op.create_table('management_tokens',
sa.Column('id', sa.String(length=36), nullable=False),
sa.Column('user_id', sa.String(length=36), nullable=False),
sa.Column('token_hash', sa.String(length=64), nullable=False),
sa.Column('token_prefix', sa.String(length=12), nullable=True),
sa.Column('name', sa.String(length=100), nullable=False),
sa.Column('description', sa.Text(), nullable=True),
sa.Column('allowed_ips', sa.JSON(), nullable=True),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('last_used_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('last_used_ip', sa.String(length=45), nullable=True),
sa.Column('usage_count', sa.Integer(), server_default='0', nullable=False),
sa.Column('is_active', sa.Boolean(), server_default='true', nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index('idx_management_tokens_is_active', 'management_tokens', ['is_active'], unique=False)
op.create_index('idx_management_tokens_user_id', 'management_tokens', ['user_id'], unique=False)
op.create_index(op.f('ix_management_tokens_token_hash'), 'management_tokens', ['token_hash'], unique=True)
# 添加用户名称唯一约束
op.create_unique_constraint(
"uq_management_tokens_user_name",
"management_tokens",
["user_id", "name"],
)
# 添加 IP 白名单非空检查约束
# 注意:JSON 类型的 NULL 可能被序列化为 JSON 'null',需要同时处理
op.create_check_constraint(
"check_allowed_ips_not_empty",
"management_tokens",
"allowed_ips IS NULL OR allowed_ips::text = 'null' OR json_array_length(allowed_ips) > 0",
)
def downgrade() -> None:
"""回滚迁移:删除 management_tokens 表"""
# 幂等性检查
if not table_exists("management_tokens"):
return
# 删除约束
if constraint_exists("management_tokens", "check_allowed_ips_not_empty"):
op.drop_constraint("check_allowed_ips_not_empty", "management_tokens", type_="check")
if constraint_exists("management_tokens", "uq_management_tokens_user_name"):
op.drop_constraint("uq_management_tokens_user_name", "management_tokens", type_="unique")
# 删除索引
if index_exists("management_tokens", "ix_management_tokens_token_hash"):
op.drop_index(op.f('ix_management_tokens_token_hash'), table_name='management_tokens')
if index_exists("management_tokens", "idx_management_tokens_user_id"):
op.drop_index('idx_management_tokens_user_id', table_name='management_tokens')
if index_exists("management_tokens", "idx_management_tokens_is_active"):
op.drop_index('idx_management_tokens_is_active', table_name='management_tokens')
# 删除表
op.drop_table('management_tokens')
@@ -1,73 +0,0 @@
"""cleanup ambiguous database fields
Revision ID: 02a45b66b7c4
Revises: ad55f1d008b7
Create Date: 2026-01-07 11:20:12.684426+00:00
变更内容:
1. users 表:重命名 allowed_endpoints 为 allowed_api_formats(修正历史命名错误)
2. api_keys 表:删除 allowed_endpoints 字段(未使用的功能)
3. providers 表:删除 rate_limit 字段(与 rpm_limit 功能重复,且未使用)
4. usage 表:重命名 provider 为 provider_name(避免与 provider_id 外键混淆)
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = '02a45b66b7c4'
down_revision = 'ad55f1d008b7'
branch_labels = None
depends_on = None
def _column_exists(table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col['name'] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
"""
1. users.allowed_endpoints -> allowed_api_formats(重命名)
2. api_keys.allowed_endpoints 删除
3. providers.rate_limit 删除(与 rpm_limit 重复)
4. usage.provider -> provider_name(重命名)
"""
# 1. users 表:重命名 allowed_endpoints 为 allowed_api_formats
if _column_exists('users', 'allowed_endpoints'):
op.alter_column('users', 'allowed_endpoints', new_column_name='allowed_api_formats')
# 2. api_keys 表:删除 allowed_endpoints 字段
if _column_exists('api_keys', 'allowed_endpoints'):
op.drop_column('api_keys', 'allowed_endpoints')
# 3. providers 表:删除 rate_limit 字段(与 rpm_limit 功能重复)
if _column_exists('providers', 'rate_limit'):
op.drop_column('providers', 'rate_limit')
# 4. usage 表:重命名 provider 为 provider_name
if _column_exists('usage', 'provider'):
op.alter_column('usage', 'provider', new_column_name='provider_name')
def downgrade() -> None:
"""回滚:恢复原字段"""
# 4. usage 表:将 provider_name 改回 provider
if _column_exists('usage', 'provider_name'):
op.alter_column('usage', 'provider_name', new_column_name='provider')
# 3. providers 表:恢复 rate_limit 字段
if not _column_exists('providers', 'rate_limit'):
op.add_column('providers', sa.Column('rate_limit', sa.Integer(), nullable=True))
# 2. api_keys 表:恢复 allowed_endpoints 字段
if not _column_exists('api_keys', 'allowed_endpoints'):
op.add_column('api_keys', sa.Column('allowed_endpoints', sa.JSON(), nullable=True))
# 1. users 表:将 allowed_api_formats 改回 allowed_endpoints
if _column_exists('users', 'allowed_api_formats'):
op.alter_column('users', 'allowed_api_formats', new_column_name='allowed_endpoints')
@@ -1,604 +0,0 @@
"""consolidated schema updates
Revision ID: m4n5o6p7q8r9
Revises: 02a45b66b7c4
Create Date: 2026-01-10 20:00:00.000000
This migration consolidates all schema changes from 2026-01-08 to 2026-01-10:
1. provider_api_keys: Key 直接关联 Provider (provider_id, api_formats)
2. provider_api_keys: 添加 rate_multipliers JSON 字段(按格式费率)
3. models: global_model_id 改为可空(支持独立 ProviderModel)
4. providers: 添加 timeout, max_retries, proxy(从 endpoint 迁移)
5. providers: display_name 重命名为 name,删除原 name
6. provider_api_keys: max_concurrent -> rpm_limit(并发改 RPM)
7. provider_api_keys: 健康度改为按格式存储(health_by_format, circuit_breaker_by_format)
8. provider_endpoints: 删除废弃的 rate_limit 列
9. usage: 添加 client_response_headers 字段
10. provider_api_keys: 删除 endpoint_id(Key 不再与 Endpoint 绑定)
11. provider_endpoints: 删除废弃的 max_concurrent 列
12. providers: 删除废弃的 rpm_limit, rpm_used, rpm_reset_at 列
"""
import logging
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
from sqlalchemy.exc import ProgrammingError
from alembic import op
# 配置日志
alembic_logger = logging.getLogger("alembic.runtime.migration")
revision = "m4n5o6p7q8r9"
down_revision = "02a45b66b7c4"
branch_labels = None
depends_on = None
def _column_exists(table_name: str, column_name: str) -> bool:
"""Check if a column exists in the table (bypasses inspector cache)"""
bind = op.get_bind()
result = bind.execute(
sa.text(
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :col"
),
{"table": table_name, "col": column_name},
)
return result.scalar() is not None
def _constraint_exists(table_name: str, constraint_name: str) -> bool:
"""Check if a constraint exists (bypasses inspector cache)"""
bind = op.get_bind()
result = bind.execute(
sa.text(
"SELECT 1 FROM information_schema.table_constraints "
"WHERE table_name = :table AND constraint_name = :name"
),
{"table": table_name, "name": constraint_name},
)
return result.scalar() is not None
def _index_exists(table_name: str, index_name: str) -> bool:
"""Check if an index exists (bypasses inspector cache)"""
bind = op.get_bind()
result = bind.execute(
sa.text("SELECT 1 FROM pg_indexes WHERE indexname = :name"),
{"name": index_name},
)
return result.scalar() is not None
def upgrade() -> None:
"""Apply all consolidated schema changes"""
bind = op.get_bind()
# ========== 1. provider_api_keys: 添加 provider_id 和 api_formats ==========
if not _column_exists("provider_api_keys", "provider_id"):
conn = op.get_bind()
conn.execute(sa.text("SAVEPOINT sp_add_provider_id"))
try:
op.add_column(
"provider_api_keys", sa.Column("provider_id", sa.String(36), nullable=True)
)
conn.execute(sa.text("RELEASE SAVEPOINT sp_add_provider_id"))
except ProgrammingError as exc:
if getattr(getattr(exc, "orig", None), "pgcode", None) == "42701":
conn.execute(sa.text("ROLLBACK TO SAVEPOINT sp_add_provider_id"))
alembic_logger.warning("provider_api_keys.provider_id already exists; skipping add")
else:
conn.execute(sa.text("ROLLBACK TO SAVEPOINT sp_add_provider_id"))
raise
# 数据迁移:从 endpoint 获取 provider_id(如果 endpoint_id 仍存在)
if _column_exists("provider_api_keys", "endpoint_id"):
op.execute("""
UPDATE provider_api_keys k
SET provider_id = e.provider_id
FROM provider_endpoints e
WHERE k.endpoint_id = e.id AND k.provider_id IS NULL
""")
# 检查无法关联的孤儿 Key
result = bind.execute(
sa.text("SELECT COUNT(*) FROM provider_api_keys WHERE provider_id IS NULL")
)
orphan_count = result.scalar() or 0
if orphan_count > 0:
# 使用 logger 记录更明显的告警
alembic_logger.warning("=" * 60)
alembic_logger.warning(
f"[MIGRATION WARNING] 发现 {orphan_count} 个无法关联 Provider 的孤儿 Key"
)
alembic_logger.warning("=" * 60)
alembic_logger.info("正在备份孤儿 Key 到 _orphan_api_keys_backup 表...")
# 先备份孤儿数据到临时表,避免数据丢失
op.execute("""
CREATE TABLE IF NOT EXISTS _orphan_api_keys_backup AS
SELECT *, NOW() as backup_at
FROM provider_api_keys
WHERE provider_id IS NULL
""")
# 记录备份的 Key ID
orphan_ids = bind.execute(
sa.text("SELECT id, name FROM provider_api_keys WHERE provider_id IS NULL")
).fetchall()
alembic_logger.info("备份的孤儿 Key 列表:")
for key_id, key_name in orphan_ids:
alembic_logger.info(f" - Key: {key_name} (ID: {key_id})")
# 删除孤儿数据
op.execute("DELETE FROM provider_api_keys WHERE provider_id IS NULL")
alembic_logger.info(f"已备份并删除 {orphan_count} 个孤儿 Key")
# 提供恢复指南
alembic_logger.warning("-" * 60)
alembic_logger.warning("[恢复指南] 如需恢复孤儿 Key:")
alembic_logger.warning(" 1. 查询备份表: SELECT * FROM _orphan_api_keys_backup;")
alembic_logger.warning(" 2. 确定正确的 provider_id")
alembic_logger.warning(" 3. 执行恢复:")
alembic_logger.warning(" INSERT INTO provider_api_keys (...)")
alembic_logger.warning(" SELECT ... FROM _orphan_api_keys_backup WHERE ...;")
alembic_logger.warning("-" * 60)
# 设置 NOT NULL 并创建外键
op.alter_column("provider_api_keys", "provider_id", nullable=False)
if not _constraint_exists("provider_api_keys", "fk_provider_api_keys_provider"):
op.create_foreign_key(
"fk_provider_api_keys_provider",
"provider_api_keys",
"providers",
["provider_id"],
["id"],
ondelete="CASCADE",
)
if not _index_exists("provider_api_keys", "idx_provider_api_keys_provider_id"):
op.create_index("idx_provider_api_keys_provider_id", "provider_api_keys", ["provider_id"])
if not _column_exists("provider_api_keys", "api_formats"):
op.add_column("provider_api_keys", sa.Column("api_formats", sa.JSON(), nullable=True))
# 数据迁移:从 endpoint 获取 api_format
op.execute("""
UPDATE provider_api_keys k
SET api_formats = json_build_array(e.api_format)
FROM provider_endpoints e
WHERE k.endpoint_id = e.id AND k.api_formats IS NULL
""")
op.alter_column("provider_api_keys", "api_formats", nullable=False, server_default="[]")
# 修改 endpoint_id 为可空,外键改为 SET NULL
if _constraint_exists("provider_api_keys", "provider_api_keys_endpoint_id_fkey"):
op.drop_constraint(
"provider_api_keys_endpoint_id_fkey", "provider_api_keys", type_="foreignkey"
)
op.alter_column("provider_api_keys", "endpoint_id", nullable=True)
# 不再重建外键,因为后面会删除这个字段
# ========== 2. provider_api_keys: 添加 rate_multipliers ==========
if not _column_exists("provider_api_keys", "rate_multipliers"):
op.add_column(
"provider_api_keys",
sa.Column("rate_multipliers", postgresql.JSON(astext_type=sa.Text()), nullable=True),
)
# 数据迁移:将 rate_multiplier 按 api_formats 转换
op.execute("""
UPDATE provider_api_keys
SET rate_multipliers = (
SELECT jsonb_object_agg(elem, rate_multiplier)
FROM jsonb_array_elements_text(api_formats::jsonb) AS elem
)
WHERE api_formats IS NOT NULL
AND api_formats::text != '[]'
AND api_formats::text != 'null'
AND rate_multipliers IS NULL
""")
# ========== 3. models: global_model_id 改为可空 ==========
op.alter_column("models", "global_model_id", existing_type=sa.String(36), nullable=True)
# ========== 4. providers: 添加 timeout, max_retries, proxy ==========
if not _column_exists("providers", "timeout"):
op.add_column(
"providers",
sa.Column("timeout", sa.Integer(), nullable=True, comment="请求超时(秒)"),
)
if not _column_exists("providers", "max_retries"):
op.add_column(
"providers",
sa.Column("max_retries", sa.Integer(), nullable=True, comment="最大重试次数"),
)
if not _column_exists("providers", "proxy"):
op.add_column(
"providers",
sa.Column("proxy", postgresql.JSONB(), nullable=True, comment="代理配置"),
)
# 从端点迁移数据到 provider(动态构建 SQL,仅引用存在的列)
ep_has_timeout = _column_exists("provider_endpoints", "timeout")
ep_has_max_retries = _column_exists("provider_endpoints", "max_retries")
ep_has_proxy = _column_exists("provider_endpoints", "proxy")
set_clauses = []
if _column_exists("providers", "timeout"):
if ep_has_timeout:
set_clauses.append("""
timeout = COALESCE(
p.timeout,
(SELECT MAX(e.timeout) FROM provider_endpoints e WHERE e.provider_id = p.id AND e.timeout IS NOT NULL),
300
)""")
else:
set_clauses.append("timeout = COALESCE(p.timeout, 300)")
if _column_exists("providers", "max_retries"):
if ep_has_max_retries:
set_clauses.append("""
max_retries = COALESCE(
p.max_retries,
(SELECT MAX(e.max_retries) FROM provider_endpoints e WHERE e.provider_id = p.id AND e.max_retries IS NOT NULL),
2
)""")
else:
set_clauses.append("max_retries = COALESCE(p.max_retries, 2)")
if _column_exists("providers", "proxy") and ep_has_proxy:
set_clauses.append("""
proxy = COALESCE(
p.proxy,
(SELECT e.proxy FROM provider_endpoints e WHERE e.provider_id = p.id AND e.proxy IS NOT NULL ORDER BY e.created_at LIMIT 1)
)""")
if set_clauses:
where_parts = []
if _column_exists("providers", "timeout"):
where_parts.append("p.timeout IS NULL")
if _column_exists("providers", "max_retries"):
where_parts.append("p.max_retries IS NULL")
where_clause = " OR ".join(where_parts) if where_parts else "TRUE"
sql = "UPDATE providers p SET " + ", ".join(set_clauses) + " WHERE " + where_clause
op.execute(sql)
# ========== 5. providers: display_name -> name ==========
# 注意:这里假设 display_name 已经被重命名为 name
# 如果 display_name 仍然存在,则需要执行重命名
if _column_exists("providers", "display_name"):
# 删除旧的 name 索引
if _index_exists("providers", "ix_providers_name"):
op.drop_index("ix_providers_name", table_name="providers")
# 如果存在旧的 name 列,先删除
if _column_exists("providers", "name"):
op.drop_column("providers", "name")
# 重命名 display_name 为 name
op.alter_column("providers", "display_name", new_column_name="name")
# 创建新索引
op.create_index("ix_providers_name", "providers", ["name"], unique=True)
# ========== 6. provider_api_keys: max_concurrent -> rpm_limit ==========
if _column_exists("provider_api_keys", "max_concurrent"):
op.alter_column("provider_api_keys", "max_concurrent", new_column_name="rpm_limit")
if _column_exists("provider_api_keys", "learned_max_concurrent"):
op.alter_column(
"provider_api_keys", "learned_max_concurrent", new_column_name="learned_rpm_limit"
)
if _column_exists("provider_api_keys", "last_concurrent_peak"):
op.alter_column(
"provider_api_keys", "last_concurrent_peak", new_column_name="last_rpm_peak"
)
# 删除废弃字段
for col in ["rate_limit", "daily_limit", "monthly_limit"]:
if _column_exists("provider_api_keys", col):
op.drop_column("provider_api_keys", col)
# ========== 7. provider_api_keys: 健康度改为按格式存储 ==========
if not _column_exists("provider_api_keys", "health_by_format"):
op.add_column(
"provider_api_keys",
sa.Column(
"health_by_format",
postgresql.JSONB(astext_type=sa.Text()),
nullable=True,
comment="按API格式存储的健康度数据",
),
)
if not _column_exists("provider_api_keys", "circuit_breaker_by_format"):
op.add_column(
"provider_api_keys",
sa.Column(
"circuit_breaker_by_format",
postgresql.JSONB(astext_type=sa.Text()),
nullable=True,
comment="按API格式存储的熔断器状态",
),
)
# 数据迁移:如果存在旧字段,迁移数据到新结构
if _column_exists("provider_api_keys", "health_score"):
op.execute("""
UPDATE provider_api_keys
SET health_by_format = (
SELECT jsonb_object_agg(
elem,
jsonb_build_object(
'health_score', COALESCE(health_score, 1.0),
'consecutive_failures', COALESCE(consecutive_failures, 0),
'last_failure_at', last_failure_at,
'request_results_window', COALESCE(request_results_window::jsonb, '[]'::jsonb)
)
)
FROM jsonb_array_elements_text(api_formats::jsonb) AS elem
)
WHERE api_formats IS NOT NULL
AND api_formats::text != '[]'
AND health_by_format IS NULL
""")
# Circuit Breaker 迁移策略:
# 不复制旧的 circuit_breaker_open 状态到所有 format,而是全部重置为 closed
# 原因:旧的单一 circuit breaker 状态可能因某一个 format 失败而打开,
# 如果复制到所有 format,会导致其他正常工作的 format 被错误标记为不可用
if _column_exists("provider_api_keys", "circuit_breaker_open"):
op.execute("""
UPDATE provider_api_keys
SET circuit_breaker_by_format = (
SELECT jsonb_object_agg(
elem,
jsonb_build_object(
'open', false,
'open_at', NULL,
'next_probe_at', NULL,
'half_open_until', NULL,
'half_open_successes', 0,
'half_open_failures', 0
)
)
FROM jsonb_array_elements_text(api_formats::jsonb) AS elem
)
WHERE api_formats IS NOT NULL
AND api_formats::text != '[]'
AND circuit_breaker_by_format IS NULL
""")
# 设置默认空对象
op.execute("""
UPDATE provider_api_keys
SET health_by_format = '{}'::jsonb
WHERE health_by_format IS NULL
""")
op.execute("""
UPDATE provider_api_keys
SET circuit_breaker_by_format = '{}'::jsonb
WHERE circuit_breaker_by_format IS NULL
""")
# 创建 GIN 索引
if not _index_exists("provider_api_keys", "ix_provider_api_keys_health_by_format"):
op.create_index(
"ix_provider_api_keys_health_by_format",
"provider_api_keys",
["health_by_format"],
postgresql_using="gin",
)
if not _index_exists("provider_api_keys", "ix_provider_api_keys_circuit_breaker_by_format"):
op.create_index(
"ix_provider_api_keys_circuit_breaker_by_format",
"provider_api_keys",
["circuit_breaker_by_format"],
postgresql_using="gin",
)
# 删除旧字段
old_health_columns = [
"health_score",
"consecutive_failures",
"last_failure_at",
"request_results_window",
"circuit_breaker_open",
"circuit_breaker_open_at",
"next_probe_at",
"half_open_until",
"half_open_successes",
"half_open_failures",
]
for col in old_health_columns:
if _column_exists("provider_api_keys", col):
op.drop_column("provider_api_keys", col)
# ========== 8. provider_endpoints: 删除废弃的 rate_limit 列 ==========
if _column_exists("provider_endpoints", "rate_limit"):
op.drop_column("provider_endpoints", "rate_limit")
# ========== 9. usage: 添加 client_response_headers ==========
if not _column_exists("usage", "client_response_headers"):
op.add_column(
"usage",
sa.Column("client_response_headers", sa.JSON(), nullable=True),
)
# ========== 10. provider_api_keys: 删除 endpoint_id ==========
# Key 不再与 Endpoint 绑定,通过 provider_id + api_formats 关联
if _column_exists("provider_api_keys", "endpoint_id"):
# 查找 endpoint_id 上的外键并删除(用 savepoint 保护,避免事务中止)
conn = op.get_bind()
fk_rows = conn.execute(
sa.text(
"SELECT con.conname FROM pg_constraint con "
"JOIN pg_attribute att ON att.attnum = ANY(con.conkey) "
" AND att.attrelid = con.conrelid "
"WHERE con.conrelid = 'provider_api_keys'::regclass "
" AND con.contype = 'f' AND att.attname = 'endpoint_id'"
)
).fetchall()
for (fk_name,) in fk_rows:
conn.execute(sa.text(f"SAVEPOINT sp_drop_fk_{fk_name}"))
try:
op.drop_constraint(fk_name, "provider_api_keys", type_="foreignkey")
conn.execute(sa.text(f"RELEASE SAVEPOINT sp_drop_fk_{fk_name}"))
except Exception:
conn.execute(sa.text(f"ROLLBACK TO SAVEPOINT sp_drop_fk_{fk_name}"))
op.drop_column("provider_api_keys", "endpoint_id")
# ========== 11. provider_endpoints: 删除废弃的 max_concurrent 列 ==========
if _column_exists("provider_endpoints", "max_concurrent"):
op.drop_column("provider_endpoints", "max_concurrent")
# ========== 12. providers: 删除废弃的 RPM 相关字段 ==========
if _column_exists("providers", "rpm_limit"):
op.drop_column("providers", "rpm_limit")
if _column_exists("providers", "rpm_used"):
op.drop_column("providers", "rpm_used")
if _column_exists("providers", "rpm_reset_at"):
op.drop_column("providers", "rpm_reset_at")
alembic_logger.info("[OK] Consolidated migration completed successfully")
def downgrade() -> None:
"""
Downgrade is complex due to data migrations.
For safety, this only removes new columns without restoring old structure.
Manual intervention may be required for full rollback.
"""
bind = op.get_bind()
# 12. 恢复 providers RPM 相关字段
if not _column_exists("providers", "rpm_limit"):
op.add_column("providers", sa.Column("rpm_limit", sa.Integer(), nullable=True))
if not _column_exists("providers", "rpm_used"):
op.add_column(
"providers",
sa.Column("rpm_used", sa.Integer(), server_default="0", nullable=True),
)
if not _column_exists("providers", "rpm_reset_at"):
op.add_column(
"providers",
sa.Column("rpm_reset_at", sa.DateTime(timezone=True), nullable=True),
)
# 11. 恢复 provider_endpoints.max_concurrent
if not _column_exists("provider_endpoints", "max_concurrent"):
op.add_column(
"provider_endpoints", sa.Column("max_concurrent", sa.Integer(), nullable=True)
)
# 10. 恢复 endpoint_id
if not _column_exists("provider_api_keys", "endpoint_id"):
op.add_column("provider_api_keys", sa.Column("endpoint_id", sa.String(36), nullable=True))
# 9. 删除 client_response_headers
if _column_exists("usage", "client_response_headers"):
op.drop_column("usage", "client_response_headers")
# 8. 恢复 provider_endpoints.rate_limit(如果需要)
if not _column_exists("provider_endpoints", "rate_limit"):
op.add_column("provider_endpoints", sa.Column("rate_limit", sa.Integer(), nullable=True))
# 7. 删除健康度 JSON 字段
bind.execute(sa.text("DROP INDEX IF EXISTS ix_provider_api_keys_health_by_format"))
bind.execute(sa.text("DROP INDEX IF EXISTS ix_provider_api_keys_circuit_breaker_by_format"))
if _column_exists("provider_api_keys", "health_by_format"):
op.drop_column("provider_api_keys", "health_by_format")
if _column_exists("provider_api_keys", "circuit_breaker_by_format"):
op.drop_column("provider_api_keys", "circuit_breaker_by_format")
# 6. rpm_limit -> max_concurrent(简化版:仅重命名)
if _column_exists("provider_api_keys", "rpm_limit"):
op.alter_column("provider_api_keys", "rpm_limit", new_column_name="max_concurrent")
if _column_exists("provider_api_keys", "learned_rpm_limit"):
op.alter_column(
"provider_api_keys", "learned_rpm_limit", new_column_name="learned_max_concurrent"
)
if _column_exists("provider_api_keys", "last_rpm_peak"):
op.alter_column(
"provider_api_keys", "last_rpm_peak", new_column_name="last_concurrent_peak"
)
# 恢复已删除的字段
if not _column_exists("provider_api_keys", "rate_limit"):
op.add_column("provider_api_keys", sa.Column("rate_limit", sa.Integer(), nullable=True))
if not _column_exists("provider_api_keys", "daily_limit"):
op.add_column("provider_api_keys", sa.Column("daily_limit", sa.Integer(), nullable=True))
if not _column_exists("provider_api_keys", "monthly_limit"):
op.add_column("provider_api_keys", sa.Column("monthly_limit", sa.Integer(), nullable=True))
# 5. name -> display_name (需要先删除索引)
if _column_exists("providers", "name") and not _column_exists("providers", "display_name"):
if _index_exists("providers", "ix_providers_name"):
op.drop_index("ix_providers_name", table_name="providers")
op.alter_column("providers", "name", new_column_name="display_name")
if not _column_exists("providers", "name"):
op.add_column("providers", sa.Column("name", sa.String(100), nullable=True))
op.execute("""
UPDATE providers
SET name = LOWER(REPLACE(REPLACE(display_name, ' ', '_'), '-', '_'))
""")
op.alter_column("providers", "name", nullable=False)
if not _index_exists("providers", "ix_providers_name"):
op.create_index("ix_providers_name", "providers", ["name"], unique=True)
# 4. 删除 providers 的 timeout, max_retries, proxy
if _column_exists("providers", "proxy"):
op.drop_column("providers", "proxy")
if _column_exists("providers", "max_retries"):
op.drop_column("providers", "max_retries")
if _column_exists("providers", "timeout"):
op.drop_column("providers", "timeout")
# 3. models: global_model_id 改回 NOT NULL
result = bind.execute(sa.text("SELECT COUNT(*) FROM models WHERE global_model_id IS NULL"))
orphan_model_count = result.scalar() or 0
if orphan_model_count > 0:
alembic_logger.warning(
f"[WARN] 发现 {orphan_model_count} 个无 global_model_id 的独立模型,将被删除"
)
op.execute("DELETE FROM models WHERE global_model_id IS NULL")
alembic_logger.info(f"已删除 {orphan_model_count} 个独立模型")
op.alter_column("models", "global_model_id", nullable=False)
# 2. 删除 rate_multipliers
if _column_exists("provider_api_keys", "rate_multipliers"):
op.drop_column("provider_api_keys", "rate_multipliers")
# 1. 删除 provider_id 和 api_formats
if _index_exists("provider_api_keys", "idx_provider_api_keys_provider_id"):
op.drop_index("idx_provider_api_keys_provider_id", table_name="provider_api_keys")
if _constraint_exists("provider_api_keys", "fk_provider_api_keys_provider"):
op.drop_constraint("fk_provider_api_keys_provider", "provider_api_keys", type_="foreignkey")
if _column_exists("provider_api_keys", "api_formats"):
op.drop_column("provider_api_keys", "api_formats")
if _column_exists("provider_api_keys", "provider_id"):
op.drop_column("provider_api_keys", "provider_id")
# 恢复 endpoint_id 外键(简化版:仅创建外键,不强制 NOT NULL)
if _column_exists("provider_api_keys", "endpoint_id"):
if not _constraint_exists("provider_api_keys", "provider_api_keys_endpoint_id_fkey"):
op.create_foreign_key(
"provider_api_keys_endpoint_id_fkey",
"provider_api_keys",
"provider_endpoints",
["endpoint_id"],
["id"],
ondelete="SET NULL",
)
alembic_logger.info("[OK] Downgrade completed (simplified version)")
@@ -1,95 +0,0 @@
"""add auto_fetch_models and locked_models to provider_api_keys
Revision ID: e4ebe3233b40
Revises: m4n5o6p7q8r9
Create Date: 2026-01-13 17:59:53.119479+00:00
为 provider_api_keys 表添加自动获取模型相关字段:
1. auto_fetch_models: 是否启用自动获取模型
2. last_models_fetch_at: 最后获取时间
3. last_models_fetch_error: 最后获取错误信息
4. locked_models: 被锁定的模型列表(刷新时不会被删除)
注意: downgrade 操作会永久删除 auto_fetch_models 配置和 locked_models 数据
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
def _index_exists(index_name: str) -> bool:
"""Check if an index exists"""
bind = op.get_bind()
inspector = inspect(bind)
indexes = inspector.get_indexes("provider_api_keys")
return any(idx["name"] == index_name for idx in indexes)
# revision identifiers, used by Alembic.
revision = 'e4ebe3233b40'
down_revision = 'm4n5o6p7q8r9'
branch_labels = None
depends_on = None
def _column_exists(table_name: str, column_name: str) -> bool:
"""Check if a column exists in the table"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
"""添加自动获取模型相关字段"""
if not _column_exists("provider_api_keys", "auto_fetch_models"):
op.add_column(
"provider_api_keys",
sa.Column("auto_fetch_models", sa.Boolean(), nullable=False, server_default="false"),
)
if not _column_exists("provider_api_keys", "last_models_fetch_at"):
op.add_column(
"provider_api_keys",
sa.Column("last_models_fetch_at", sa.DateTime(timezone=True), nullable=True),
)
if not _column_exists("provider_api_keys", "last_models_fetch_error"):
op.add_column(
"provider_api_keys",
sa.Column("last_models_fetch_error", sa.Text(), nullable=True),
)
if not _column_exists("provider_api_keys", "locked_models"):
op.add_column(
"provider_api_keys",
sa.Column("locked_models", sa.JSON(), nullable=True),
)
# 添加复合索引以优化调度器查询
if not _index_exists("ix_provider_api_keys_auto_fetch_active"):
op.create_index(
"ix_provider_api_keys_auto_fetch_active",
"provider_api_keys",
["auto_fetch_models", "is_active"],
postgresql_where=sa.text("auto_fetch_models = true AND is_active = true"),
)
def downgrade() -> None:
"""移除自动获取模型相关字段"""
# 先删除索引
if _index_exists("ix_provider_api_keys_auto_fetch_active"):
op.drop_index("ix_provider_api_keys_auto_fetch_active", table_name="provider_api_keys")
if _column_exists("provider_api_keys", "locked_models"):
op.drop_column("provider_api_keys", "locked_models")
if _column_exists("provider_api_keys", "last_models_fetch_error"):
op.drop_column("provider_api_keys", "last_models_fetch_error")
if _column_exists("provider_api_keys", "last_models_fetch_at"):
op.drop_column("provider_api_keys", "last_models_fetch_at")
if _column_exists("provider_api_keys", "auto_fetch_models"):
op.drop_column("provider_api_keys", "auto_fetch_models")
@@ -1,104 +0,0 @@
"""add header_rules to provider_endpoints and is_locked to api_keys
Revision ID: 6d579000e511
Revises: e4ebe3233b40
Create Date: 2026-01-15 23:00:00.000000+00:00
变更:
1. provider_endpoints 表: 添加 header_rules 字段,迁移 headers 数据
2. api_keys 表: 添加 is_locked 字段(管理员锁定标志)
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import JSON
# revision identifiers, used by Alembic.
revision = '6d579000e511'
down_revision = 'e4ebe3233b40'
branch_labels = None
depends_on = None
def _column_exists(connection, table: str, column: str) -> bool:
"""检查列是否存在"""
result = connection.execute(
sa.text("""
SELECT 1 FROM information_schema.columns
WHERE table_name = :table AND column_name = :column
"""),
{"table": table, "column": column}
)
return result.fetchone() is not None
def upgrade() -> None:
"""添加 header_rules 字段并迁移现有 headers 数据;添加 is_locked 字段"""
connection = op.get_bind()
# ========== provider_endpoints.header_rules ==========
# 1. 添加 header_rules 列(幂等)
if not _column_exists(connection, 'provider_endpoints', 'header_rules'):
op.add_column('provider_endpoints', sa.Column('header_rules', JSON, nullable=True))
# 2. 批量迁移:headers -> header_rules
# 使用纯 SQL 将 {"k1":"v1", "k2":"v2"} 转换为 [{"action":"set","key":"k1","value":"v1"}, ...]
if _column_exists(connection, 'provider_endpoints', 'headers'):
connection.execute(
sa.text("""
UPDATE provider_endpoints
SET header_rules = (
SELECT jsonb_agg(
jsonb_build_object('action', 'set', 'key', key, 'value', value)
)
FROM jsonb_each_text(headers::jsonb)
)
WHERE headers IS NOT NULL
AND headers::text != '{}'
AND jsonb_typeof(headers::jsonb) = 'object'
AND header_rules IS NULL
""")
)
# 3. 删除旧列
op.drop_column('provider_endpoints', 'headers')
# ========== api_keys.is_locked ==========
if not _column_exists(connection, 'api_keys', 'is_locked'):
op.add_column(
'api_keys',
sa.Column('is_locked', sa.Boolean(), nullable=False, server_default='false')
)
def downgrade() -> None:
"""移除 header_rules 字段,恢复 headers 字段;移除 is_locked 字段"""
connection = op.get_bind()
# ========== api_keys.is_locked ==========
if _column_exists(connection, 'api_keys', 'is_locked'):
op.drop_column('api_keys', 'is_locked')
# ========== provider_endpoints.header_rules ==========
# 1. 添加 headers 列(幂等)
if not _column_exists(connection, 'provider_endpoints', 'headers'):
op.add_column('provider_endpoints', sa.Column('headers', JSON, nullable=True))
# 2. 批量迁移:header_rules -> headers(仅提取 set 操作)
if _column_exists(connection, 'provider_endpoints', 'header_rules'):
connection.execute(
sa.text("""
UPDATE provider_endpoints
SET headers = (
SELECT jsonb_object_agg(rule->>'key', rule->>'value')
FROM jsonb_array_elements(header_rules::jsonb) AS rule
WHERE rule->>'action' = 'set'
AND rule->>'key' IS NOT NULL
)
WHERE header_rules IS NOT NULL
AND jsonb_typeof(header_rules::jsonb) = 'array'
AND jsonb_array_length(header_rules::jsonb) > 0
""")
)
# 3. 删除 header_rules 列
op.drop_column('provider_endpoints', 'header_rules')
@@ -1,127 +0,0 @@
"""add global_priority_by_format and remove deprecated fields
Revision ID: ddd59cdf0349
Revises: 6d579000e511
Create Date: 2026-01-16 12:00:00.000000+00:00
变更:
1. provider_api_keys 表: 添加 global_priority_by_format 字段(按 API 格式的全局优先级)
2. 迁移现有 global_priority 数据到新字段
3. 删除已废弃的 global_priority 字段
4. 删除已废弃的 rate_multiplier 字段(已被 rate_multipliers 替代)
5. 删除已废弃的 providers.timeout 字段(由环境变量控制)
6. 删除已废弃的 provider_endpoints.timeout 字段(由环境变量控制)
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import JSON
# revision identifiers, used by Alembic.
revision = 'ddd59cdf0349'
down_revision = '6d579000e511'
branch_labels = None
depends_on = None
def _column_exists(connection, table: str, column: str) -> bool:
"""检查列是否存在"""
result = connection.execute(
sa.text("""
SELECT 1 FROM information_schema.columns
WHERE table_name = :table AND column_name = :column
"""),
{"table": table, "column": column}
)
return result.fetchone() is not None
def upgrade():
connection = op.get_bind()
# 1. 添加 global_priority_by_format 字段
if not _column_exists(connection, 'provider_api_keys', 'global_priority_by_format'):
op.add_column(
'provider_api_keys',
sa.Column('global_priority_by_format', JSON, nullable=True)
)
# 2. 迁移现有 global_priority 数据到新字段
# 对于有 global_priority 的 Key,将其值应用到所有支持的 api_formats
if _column_exists(connection, 'provider_api_keys', 'global_priority'):
# 将 JSON 数组转换为 text[] 后使用 unnest
connection.execute(sa.text("""
UPDATE provider_api_keys
SET global_priority_by_format = (
SELECT jsonb_object_agg(format, global_priority)
FROM jsonb_array_elements_text(api_formats::jsonb) AS format
)
WHERE global_priority IS NOT NULL
AND api_formats IS NOT NULL
AND jsonb_array_length(api_formats::jsonb) > 0
AND global_priority_by_format IS NULL
"""))
# 3. 删除 global_priority 字段
op.drop_column('provider_api_keys', 'global_priority')
# 4. 删除 rate_multiplier 字段(已被 rate_multipliers 替代)
if _column_exists(connection, 'provider_api_keys', 'rate_multiplier'):
op.drop_column('provider_api_keys', 'rate_multiplier')
# 5. 删除 providers.timeout 字段(由环境变量控制)
if _column_exists(connection, 'providers', 'timeout'):
op.drop_column('providers', 'timeout')
# 6. 删除 provider_endpoints.timeout 字段(由环境变量控制)
if _column_exists(connection, 'provider_endpoints', 'timeout'):
op.drop_column('provider_endpoints', 'timeout')
def downgrade():
connection = op.get_bind()
# 1. 恢复 rate_multiplier 字段
if not _column_exists(connection, 'provider_api_keys', 'rate_multiplier'):
op.add_column(
'provider_api_keys',
sa.Column('rate_multiplier', sa.Float, nullable=False, server_default='1.0')
)
# 2. 恢复 global_priority 字段并迁移数据
if not _column_exists(connection, 'provider_api_keys', 'global_priority'):
op.add_column(
'provider_api_keys',
sa.Column('global_priority', sa.Integer, nullable=True)
)
# 从 global_priority_by_format 迁移数据(取第一个格式的优先级值)
if _column_exists(connection, 'provider_api_keys', 'global_priority_by_format'):
connection.execute(sa.text("""
UPDATE provider_api_keys
SET global_priority = (
SELECT (value::text)::integer
FROM jsonb_each(global_priority_by_format::jsonb)
LIMIT 1
)
WHERE global_priority_by_format IS NOT NULL
AND jsonb_typeof(global_priority_by_format::jsonb) = 'object'
AND global_priority IS NULL
"""))
# 3. 删除 global_priority_by_format 字段
if _column_exists(connection, 'provider_api_keys', 'global_priority_by_format'):
op.drop_column('provider_api_keys', 'global_priority_by_format')
# 4. 恢复 providers.timeout 字段
if not _column_exists(connection, 'providers', 'timeout'):
op.add_column(
'providers',
sa.Column('timeout', sa.Integer, nullable=True, server_default='300')
)
# 5. 恢复 provider_endpoints.timeout 字段
if not _column_exists(connection, 'provider_endpoints', 'timeout'):
op.add_column(
'provider_endpoints',
sa.Column('timeout', sa.Integer, nullable=True, server_default='300')
)
@@ -1,223 +0,0 @@
"""make users email/password nullable add email_verified and oauth tables
Revision ID: 33e347f97c0c
Revises: ddd59cdf0349
Create Date: 2026-01-18 11:18:15.940559+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = "33e347f97c0c"
down_revision = "ddd59cdf0349"
branch_labels = None
depends_on = None
def column_exists(table_name: str, column_name: str) -> bool:
"""检查列是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def table_exists(table_name: str) -> bool:
"""检查表是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_is_nullable(table_name: str, column_name: str) -> bool:
"""检查列是否允许 NULL"""
bind = op.get_bind()
inspector = inspect(bind)
for col in inspector.get_columns(table_name):
if col["name"] == column_name:
return col["nullable"]
return False
def enum_value_exists(enum_name: str, value: str) -> bool:
"""检查 PostgreSQL ENUM 是否包含指定值"""
bind = op.get_bind()
if bind.dialect.name != "postgresql":
return True # 非 PostgreSQL 跳过检查
result = bind.execute(
sa.text(
"SELECT 1 FROM pg_enum WHERE enumlabel = :value "
"AND enumtypid = (SELECT oid FROM pg_type WHERE typname = :enum_name)"
),
{"value": value, "enum_name": enum_name},
).first()
return result is not None
def upgrade() -> None:
"""应用迁移:升级到新版本"""
bind = op.get_bind()
# ========== Part 1: users 表修改 ==========
# 1) 新增 email_verified
if not column_exists("users", "email_verified"):
op.add_column("users", sa.Column("email_verified", sa.Boolean(), nullable=True))
# 历史数据回填:已有邮箱的用户默认视为已验证
op.execute(sa.text("UPDATE users SET email_verified = true WHERE email IS NOT NULL"))
op.execute(sa.text("UPDATE users SET email_verified = false WHERE email IS NULL"))
# 收紧约束
op.alter_column("users", "email_verified", existing_type=sa.Boolean(), nullable=False)
# 2) email 放宽为可空
if not column_is_nullable("users", "email"):
op.alter_column(
"users",
"email",
existing_type=sa.String(length=255),
nullable=True,
)
# 3) password_hash 放宽为可空
if not column_is_nullable("users", "password_hash"):
op.alter_column(
"users",
"password_hash",
existing_type=sa.String(length=255),
nullable=True,
)
# ========== Part 2: OAuth 相关 ==========
# 4) 扩展 authsource enum
if bind.dialect.name == "postgresql" and not enum_value_exists("authsource", "oauth"):
ctx = op.get_context()
with ctx.autocommit_block():
op.execute("ALTER TYPE authsource ADD VALUE IF NOT EXISTS 'oauth'")
# 5) OAuth provider 配置表
if not table_exists("oauth_providers"):
op.create_table(
"oauth_providers",
sa.Column("provider_type", sa.String(length=50), primary_key=True),
sa.Column("display_name", sa.String(length=100), nullable=False),
sa.Column("client_id", sa.String(length=255), nullable=False),
sa.Column("client_secret_encrypted", sa.Text(), nullable=True),
sa.Column("authorization_url_override", sa.String(length=500), nullable=True),
sa.Column("token_url_override", sa.String(length=500), nullable=True),
sa.Column("userinfo_url_override", sa.String(length=500), nullable=True),
sa.Column("scopes", sa.JSON(), nullable=True),
sa.Column("redirect_uri", sa.String(length=500), nullable=False),
sa.Column("frontend_callback_url", sa.String(length=500), nullable=False),
sa.Column("attribute_mapping", sa.JSON(), nullable=True),
sa.Column("extra_config", sa.JSON(), nullable=True),
sa.Column(
"is_enabled", sa.Boolean(), nullable=False, server_default=sa.text("false")
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
)
# 6) 用户 OAuth 绑定关系表
if not table_exists("user_oauth_links"):
op.create_table(
"user_oauth_links",
sa.Column("id", sa.String(length=36), primary_key=True),
sa.Column(
"user_id",
sa.String(length=36),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"provider_type",
sa.String(length=50),
sa.ForeignKey("oauth_providers.provider_type", ondelete="CASCADE"),
nullable=False,
),
sa.Column("provider_user_id", sa.String(length=255), nullable=False),
sa.Column("provider_username", sa.String(length=255), nullable=True),
sa.Column("provider_email", sa.String(length=255), nullable=True),
sa.Column("extra_data", sa.JSON(), nullable=True),
sa.Column(
"linked_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.Column("last_login_at", sa.DateTime(timezone=True), nullable=True),
sa.UniqueConstraint(
"provider_type", "provider_user_id", name="uq_oauth_provider_user"
),
sa.UniqueConstraint("user_id", "provider_type", name="uq_user_oauth_provider"),
)
op.create_index("ix_user_oauth_links_user_id", "user_oauth_links", ["user_id"])
op.create_index(
"ix_user_oauth_links_provider_type", "user_oauth_links", ["provider_type"]
)
def downgrade() -> None:
"""回滚迁移:降级到旧版本"""
bind = op.get_bind()
# ========== Part 2: OAuth 相关(先删除,因为有外键依赖) ==========
if table_exists("user_oauth_links"):
op.drop_index("ix_user_oauth_links_provider_type", table_name="user_oauth_links")
op.drop_index("ix_user_oauth_links_user_id", table_name="user_oauth_links")
op.drop_table("user_oauth_links")
if table_exists("oauth_providers"):
op.drop_table("oauth_providers")
# 注意:Postgres 不支持从 ENUM 删除值,authsource 不回退
# ========== Part 1: users 表修改 ==========
# 降级前检查:避免把包含 NULL 的列强制改回 NOT NULL
has_null_email = bind.execute(
sa.text("SELECT 1 FROM users WHERE email IS NULL LIMIT 1")
).first()
if has_null_email:
raise RuntimeError("Cannot downgrade: users.email contains NULL values")
has_null_password = bind.execute(
sa.text("SELECT 1 FROM users WHERE password_hash IS NULL LIMIT 1")
).first()
if has_null_password:
raise RuntimeError("Cannot downgrade: users.password_hash contains NULL values")
# 恢复 NOT NULL 约束
if column_is_nullable("users", "email"):
op.alter_column(
"users",
"email",
existing_type=sa.String(length=255),
nullable=False,
)
if column_is_nullable("users", "password_hash"):
op.alter_column(
"users",
"password_hash",
existing_type=sa.String(length=255),
nullable=False,
)
if column_exists("users", "email_verified"):
op.drop_column("users", "email_verified")
@@ -1,65 +0,0 @@
"""add_stats_daily_provider_table
Revision ID: c868729753ad
Revises: 33e347f97c0c
Create Date: 2026-01-19 05:19:49.634662+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = 'c868729753ad'
down_revision = '33e347f97c0c'
branch_labels = None
depends_on = None
def table_exists(table_name: str) -> bool:
"""检查表是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def index_exists(table_name: str, index_name: str) -> bool:
"""检查索引是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
indexes = [idx['name'] for idx in inspector.get_indexes(table_name)]
return index_name in indexes
def upgrade() -> None:
"""应用迁移:升级到新版本"""
if not table_exists('stats_daily_provider'):
op.create_table(
'stats_daily_provider',
sa.Column('id', sa.String(length=36), nullable=False),
sa.Column('date', sa.DateTime(timezone=True), nullable=False),
sa.Column('provider_name', sa.String(length=100), nullable=False),
sa.Column('total_requests', sa.Integer(), nullable=False),
sa.Column('input_tokens', sa.BigInteger(), nullable=False),
sa.Column('output_tokens', sa.BigInteger(), nullable=False),
sa.Column('cache_creation_tokens', sa.BigInteger(), nullable=False),
sa.Column('cache_read_tokens', sa.BigInteger(), nullable=False),
sa.Column('total_cost', sa.Float(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('date', 'provider_name', name='uq_stats_daily_provider')
)
op.create_index('idx_stats_daily_provider_date', 'stats_daily_provider', ['date'], unique=False)
op.create_index('idx_stats_daily_provider_date_provider', 'stats_daily_provider', ['date', 'provider_name'], unique=False)
def downgrade() -> None:
"""回滚迁移:降级到旧版本"""
if table_exists('stats_daily_provider'):
if index_exists('stats_daily_provider', 'idx_stats_daily_provider_date_provider'):
op.drop_index('idx_stats_daily_provider_date_provider', table_name='stats_daily_provider')
if index_exists('stats_daily_provider', 'idx_stats_daily_provider_date'):
op.drop_index('idx_stats_daily_provider_date', table_name='stats_daily_provider')
op.drop_table('stats_daily_provider')
@@ -1,51 +0,0 @@
"""add_format_acceptance_config_to_provider_endpoints
Revision ID: 4b4c7b0df1a2
Revises: c868729753ad
Create Date: 2026-01-21 18:45:00+00:00
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = "4b4c7b0df1a2"
down_revision = "c868729753ad"
branch_labels = None
depends_on = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
if not table_exists("provider_endpoints"):
return
if column_exists("provider_endpoints", "format_acceptance_config"):
return
op.add_column(
"provider_endpoints",
sa.Column("format_acceptance_config", sa.JSON(), nullable=True),
)
def downgrade() -> None:
if not table_exists("provider_endpoints"):
return
if not column_exists("provider_endpoints", "format_acceptance_config"):
return
op.drop_column("provider_endpoints", "format_acceptance_config")
@@ -1,114 +0,0 @@
"""add_format_conversion_tracking_and_model_filter_patterns_and_provider_timeout
Revision ID: f7c8d9e0a1b2
Revises: 4b4c7b0df1a2
Create Date: 2026-01-27 10:00:00+00:00
Changes:
1. usage 表: 添加 endpoint_api_format 和 has_format_conversion 字段
2. provider_api_keys 表: 添加 model_include_patterns 和 model_exclude_patterns 字段
- 支持通配符规则自动过滤从上游获取的模型列表
- 包含规则和排除规则(支持 * 和 ? 通配符)
3. providers 表: 添加 stream_first_byte_timeout 和 request_timeout 字段
- 允许每个提供商单独配置超时时间
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = "f7c8d9e0a1b2"
down_revision = "4b4c7b0df1a2"
branch_labels = None
depends_on = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
# === usage 表: 格式转换追踪 ===
if table_exists("usage"):
# 添加 endpoint_api_format 字段(端点原生 API 格式)
if not column_exists("usage", "endpoint_api_format"):
op.add_column(
"usage",
sa.Column("endpoint_api_format", sa.String(50), nullable=True),
)
# 添加 has_format_conversion 字段(是否发生了格式转换)
if not column_exists("usage", "has_format_conversion"):
op.add_column(
"usage",
sa.Column("has_format_conversion", sa.Boolean(), nullable=True, server_default="false"),
)
# === provider_api_keys 表: 模型过滤规则 ===
if table_exists("provider_api_keys"):
# 添加 model_include_patterns 字段(包含规则,支持 * 和 ? 通配符)
if not column_exists("provider_api_keys", "model_include_patterns"):
op.add_column(
"provider_api_keys",
sa.Column("model_include_patterns", sa.JSON(), nullable=True),
)
# 添加 model_exclude_patterns 字段(排除规则,支持 * 和 ? 通配符)
if not column_exists("provider_api_keys", "model_exclude_patterns"):
op.add_column(
"provider_api_keys",
sa.Column("model_exclude_patterns", sa.JSON(), nullable=True),
)
# === providers 表: 超时配置 ===
if table_exists("providers"):
# 添加 stream_first_byte_timeout 字段(流式请求首字节超时)
if not column_exists("providers", "stream_first_byte_timeout"):
op.add_column(
"providers",
sa.Column("stream_first_byte_timeout", sa.Float(), nullable=True),
)
# 添加 request_timeout 字段(非流式请求整体超时)
if not column_exists("providers", "request_timeout"):
op.add_column(
"providers",
sa.Column("request_timeout", sa.Float(), nullable=True),
)
def downgrade() -> None:
# === providers 表: 移除超时配置 ===
if table_exists("providers"):
if column_exists("providers", "request_timeout"):
op.drop_column("providers", "request_timeout")
if column_exists("providers", "stream_first_byte_timeout"):
op.drop_column("providers", "stream_first_byte_timeout")
# === provider_api_keys 表: 移除模型过滤规则 ===
if table_exists("provider_api_keys"):
if column_exists("provider_api_keys", "model_exclude_patterns"):
op.drop_column("provider_api_keys", "model_exclude_patterns")
if column_exists("provider_api_keys", "model_include_patterns"):
op.drop_column("provider_api_keys", "model_include_patterns")
# === usage 表: 移除格式转换追踪 ===
if table_exists("usage"):
if column_exists("usage", "has_format_conversion"):
op.drop_column("usage", "has_format_conversion")
if column_exists("usage", "endpoint_api_format"):
op.drop_column("usage", "endpoint_api_format")
@@ -1,58 +0,0 @@
"""add_keep_priority_on_conversion_to_providers
Revision ID: 364680d1bc99
Revises: f7c8d9e0a1b2
Create Date: 2026-01-28 12:00:00+00:00
Changes:
1. providers 表: 添加 keep_priority_on_conversion 字段
- 格式转换时是否保持提供商原优先级
- 默认 False:需要格式转换时,候选会被降级到不需要转换的候选之后
- 设为 True:即使需要格式转换,也保持原优先级排名
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision = "364680d1bc99"
down_revision = "f7c8d9e0a1b2"
branch_labels = None
depends_on = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
# === providers 表: 添加格式转换优先级保持配置 ===
if table_exists("providers"):
if not column_exists("providers", "keep_priority_on_conversion"):
op.add_column(
"providers",
sa.Column(
"keep_priority_on_conversion",
sa.Boolean(),
nullable=False,
server_default="false",
),
)
def downgrade() -> None:
# === providers 表: 移除格式转换优先级保持配置 ===
if table_exists("providers"):
if column_exists("providers", "keep_priority_on_conversion"):
op.drop_column("providers", "keep_priority_on_conversion")
@@ -1,51 +0,0 @@
"""Add auth_type and auth_config fields to provider_api_keys table
Revision ID: 7f6f8065f517
Revises: 364680d1bc99
Create Date: 2026-01-30 10:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect
# revision identifiers, used by Alembic.
revision: str = "7f6f8065f517"
down_revision: Union[str, None] = "364680d1bc99"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def column_exists(table_name: str, column_name: str) -> bool:
"""检查列是否已存在"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
# 添加 auth_type 字段,默认值为 "api_key"
if not column_exists("provider_api_keys", "auth_type"):
op.add_column(
"provider_api_keys",
sa.Column("auth_type", sa.String(20), nullable=False, server_default="api_key"),
)
# 添加 auth_config 字段(Text,存储加密后的认证配置)
if not column_exists("provider_api_keys", "auth_config"):
op.add_column(
"provider_api_keys",
sa.Column("auth_config", sa.Text, nullable=True),
)
def downgrade() -> None:
if column_exists("provider_api_keys", "auth_config"):
op.drop_column("provider_api_keys", "auth_config")
if column_exists("provider_api_keys", "auth_type"):
op.drop_column("provider_api_keys", "auth_type")
@@ -1,112 +0,0 @@
"""Add video_tasks table
Revision ID: b6f1a2c5d8e9
Revises: 7f6f8065f517
Create Date: 2026-01-30 18:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "b6f1a2c5d8e9"
down_revision: Union[str, None] = "7f6f8065f517"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def upgrade() -> None:
if table_exists("video_tasks"):
return
op.create_table(
"video_tasks",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("external_task_id", sa.String(200), nullable=True, index=False),
sa.Column("user_id", sa.String(36), sa.ForeignKey("users.id"), nullable=False),
sa.Column("api_key_id", sa.String(36), sa.ForeignKey("api_keys.id"), nullable=True),
sa.Column("provider_id", sa.String(36), sa.ForeignKey("providers.id"), nullable=True),
sa.Column(
"endpoint_id", sa.String(36), sa.ForeignKey("provider_endpoints.id"), nullable=True
),
sa.Column("key_id", sa.String(36), sa.ForeignKey("provider_api_keys.id"), nullable=True),
sa.Column("client_api_format", sa.String(50), nullable=False),
sa.Column("provider_api_format", sa.String(50), nullable=False),
sa.Column("format_converted", sa.Boolean(), server_default=sa.false()),
sa.Column("model", sa.String(100), nullable=False),
sa.Column("prompt", sa.Text(), nullable=False),
sa.Column("original_request_body", sa.JSON(), nullable=True),
sa.Column("converted_request_body", sa.JSON(), nullable=True),
sa.Column("duration_seconds", sa.Integer(), server_default=sa.text("4")),
sa.Column("resolution", sa.String(20), server_default=sa.text("'720p'")),
sa.Column("aspect_ratio", sa.String(10), server_default=sa.text("'16:9'")),
sa.Column("size", sa.String(20), nullable=True),
sa.Column("status", sa.String(20), server_default=sa.text("'pending'")),
sa.Column("progress_percent", sa.Integer(), server_default=sa.text("0")),
sa.Column("progress_message", sa.String(500), nullable=True),
sa.Column("video_url", sa.String(2000), nullable=True),
sa.Column("video_urls", sa.JSON(), nullable=True),
sa.Column("thumbnail_url", sa.String(2000), nullable=True),
sa.Column("video_size_bytes", sa.BigInteger(), nullable=True),
sa.Column("video_expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("stored_video_path", sa.String(500), nullable=True),
sa.Column("storage_provider", sa.String(50), nullable=True),
sa.Column("error_code", sa.String(50), nullable=True),
sa.Column("error_message", sa.Text(), nullable=True),
sa.Column("retry_count", sa.Integer(), server_default=sa.text("0")),
sa.Column("max_retries", sa.Integer(), server_default=sa.text("3")),
sa.Column("poll_interval_seconds", sa.Integer(), server_default=sa.text("10")),
sa.Column("next_poll_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("poll_count", sa.Integer(), server_default=sa.text("0")),
sa.Column("max_poll_count", sa.Integer(), server_default=sa.text("360")),
sa.Column(
"remixed_from_task_id",
sa.String(36),
sa.ForeignKey("video_tasks.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.Column("submitted_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
),
)
op.create_index("idx_video_tasks_user_status", "video_tasks", ["user_id", "status"])
op.create_index("idx_video_tasks_next_poll", "video_tasks", ["next_poll_at"])
op.create_index("idx_video_tasks_external_id", "video_tasks", ["external_task_id"])
# 唯一约束:同一用户不能有重复的 external_task_id
op.create_unique_constraint(
"uq_video_tasks_user_external_id",
"video_tasks",
["user_id", "external_task_id"],
)
def downgrade() -> None:
if not table_exists("video_tasks"):
return
op.drop_constraint("uq_video_tasks_user_external_id", "video_tasks", type_="unique")
op.drop_index("idx_video_tasks_external_id", table_name="video_tasks")
op.drop_index("idx_video_tasks_next_poll", table_name="video_tasks")
op.drop_index("idx_video_tasks_user_status", table_name="video_tasks")
op.drop_table("video_tasks")
@@ -1,180 +0,0 @@
"""Add billing system tables and video_tasks.request_metadata
Revision ID: c8d2e4f6a1b3
Revises: b6f1a2c5d8e9
Create Date: 2026-01-31 12:00:00.000000
"""
from __future__ import annotations
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy import inspect
from sqlalchemy.dialects.postgresql import JSONB
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "c8d2e4f6a1b3"
down_revision: Union[str, None] = "b6f1a2c5d8e9"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def index_exists(table_name: str, index_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
try:
indexes = inspector.get_indexes(table_name)
except Exception:
return False
return any(idx.get("name") == index_name for idx in indexes)
def upgrade() -> None:
# ==================== video_tasks.request_metadata ====================
if not column_exists("video_tasks", "request_metadata"):
op.add_column(
"video_tasks",
sa.Column("request_metadata", sa.JSON(), nullable=True),
)
# ==================== billing_rules ====================
if not table_exists("billing_rules"):
op.create_table(
"billing_rules",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column(
"global_model_id",
sa.String(36),
sa.ForeignKey("global_models.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column(
"model_id",
sa.String(36),
sa.ForeignKey("models.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column("name", sa.String(100), nullable=False),
sa.Column("task_type", sa.String(20), nullable=False, server_default="chat"),
sa.Column("expression", sa.Text(), nullable=False),
sa.Column("variables", JSONB, nullable=False, server_default=sa.text("'{}'::jsonb")),
sa.Column(
"dimension_mappings", JSONB, nullable=False, server_default=sa.text("'{}'::jsonb")
),
sa.Column("is_enabled", sa.Boolean(), nullable=False, server_default=sa.text("true")),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.CheckConstraint(
"(global_model_id IS NOT NULL AND model_id IS NULL) OR "
"(global_model_id IS NULL AND model_id IS NOT NULL)",
name="chk_billing_rules_model_ref",
),
)
# Partial unique indexes for enabled rules
if table_exists("billing_rules"):
if not index_exists("billing_rules", "uq_billing_rules_global_model_task"):
op.create_index(
"uq_billing_rules_global_model_task",
"billing_rules",
["global_model_id", "task_type"],
unique=True,
postgresql_where=sa.text("is_enabled = TRUE AND global_model_id IS NOT NULL"),
)
if not index_exists("billing_rules", "uq_billing_rules_model_task"):
op.create_index(
"uq_billing_rules_model_task",
"billing_rules",
["model_id", "task_type"],
unique=True,
postgresql_where=sa.text("is_enabled = TRUE AND model_id IS NOT NULL"),
)
# ==================== dimension_collectors ====================
if not table_exists("dimension_collectors"):
op.create_table(
"dimension_collectors",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("api_format", sa.String(50), nullable=False),
sa.Column("task_type", sa.String(20), nullable=False),
sa.Column("dimension_name", sa.String(100), nullable=False),
sa.Column("source_type", sa.String(20), nullable=False),
sa.Column("source_path", sa.String(200), nullable=True),
sa.Column("value_type", sa.String(20), nullable=False, server_default="float"),
sa.Column("transform_expression", sa.Text(), nullable=True),
sa.Column("default_value", sa.String(100), nullable=True),
sa.Column("priority", sa.Integer(), nullable=False, server_default="0"),
sa.Column("is_enabled", sa.Boolean(), nullable=False, server_default=sa.text("true")),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("now()"),
),
sa.CheckConstraint(
"(source_type = 'computed' AND source_path IS NULL AND transform_expression IS NOT NULL) OR "
"(source_type != 'computed' AND source_path IS NOT NULL)",
name="chk_dimension_collectors_source_config",
),
)
if table_exists("dimension_collectors"):
if not index_exists("dimension_collectors", "uq_dimension_collectors_enabled"):
op.create_index(
"uq_dimension_collectors_enabled",
"dimension_collectors",
["api_format", "task_type", "dimension_name", "priority"],
unique=True,
postgresql_where=sa.text("is_enabled = TRUE"),
)
def downgrade() -> None:
# Drop in reverse order
if table_exists("dimension_collectors"):
if index_exists("dimension_collectors", "uq_dimension_collectors_enabled"):
op.drop_index("uq_dimension_collectors_enabled", table_name="dimension_collectors")
op.drop_table("dimension_collectors")
if table_exists("billing_rules"):
if index_exists("billing_rules", "uq_billing_rules_model_task"):
op.drop_index("uq_billing_rules_model_task", table_name="billing_rules")
if index_exists("billing_rules", "uq_billing_rules_global_model_task"):
op.drop_index("uq_billing_rules_global_model_task", table_name="billing_rules")
op.drop_table("billing_rules")
if column_exists("video_tasks", "request_metadata"):
op.drop_column("video_tasks", "request_metadata")
@@ -1,462 +0,0 @@
"""Add api_family/endpoint_kind and migrate api_format to endpoint signature keys
Revision ID: cf40e6a5c5b1
Revises: c8d2e4f6a1b3
Create Date: 2026-01-31 15:30:00.000000
"""
from __future__ import annotations
import json
from datetime import datetime, timezone
from typing import Sequence, Union
from uuid import uuid4
import sqlalchemy as sa
from sqlalchemy import inspect, text
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "cf40e6a5c5b1"
down_revision: Union[str, None] = "c8d2e4f6a1b3"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _json_loads(val):
if val is None:
return None
if isinstance(val, (dict, list)):
return val
if isinstance(val, str):
try:
return json.loads(val)
except Exception:
return None
return None
def _json_dumps(val):
"""将 dict/list 转为 JSON 字符串,None 保持 None"""
if val is None:
return None
if isinstance(val, str):
return val
return json.dumps(val)
def _normalize_signature(value: str | None) -> str | None:
"""
Normalize legacy api_format / signature-ish strings to canonical signature key.
- canonical: `<family>:<kind>` (lowercase)
- legacy examples: "OPENAI", "OPENAI_CLI", "GEMINI_VIDEO"
"""
if value is None:
return None
raw = str(value).strip()
if not raw:
return None
if ":" in raw:
fam, kind = raw.split(":", 1)
fam = fam.strip().lower()
kind = kind.strip().lower()
if not fam or not kind:
return None
return f"{fam}:{kind}"
upper = raw.upper()
if upper.startswith("CLAUDE"):
fam = "claude"
elif upper.startswith("OPENAI"):
fam = "openai"
elif upper.startswith("GEMINI"):
fam = "gemini"
else:
return None
kind = "chat"
if upper.endswith("_CLI"):
kind = "cli"
elif upper.endswith("_VIDEO"):
kind = "video"
return f"{fam}:{kind}"
def _normalize_signature_list(values) -> list[str] | None:
if values is None:
return None
if isinstance(values, str):
values = _json_loads(values)
if not isinstance(values, list):
return None
out: list[str] = []
seen: set[str] = set()
for v in values:
sig = _normalize_signature(str(v) if v is not None else None)
if not sig:
continue
if sig in seen:
continue
seen.add(sig)
out.append(sig)
return out
def _normalize_signature_dict(values) -> dict | None:
if values is None:
return None
if isinstance(values, str):
values = _json_loads(values)
if not isinstance(values, dict):
return None
out: dict = {}
for k, v in values.items():
sig = _normalize_signature(str(k) if k is not None else None)
if not sig:
continue
out[sig] = v
return out
def _add_video_variants(formats: list[str]) -> list[str]:
"""
保持原有格式,不自动补齐 video 变体。
"""
return formats
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def index_exists(table_name: str, index_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
try:
indexes = inspector.get_indexes(table_name)
except Exception:
return False
return any(idx.get("name") == index_name for idx in indexes)
def _migrate_format_acceptance_config(cfg) -> dict | None:
cfg_obj = _json_loads(cfg)
if not isinstance(cfg_obj, dict):
return cfg_obj if cfg_obj is None else None
for key in ("accept_formats", "reject_formats"):
raw = cfg_obj.get(key)
if not isinstance(raw, list):
continue
normalized = _normalize_signature_list(raw) or []
cfg_obj[key] = normalized
return cfg_obj
def migrate_provider_endpoints(connection) -> None:
"""
- 将 provider_endpoints.api_format 统一迁移为 signature key(小写)
- 填充/校准 api_family / endpoint_kind
- 迁移 format_acceptance_config 中的 accept/reject formats
"""
rows = connection.execute(text("""
SELECT
id,
api_format,
api_family,
endpoint_kind,
format_acceptance_config
FROM provider_endpoints
""")).fetchall()
for row in rows:
sig = _normalize_signature(row.api_format)
if not sig:
continue
fam, kind = sig.split(":", 1)
cfg = _migrate_format_acceptance_config(row.format_acceptance_config)
connection.execute(
text("""
UPDATE provider_endpoints
SET
api_format = :api_format,
api_family = :api_family,
endpoint_kind = :endpoint_kind,
format_acceptance_config = CAST(:format_acceptance_config AS json)
WHERE id = :id
"""),
{
"id": row.id,
"api_format": sig,
"api_family": fam,
"endpoint_kind": kind,
"format_acceptance_config": _json_dumps(cfg),
},
)
def create_video_endpoints(connection) -> None:
"""
不再自动创建 video endpoint,保持原有配置。
"""
pass
def migrate_provider_api_keys(connection) -> None:
"""
迁移 provider_api_keys:
- api_formats -> signature keys(并补齐 video 变体)
- dict 字段 key -> signature keys(rate_multipliers/global_priority/health/circuit_breaker)
- rate_multipliers/global_priority_by_format 复制 chat -> video(如 openai:chat -> openai:video)
"""
rows = connection.execute(text("""
SELECT
id,
api_formats,
rate_multipliers,
global_priority_by_format,
health_by_format,
circuit_breaker_by_format
FROM provider_api_keys
""")).fetchall()
for row in rows:
api_formats = _normalize_signature_list(row.api_formats)
if api_formats is not None:
api_formats = _add_video_variants(api_formats)
rate_multipliers = _normalize_signature_dict(row.rate_multipliers)
global_priority_by_format = _normalize_signature_dict(row.global_priority_by_format)
health_by_format = _normalize_signature_dict(row.health_by_format)
circuit_breaker_by_format = _normalize_signature_dict(row.circuit_breaker_by_format)
connection.execute(
text("""
UPDATE provider_api_keys
SET
api_formats = CAST(:api_formats AS json),
rate_multipliers = CAST(:rate_multipliers AS json),
global_priority_by_format = CAST(:global_priority_by_format AS json),
health_by_format = CAST(:health_by_format AS json),
circuit_breaker_by_format = CAST(:circuit_breaker_by_format AS json)
WHERE id = :id
"""),
{
"id": row.id,
"api_formats": _json_dumps(api_formats),
"rate_multipliers": _json_dumps(rate_multipliers),
"global_priority_by_format": _json_dumps(global_priority_by_format),
"health_by_format": _json_dumps(health_by_format),
"circuit_breaker_by_format": _json_dumps(circuit_breaker_by_format),
},
)
def migrate_allowed_api_formats(connection, *, table_name: str) -> None:
"""迁移 users/api_keys.allowed_api_formats 为 signature keys(并补齐 video 变体)。"""
if not table_exists(table_name):
return
rows = connection.execute(text(f"""
SELECT id, allowed_api_formats
FROM {table_name}
""")).fetchall()
for row in rows:
allowed = _normalize_signature_list(row.allowed_api_formats)
if allowed is None:
continue
allowed = _add_video_variants(allowed)
connection.execute(
text(f"""
UPDATE {table_name}
SET allowed_api_formats = CAST(:allowed_api_formats AS json)
WHERE id = :id
"""),
{"id": row.id, "allowed_api_formats": _json_dumps(allowed)},
)
def migrate_video_tasks(connection) -> None:
"""
video_tasks.*_api_format 迁移为 signature keys。
注意:video_tasks 表天然是 video 任务,因此将 openai/gemini 的 kind 强制归一为 video,
以兼容历史上复用 chat 格式存储的旧记录。
"""
if not table_exists("video_tasks"):
return
rows = connection.execute(text("""
SELECT id, client_api_format, provider_api_format
FROM video_tasks
""")).fetchall()
for row in rows:
client_sig = _normalize_signature(row.client_api_format) or ""
provider_sig = _normalize_signature(row.provider_api_format) or ""
def _force_video(sig: str) -> str:
if not sig or ":" not in sig:
return sig
fam, _kind = sig.split(":", 1)
fam = fam.strip().lower()
if fam in ("openai", "gemini"):
return f"{fam}:video"
return sig
client_sig = _force_video(client_sig)
provider_sig = _force_video(provider_sig)
if not client_sig or not provider_sig:
continue
connection.execute(
text("""
UPDATE video_tasks
SET client_api_format = :client_api_format,
provider_api_format = :provider_api_format
WHERE id = :id
"""),
{
"id": row.id,
"client_api_format": client_sig,
"provider_api_format": provider_sig,
},
)
def migrate_model_provider_mappings(connection) -> None:
"""迁移 models.provider_model_mappings[*].api_formats 为 signature keys。"""
if not table_exists("models"):
return
rows = connection.execute(text("""
SELECT id, provider_model_mappings
FROM models
WHERE provider_model_mappings IS NOT NULL
""")).fetchall()
for row in rows:
mappings = _json_loads(row.provider_model_mappings)
if not isinstance(mappings, list):
continue
changed = False
new_mappings: list = []
for item in mappings:
if not isinstance(item, dict):
new_mappings.append(item)
continue
raw_formats = item.get("api_formats")
if isinstance(raw_formats, list):
normalized = _normalize_signature_list(raw_formats) or []
# 内容比较(而非引用比较),避免已迁移数据被无意义地重复 UPDATE
if set(normalized) != set(raw_formats):
changed = True
item = dict(item)
item["api_formats"] = normalized
new_mappings.append(item)
if not changed:
continue
connection.execute(
text("""
UPDATE models
SET provider_model_mappings = CAST(:provider_model_mappings AS json)
WHERE id = :id
"""),
{"id": row.id, "provider_model_mappings": _json_dumps(new_mappings)},
)
def migrate_dimension_collectors(connection) -> None:
"""迁移 dimension_collectors.api_format 为 signature keys(如果存在历史数据)。"""
if not table_exists("dimension_collectors"):
return
rows = connection.execute(text("""
SELECT id, api_format
FROM dimension_collectors
WHERE api_format IS NOT NULL
""")).fetchall()
for row in rows:
sig = _normalize_signature(row.api_format)
if not sig:
continue
connection.execute(
text("""
UPDATE dimension_collectors
SET api_format = :api_format
WHERE id = :id
"""),
{"id": row.id, "api_format": sig},
)
def upgrade() -> None:
if not table_exists("provider_endpoints"):
return
# ==================== provider_endpoints.api_family / endpoint_kind ====================
if not column_exists("provider_endpoints", "api_family"):
op.add_column("provider_endpoints", sa.Column("api_family", sa.String(50), nullable=True))
if not column_exists("provider_endpoints", "endpoint_kind"):
op.add_column(
"provider_endpoints", sa.Column("endpoint_kind", sa.String(50), nullable=True)
)
# ==================== idx_provider_family_kind ====================
if not index_exists("provider_endpoints", "idx_provider_family_kind"):
op.create_index(
"idx_provider_family_kind",
"provider_endpoints",
["provider_id", "api_family", "endpoint_kind"],
)
# ==================== data migrations (idempotent) ====================
conn = op.get_bind()
migrate_provider_endpoints(conn)
create_video_endpoints(conn)
if table_exists("provider_api_keys"):
migrate_provider_api_keys(conn)
migrate_allowed_api_formats(conn, table_name="users")
migrate_allowed_api_formats(conn, table_name="api_keys")
migrate_video_tasks(conn)
migrate_model_provider_mappings(conn)
migrate_dimension_collectors(conn)
def downgrade() -> None:
# Drop index/columns only; data changes are intentionally kept (safe rollback strategy).
if table_exists("provider_endpoints"):
if index_exists("provider_endpoints", "idx_provider_family_kind"):
op.drop_index("idx_provider_family_kind", table_name="provider_endpoints")
if column_exists("provider_endpoints", "endpoint_kind"):
op.drop_column("provider_endpoints", "endpoint_kind")
if column_exists("provider_endpoints", "api_family"):
op.drop_column("provider_endpoints", "api_family")
@@ -1,329 +0,0 @@
"""Add usage billing, video_tasks fields, gemini_file_mappings, provider format conversion, and indexes
Revision ID: a2f1b3c4d5e6
Revises: cf40e6a5c5b1
Create Date: 2026-02-01 12:00:00+00:00
Changes:
1. usage 表:
- 添加 billing_status (pending/settled/void),用于表示结算状态
- 添加 finalized_at,用于记录结算完成时间
- 添加 (provider_name, created_at) 和 (model, created_at) 索引
2. video_tasks 表:
- 添加 request_id(全局唯一),用于与 Usage/RequestCandidate 建立稳定关联
- 添加 short_id (Gemini-style short ID)
3. gemini_file_mappings 表:
- 创建新表用于文件映射
- 添加 source_hash 字段用于关联相同源文件
4. providers 表:
- 添加 enable_format_conversion 开关字段
5. request_candidates 表:
- 添加 created_at 索引
"""
from __future__ import annotations
import secrets
import string
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy import inspect, text
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "a2f1b3c4d5e6"
down_revision: Union[str, None] = "cf40e6a5c5b1"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
inspector.clear_cache()
return table_name in inspector.get_table_names()
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
# Clear cached schema info to get fresh data
inspector.clear_cache()
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def index_exists(table_name: str, index_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
inspector.clear_cache()
indexes = inspector.get_indexes(table_name)
return any(idx.get("name") == index_name for idx in indexes)
def unique_constraint_exists(table_name: str, constraint_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
inspector.clear_cache()
constraints = inspector.get_unique_constraints(table_name)
return any(c.get("name") == constraint_name for c in constraints)
def generate_short_id(length: int = 12) -> str:
"""Generate a Gemini-style short ID (lowercase letters + digits)"""
alphabet = string.ascii_lowercase + string.digits
return "".join(secrets.choice(alphabet) for _ in range(length))
def upgrade() -> None:
bind = op.get_bind()
dialect = bind.dialect.name
# =========================================================================
# 1. usage 表: billing_status + finalized_at + 索引
# =========================================================================
if table_exists("usage"):
if not column_exists("usage", "billing_status"):
op.add_column(
"usage",
sa.Column(
"billing_status",
sa.String(20),
nullable=False,
server_default="settled",
),
)
if not column_exists("usage", "finalized_at"):
op.add_column(
"usage",
sa.Column("finalized_at", sa.DateTime(timezone=True), nullable=True),
)
if not index_exists("usage", "idx_usage_billing_status"):
op.create_index("idx_usage_billing_status", "usage", ["billing_status"])
# (provider_name, created_at) — provider list / dashboard queries
if (
column_exists("usage", "provider_name")
and column_exists("usage", "created_at")
and not index_exists("usage", "idx_usage_provider_created")
):
op.create_index("idx_usage_provider_created", "usage", ["provider_name", "created_at"])
# (model, created_at) — model analytics / recent requests queries
if (
column_exists("usage", "model")
and column_exists("usage", "created_at")
and not index_exists("usage", "idx_usage_model_created")
):
op.create_index("idx_usage_model_created", "usage", ["model", "created_at"])
# =========================================================================
# 2. video_tasks 表: request_id + short_id
# =========================================================================
if table_exists("video_tasks"):
# --- request_id ---
if not column_exists("video_tasks", "request_id"):
op.add_column(
"video_tasks",
sa.Column("request_id", sa.String(100), nullable=True),
)
# 回填 request_id
if dialect == "postgresql":
op.execute("""
UPDATE video_tasks
SET request_id = COALESCE(request_metadata->>'request_id', id)
WHERE request_id IS NULL
""")
elif dialect == "sqlite":
op.execute("""
UPDATE video_tasks
SET request_id = COALESCE(json_extract(request_metadata, '$.request_id'), id)
WHERE request_id IS NULL
""")
else:
op.execute("""
UPDATE video_tasks
SET request_id = id
WHERE request_id IS NULL
""")
if dialect == "postgresql":
op.alter_column("video_tasks", "request_id", nullable=False)
if not index_exists("video_tasks", "idx_video_tasks_request_id"):
op.create_index("idx_video_tasks_request_id", "video_tasks", ["request_id"])
if not unique_constraint_exists("video_tasks", "uq_video_tasks_request_id"):
op.create_unique_constraint(
"uq_video_tasks_request_id",
"video_tasks",
["request_id"],
)
# --- short_id ---
if not column_exists("video_tasks", "short_id"):
op.add_column(
"video_tasks",
sa.Column("short_id", sa.String(16), nullable=True),
)
# Populate existing rows with unique short_ids
result = bind.execute(text("SELECT id FROM video_tasks WHERE short_id IS NULL"))
for row in result:
short_id = generate_short_id()
bind.execute(
text("UPDATE video_tasks SET short_id = :short_id WHERE id = :id"),
{"short_id": short_id, "id": row[0]},
)
op.alter_column("video_tasks", "short_id", nullable=False)
op.create_index("ix_video_tasks_short_id", "video_tasks", ["short_id"], unique=True)
# =========================================================================
# 3. gemini_file_mappings 表
# =========================================================================
if not table_exists("gemini_file_mappings"):
op.create_table(
"gemini_file_mappings",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("file_name", sa.String(255), nullable=False, unique=True),
sa.Column(
"key_id",
sa.String(36),
sa.ForeignKey("provider_api_keys.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column(
"user_id",
sa.String(36),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=True,
),
sa.Column("display_name", sa.String(255), nullable=True),
sa.Column("mime_type", sa.String(100), nullable=True),
sa.Column("source_hash", sa.String(64), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
)
op.create_index("ix_gemini_file_mappings_id", "gemini_file_mappings", ["id"])
op.create_index(
"ix_gemini_file_mappings_file_name", "gemini_file_mappings", ["file_name"], unique=True
)
op.create_index("ix_gemini_file_mappings_key_id", "gemini_file_mappings", ["key_id"])
op.create_index("ix_gemini_file_mappings_user_id", "gemini_file_mappings", ["user_id"])
op.create_index("idx_gemini_file_mappings_expires", "gemini_file_mappings", ["expires_at"])
op.create_index(
"idx_gemini_file_mappings_source_hash", "gemini_file_mappings", ["source_hash"]
)
else:
# 表已存在,只添加 source_hash
if not column_exists("gemini_file_mappings", "source_hash"):
op.add_column(
"gemini_file_mappings",
sa.Column("source_hash", sa.String(64), nullable=True),
)
op.create_index(
"idx_gemini_file_mappings_source_hash",
"gemini_file_mappings",
["source_hash"],
)
# =========================================================================
# 4. providers 表: enable_format_conversion
# =========================================================================
if table_exists("providers") and not column_exists("providers", "enable_format_conversion"):
op.add_column(
"providers",
sa.Column(
"enable_format_conversion",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
),
)
# =========================================================================
# 5. request_candidates 表: created_at 索引
# =========================================================================
if table_exists("request_candidates"):
if not index_exists("request_candidates", "idx_request_candidates_created_at"):
op.create_index(
"idx_request_candidates_created_at",
"request_candidates",
["created_at"],
unique=False,
)
def downgrade() -> None:
bind = op.get_bind()
dialect = bind.dialect.name
# =========================================================================
# 5. request_candidates 表回滚
# =========================================================================
if table_exists("request_candidates"):
if index_exists("request_candidates", "idx_request_candidates_created_at"):
op.drop_index("idx_request_candidates_created_at", table_name="request_candidates")
# =========================================================================
# 4. providers 表回滚
# =========================================================================
if table_exists("providers") and column_exists("providers", "enable_format_conversion"):
op.drop_column("providers", "enable_format_conversion")
# =========================================================================
# 3. gemini_file_mappings 表回滚
# =========================================================================
if table_exists("gemini_file_mappings"):
op.drop_index("idx_gemini_file_mappings_source_hash", table_name="gemini_file_mappings")
op.drop_index("idx_gemini_file_mappings_expires", table_name="gemini_file_mappings")
op.drop_index("ix_gemini_file_mappings_user_id", table_name="gemini_file_mappings")
op.drop_index("ix_gemini_file_mappings_key_id", table_name="gemini_file_mappings")
op.drop_index("ix_gemini_file_mappings_file_name", table_name="gemini_file_mappings")
op.drop_index("ix_gemini_file_mappings_id", table_name="gemini_file_mappings")
op.drop_table("gemini_file_mappings")
# =========================================================================
# 2. video_tasks 表回滚
# =========================================================================
if table_exists("video_tasks"):
# short_id
if column_exists("video_tasks", "short_id"):
if index_exists("video_tasks", "ix_video_tasks_short_id"):
op.drop_index("ix_video_tasks_short_id", table_name="video_tasks")
op.drop_column("video_tasks", "short_id")
# request_id
if column_exists("video_tasks", "request_id"):
if dialect == "postgresql":
if unique_constraint_exists("video_tasks", "uq_video_tasks_request_id"):
op.drop_constraint("uq_video_tasks_request_id", "video_tasks", type_="unique")
if index_exists("video_tasks", "idx_video_tasks_request_id"):
op.drop_index("idx_video_tasks_request_id", table_name="video_tasks")
op.drop_column("video_tasks", "request_id")
# =========================================================================
# 1. usage 表回滚
# =========================================================================
if table_exists("usage"):
if index_exists("usage", "idx_usage_model_created"):
op.drop_index("idx_usage_model_created", table_name="usage")
if index_exists("usage", "idx_usage_provider_created"):
op.drop_index("idx_usage_provider_created", table_name="usage")
if index_exists("usage", "idx_usage_billing_status"):
op.drop_index("idx_usage_billing_status", table_name="usage")
if column_exists("usage", "finalized_at"):
op.drop_column("usage", "finalized_at")
if column_exists("usage", "billing_status"):
op.drop_column("usage", "billing_status")
@@ -1,60 +0,0 @@
"""Add video_duration_seconds to video_tasks and body_rules to provider_endpoints
Revision ID: b3c4d5e6f7a8
Revises: a2f1b3c4d5e6
Create Date: 2026-02-03 15:00:00.000000
"""
from __future__ import annotations
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "b3c4d5e6f7a8"
down_revision: Union[str, None] = "a2f1b3c4d5e6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _column_exists(table_name: str, column_name: str) -> bool:
"""Check if a column exists in a table."""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
# 1. Add video_duration_seconds to video_tasks
if not _column_exists("video_tasks", "video_duration_seconds"):
op.add_column(
"video_tasks",
sa.Column("video_duration_seconds", sa.Float(), nullable=True),
)
# 2. Add body_rules to provider_endpoints
# 请求体规则支持三种操作:
# - set: 设置/覆盖字段 {"action": "set", "path": "metadata", "value": {"custom": "val"}}
# - drop: 删除字段 {"action": "drop", "path": "unwanted_field"}
# - rename: 重命名字段 {"action": "rename", "from": "old_key", "to": "new_key"}
if not _column_exists("provider_endpoints", "body_rules"):
op.add_column(
"provider_endpoints",
sa.Column("body_rules", sa.JSON(), nullable=True),
)
def downgrade() -> None:
# Remove body_rules from provider_endpoints
if _column_exists("provider_endpoints", "body_rules"):
op.drop_column("provider_endpoints", "body_rules")
# Remove video_duration_seconds from video_tasks
if _column_exists("video_tasks", "video_duration_seconds"):
op.drop_column("video_tasks", "video_duration_seconds")
@@ -1,347 +0,0 @@
"""add_stats_hourly_and_daily_complete_flag
Revision ID: c4e8f9a1b2c3
Revises: b3c4d5e6f7a8
Create Date: 2026-02-04 12:00:00.000000
"""
from __future__ import annotations
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "c4e8f9a1b2c3"
down_revision: Union[str, None] = "b3c4d5e6f7a8"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def _index_exists(table_name: str, index_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
indexes = [idx["name"] for idx in inspector.get_indexes(table_name)]
return index_name in indexes
def _column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
# Use information_schema for more reliable detection (inspector can have caching issues)
result = bind.execute(
sa.text(
"SELECT EXISTS ("
"SELECT 1 FROM information_schema.columns "
"WHERE table_name = :table AND column_name = :column"
")"
),
{"table": table_name, "column": column_name},
)
return bool(result.scalar())
def upgrade() -> None:
if _table_exists("stats_daily"):
if not _column_exists("stats_daily", "is_complete"):
op.add_column(
"stats_daily",
sa.Column("is_complete", sa.Boolean(), nullable=False, server_default=sa.false()),
)
op.execute("UPDATE stats_daily SET is_complete = true")
if not _column_exists("stats_daily", "aggregated_at"):
op.add_column(
"stats_daily",
sa.Column("aggregated_at", sa.DateTime(timezone=True), nullable=True),
)
if not _table_exists("stats_hourly"):
op.create_table(
"stats_hourly",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("hour_utc", sa.DateTime(timezone=True), nullable=False),
sa.Column("total_requests", sa.Integer(), nullable=False),
sa.Column("success_requests", sa.Integer(), nullable=False),
sa.Column("error_requests", sa.Integer(), nullable=False),
sa.Column("input_tokens", sa.BigInteger(), nullable=False),
sa.Column("output_tokens", sa.BigInteger(), nullable=False),
sa.Column("cache_creation_tokens", sa.BigInteger(), nullable=False),
sa.Column("cache_read_tokens", sa.BigInteger(), nullable=False),
sa.Column("total_cost", sa.Float(), nullable=False),
sa.Column("actual_total_cost", sa.Float(), nullable=False),
sa.Column("avg_response_time_ms", sa.Float(), nullable=False),
sa.Column("is_complete", sa.Boolean(), nullable=False),
sa.Column("aggregated_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("hour_utc", name="uq_stats_hourly_hour"),
)
op.create_index("idx_stats_hourly_hour", "stats_hourly", ["hour_utc"], unique=False)
if not _table_exists("stats_hourly_user"):
op.create_table(
"stats_hourly_user",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("hour_utc", sa.DateTime(timezone=True), nullable=False),
sa.Column("user_id", sa.String(length=36), nullable=False),
sa.Column("total_requests", sa.Integer(), nullable=False),
sa.Column("success_requests", sa.Integer(), nullable=False),
sa.Column("error_requests", sa.Integer(), nullable=False),
sa.Column("input_tokens", sa.BigInteger(), nullable=False),
sa.Column("output_tokens", sa.BigInteger(), nullable=False),
sa.Column("total_cost", sa.Float(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("hour_utc", "user_id", name="uq_stats_hourly_user"),
)
op.create_index(
"idx_stats_hourly_user_hour", "stats_hourly_user", ["hour_utc"], unique=False
)
op.create_index(
"idx_stats_hourly_user_user_hour",
"stats_hourly_user",
["user_id", "hour_utc"],
unique=False,
)
if not _table_exists("stats_hourly_model"):
op.create_table(
"stats_hourly_model",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("hour_utc", sa.DateTime(timezone=True), nullable=False),
sa.Column("model", sa.String(length=100), nullable=False),
sa.Column("total_requests", sa.Integer(), nullable=False),
sa.Column("input_tokens", sa.BigInteger(), nullable=False),
sa.Column("output_tokens", sa.BigInteger(), nullable=False),
sa.Column("total_cost", sa.Float(), nullable=False),
sa.Column("avg_response_time_ms", sa.Float(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("hour_utc", "model", name="uq_stats_hourly_model"),
)
op.create_index(
"idx_stats_hourly_model_hour", "stats_hourly_model", ["hour_utc"], unique=False
)
op.create_index(
"idx_stats_hourly_model_model_hour",
"stats_hourly_model",
["model", "hour_utc"],
unique=False,
)
if not _table_exists("stats_hourly_provider"):
op.create_table(
"stats_hourly_provider",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("hour_utc", sa.DateTime(timezone=True), nullable=False),
sa.Column("provider_name", sa.String(length=100), nullable=False),
sa.Column("total_requests", sa.Integer(), nullable=False),
sa.Column("input_tokens", sa.BigInteger(), nullable=False),
sa.Column("output_tokens", sa.BigInteger(), nullable=False),
sa.Column("total_cost", sa.Float(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("hour_utc", "provider_name", name="uq_stats_hourly_provider"),
)
op.create_index(
"idx_stats_hourly_provider_hour",
"stats_hourly_provider",
["hour_utc"],
unique=False,
)
if not _table_exists("stats_daily_api_key"):
op.create_table(
"stats_daily_api_key",
sa.Column("id", sa.String(length=36), primary_key=True),
sa.Column(
"api_key_id",
sa.String(length=36),
sa.ForeignKey("api_keys.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("date", sa.DateTime(timezone=True), nullable=False),
sa.Column("total_requests", sa.Integer(), nullable=False, server_default="0"),
sa.Column("success_requests", sa.Integer(), nullable=False, server_default="0"),
sa.Column("error_requests", sa.Integer(), nullable=False, server_default="0"),
sa.Column("input_tokens", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("output_tokens", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("cache_creation_tokens", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("cache_read_tokens", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("total_cost", sa.Float(), nullable=False, server_default="0"),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.UniqueConstraint("api_key_id", "date", name="uq_stats_daily_api_key"),
)
if _table_exists("stats_daily_api_key"):
if not _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_date"):
op.create_index("idx_stats_daily_api_key_date", "stats_daily_api_key", ["date"])
if not _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_key_date"):
op.create_index(
"idx_stats_daily_api_key_key_date",
"stats_daily_api_key",
["api_key_id", "date"],
)
if not _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_date_requests"):
op.create_index(
"idx_stats_daily_api_key_date_requests",
"stats_daily_api_key",
["date", "total_requests"],
)
if not _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_date_cost"):
op.create_index(
"idx_stats_daily_api_key_date_cost",
"stats_daily_api_key",
["date", "total_cost"],
)
if _table_exists("usage"):
if not _column_exists("usage", "error_category"):
op.add_column(
"usage",
sa.Column("error_category", sa.String(length=50), nullable=True),
)
op.create_index("idx_usage_error_category", "usage", ["error_category"], unique=False)
if _table_exists("stats_daily"):
for name in (
"p50_response_time_ms",
"p90_response_time_ms",
"p99_response_time_ms",
"p50_first_byte_time_ms",
"p90_first_byte_time_ms",
"p99_first_byte_time_ms",
):
if not _column_exists("stats_daily", name):
op.add_column("stats_daily", sa.Column(name, sa.Integer(), nullable=True))
if not _table_exists("stats_daily_error"):
op.create_table(
"stats_daily_error",
sa.Column("id", sa.String(length=36), primary_key=True),
sa.Column("date", sa.DateTime(timezone=True), nullable=False),
sa.Column("error_category", sa.String(length=50), nullable=False),
sa.Column("provider_name", sa.String(length=100), nullable=True),
sa.Column("model", sa.String(length=100), nullable=True),
sa.Column("count", sa.Integer(), nullable=False, server_default="0"),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.text("CURRENT_TIMESTAMP"),
),
sa.UniqueConstraint(
"date",
"error_category",
"provider_name",
"model",
name="uq_stats_daily_error",
),
)
if _table_exists("stats_daily_error"):
if not _index_exists("stats_daily_error", "idx_stats_daily_error_date"):
op.create_index("idx_stats_daily_error_date", "stats_daily_error", ["date"])
if not _index_exists("stats_daily_error", "idx_stats_daily_error_category"):
op.create_index(
"idx_stats_daily_error_category",
"stats_daily_error",
["date", "error_category"],
)
def downgrade() -> None:
if _table_exists("stats_daily_error"):
if _index_exists("stats_daily_error", "idx_stats_daily_error_category"):
op.drop_index("idx_stats_daily_error_category", table_name="stats_daily_error")
if _index_exists("stats_daily_error", "idx_stats_daily_error_date"):
op.drop_index("idx_stats_daily_error_date", table_name="stats_daily_error")
op.drop_table("stats_daily_error")
if _table_exists("stats_daily"):
for name in (
"p50_response_time_ms",
"p90_response_time_ms",
"p99_response_time_ms",
"p50_first_byte_time_ms",
"p90_first_byte_time_ms",
"p99_first_byte_time_ms",
):
if _column_exists("stats_daily", name):
op.drop_column("stats_daily", name)
if _table_exists("usage") and _column_exists("usage", "error_category"):
if _index_exists("usage", "idx_usage_error_category"):
op.drop_index("idx_usage_error_category", table_name="usage")
op.drop_column("usage", "error_category")
if _table_exists("stats_daily_api_key"):
if _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_date_cost"):
op.drop_index("idx_stats_daily_api_key_date_cost", table_name="stats_daily_api_key")
if _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_date_requests"):
op.drop_index("idx_stats_daily_api_key_date_requests", table_name="stats_daily_api_key")
if _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_key_date"):
op.drop_index("idx_stats_daily_api_key_key_date", table_name="stats_daily_api_key")
if _index_exists("stats_daily_api_key", "idx_stats_daily_api_key_date"):
op.drop_index("idx_stats_daily_api_key_date", table_name="stats_daily_api_key")
op.drop_table("stats_daily_api_key")
if _table_exists("stats_hourly_provider"):
if _index_exists("stats_hourly_provider", "idx_stats_hourly_provider_hour"):
op.drop_index("idx_stats_hourly_provider_hour", table_name="stats_hourly_provider")
op.drop_table("stats_hourly_provider")
if _table_exists("stats_hourly_model"):
if _index_exists("stats_hourly_model", "idx_stats_hourly_model_model_hour"):
op.drop_index("idx_stats_hourly_model_model_hour", table_name="stats_hourly_model")
if _index_exists("stats_hourly_model", "idx_stats_hourly_model_hour"):
op.drop_index("idx_stats_hourly_model_hour", table_name="stats_hourly_model")
op.drop_table("stats_hourly_model")
if _table_exists("stats_hourly_user"):
if _index_exists("stats_hourly_user", "idx_stats_hourly_user_user_hour"):
op.drop_index("idx_stats_hourly_user_user_hour", table_name="stats_hourly_user")
if _index_exists("stats_hourly_user", "idx_stats_hourly_user_hour"):
op.drop_index("idx_stats_hourly_user_hour", table_name="stats_hourly_user")
op.drop_table("stats_hourly_user")
if _table_exists("stats_hourly"):
if _index_exists("stats_hourly", "idx_stats_hourly_hour"):
op.drop_index("idx_stats_hourly_hour", table_name="stats_hourly")
op.drop_table("stats_hourly")
if _table_exists("stats_daily"):
if _column_exists("stats_daily", "aggregated_at"):
op.drop_column("stats_daily", "aggregated_at")
if _column_exists("stats_daily", "is_complete"):
op.drop_column("stats_daily", "is_complete")
@@ -1,205 +0,0 @@
"""Add provider_type, upstream_metadata, oauth_invalid fields and expand string columns to TEXT
- Add providers.provider_type (String(20), server_default="custom")
- Add provider_api_keys.upstream_metadata (JSON, nullable)
- Add provider_api_keys.oauth_invalid_at (DateTime, nullable) - OAuth Token 失效时间
- Add provider_api_keys.oauth_invalid_reason (String(255), nullable) - OAuth Token 失效原因
- Expand multiple VARCHAR columns to TEXT for long values (OAuth tokens, LDAP DN, URLs, etc.)
Revision ID: b5c6d7e8f9a0
Revises: c4e8f9a1b2c3
Create Date: 2026-02-04 15:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "b5c6d7e8f9a0"
down_revision: Union[str, None] = "c4e8f9a1b2c3"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
# 需要扩展为 TEXT 的列(表名, 列名, 原始类型长度)
COLUMNS_TO_EXPAND = [
("provider_api_keys", "api_key", 500), # OAuth tokens can be very long
(
"provider_api_keys",
"auth_config",
None,
), # 确保 auth_config 是 TEXT 类型(可能从 JSON 迁移过来)
("ldap_configs", "bind_dn", 255), # LDAP DN can be deeply nested
("ldap_configs", "base_dn", 255), # LDAP DN can be deeply nested
("ldap_configs", "user_search_filter", 500), # Complex LDAP filters
("oauth_providers", "client_id", 255), # Some OAuth providers use JWT client_id
]
def column_exists(table_name: str, column_name: str) -> bool:
"""检查列是否已存在"""
bind = op.get_bind()
inspector = inspect(bind)
columns = [col["name"] for col in inspector.get_columns(table_name)]
return column_name in columns
def table_exists(table_name: str) -> bool:
"""检查表是否存在"""
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def is_sqlite() -> bool:
"""检查是否为 SQLite 数据库"""
bind = op.get_bind()
return bind.dialect.name == "sqlite"
def get_column_type(table_name: str, column_name: str) -> str | None:
"""获取列的数据类型"""
bind = op.get_bind()
inspector = inspect(bind)
for col in inspector.get_columns(table_name):
if col["name"] == column_name:
return str(col["type"]).upper()
return None
def expand_column_to_text(table_name: str, column_name: str, original_length: int | None) -> None:
"""将 VARCHAR 列扩展为 TEXT(兼容 SQLite)"""
if not table_exists(table_name):
return
if not column_exists(table_name, column_name):
return
# 检查当前列类型,如果已经是 TEXT 则跳过
col_type = get_column_type(table_name, column_name)
if col_type and "TEXT" in col_type:
return
# 如果是 JSON 类型(可能是历史遗留),先将 JSON 数据转为文本表示再变更类型
is_json_col = col_type and "JSON" in col_type
if is_json_col and not is_sqlite():
# PostgreSQL: 先用 CAST 把 JSON 值转为 TEXT,保留数据
op.execute(
sa.text(
f"ALTER TABLE {table_name} ALTER COLUMN {column_name} "
f"TYPE TEXT USING {column_name}::TEXT"
)
)
return
if is_sqlite():
# SQLite 不支持直接 ALTER COLUMN,需要用 batch 模式
# batch 模式会自动处理 JSON->TEXT 的数据迁移
with op.batch_alter_table(table_name) as batch_op:
batch_op.alter_column(
column_name,
type_=sa.Text(),
existing_type=sa.String(original_length) if original_length else sa.Text(),
)
else:
op.alter_column(
table_name,
column_name,
type_=sa.Text(),
existing_type=sa.String(original_length) if original_length else sa.Text(),
existing_nullable=True,
)
def shrink_column_to_varchar(
table_name: str, column_name: str, target_length: int, nullable: bool = False
) -> None:
"""将 TEXT 列缩小为 VARCHAR(兼容 SQLite)
WARNING: 如果数据超过 target_length 会失败
"""
if not table_exists(table_name):
return
if not column_exists(table_name, column_name):
return
if is_sqlite():
with op.batch_alter_table(table_name) as batch_op:
batch_op.alter_column(
column_name,
type_=sa.String(target_length),
existing_type=sa.Text(),
existing_nullable=nullable,
)
else:
op.alter_column(
table_name,
column_name,
type_=sa.String(target_length),
existing_type=sa.Text(),
existing_nullable=nullable,
)
def upgrade() -> None:
# Add providers.provider_type
if not column_exists("providers", "provider_type"):
op.add_column(
"providers",
sa.Column("provider_type", sa.String(20), nullable=False, server_default="custom"),
)
# Add provider_api_keys.upstream_metadata
if not column_exists("provider_api_keys", "upstream_metadata"):
op.add_column(
"provider_api_keys",
sa.Column("upstream_metadata", sa.JSON(), nullable=True),
)
# Add provider_api_keys.oauth_invalid_at
if not column_exists("provider_api_keys", "oauth_invalid_at"):
op.add_column(
"provider_api_keys",
sa.Column("oauth_invalid_at", sa.DateTime(timezone=True), nullable=True),
)
# Add provider_api_keys.oauth_invalid_reason
if not column_exists("provider_api_keys", "oauth_invalid_reason"):
op.add_column(
"provider_api_keys",
sa.Column("oauth_invalid_reason", sa.String(255), nullable=True),
)
# Expand VARCHAR columns to TEXT
for table_name, column_name, original_length in COLUMNS_TO_EXPAND:
expand_column_to_text(table_name, column_name, original_length)
def downgrade() -> None:
# Shrink TEXT columns back to VARCHAR
# WARNING: Downgrade may fail if any values exceed original length
for table_name, column_name, original_length in reversed(COLUMNS_TO_EXPAND):
# 跳过没有原始长度的列(如 auth_config,由其他迁移创建)
if original_length is None:
continue
shrink_column_to_varchar(table_name, column_name, original_length)
# Drop provider_api_keys.oauth_invalid_reason
if column_exists("provider_api_keys", "oauth_invalid_reason"):
op.drop_column("provider_api_keys", "oauth_invalid_reason")
# Drop provider_api_keys.oauth_invalid_at
if column_exists("provider_api_keys", "oauth_invalid_at"):
op.drop_column("provider_api_keys", "oauth_invalid_at")
# Drop provider_api_keys.upstream_metadata
if column_exists("provider_api_keys", "upstream_metadata"):
op.drop_column("provider_api_keys", "upstream_metadata")
# Drop providers.provider_type
if column_exists("providers", "provider_type"):
op.drop_column("providers", "provider_type")
@@ -1,254 +0,0 @@
"""Antigravity endpoint signature to gemini:chat & add proxy_nodes table (with manual fields)
Revision ID: e1b2c3d4f5a6
Revises: b5c6d7e8f9a0
Create Date: 2026-02-06 23:45:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect, text
from sqlalchemy.dialects import postgresql
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "e1b2c3d4f5a6"
down_revision: str | None = "b5c6d7e8f9a0"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
return table_name in inspector.get_table_names()
def upgrade() -> None:
conn = op.get_bind()
# =========================================================================
# Part 1: Antigravity endpoint signature migration (gemini:cli -> gemini:chat)
# =========================================================================
# --- provider_endpoints ---
# Update only when there is no conflicting gemini:chat endpoint for the same provider
# (provider_endpoints has a unique constraint on (provider_id, api_format)).
conn.execute(text("""
UPDATE provider_endpoints pe
SET
api_format = 'gemini:chat',
api_family = 'gemini',
endpoint_kind = 'chat'
WHERE pe.api_format = 'gemini:cli'
AND pe.provider_id IN (
SELECT id FROM providers WHERE lower(provider_type) = 'antigravity'
)
AND NOT EXISTS (
SELECT 1 FROM provider_endpoints pe2
WHERE pe2.provider_id = pe.provider_id
AND pe2.api_format = 'gemini:chat'
)
"""))
# Best-effort normalization for already-existing Antigravity gemini:chat endpoints.
conn.execute(text("""
UPDATE provider_endpoints pe
SET
api_family = 'gemini',
endpoint_kind = 'chat'
WHERE pe.api_format = 'gemini:chat'
AND pe.provider_id IN (
SELECT id FROM providers WHERE lower(provider_type) = 'antigravity'
)
"""))
# --- provider_api_keys.api_formats (JSON array) ---
# Replace "gemini:cli" with "gemini:chat" in the JSON array for Antigravity keys.
# Uses text-level replace on the serialized JSON -- safe because the value is a
# simple string with no special characters that could cause ambiguous replacements.
conn.execute(text("""
UPDATE provider_api_keys pak
SET api_formats = replace(pak.api_formats::text, '"gemini:cli"', '"gemini:chat"')::json
WHERE pak.provider_id IN (
SELECT id FROM providers WHERE lower(provider_type) = 'antigravity'
)
AND pak.api_formats IS NOT NULL
AND pak.api_formats::text LIKE '%"gemini:cli"%'
"""))
# =========================================================================
# Part 2: Create proxy_nodes table with manual proxy fields (idempotent)
# =========================================================================
# Create ENUM type (idempotent)
op.execute(
"DO $$ BEGIN "
"CREATE TYPE proxynodestatus AS ENUM ('online', 'unhealthy', 'offline'); "
"EXCEPTION WHEN duplicate_object THEN NULL; "
"END $$"
)
if table_exists("proxy_nodes"):
# Table already exists — ensure manual proxy columns are present
inspector = inspect(conn)
existing_columns = {c["name"] for c in inspector.get_columns("proxy_nodes")}
# ip 列扩容:手动节点的 ip 存储 "socks5://hostname" 形式,45 字符可能不够
ip_col = next((c for c in inspector.get_columns("proxy_nodes") if c["name"] == "ip"), None)
if ip_col and hasattr(ip_col["type"], "length") and (ip_col["type"].length or 0) < 512:
op.alter_column("proxy_nodes", "ip", type_=sa.String(512), existing_nullable=False)
manual_columns = [
("is_manual", sa.Boolean(), False, sa.text("false"), "是否为手动添加的代理节点"),
("proxy_url", sa.String(500), True, None, "手动节点的完整代理 URL"),
("proxy_username", sa.String(255), True, None, "手动节点的代理用户名"),
("proxy_password", sa.String(500), True, None, "手动节点的代理密码"),
]
for col_name, col_type, nullable, default, comment in manual_columns:
if col_name not in existing_columns:
op.add_column(
"proxy_nodes",
sa.Column(
col_name,
col_type, # type: ignore[arg-type]
nullable=nullable,
server_default=default,
comment=comment,
),
)
return
op.create_table(
"proxy_nodes",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column("name", sa.String(100), nullable=False),
sa.Column("ip", sa.String(512), nullable=False),
sa.Column("port", sa.Integer(), nullable=False),
sa.Column("region", sa.String(100), nullable=True),
sa.Column(
"status",
postgresql.ENUM(
"online",
"unhealthy",
"offline",
name="proxynodestatus",
create_type=False,
),
nullable=False,
server_default=sa.text("'online'"),
),
sa.Column(
"registered_by",
sa.String(36),
sa.ForeignKey("users.id", ondelete="SET NULL"),
nullable=True,
),
sa.Column("last_heartbeat_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("heartbeat_interval", sa.Integer(), nullable=False, server_default=sa.text("30")),
sa.Column("active_connections", sa.Integer(), nullable=False, server_default=sa.text("0")),
sa.Column("total_requests", sa.BigInteger(), nullable=False, server_default=sa.text("0")),
sa.Column("avg_latency_ms", sa.Float(), nullable=True),
# --- Manual proxy node fields ---
sa.Column(
"is_manual",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
comment="是否为手动添加的代理节点",
),
sa.Column(
"proxy_url",
sa.String(500),
nullable=True,
comment="手动节点的完整代理 URL",
),
sa.Column(
"proxy_username",
sa.String(255),
nullable=True,
comment="手动节点的代理用户名",
),
sa.Column(
"proxy_password",
sa.String(500),
nullable=True,
comment="手动节点的代理密码",
),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.text("CURRENT_TIMESTAMP"),
nullable=False,
),
sa.UniqueConstraint("ip", "port", name="uq_proxy_node_ip_port"),
)
def downgrade() -> None:
conn = op.get_bind()
# =========================================================================
# Part 2 rollback: Drop proxy_nodes table (and manual columns if present)
# =========================================================================
if table_exists("proxy_nodes"):
op.drop_table("proxy_nodes")
# Best-effort: drop type (only used by proxy_nodes)
op.execute("DROP TYPE IF EXISTS proxynodestatus")
# =========================================================================
# Part 1 rollback: Revert Antigravity endpoint signature (gemini:chat -> gemini:cli)
# =========================================================================
# --- provider_endpoints ---
conn.execute(text("""
UPDATE provider_endpoints pe
SET
api_format = 'gemini:cli',
api_family = 'gemini',
endpoint_kind = 'cli'
WHERE pe.api_format = 'gemini:chat'
AND pe.provider_id IN (
SELECT id FROM providers WHERE lower(provider_type) = 'antigravity'
)
AND NOT EXISTS (
SELECT 1 FROM provider_endpoints pe2
WHERE pe2.provider_id = pe.provider_id
AND pe2.api_format = 'gemini:cli'
)
"""))
# Best-effort normalization for already-existing Antigravity gemini:cli endpoints.
conn.execute(text("""
UPDATE provider_endpoints pe
SET
api_family = 'gemini',
endpoint_kind = 'cli'
WHERE pe.api_format = 'gemini:cli'
AND pe.provider_id IN (
SELECT id FROM providers WHERE lower(provider_type) = 'antigravity'
)
"""))
# --- provider_api_keys.api_formats (JSON array) ---
conn.execute(text("""
UPDATE provider_api_keys pak
SET api_formats = replace(pak.api_formats::text, '"gemini:chat"', '"gemini:cli"')::json
WHERE pak.provider_id IN (
SELECT id FROM providers WHERE lower(provider_type) = 'antigravity'
)
AND pak.api_formats IS NOT NULL
AND pak.api_formats::text LIKE '%"gemini:chat"%'
"""))
@@ -1,61 +0,0 @@
"""Add remote_config and config_version to proxy_nodes
Revision ID: 3aff3ffc4a0e
Revises: e1b2c3d4f5a6
Create Date: 2026-02-07 15:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "3aff3ffc4a0e"
down_revision: str | None = "e1b2c3d4f5a6"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [c["name"] for c in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
if not column_exists("proxy_nodes", "remote_config"):
op.add_column(
"proxy_nodes",
sa.Column(
"remote_config",
sa.JSON(),
nullable=True,
comment="管理端下发的远程配置 (allowed_ports, log_level, heartbeat_interval, timestamp_tolerance)",
),
)
if not column_exists("proxy_nodes", "config_version"):
op.add_column(
"proxy_nodes",
sa.Column(
"config_version",
sa.Integer(),
nullable=False,
server_default="0",
comment="远程配置版本号,每次更新 +1",
),
)
def downgrade() -> None:
if column_exists("proxy_nodes", "config_version"):
op.drop_column("proxy_nodes", "config_version")
if column_exists("proxy_nodes", "remote_config"):
op.drop_column("proxy_nodes", "remote_config")
@@ -1,61 +0,0 @@
"""Add tls_enabled and tls_cert_fingerprint to proxy_nodes
Revision ID: 4b5c6d7e8f9a
Revises: 3aff3ffc4a0e
Create Date: 2026-02-07 18:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "4b5c6d7e8f9a"
down_revision: str | None = "3aff3ffc4a0e"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [c["name"] for c in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
if not column_exists("proxy_nodes", "tls_enabled"):
op.add_column(
"proxy_nodes",
sa.Column(
"tls_enabled",
sa.Boolean(),
nullable=False,
server_default="false",
comment="是否启用 TLS 加密",
),
)
if not column_exists("proxy_nodes", "tls_cert_fingerprint"):
op.add_column(
"proxy_nodes",
sa.Column(
"tls_cert_fingerprint",
sa.String(128),
nullable=True,
comment="TLS 证书 SHA-256 指纹(hex)",
),
)
def downgrade() -> None:
if column_exists("proxy_nodes", "tls_cert_fingerprint"):
op.drop_column("proxy_nodes", "tls_cert_fingerprint")
if column_exists("proxy_nodes", "tls_enabled"):
op.drop_column("proxy_nodes", "tls_enabled")
@@ -1,60 +0,0 @@
"""Add hardware_info and estimated_max_concurrency to proxy_nodes
Revision ID: 5c6d7e8f9a0b
Revises: 4b5c6d7e8f9a
Create Date: 2026-02-08 12:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "5c6d7e8f9a0b"
down_revision: str | None = "4b5c6d7e8f9a"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [c["name"] for c in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
if not column_exists("proxy_nodes", "hardware_info"):
op.add_column(
"proxy_nodes",
sa.Column(
"hardware_info",
sa.JSON(),
nullable=True,
comment="硬件信息 (cpu_cores, total_memory_mb, os_info, fd_limit, ...)",
),
)
if not column_exists("proxy_nodes", "estimated_max_concurrency"):
op.add_column(
"proxy_nodes",
sa.Column(
"estimated_max_concurrency",
sa.Integer(),
nullable=True,
comment="基于硬件估算的最大并发连接数",
),
)
def downgrade() -> None:
if column_exists("proxy_nodes", "estimated_max_concurrency"):
op.drop_column("proxy_nodes", "estimated_max_concurrency")
if column_exists("proxy_nodes", "hardware_info"):
op.drop_column("proxy_nodes", "hardware_info")
@@ -1,47 +0,0 @@
"""Add proxy column to provider_api_keys for per-key proxy configuration
Revision ID: 6d7e8f9a0b1c
Revises: 5c6d7e8f9a0b
Create Date: 2026-02-08 15:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "6d7e8f9a0b1c"
down_revision: str | None = "5c6d7e8f9a0b"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [c["name"] for c in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
if not column_exists("provider_api_keys", "proxy"):
op.add_column(
"provider_api_keys",
sa.Column(
"proxy",
sa.JSON(),
nullable=True,
comment="Key 级别代理配置(覆盖 Provider 级别代理),如 {node_id, enabled}",
),
)
def downgrade() -> None:
if column_exists("provider_api_keys", "proxy"):
op.drop_column("provider_api_keys", "proxy")
@@ -1,46 +0,0 @@
"""Add provider_request_body and client_response_body columns to usage table
Revision ID: 7e8f9a0b1c2d
Revises: 6d7e8f9a0b1c
Create Date: 2026-02-20 18:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
from sqlalchemy import text
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "7e8f9a0b1c2d"
down_revision: str | None = "6d7e8f9a0b1c"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
conn = op.get_bind()
# Use PostgreSQL native IF NOT EXISTS to avoid duplicate-column races
# when migrations are triggered concurrently (e.g. startup + manual run).
conn.execute(text("ALTER TABLE usage ADD COLUMN IF NOT EXISTS provider_request_body JSON"))
conn.execute(
text("ALTER TABLE usage ADD COLUMN IF NOT EXISTS provider_request_body_compressed BYTEA")
)
conn.execute(text("ALTER TABLE usage ADD COLUMN IF NOT EXISTS client_response_body JSON"))
conn.execute(
text("ALTER TABLE usage ADD COLUMN IF NOT EXISTS client_response_body_compressed BYTEA")
)
def downgrade() -> None:
conn = op.get_bind()
for col in (
"client_response_body_compressed",
"client_response_body",
"provider_request_body_compressed",
"provider_request_body",
):
conn.execute(text(f"ALTER TABLE usage DROP COLUMN IF EXISTS {col}"))
@@ -1,91 +0,0 @@
"""Add api_family and endpoint_kind columns to usage table
Revision ID: 8f9a0b1c2d3e
Revises: 7e8f9a0b1c2d
Create Date: 2026-02-21 15:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect, text
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "8f9a0b1c2d3e"
down_revision: str | None = "7e8f9a0b1c2d"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
# Usage 表新增列
NEW_COLUMNS = [
("api_family", sa.String(50)),
("endpoint_kind", sa.String(50)),
("provider_api_family", sa.String(50)),
("provider_endpoint_kind", sa.String(50)),
]
# 新增索引
NEW_INDEXES = [
("idx_usage_api_family", "usage", ["api_family"]),
("idx_usage_endpoint_kind", "usage", ["endpoint_kind"]),
("idx_usage_family_kind", "usage", ["api_family", "endpoint_kind"]),
]
def upgrade() -> None:
conn = op.get_bind()
# 使用 PostgreSQL 原生 IF NOT EXISTS,比 inspect 更可靠(避免同一事务内缓存问题)
col_definitions = {
"api_family": "VARCHAR(50)",
"endpoint_kind": "VARCHAR(50)",
"provider_api_family": "VARCHAR(50)",
"provider_endpoint_kind": "VARCHAR(50)",
}
for col_name, col_type_sql in col_definitions.items():
conn.execute(text(f"ALTER TABLE usage ADD COLUMN IF NOT EXISTS {col_name} {col_type_sql}"))
# 数据迁移:从 api_format 解析 api_family + endpoint_kind
conn.execute(text("""
UPDATE usage SET
api_family = lower(split_part(api_format, ':', 1)),
endpoint_kind = lower(split_part(api_format, ':', 2))
WHERE api_format IS NOT NULL
AND api_format LIKE '%%:%%'
AND api_family IS NULL
"""))
conn.execute(text("""
UPDATE usage SET
provider_api_family = lower(split_part(endpoint_api_format, ':', 1)),
provider_endpoint_kind = lower(split_part(endpoint_api_format, ':', 2))
WHERE endpoint_api_format IS NOT NULL
AND endpoint_api_format LIKE '%%:%%'
AND provider_api_family IS NULL
"""))
# 创建索引
inspector = inspect(conn)
existing_indexes = {idx["name"] for idx in inspector.get_indexes("usage")}
for idx_name, table, columns in NEW_INDEXES:
if idx_name not in existing_indexes:
op.create_index(idx_name, table, columns)
def downgrade() -> None:
conn = op.get_bind()
inspector = inspect(conn)
existing_indexes = {idx["name"] for idx in inspector.get_indexes("usage")}
for idx_name, _, _ in reversed(NEW_INDEXES):
if idx_name in existing_indexes:
op.drop_index(idx_name, table_name="usage")
existing_columns = {col["name"] for col in inspector.get_columns("usage")}
for col_name, _ in reversed(NEW_COLUMNS):
if col_name in existing_columns:
op.drop_column("usage", col_name)
@@ -1,106 +0,0 @@
"""Add tunnel mode fields and remove IP forwarding fields
Revision ID: 9a0b1c2d3e4f
Revises: 8f9a0b1c2d3e
Create Date: 2026-02-24 17:00:00.000000
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
revision: str = "9a0b1c2d3e4f"
down_revision: str | None = "8f9a0b1c2d3e"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
inspector = inspect(bind)
columns = [c["name"] for c in inspector.get_columns(table_name)]
return column_name in columns
def upgrade() -> None:
# 添加 tunnel 模式字段
if not column_exists("proxy_nodes", "tunnel_mode"):
op.add_column(
"proxy_nodes",
sa.Column(
"tunnel_mode",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
comment="是否使用 WebSocket 隧道模式",
),
)
if not column_exists("proxy_nodes", "tunnel_connected"):
op.add_column(
"proxy_nodes",
sa.Column(
"tunnel_connected",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
comment="隧道是否已连接",
),
)
if not column_exists("proxy_nodes", "tunnel_connected_at"):
op.add_column(
"proxy_nodes",
sa.Column(
"tunnel_connected_at",
sa.DateTime(timezone=True),
nullable=True,
comment="隧道最近一次建立时间",
),
)
# tunnel 模式节点不需要 port,将其置零
op.execute("UPDATE proxy_nodes SET port = 0 WHERE tunnel_mode = true")
# 移除旧的 IP 转发字段
if column_exists("proxy_nodes", "tls_enabled"):
op.drop_column("proxy_nodes", "tls_enabled")
if column_exists("proxy_nodes", "tls_cert_fingerprint"):
op.drop_column("proxy_nodes", "tls_cert_fingerprint")
def downgrade() -> None:
# 恢复 IP 转发字段
if not column_exists("proxy_nodes", "tls_cert_fingerprint"):
op.add_column(
"proxy_nodes",
sa.Column(
"tls_cert_fingerprint",
sa.String(128),
nullable=True,
comment="TLS 证书 SHA-256 指纹(hex)",
),
)
if not column_exists("proxy_nodes", "tls_enabled"):
op.add_column(
"proxy_nodes",
sa.Column(
"tls_enabled",
sa.Boolean(),
nullable=False,
server_default=sa.text("false"),
comment="是否启用 TLS 加密",
),
)
# 移除 tunnel 模式字段
if column_exists("proxy_nodes", "tunnel_connected_at"):
op.drop_column("proxy_nodes", "tunnel_connected_at")
if column_exists("proxy_nodes", "tunnel_connected"):
op.drop_column("proxy_nodes", "tunnel_connected")
if column_exists("proxy_nodes", "tunnel_mode"):
op.drop_column("proxy_nodes", "tunnel_mode")
@@ -1,224 +0,0 @@
"""Add cache_creation columns, clean up capability settings, add user_model_usage_counts,
enforce global_model_id NOT NULL
1. Add cache_creation_input_tokens_5m and cache_creation_input_tokens_1h to usage table.
2. Clean up cache_1h/context_1m/gemini_files from user-configurable settings
(now auto-detected via REQUEST_PARAM mode).
3. Create user_model_usage_counts table for per-user per-model atomic usage counters.
4. Enforce models.global_model_id NOT NULL (delete orphan models without global model).
Revision ID: b2c3d4e5f6a7
Revises: 9a0b1c2d3e4f
Create Date: 2026-02-28 14:00:00.000000
"""
from __future__ import annotations
import json
import uuid
from collections.abc import Sequence
from datetime import datetime, timezone
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
revision: str = "b2c3d4e5f6a7"
down_revision: str | None = "9a0b1c2d3e4f"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
insp = inspect(bind)
columns = [c["name"] for c in insp.get_columns(table_name)]
return column_name in columns
def table_exists(table_name: str) -> bool:
bind = op.get_bind()
insp = inspect(bind)
return table_name in insp.get_table_names()
def index_exists(table_name: str, index_name: str) -> bool:
bind = op.get_bind()
insp = inspect(bind)
return any(idx["name"] == index_name for idx in insp.get_indexes(table_name))
def upgrade() -> None:
# --- 1. Add cache_creation columns ---
if not column_exists("usage", "cache_creation_input_tokens_5m"):
op.add_column(
"usage",
sa.Column(
"cache_creation_input_tokens_5m",
sa.Integer(),
nullable=False,
server_default=sa.text("0"),
comment="5min TTL cache creation input tokens",
),
)
if not column_exists("usage", "cache_creation_input_tokens_1h"):
op.add_column(
"usage",
sa.Column(
"cache_creation_input_tokens_1h",
sa.Integer(),
nullable=False,
server_default=sa.text("0"),
comment="1h TTL cache creation input tokens",
),
)
# --- 2. Clean up stale capability settings (pure Python, DB-agnostic) ---
stale_keys = {"cache_1h", "context_1m", "gemini_files"}
conn = op.get_bind()
# ApiKey.force_capabilities: dict-like JSON, remove stale keys
rows = conn.execute(
sa.text("SELECT id, force_capabilities FROM api_keys WHERE force_capabilities IS NOT NULL")
).fetchall()
for row in rows:
raw = row[1]
if raw is None:
continue
data = raw if isinstance(raw, dict) else json.loads(raw)
cleaned = {k: v for k, v in data.items() if k not in stale_keys}
new_val = json.dumps(cleaned) if cleaned else None
conn.execute(
sa.text("UPDATE api_keys SET force_capabilities = :val WHERE id = :id"),
{"val": new_val, "id": row[0]},
)
# User.model_capability_settings: nested dict {model_key: {cap: val}}, remove stale keys
rows = conn.execute(
sa.text(
"SELECT id, model_capability_settings FROM users"
" WHERE model_capability_settings IS NOT NULL"
)
).fetchall()
for row in rows:
raw = row[1]
if raw is None:
continue
data = raw if isinstance(raw, dict) else json.loads(raw)
cleaned = {}
for model_key, caps in data.items():
cap_cleaned = {k: v for k, v in caps.items() if k not in stale_keys}
if cap_cleaned:
cleaned[model_key] = cap_cleaned
new_val = json.dumps(cleaned) if cleaned else None
conn.execute(
sa.text("UPDATE users SET model_capability_settings = :val WHERE id = :id"),
{"val": new_val, "id": row[0]},
)
# GlobalModel.supported_capabilities: JSON array, remove stale entries
rows = conn.execute(
sa.text(
"SELECT id, supported_capabilities FROM global_models"
" WHERE supported_capabilities IS NOT NULL"
)
).fetchall()
for row in rows:
raw = row[1]
if raw is None:
continue
data = raw if isinstance(raw, list) else json.loads(raw)
cleaned = [c for c in data if c not in stale_keys]
new_val = json.dumps(cleaned) if cleaned else None
conn.execute(
sa.text("UPDATE global_models SET supported_capabilities = :val WHERE id = :id"),
{"val": new_val, "id": row[0]},
)
# --- 3. Create user_model_usage_counts table ---
if not table_exists("user_model_usage_counts"):
op.create_table(
"user_model_usage_counts",
sa.Column("id", sa.String(36), primary_key=True),
sa.Column(
"user_id",
sa.String(36),
sa.ForeignKey("users.id", ondelete="CASCADE"),
nullable=False,
),
sa.Column("model", sa.String(100), nullable=False),
sa.Column("usage_count", sa.Integer, nullable=False, server_default="0"),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
nullable=False,
server_default=sa.func.now(),
),
sa.UniqueConstraint("user_id", "model", name="uq_user_model_usage_count"),
)
if not index_exists("user_model_usage_counts", "idx_user_model_usage_user"):
op.create_index("idx_user_model_usage_user", "user_model_usage_counts", ["user_id"])
if not index_exists("user_model_usage_counts", "idx_user_model_usage_model"):
op.create_index("idx_user_model_usage_model", "user_model_usage_counts", ["model"])
# Backfill from existing usage records (truncate first for idempotency)
conn.execute(sa.text("DELETE FROM user_model_usage_counts"))
rows = conn.execute(
sa.text(
"SELECT user_id, model, COUNT(*) AS cnt FROM usage"
" WHERE user_id IS NOT NULL GROUP BY user_id, model"
)
).fetchall()
now = datetime.now(timezone.utc)
for row in rows:
conn.execute(
sa.text(
"INSERT INTO user_model_usage_counts"
" (id, user_id, model, usage_count, created_at, updated_at)"
" VALUES (:id, :user_id, :model, :cnt, :now, :now)"
),
{
"id": str(uuid.uuid4()),
"user_id": row[0],
"model": row[1],
"cnt": row[2],
"now": now,
},
)
# --- 4. Enforce models.global_model_id NOT NULL ---
conn = op.get_bind()
insp = inspect(conn)
model_cols = {c["name"]: c for c in insp.get_columns("models")}
if model_cols.get("global_model_id", {}).get("nullable", True):
op.execute("DELETE FROM models WHERE global_model_id IS NULL")
op.alter_column("models", "global_model_id", existing_type=sa.String(36), nullable=False)
def downgrade() -> None:
# Revert models.global_model_id to nullable
if column_exists("models", "global_model_id"):
op.alter_column("models", "global_model_id", existing_type=sa.String(36), nullable=True)
# Drop user_model_usage_counts
if table_exists("user_model_usage_counts"):
if index_exists("user_model_usage_counts", "idx_user_model_usage_model"):
op.drop_index("idx_user_model_usage_model", table_name="user_model_usage_counts")
if index_exists("user_model_usage_counts", "idx_user_model_usage_user"):
op.drop_index("idx_user_model_usage_user", table_name="user_model_usage_counts")
op.drop_table("user_model_usage_counts")
# Drop cache_creation columns
if column_exists("usage", "cache_creation_input_tokens_1h"):
op.drop_column("usage", "cache_creation_input_tokens_1h")
if column_exists("usage", "cache_creation_input_tokens_5m"):
op.drop_column("usage", "cache_creation_input_tokens_5m")
# capability settings cleanup is not reversible
@@ -1,153 +0,0 @@
"""proxy_node_metrics_and_events
Revision ID: 48afe197cc15
Revises: b2c3d4e5f6a7
Create Date: 2026-02-28 04:33:11.201185+00:00
"""
import sqlalchemy as sa
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision = "48afe197cc15"
down_revision = "b2c3d4e5f6a7"
branch_labels = None
depends_on = None
def _column_exists(table_name: str, column_name: str) -> bool:
bind = op.get_bind()
insp = inspect(bind)
columns = [c["name"] for c in insp.get_columns(table_name)]
return column_name in columns
def _table_exists(table_name: str) -> bool:
bind = op.get_bind()
insp = inspect(bind)
return table_name in insp.get_table_names()
def _enum_has_value(enum_name: str, value: str) -> bool:
"""检查 PostgreSQL 枚举类型是否包含指定值"""
bind = op.get_bind()
result = bind.execute(
sa.text(
"SELECT 1 FROM pg_enum e JOIN pg_type t ON e.enumtypid = t.oid"
" WHERE t.typname = :enum_name AND e.enumlabel = :value"
),
{"enum_name": enum_name, "value": value},
)
return result.fetchone() is not None
def upgrade() -> None:
# proxy_nodes: 将已废弃的 unhealthy 状态迁移为 offline,然后从枚举中移除
if _enum_has_value("proxynodestatus", "unhealthy"):
op.execute("UPDATE proxy_nodes SET status = 'offline' WHERE status = 'unhealthy'")
op.execute("ALTER TYPE proxynodestatus RENAME TO proxynodestatus_old")
op.execute("CREATE TYPE proxynodestatus AS ENUM ('online', 'offline')")
# 必须先移除旧枚举类型的 DEFAULT,否则 ALTER TYPE 会因无法转换默认值而报错
op.execute("ALTER TABLE proxy_nodes ALTER COLUMN status DROP DEFAULT")
op.execute(
"ALTER TABLE proxy_nodes ALTER COLUMN status TYPE proxynodestatus"
" USING status::text::proxynodestatus"
)
op.execute("ALTER TABLE proxy_nodes ALTER COLUMN status SET DEFAULT 'online'::proxynodestatus")
op.execute("DROP TYPE proxynodestatus_old")
# proxy_nodes: 新增错误指标字段
if not _column_exists("proxy_nodes", "failed_requests"):
op.add_column(
"proxy_nodes",
sa.Column(
"failed_requests",
sa.BigInteger(),
nullable=False,
server_default="0",
comment="累计失败请求数",
),
)
if not _column_exists("proxy_nodes", "dns_failures"):
op.add_column(
"proxy_nodes",
sa.Column(
"dns_failures",
sa.BigInteger(),
nullable=False,
server_default="0",
comment="累计 DNS 失败数",
),
)
if not _column_exists("proxy_nodes", "stream_errors"):
op.add_column(
"proxy_nodes",
sa.Column(
"stream_errors",
sa.BigInteger(),
nullable=False,
server_default="0",
comment="累计流错误数",
),
)
# proxy_node_events: 连接事件表
if not _table_exists("proxy_node_events"):
op.create_table(
"proxy_node_events",
sa.Column("id", sa.BigInteger(), autoincrement=True, nullable=False),
sa.Column("node_id", sa.String(length=36), nullable=False),
sa.Column(
"event_type",
sa.String(length=20),
nullable=False,
comment="事件类型: connected, disconnected, error",
),
sa.Column(
"detail",
sa.String(length=500),
nullable=True,
comment="事件详情(如断开原因)",
),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(["node_id"], ["proxy_nodes.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"idx_proxy_node_events_node_created",
"proxy_node_events",
["node_id", "created_at"],
)
op.create_index(
op.f("ix_proxy_node_events_node_id"),
"proxy_node_events",
["node_id"],
)
def downgrade() -> None:
# 恢复 proxynodestatus 枚举,加回 unhealthy
if not _enum_has_value("proxynodestatus", "unhealthy"):
op.execute("ALTER TYPE proxynodestatus RENAME TO proxynodestatus_old")
op.execute("CREATE TYPE proxynodestatus AS ENUM ('online', 'unhealthy', 'offline')")
op.execute("ALTER TABLE proxy_nodes ALTER COLUMN status DROP DEFAULT")
op.execute(
"ALTER TABLE proxy_nodes ALTER COLUMN status TYPE proxynodestatus"
" USING status::text::proxynodestatus"
)
op.execute("ALTER TABLE proxy_nodes ALTER COLUMN status SET DEFAULT 'online'::proxynodestatus")
op.execute("DROP TYPE proxynodestatus_old")
if _table_exists("proxy_node_events"):
op.drop_index(op.f("ix_proxy_node_events_node_id"), table_name="proxy_node_events")
op.drop_index("idx_proxy_node_events_node_created", table_name="proxy_node_events")
op.drop_table("proxy_node_events")
if _column_exists("proxy_nodes", "stream_errors"):
op.drop_column("proxy_nodes", "stream_errors")
if _column_exists("proxy_nodes", "dns_failures"):
op.drop_column("proxy_nodes", "dns_failures")
if _column_exists("proxy_nodes", "failed_requests"):
op.drop_column("proxy_nodes", "failed_requests")
@@ -1,49 +0,0 @@
"""add_request_candidates_composite_indexes
Revision ID: 00b9161b8729
Revises: 48afe197cc15
Create Date: 2026-02-28 14:48:00.000000+00:00
"""
from sqlalchemy import inspect
from alembic import op
# revision identifiers, used by Alembic.
revision = "00b9161b8729"
down_revision = "48afe197cc15"
branch_labels = None
depends_on = None
def _index_exists(index_name: str) -> bool:
bind = op.get_bind()
insp = inspect(bind)
indexes = insp.get_indexes("request_candidates")
return any(idx["name"] == index_name for idx in indexes)
def upgrade() -> None:
# (request_id, status) - fallback/retry 查询优化
if not _index_exists("idx_rc_request_id_status"):
op.create_index(
"idx_rc_request_id_status",
"request_candidates",
["request_id", "status"],
)
# (provider_id, status, created_at) - provider 聚合统计优化
if not _index_exists("idx_rc_provider_status_created"):
op.create_index(
"idx_rc_provider_status_created",
"request_candidates",
["provider_id", "status", "created_at"],
)
def downgrade() -> None:
if _index_exists("idx_rc_provider_status_created"):
op.drop_index("idx_rc_provider_status_created", table_name="request_candidates")
if _index_exists("idx_rc_request_id_status"):
op.drop_index("idx_rc_request_id_status", table_name="request_candidates")
@@ -1,263 +0,0 @@
"""vertex_ai_provider_type
Migrate legacy Vertex auth_type/provider_type into the new model:
- provider_type=vertex_ai
- auth_type=service_account (legacy vertex_ai renamed)
- fixed Vertex endpoints: gemini:chat + claude:chat
Revision ID: 2a624af8dd3a
Revises: 00b9161b8729
Create Date: 2026-02-28 15:00:00.000000+00:00
"""
from __future__ import annotations
import uuid
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "2a624af8dd3a"
down_revision = "00b9161b8729"
branch_labels = None
depends_on = None
_VERTEX_BASE_URL = "https://aiplatform.googleapis.com"
_VERTEX_ENDPOINTS: tuple[tuple[str, str, str], ...] = (
("gemini:chat", "gemini", "chat"),
("claude:chat", "claude", "chat"),
)
_VERTEX_KEY_FORMATS_SA = '["gemini:chat","claude:chat"]'
_VERTEX_KEY_FORMATS_API_KEY = '["gemini:chat"]'
def _select_vertex_provider_ids(conn: sa.Connection) -> list[str]:
"""Collect providers that should be treated as Vertex after migration."""
rows = conn.execute(sa.text("""
SELECT DISTINCT p.id
FROM providers p
LEFT JOIN provider_api_keys pak ON pak.provider_id = p.id
WHERE lower(COALESCE(p.provider_type, '')) = 'vertex_ai'
OR pak.auth_type = 'vertex_ai'
"""))
return [str(row[0]) for row in rows if row[0]]
def _ensure_fixed_vertex_endpoints(conn: sa.Connection, provider_ids: list[str]) -> None:
"""Ensure every Vertex provider has fixed gemini:chat + claude:chat endpoints."""
for provider_id in provider_ids:
provider_max_retries = (
conn.execute(
sa.text("""
SELECT COALESCE(max_retries, 2)
FROM providers
WHERE id = :provider_id
"""),
{"provider_id": provider_id},
).scalar()
or 2
)
for api_format, api_family, endpoint_kind in _VERTEX_ENDPOINTS:
# Normalize existing fixed endpoint fields.
conn.execute(
sa.text("""
UPDATE provider_endpoints
SET
api_family = :api_family,
endpoint_kind = :endpoint_kind,
base_url = :base_url,
custom_path = NULL,
is_active = TRUE,
updated_at = CURRENT_TIMESTAMP
WHERE provider_id = :provider_id
AND api_format = :api_format
"""),
{
"provider_id": provider_id,
"api_format": api_format,
"api_family": api_family,
"endpoint_kind": endpoint_kind,
"base_url": _VERTEX_BASE_URL,
},
)
exists = conn.execute(
sa.text("""
SELECT 1
FROM provider_endpoints
WHERE provider_id = :provider_id
AND api_format = :api_format
LIMIT 1
"""),
{"provider_id": provider_id, "api_format": api_format},
).first()
if not exists:
conn.execute(
sa.text("""
INSERT INTO provider_endpoints (
id,
provider_id,
api_format,
api_family,
endpoint_kind,
base_url,
custom_path,
header_rules,
body_rules,
max_retries,
is_active,
config,
format_acceptance_config,
proxy,
created_at,
updated_at
)
VALUES (
:id,
:provider_id,
:api_format,
:api_family,
:endpoint_kind,
:base_url,
NULL,
NULL,
NULL,
:max_retries,
TRUE,
NULL,
NULL,
NULL,
CURRENT_TIMESTAMP,
CURRENT_TIMESTAMP
)
"""),
{
"id": str(uuid.uuid4()),
"provider_id": provider_id,
"api_format": api_format,
"api_family": api_family,
"endpoint_kind": endpoint_kind,
"base_url": _VERTEX_BASE_URL,
"max_retries": int(provider_max_retries),
},
)
# Vertex fixed-provider model: disable non-fixed endpoints.
conn.execute(
sa.text("""
UPDATE provider_endpoints
SET
is_active = FALSE,
updated_at = CURRENT_TIMESTAMP
WHERE provider_id = :provider_id
AND api_format NOT IN ('gemini:chat', 'claude:chat')
"""),
{"provider_id": provider_id},
)
def _normalize_vertex_key_formats(conn: sa.Connection, provider_ids: list[str]) -> None:
"""Normalize key.api_formats for Vertex keys by auth type."""
for provider_id in provider_ids:
# Service Account (and legacy vertex_ai) keys: allow Gemini + Claude models.
conn.execute(
sa.text("""
UPDATE provider_api_keys
SET
api_formats = CAST(:api_formats AS json),
updated_at = CURRENT_TIMESTAMP
WHERE provider_id = :provider_id
AND auth_type IN ('service_account', 'vertex_ai')
"""),
{
"provider_id": provider_id,
"api_formats": _VERTEX_KEY_FORMATS_SA,
},
)
# API Key mode on Vertex 仅支持 Gemini(Google publisher)。
conn.execute(
sa.text("""
UPDATE provider_api_keys
SET
api_formats = CAST(:api_formats AS json),
updated_at = CURRENT_TIMESTAMP
WHERE provider_id = :provider_id
AND auth_type = 'api_key'
"""),
{
"provider_id": provider_id,
"api_formats": _VERTEX_KEY_FORMATS_API_KEY,
},
)
def upgrade() -> None:
conn = op.get_bind()
# 1) 收集目标 Provider(兼容重复执行,先识别 legacy/new 两种来源)。
provider_ids = _select_vertex_provider_ids(conn)
# 2) 先重命名 auth_type(legacy vertex_ai -> service_account)。
conn.execute(sa.text("""
UPDATE provider_api_keys
SET auth_type = 'service_account'
WHERE auth_type = 'vertex_ai'
"""))
if not provider_ids:
return
# 3) 归一 provider_type,并启用格式转换(Vertex 同时承载 Gemini/Claude)。
for provider_id in provider_ids:
conn.execute(
sa.text("""
UPDATE providers
SET
provider_type = 'vertex_ai',
enable_format_conversion = TRUE
WHERE id = :provider_id
"""),
{"provider_id": provider_id},
)
# 4) 固定端点落地:gemini:chat + claude:chat。
_ensure_fixed_vertex_endpoints(conn, provider_ids)
# 5) 归一 key 的 api_formats,避免调度命中旧格式。
_normalize_vertex_key_formats(conn, provider_ids)
def downgrade() -> None:
conn = op.get_bind()
provider_rows = conn.execute(sa.text("""
SELECT id
FROM providers
WHERE lower(COALESCE(provider_type, '')) = 'vertex_ai'
"""))
provider_ids = [str(row[0]) for row in provider_rows if row[0]]
if provider_ids:
for provider_id in provider_ids:
conn.execute(
sa.text("""
UPDATE provider_api_keys
SET auth_type = 'vertex_ai'
WHERE provider_id = :provider_id
AND auth_type = 'service_account'
"""),
{"provider_id": provider_id},
)
conn.execute(
sa.text("""
UPDATE providers
SET provider_type = 'custom'
WHERE id = :provider_id
"""),
{"provider_id": provider_id},
)
@@ -1,199 +0,0 @@
"""backfill_codex_compact_endpoint
Backfill Codex reverse-proxy endpoints:
- ensure `openai:cli` endpoint is pinned to force_stream
- ensure `openai:compact` endpoint exists
Revision ID: f0c3a7b9d1e2
Revises: 2a624af8dd3a
Create Date: 2026-03-01 17:00:00.000000+00:00
"""
from __future__ import annotations
import json
import uuid
from typing import Any
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "f0c3a7b9d1e2"
down_revision = "2a624af8dd3a"
branch_labels = None
depends_on = None
_CODEX_BASE_URL = "https://chatgpt.com/backend-api/codex"
_COMPACT_FORMAT = "openai:compact"
_CLI_FORMAT = "openai:cli"
_FORCE_STREAM = "force_stream"
def _find_codex_provider_ids(conn: sa.Connection) -> list[str]:
"""Find Codex providers (by provider_type or legacy base_url pattern)."""
rows = conn.execute(sa.text("""
SELECT DISTINCT p.id
FROM providers p
LEFT JOIN provider_endpoints pe ON pe.provider_id = p.id
WHERE lower(COALESCE(p.provider_type, '')) = 'codex'
OR (
lower(COALESCE(pe.api_format, '')) = 'openai:cli'
AND lower(COALESCE(pe.base_url, '')) LIKE '%/backend-api/codex%'
)
"""))
return [str(r[0]) for r in rows if r[0]]
def _get_cli_endpoint(conn: sa.Connection, provider_id: str) -> dict[str, Any] | None:
"""Load existing openai:cli endpoint for the provider."""
row = (
conn.execute(
sa.text("""
SELECT base_url, header_rules, body_rules, max_retries, proxy, config
FROM provider_endpoints
WHERE provider_id = :pid AND api_format = :fmt
LIMIT 1
"""),
{"pid": provider_id, "fmt": _CLI_FORMAT},
)
.mappings()
.first()
)
return dict(row) if row else None
def _pin_cli_force_stream(conn: sa.Connection, provider_id: str, cli: dict[str, Any]) -> None:
"""Set upstream_stream_policy=force_stream on existing cli endpoint."""
cfg = dict(cli.get("config") or {}) if isinstance(cli.get("config"), dict) else {}
cfg.pop("upstreamStreamPolicy", None)
cfg.pop("upstream_stream", None)
cfg["upstream_stream_policy"] = _FORCE_STREAM
conn.execute(
sa.text("""
UPDATE provider_endpoints
SET api_family = 'openai',
endpoint_kind = 'cli',
config = CAST(:config AS json),
updated_at = CURRENT_TIMESTAMP
WHERE provider_id = :pid AND api_format = :fmt
"""),
{
"pid": provider_id,
"fmt": _CLI_FORMAT,
"config": json.dumps(cfg, ensure_ascii=False),
},
)
def _ensure_compact_endpoint(conn: sa.Connection, provider_id: str, cli: dict[str, Any]) -> None:
"""Create openai:compact endpoint if missing (clone from cli)."""
exists = conn.execute(
sa.text(
"SELECT 1 FROM provider_endpoints WHERE provider_id = :pid AND api_format = :fmt LIMIT 1"
),
{"pid": provider_id, "fmt": _COMPACT_FORMAT},
).first()
if exists:
# Already exists, just ensure api_family/endpoint_kind are set.
conn.execute(
sa.text("""
UPDATE provider_endpoints
SET api_family = 'openai', endpoint_kind = 'compact',
updated_at = CURRENT_TIMESTAMP
WHERE provider_id = :pid AND api_format = :fmt
"""),
{"pid": provider_id, "fmt": _COMPACT_FORMAT},
)
return
# Clone from cli endpoint, strip stream policy.
cfg = dict(cli.get("config") or {}) if isinstance(cli.get("config"), dict) else {}
for k in ("upstream_stream_policy", "upstreamStreamPolicy", "upstream_stream"):
cfg.pop(k, None)
def _json(val: Any) -> str | None:
return json.dumps(val, ensure_ascii=False) if val is not None else None
conn.execute(
sa.text("""
INSERT INTO provider_endpoints (
id, provider_id, api_format, api_family, endpoint_kind,
base_url, custom_path, header_rules, body_rules,
max_retries, is_active, config, format_acceptance_config,
proxy, created_at, updated_at
) VALUES (
:id, :pid, :fmt, 'openai', 'compact',
:base_url, NULL, CAST(:header_rules AS json), CAST(:body_rules AS json),
:max_retries, TRUE, CAST(:config AS json), NULL,
CAST(:proxy AS jsonb), CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
)
"""),
{
"id": str(uuid.uuid4()),
"pid": provider_id,
"fmt": _COMPACT_FORMAT,
"base_url": cli.get("base_url") or _CODEX_BASE_URL,
"header_rules": _json(cli.get("header_rules")),
"body_rules": _json(cli.get("body_rules")),
"max_retries": cli.get("max_retries") or 2,
"config": _json(cfg or None),
"proxy": _json(cli.get("proxy")),
},
)
def _add_compact_to_key_formats(conn: sa.Connection, provider_id: str) -> None:
"""Ensure provider keys include openai:compact in api_formats."""
rows = (
conn.execute(
sa.text("SELECT id, api_formats FROM provider_api_keys WHERE provider_id = :pid"),
{"pid": provider_id},
)
.mappings()
.all()
)
for row in rows:
raw = row["api_formats"]
formats: list[str] = []
if isinstance(raw, list):
for item in raw:
v = str(item or "").strip().lower()
if v and v not in formats:
formats.append(v)
if _COMPACT_FORMAT in formats:
continue
# Insert compact right after cli, or at end.
if _CLI_FORMAT in formats:
idx = formats.index(_CLI_FORMAT) + 1
formats.insert(idx, _COMPACT_FORMAT)
else:
formats.append(_COMPACT_FORMAT)
conn.execute(
sa.text("""
UPDATE provider_api_keys
SET api_formats = CAST(:fmts AS json), updated_at = CURRENT_TIMESTAMP
WHERE id = :id
"""),
{"id": row["id"], "fmts": json.dumps(formats, ensure_ascii=False)},
)
def upgrade() -> None:
conn = op.get_bind()
for provider_id in _find_codex_provider_ids(conn):
cli = _get_cli_endpoint(conn, provider_id)
if not cli:
continue # No cli endpoint to clone from; skip.
_pin_cli_force_stream(conn, provider_id, cli)
_ensure_compact_endpoint(conn, provider_id, cli)
_add_compact_to_key_formats(conn, provider_id)
def downgrade() -> None:
# Data backfill: no-op to avoid deleting user-managed data.
return
@@ -1,29 +0,0 @@
"""add_proxy_metadata_to_proxy_nodes
Revision ID: 1d2e3f4a5b6c
Revises: f0c3a7b9d1e2
Create Date: 2026-03-02 13:00:00.000000+00:00
"""
from __future__ import annotations
from collections.abc import Sequence
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "1d2e3f4a5b6c"
down_revision: str | None = "f0c3a7b9d1e2"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
op.execute("ALTER TABLE public.proxy_nodes ADD COLUMN IF NOT EXISTS proxy_metadata json")
op.execute(
"COMMENT ON COLUMN public.proxy_nodes.proxy_metadata IS 'aether-proxy 上报元数据(版本等)'"
)
def downgrade() -> None:
op.execute("ALTER TABLE public.proxy_nodes DROP COLUMN IF EXISTS proxy_metadata")
@@ -1,72 +0,0 @@
"""backfill_codex_default_body_rules
Backfill default body_rules for codex providers with openai:cli endpoints
that currently have body_rules IS NULL.
Rules:
- drop max_output_tokens
- drop temperature
- drop top_p
- set store = false
- set instructions = "You are GPT-5." (when instructions not exists)
Revision ID: dd0278c0a28c
Revises: 1d2e3f4a5b6c
Create Date: 2026-03-02 15:00:00.000000+00:00
"""
from __future__ import annotations
import json
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision = "dd0278c0a28c"
down_revision = "1d2e3f4a5b6c"
branch_labels = None
depends_on = None
_TARGET_FORMATS = ("openai:cli",)
_DEFAULT_BODY_RULES = [
{"action": "drop", "path": "max_output_tokens"},
{"action": "drop", "path": "temperature"},
{"action": "drop", "path": "top_p"},
{"action": "set", "path": "store", "value": False},
{
"action": "set",
"path": "instructions",
"value": "You are GPT-5.",
"condition": {"path": "instructions", "op": "not_exists"},
},
]
def upgrade() -> None:
conn = op.get_bind()
# 幂等性: 仅回填 codex 提供商中 body_rules 为空(SQL NULL 或 JSON null)的记录
rules_json = json.dumps(_DEFAULT_BODY_RULES, ensure_ascii=False)
result = conn.execute(
sa.text("""
UPDATE provider_endpoints pe
SET body_rules = CAST(:rules AS json),
updated_at = CURRENT_TIMESTAMP
FROM providers p
WHERE pe.provider_id = p.id
AND p.provider_type = :ptype
AND pe.api_format = :fmt
AND (pe.body_rules IS NULL OR pe.body_rules::text = 'null')
"""),
{"rules": rules_json, "ptype": "codex", "fmt": _TARGET_FORMATS[0]},
)
if result.rowcount:
print(f" backfilled body_rules for {result.rowcount} endpoint(s)")
def downgrade() -> None:
# Data backfill: no-op to avoid removing user-customized rules.
return

Some files were not shown because too many files have changed in this diff Show More