i/Aether
1
0
Fork 0
mirror of https://github.com/fawney19/Aether.git synced 2026-10-11 19:59:50 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(ws): bill a provider-reached terminal even when client delivery fails

Browse Source 
评审第 5 条后半：provider 终态已经到达、只是 gateway 写客户端 socket 失败时，
relay loop 用 client_disconnected() 覆盖了结算信号，于是一条供应商已经完成推理
并消耗了 token 的响应被记成 void billing、candidate 记 Cancelled、不投射供应商
效果、也不提交 execution report。上游成本凭空消失。

结算表只改一行：作废账单的条件从
    provider.cancelled_by_provider() || delivery.is_aborted()
收紧为
    provider.cancelled_by_provider() || (delivery.is_aborted() && !provider.is_terminal())

于是 Terminal{cancelled=false} + delivery Aborted 与 delivery Complete 落在同一侧：
Billed、candidate Success 或 Failed、投射供应商效果、提交 execution report。
状态码随之变成纯 provider 事实（不再把 200 改写成 499）；作废分支的 provider
状态码本身就是 499，取值不变。

依据：供应商已经完成推理并消耗 token，客户端还能用 previous_response_id 续取
这条响应。供应商没给出终态时（客户端先走了）仍然作废，这一侧未改。

配套改动：
- connection.rs 写客户端失败处改为 record_client_delivery_aborted(reason) +
  settle_signal_for_client_delivery_failure(terminal_outcome)：provider 终态已到达
  就用那条终态作结算信号，不再无条件覆盖。投递失败原因也不再谎称
  「客户端在终态前断开」。
- 投递结果记在 attempt 上而非 logical turn 上：结算按 attempt 进行，且配额透明
  重试时各 attempt 的投递结果彼此独立。
- report_context 新增 websocket_client_delivery="aborted" 与
  websocket_client_delivery_reason，只增字段不改既有字段，便于事后区分
  「客户端拿到了」和「客户端没拿到但已计费」。
- candidate error_type 新增 client_delivery_failed（原先这个场景写的是
  websocket_cancelled）。它排在供应商侧分类之前：这条记录之所以特别正是因为
  内容没送到客户端，供应商侧判定仍由 candidate_status 与 error_message 保留。
- finish_summary 改用作废判定而非「投递失败」判定：provider 终态已到达时摘要
  必须保留真实的 finish_reason 与 usage，否则计费记录会被写坏。

e2e 期望值变化：client_disconnect_mid_turn_still_settles_the_usage_row 改名为
client_disconnect_before_any_provider_output_settles_a_void_row，并补上
「不计费 + status=cancelled + status_code=499」的断言。原用例的 mock 行为是
StallAfterCreated（只发 response.created 就静默），provider 从未给出终态，所以
它走的是未改动的作废一侧；原来的文档注释说「must still be billed」与实际语义
不符，一并纠正。真正被修正的那一行无法在 e2e 里确定性触发——它取决于 relay
loop 的 select! 先观察到上游终态帧还是先观察到已关闭的客户端 socket，是构造性
竞态——因此由 relay 级单测确定性覆盖，e2e 里以注释指向这两个单测。

新增 7 个测试：结算表修正行（并与「投递成功」逐字段对照，只有 candidate 错误
分类不同）、无终态时仍作废、供应商声明取消即使送达也不计费、结算信号选择、
已记录的投递失败不被结算信号覆盖、relay 级「终态到达 + 客户端已关闭 ⇒ Billed /
Success / ProviderSuccess / 已提交 report 且 usage 完整保留」及其镜像、
report_context 只增不改。
This commit is contained in:
AAEE86
2026-08-17 14:53:12 +08:00
committed by ZheFox
parent dc3743aecf
commit 247e7105a2
5 changed files with 425 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/aether-gateway/src/handlers/proxy/websocket/responses/settlement.rs
+174 -39
View File
@@ -155,6 +155,9 @@ pub(super) enum AttemptCandidateStatus {
pub(super) enum AttemptCandidateError {
None,
Cancelled,
/// 供应商已经给出终态，但这一轮内容没能完整交付给客户端。账单照记，
/// candidate 行上留下这条事实。
ClientDeliveryFailed,
MissingTerminal,
TerminalError,
}
@@ -205,7 +208,8 @@ pub(super) const fn classify_responses_websocket_turn_effect(
}
}
/// 把「结算触发信号」+「已观察到的 provider 终态」映射成两个正交事实。
/// 把「结算触发信号」+「已观察到的 provider 终态」+「已记录的投递结果」映射成
/// 两个正交事实。
///
/// `ResponsesWebSocketTurnOutcome` 描述的是 relay loop 为什么现在结算这一
/// attempt，它对 provider 的信息量并不总是完整的：
@@ -214,11 +218,16 @@ pub(super) const fn classify_responses_websocket_turn_effect(
/// - `Cancelled` 只说明「我们为客户端或连接层面的原因停下了」，不携带任何
/// provider 信息。已经观察到的 provider 终态是独立事实，不能被它覆盖——
/// 这正是评审第 5 条要求分开记录的那一处。
///
/// `recorded_delivery` 是 relay loop 明确记下的投递失败（写客户端 socket 失败）。
/// 它与结算信号推出的投递结果取「只要有一侧失败就是失败」，并优先保留明确记录
/// 的原因。
pub(super) fn attempt_facts_for_outcome(
observed_provider_terminal: Option<AttemptProviderOutcome>,
recorded_delivery: AttemptClientDelivery,
settling: ResponsesWebSocketTurnOutcome,
) -> AttemptTerminalFacts {
match settling {
let facts = match settling {
ResponsesWebSocketTurnOutcome::ProviderTerminal {
status_code,
cancelled,
@@ -249,19 +258,46 @@ pub(super) fn attempt_facts_for_outcome(
}),
delivery: AttemptClientDelivery::Aborted { reason },
},
};
AttemptTerminalFacts {
delivery: match recorded_delivery {
AttemptClientDelivery::Aborted { .. } => recorded_delivery,
AttemptClientDelivery::Complete => facts.delivery,
},
..facts
}
}
/// 客户端投递失败时应该用哪个结算信号。
///
/// provider 终态已经到达就用那条终态：它是权威的 provider 事实，绝不能被
/// `client_disconnected()` 覆盖掉——那正是把已完成响应记成 void billing 的原因。
/// 供应商还没给出终态时，客户端断开才是这一 attempt 的全部结论。
pub(super) fn settle_signal_for_client_delivery_failure(
terminal_outcome: Option<ResponsesWebSocketTurnOutcome>,
) -> ResponsesWebSocketTurnOutcome {
terminal_outcome.unwrap_or_else(ResponsesWebSocketTurnOutcome::client_disconnected)
}
/// 这一个 attempt 的账单是否作废。
///
/// 只有两种情况作废：供应商自己声明取消，或者供应商根本没给出终态而客户端
/// 又已经走了。**供应商已经给出终态时，客户端最后一跳投递失败不作废账单**：
/// 供应商已经完成推理并消耗了 token，客户端还能用 `previous_response_id`
/// 续取这条响应，把成本记成 0 等于让上游账单凭空消失。
pub(super) const fn attempt_billing_is_void(facts: AttemptTerminalFacts) -> bool {
facts.provider.cancelled_by_provider()
|| (facts.delivery.is_aborted() && !facts.provider.is_terminal())
}
/// attempt 对外记录的状态码。
///
/// 投递失败一律记 499：与拆分前 `ResponsesWebSocketTurnOutcome::Cancelled`
/// 走的分支一致。payload 需要在结算判定之前就知道状态码，所以单独暴露。
/// 状态码现在纯粹是 provider 事实：客户端投递失败不再把一条已经拿到 200
/// 终态的记录改写成 499。作废分支的 provider 状态码本身就是 499
/// （`response.cancelled` 映射 499，`Cancelled` 信号的兜底也是 499），
/// 所以这些行的取值不变。
pub(super) const fn attempt_status_code(facts: AttemptTerminalFacts) -> u16 {
if facts.delivery.is_aborted() {
CLIENT_CANCELLED_STATUS_CODE
} else {
facts.provider.status_code()
}
facts.provider.status_code()
}
/// 结算判定的输入：两个正交事实 + 记账层对这条 report 的判定 + 终态摘要事实。
@@ -289,10 +325,8 @@ pub(super) struct AttemptSettlement {
/// 由两个正交事实推出结算动作。唯一的判定入口，表驱动测试逐行锁死。
///
/// 当前口径与拆分前的 `finalize()` 完全一致：客户端投递失败与「供应商声明取消」
/// 落在同一侧（作废账单、candidate 记 Cancelled、只释放 lease、不提交 execution
/// report）。即使已经观察到 provider 终态也是如此——这一行是刻意保留的现状，
/// 修正它是独立的一步。
/// provider 终态已到达时，客户端投递失败只影响 candidate 的错误分类，不再作废
/// 账单、不再把状态码改成 499、也不再跳过供应商效果和 execution report。
pub(super) const fn classify_attempt_settlement(
inputs: AttemptSettlementInputs,
) -> AttemptSettlement {
@@ -303,25 +337,30 @@ pub(super) const fn classify_attempt_settlement(
has_parser_error,
} = inputs;
let cancelled = facts.provider.cancelled_by_provider() || facts.delivery.is_aborted();
let void = attempt_billing_is_void(facts);
let status_code = attempt_status_code(facts);
let failed = !cancelled && report_represents_failure;
let missing_terminal = !cancelled && !observed_finish;
let projects_provider_failure = !cancelled
let failed = !void && report_represents_failure;
let missing_terminal = !void && !observed_finish;
let projects_provider_failure = !void
&& (status_code >= 400
|| facts.forced_error().is_some()
|| has_parser_error
|| missing_terminal);
let candidate_status = if cancelled {
let candidate_status = if void {
AttemptCandidateStatus::Cancelled
} else if failed {
AttemptCandidateStatus::Failed
} else {
AttemptCandidateStatus::Success
};
let candidate_error = if cancelled {
// 投递失败排在供应商侧分类之前：这条记录之所以特别，正是因为内容没送到
// 客户端手上。供应商侧的判定仍然通过 candidate_status 和 error_message
// 保留下来。
let candidate_error = if void {
AttemptCandidateError::Cancelled
} else if facts.delivery.is_aborted() {
AttemptCandidateError::ClientDeliveryFailed
} else if missing_terminal {
AttemptCandidateError::MissingTerminal
} else if failed {
@@ -332,7 +371,7 @@ pub(super) const fn classify_attempt_settlement(
AttemptSettlement {
status_code,
billing: if cancelled {
billing: if void {
AttemptBilling::Void
} else {
AttemptBilling::Billed
@@ -340,11 +379,11 @@ pub(super) const fn classify_attempt_settlement(
candidate_status,
candidate_error,
provider_effect: classify_responses_websocket_turn_effect(
cancelled,
void,
projects_provider_failure,
failed,
),
submit_execution_report: !cancelled,
submit_execution_report: !void,
}
}
@@ -352,9 +391,10 @@ pub(super) const fn classify_attempt_settlement(
mod tests {
use super::{
attempt_facts_for_outcome, classify_attempt_settlement,
classify_responses_websocket_turn_effect, AttemptBilling, AttemptCandidateError,
AttemptCandidateStatus, AttemptClientDelivery, AttemptProviderOutcome, AttemptSettlement,
AttemptSettlementInputs, AttemptTerminalFacts, ResponsesWebSocketTurnEffect,
classify_responses_websocket_turn_effect, settle_signal_for_client_delivery_failure,
AttemptBilling, AttemptCandidateError, AttemptCandidateStatus, AttemptClientDelivery,
AttemptProviderOutcome, AttemptSettlement, AttemptSettlementInputs, AttemptTerminalFacts,
ResponsesWebSocketTurnEffect,
};
use super::super::turn::ResponsesWebSocketTurnOutcome;
@@ -401,6 +441,7 @@ mod tests {
assert_eq!(
attempt_facts_for_outcome(
None,
AttemptClientDelivery::Complete,
ResponsesWebSocketTurnOutcome::ProviderTerminal {
status_code: 200,
cancelled: false,
@@ -414,6 +455,7 @@ mod tests {
assert_eq!(
attempt_facts_for_outcome(
None,
AttemptClientDelivery::Complete,
ResponsesWebSocketTurnOutcome::ProviderTerminal {
status_code: 499,
cancelled: true,
@@ -425,7 +467,7 @@ mod tests {
}
);
assert_eq!(
attempt_facts_for_outcome(None, ResponsesWebSocketTurnOutcome::upstream_closed()),
attempt_facts_for_outcome(None, AttemptClientDelivery::Complete, ResponsesWebSocketTurnOutcome::upstream_closed()),
AttemptTerminalFacts {
provider: aborted(
502,
@@ -435,7 +477,7 @@ mod tests {
}
);
assert_eq!(
attempt_facts_for_outcome(None, ResponsesWebSocketTurnOutcome::client_disconnected()),
attempt_facts_for_outcome(None, AttemptClientDelivery::Complete, ResponsesWebSocketTurnOutcome::client_disconnected()),
AttemptTerminalFacts {
provider: aborted(499, "client disconnected before provider terminal event"),
delivery: AttemptClientDelivery::Aborted {
@@ -446,14 +488,15 @@ mod tests {
// 超时一族必须保留 stream_timeout 标记，否则 pool stream timeout 效果丢失。
let first_event_timeout =
attempt_facts_for_outcome(None, ResponsesWebSocketTurnOutcome::first_event_timeout());
attempt_facts_for_outcome(None, AttemptClientDelivery::Complete, ResponsesWebSocketTurnOutcome::first_event_timeout());
assert!(first_event_timeout.provider.stream_timeout());
let terminal_timeout =
attempt_facts_for_outcome(None, ResponsesWebSocketTurnOutcome::terminal_timeout());
attempt_facts_for_outcome(None, AttemptClientDelivery::Complete, ResponsesWebSocketTurnOutcome::terminal_timeout());
assert!(terminal_timeout.provider.stream_timeout());
// 非 504 的失败不得被当成流式超时。
assert!(!attempt_facts_for_outcome(
None,
AttemptClientDelivery::Complete,
ResponsesWebSocketTurnOutcome::upstream_closed()
)
.provider
@@ -462,6 +505,7 @@ mod tests {
// `stream_timeout()` 只匹配 Failure 分支。
assert!(!attempt_facts_for_outcome(
None,
AttemptClientDelivery::Complete,
ResponsesWebSocketTurnOutcome::ProviderTerminal {
status_code: 504,
cancelled: false,
@@ -479,6 +523,7 @@ mod tests {
let facts = attempt_facts_for_outcome(
Some(observed),
AttemptClientDelivery::Complete,
ResponsesWebSocketTurnOutcome::client_disconnected(),
);
assert_eq!(facts.provider, observed);
@@ -492,6 +537,7 @@ mod tests {
// 权威信号不被已记录事实改写。
let facts = attempt_facts_for_outcome(
Some(observed),
AttemptClientDelivery::Complete,
ResponsesWebSocketTurnOutcome::upstream_closed(),
);
assert_eq!(
@@ -678,14 +724,17 @@ mod tests {
);
}
/// ✱ C2 保留现状的那一行：provider 终态已到达，但客户端投递失败 ⇒ 仍作废账单。
/// 这一行是下一步唯一要改的地方，先在这里锁住现状。
/// ✱ 修正后的那一行：provider 终态已到达，客户端投递失败不再作废账单。
///
/// 供应商已经完成推理并消耗 token，客户端还能用 `previous_response_id`
/// 续取这条响应；把成本记成 0 等于让上游账单凭空消失。投递失败作为独立
/// 事实留在 candidate 的错误分类里。
#[test]
fn settlement_table_row_client_delivery_failure_currently_voids_a_reached_terminal() {
fn settlement_table_row_client_delivery_failure_keeps_a_reached_terminal_billed() {
let settlement = settle(
terminal(200),
AttemptClientDelivery::Aborted {
reason: "client disconnected before provider terminal event",
reason: "gateway could not relay the provider event to the client",
},
false,
true,
@@ -694,14 +743,100 @@ mod tests {
assert_eq!(
settlement,
AttemptSettlement {
status_code: 499,
billing: AttemptBilling::Void,
candidate_status: AttemptCandidateStatus::Cancelled,
candidate_error: AttemptCandidateError::Cancelled,
provider_effect: ResponsesWebSocketTurnEffect::ReleasePoolKeyLease,
submit_execution_report: false,
status_code: 200,
billing: AttemptBilling::Billed,
candidate_status: AttemptCandidateStatus::Success,
candidate_error: AttemptCandidateError::ClientDeliveryFailed,
provider_effect: ResponsesWebSocketTurnEffect::ProviderSuccess,
submit_execution_report: true,
}
);
// 除了 candidate 的错误分类，其余判定与「投递成功」完全一致。
let delivered = settle(terminal(200), AttemptClientDelivery::Complete, false, true, false);
assert_eq!(settlement.status_code, delivered.status_code);
assert_eq!(settlement.billing, delivered.billing);
assert_eq!(settlement.candidate_status, delivered.candidate_status);
assert_eq!(settlement.provider_effect, delivered.provider_effect);
assert_eq!(
settlement.submit_execution_report,
delivered.submit_execution_report
);
assert_ne!(settlement.candidate_error, delivered.candidate_error);
}
/// 供应商还没给出终态时，客户端投递失败仍然作废账单：这一轮确实没有产出。
#[test]
fn a_delivery_failure_without_a_provider_terminal_still_voids_the_bill() {
let settlement = settle(
aborted(499, "client went away"),
AttemptClientDelivery::Aborted {
reason: "client went away",
},
false,
false,
false,
);
assert_eq!(settlement.status_code, 499);
assert_eq!(settlement.billing, AttemptBilling::Void);
assert_eq!(
settlement.candidate_status,
AttemptCandidateStatus::Cancelled
);
assert_eq!(settlement.candidate_error, AttemptCandidateError::Cancelled);
assert!(!settlement.submit_execution_report);
}
/// 供应商自己声明取消时，即使内容送到了客户端也不计费。
#[test]
fn a_provider_declared_cancellation_is_void_even_when_delivered() {
let settlement =
settle(provider_cancelled(), AttemptClientDelivery::Complete, false, true, false);
assert_eq!(settlement.billing, AttemptBilling::Void);
assert_eq!(settlement.candidate_error, AttemptCandidateError::Cancelled);
}
/// 结算信号的选择：provider 终态已到达就用它，否则才是 client 断开。
/// 这是修正的核心——旧实现无条件用 client_disconnected() 覆盖，
/// 于是已完成的响应被记成 void billing。
#[test]
fn a_reached_terminal_is_the_settle_signal_for_a_delivery_failure() {
let terminal_outcome = ResponsesWebSocketTurnOutcome::ProviderTerminal {
status_code: 200,
cancelled: false,
};
assert_eq!(
settle_signal_for_client_delivery_failure(Some(terminal_outcome)),
terminal_outcome
);
assert_eq!(
settle_signal_for_client_delivery_failure(None),
ResponsesWebSocketTurnOutcome::client_disconnected()
);
}
/// 明确记录的投递失败不会被结算信号推出的「投递成功」覆盖。
#[test]
fn a_recorded_delivery_failure_survives_a_provider_terminal_settle_signal() {
let facts = attempt_facts_for_outcome(
Some(terminal(200)),
AttemptClientDelivery::Aborted {
reason: "write failed"
},
ResponsesWebSocketTurnOutcome::ProviderTerminal {
status_code: 200,
cancelled: false,
},
);
assert_eq!(facts.provider, terminal(200));
assert_eq!(
facts.delivery,
AttemptClientDelivery::Aborted {
reason: "write failed"
}
);
// 投递失败不是供应商的错误，摘要不该因此补 parser_error。
assert_eq!(facts.forced_error(), None);
}
/// 记账层判 Success，但摘要没观察到 finish：现状会写出