mirror of
https://github.com/fawney19/Aether.git
synced 2026-09-09 12:40:20 +08:00
feat(gateway): Codex/OpenAI Responses WebSocket 代理模式
在 /v1/responses 上支持 WebSocket 升级，把客户端帧中继到上游 Codex / OpenAI Responses WebSocket 端点，同时保持既有的路由、鉴权、配额与用量 语义： - 路由与准入：control/route/ai.rs 识别 WebSocket 升级请求； websocket/ingress.rs 复用 API Key 鉴权、IP 规则与并发许可，并引入 独立的 WebSocket 连接许可 - 中继：websocket/responses/* 按 connection / session / turn 分层， 帧解析归一化、socket 写入有界、continuation 保持调度亲和性 - 配额：orchestration/codex_quota_breaker.rs 在账号配额耗尽时熔断并 自动恢复，不再直接断开客户端连接 - 用量：每个 turn 的终态用量落库，request_metadata 记录 websocket_mode / websocket_transport，管理端与 usage 视图暴露 is_websocket - 管理端：provider 可配置 Responses WebSocket 开关
This commit is contained in:
AAEE86
@@ -0,0 +1,194 @@
|
||||
# WebSocket Mode
|
||||
|
||||
The Responses API supports a WebSocket mode for long-running, tool-call-heavy workflows. In this mode, you keep a persistent connection to `/v1/responses` and continue each turn by sending only new input items plus `previous_response_id`.
|
||||
|
||||
WebSocket mode is compatible with both Zero Data Retention (ZDR) and `store=false`.
|
||||
|
||||
## Why use WebSocket mode
|
||||
|
||||
WebSocket mode is most useful when a workflow involves many model-tool round trips (for example, agentic coding or orchestration loops with repeated tool calls).
|
||||
|
||||
Because the connection stays open and each turn sends only incremental input, WebSocket mode reduces per-turn continuation overhead and improves end-to-end latency across long chains. For rollouts with 20+ tool calls, we have seen up to roughly 40% faster end-to-end execution.
|
||||
|
||||
## Connect and create responses
|
||||
|
||||
In WebSocket mode, start each turn by sending a `response.create` event from the client. The payload mirrors the normal [Responses create body](https://developers.openai.com/api/reference/resources/responses/methods/create), except that transport-specific fields like `stream` and `background` are not used.
|
||||
|
||||
```python
|
||||
from websocket import create_connection
|
||||
import json
|
||||
import os
|
||||
|
||||
ws = create_connection(
|
||||
"wss://api.openai.com/v1/responses",
|
||||
header=[
|
||||
f"Authorization: Bearer {os.environ['OPENAI_API_KEY']}",
|
||||
],
|
||||
)
|
||||
|
||||
ws.send(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "response.create",
|
||||
"model": "gpt-5.6",
|
||||
"store": False,
|
||||
"input": [
|
||||
{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": "Find fizz_buzz()"}],
|
||||
}
|
||||
],
|
||||
"tools": [],
|
||||
}
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
Clients can optionally warm up request state by sending `response.create` with `generate: false`. This is useful when you already know the tools, instructions, and/or custom messages you plan to send with an upcoming turn. `generate: false` does not return a model output, but prepares request state so the next generated turn can start faster. The warmup request returns a response ID that you can chain from with `previous_response_id`, including on later turns in a response chain. The next section explains how to continue a session using `previous_response_id` and incremental inputs.
|
||||
|
||||
## Continue with incremental inputs
|
||||
|
||||
To continue a run, send another `response.create` with:
|
||||
|
||||
- `previous_response_id` set to the prior response ID.
|
||||
- `input` containing only new items (for example, tool outputs and the next user message).
|
||||
|
||||
```python
|
||||
ws.send(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "response.create",
|
||||
"model": "gpt-5.6",
|
||||
"store": False,
|
||||
"previous_response_id": "resp_123",
|
||||
"input": [
|
||||
{
|
||||
"type": "function_call_output",
|
||||
"call_id": "call_123",
|
||||
"output": "tool result",
|
||||
},
|
||||
{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": "Now optimize it."}],
|
||||
},
|
||||
],
|
||||
"tools": [],
|
||||
}
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## How continuation works
|
||||
|
||||
WebSocket mode uses the same `previous_response_id` chaining semantics as HTTP mode, but it adds a lower-latency continuation path on the active socket.
|
||||
|
||||
On an active WebSocket connection, the service keeps one previous-response state in a connection-local in-memory cache (the most recent response). Continuing from that most recent response is fast because the service can reuse connection-local state. Because the previous-response state is retained only in memory and is not written to disk, you can use WebSocket mode in a way that is compatible with `store=false` and Zero Data Retention (ZDR).
|
||||
|
||||
If a `previous_response_id` is not in the in-memory cache, behavior depends on whether you store responses:
|
||||
|
||||
- With `store=true`, the service may hydrate older response IDs from persisted state when available. Continuation can still work, but it usually loses the in-memory latency benefit.
|
||||
- With `store=false` (including ZDR), there is no persisted fallback. If the ID is uncached, the request returns `previous_response_not_found`.
|
||||
|
||||
If a turn fails (`4xx` or `5xx`), the service evicts the referenced `previous_response_id` from the connection-local cache. This prevents reusing stale cached state for that failed continuation.
|
||||
|
||||
## Compaction and creating new responses
|
||||
|
||||
If you are using compaction, there are two different continuation patterns:
|
||||
|
||||
### Server-side compaction (`context_management`)
|
||||
|
||||
When you enable server-side compaction (`context_management` with `compact_threshold`), compaction happens during normal `/responses` generation. In WebSocket mode, you continue the same way you normally do: send the next `response.create` with the latest `previous_response_id` and only new input items.
|
||||
|
||||
### Standalone `/responses/compact`
|
||||
|
||||
The standalone [`/responses/compact` endpoint](https://developers.openai.com/api/docs/api-reference/responses/compact) returns a new compacted input window, not a response ID. After compaction, create a new response on your WebSocket connection using the compacted window as `input` (plus the next user/tool items).
|
||||
|
||||
Start a new chain by omitting `previous_response_id` or setting it to `null`. Pass the compacted output as-is; do not prune the returned window.
|
||||
|
||||
```python
|
||||
# Compact your current window (HTTP call)
|
||||
compacted = client.responses.compact(
|
||||
model="gpt-5.6",
|
||||
input=long_input_items_array,
|
||||
)
|
||||
|
||||
# Start a new response on the WebSocket using the compacted window
|
||||
ws.send(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "response.create",
|
||||
"model": "gpt-5.6",
|
||||
"store": False,
|
||||
"input": [
|
||||
*compacted.output,
|
||||
{
|
||||
"type": "message",
|
||||
"role": "user",
|
||||
"content": [{"type": "input_text", "text": "Continue from here."}],
|
||||
},
|
||||
],
|
||||
"tools": [],
|
||||
}
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Connection behavior and limits
|
||||
|
||||
- Server events and ordering match the existing Responses streaming event model.
|
||||
- A single WebSocket connection can receive multiple `response.create` messages, but it runs them sequentially (one in-flight response at a time).
|
||||
- No multiplexing support today. Use multiple connections if you need parallel runs.
|
||||
- Connection duration is limited to 60 minutes. Reconnect when the limit is reached.
|
||||
- Aether binds each upstream WebSocket to one selected provider key. A provider must explicitly enable the standard Responses WebSocket capability and expose an `openai:responses` endpoint before it is eligible for this bridge.
|
||||
- The Codex adapter additionally watches Codex quota events. A `usage_limit_reached` terminal error immediately marks the bound account unavailable. If the client has not received a standard `response.*` event and the request has no `previous_response_id`, Aether retries that one turn once on another eligible key without closing the public socket.
|
||||
- After a standard response event has reached the client, after a retry has already been attempted, or for a request using `previous_response_id`, Aether forwards the provider terminal error and detaches only the exhausted upstream. If the upstream closes immediately after the quota signal, Aether emits a recoverable gateway error instead. The public WebSocket stays open so a later independent `response.create` can select another key.
|
||||
- Aether does not transparently move an existing response chain to another provider key. Connection-local `previous_response_id` state cannot be transferred safely, especially with `store=false`/ZDR; send a new request with complete input after an exhausted continuation.
|
||||
|
||||
## Reconnect and recover
|
||||
|
||||
When a connection closes (or hits the 60-minute limit), open a new WebSocket connection and continue with one of these patterns:
|
||||
|
||||
1. If your prior response is persisted (`store=true`) and you have a valid response ID, continue with `previous_response_id` and new input items.
|
||||
2. If you cannot continue the chain (for example, `store=false`/ZDR or `previous_response_not_found`), start a new response by setting `previous_response_id` to `null` (or omitting it) and send the full input context for the next turn.
|
||||
3. If you compacted context with `/responses/compact`, use the returned compacted window as the base `input` for that new response, then append the latest user/tool items.
|
||||
|
||||
## Errors to handle
|
||||
|
||||
`previous_response_not_found`
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"status": 400,
|
||||
"error": {
|
||||
"code": "previous_response_not_found",
|
||||
"message": "Previous response with id 'resp_abc' not found.",
|
||||
"param": "previous_response_id"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`websocket_connection_limit_reached`
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "invalid_request_error",
|
||||
"code": "websocket_connection_limit_reached",
|
||||
"message": "Responses websocket connection limit reached (60 minutes). Create a new websocket connection to continue."
|
||||
},
|
||||
"status": 400
|
||||
}
|
||||
```
|
||||
|
||||
## Related guides
|
||||
|
||||
- [Conversation state](https://developers.openai.com/api/docs/guides/conversation-state)
|
||||
- [Streaming API responses](https://developers.openai.com/api/docs/guides/streaming-responses)
|
||||
- [Responses streaming events reference](https://developers.openai.com/api/docs/api-reference/responses-streaming)
|
||||
Reference in New Issue
Block a user