fix: raise the output budget so reasoning models reach the tool call

A reasoning model spends the output budget in order: thinking first, then prose,
then the tool call. With 16000 the thinking alone can consume all of it, so the
turn ends with finishReason "length" before display_diagram is ever called. The
canvas stays empty and nothing surfaces in the UI, because no tool call means no
tool error, and the client never reads finishReason.

Measured on openrouter deepseek/deepseek-v4-flash, the model from the report:
- max_tokens=800 with reasoning on returns reasoning_tokens=800, empty content,
  finish_reason length. So reasoning is billed against this budget, not exempt.
- refining an existing diagram (19k chars of XML in the input) produced 49142
  chars of reasoning, zero tool calls, finishReason "length" at 16000
- the same request at 40000 finished and called edit_diagram with 12 operations

64000 cannot just be sent to every model: bedrock claude-3-haiku caps at 4096,
nova-lite at 10000, and the openrouter deepseek-r1 endpoint counts input and
output against one 64000 ceiling. All three name the real limit in the 400, so
parse it and retry once. Verified: nova-lite logs "64000 rejected, retrying with
10000" and then completes its tool call.

Also expose the budget in Settings. It is sent as a header rather than read from
env only, so desktop users can raise it themselves without an env file.

vercel.json goes back to the 300s it had before #238 traded it for $2-4/month.
That is now Vercel's own default, and billing pauses while the function waits on
the model, so the saving that motivated 120s no longer applies. edgeone.json is
left alone: its 120 may be that platform's actual ceiling.
This commit is contained in:
dayuan.jiang
2026-08-22 11:45:09 +09:00
parent 12903cd516
commit 8fb9ef20bd
13 changed files with 288 additions and 8 deletions

View File

@@ -132,6 +132,8 @@
"customSystemMessage": "Custom System Message",
"customSystemMessageDescription": "Add custom instructions appended to the AI's system prompt.",
"customSystemMessagePlaceholder": "e.g., Always use blue color scheme for diagrams...",
"maxOutputTokens": "Max Output Tokens",
"maxOutputTokensDescription": "Budget for one reply, shared by thinking and the diagram XML. Raise it if the AI keeps thinking and no diagram appears. Leave empty for the default.",
"panelVisibility": "Lobby Panels",
"panelVisibilityDescription": "Choose which panels to show on the chat lobby.",
"showRecentChats": "Recent Chats",

View File

@@ -132,6 +132,8 @@
"customSystemMessage": "カスタムシステムメッセージ",
"customSystemMessageDescription": "AIのシステムプロンプトに追加されるカスタム指示を入力します。",
"customSystemMessagePlaceholder": "例:ダイアグラムには常に青色のカラースキームを使用...",
"maxOutputTokens": "最大出力トークン数",
"maxOutputTokensDescription": "1回の応答の予算で、思考過程とダイアグラムの XML が共有します。AI が考え続けてダイアグラムが生成されない場合は大きくしてください。空欄ならデフォルト値を使います。",
"panelVisibility": "ロビーパネル",
"panelVisibilityDescription": "チャットロビーに表示するパネルを選択します。",
"showRecentChats": "最近のチャット",

View File

@@ -132,6 +132,8 @@
"customSystemMessage": "自訂系統訊息",
"customSystemMessageDescription": "新增自訂指示,將附加到 AI 的系統提示末尾。",
"customSystemMessagePlaceholder": "例如:圖表始終使用藍色配色方案...",
"maxOutputTokens": "最大輸出 token 數",
"maxOutputTokensDescription": "單次回覆的額度,思考過程與圖表 XML 共用。若 AI 一直在思考卻沒有產生圖表,請將它調大。留空則使用預設值。",
"panelVisibility": "大廳面板",
"panelVisibilityDescription": "選擇在聊天大廳顯示哪些面板。",
"showRecentChats": "最近聊天",

View File

@@ -132,6 +132,8 @@
"customSystemMessage": "自定义系统消息",
"customSystemMessageDescription": "添加自定义指令,将附加到 AI 的系统提示末尾。",
"customSystemMessagePlaceholder": "例如:图表始终使用蓝色配色方案...",
"maxOutputTokens": "最大输出 token 数",
"maxOutputTokensDescription": "单次回复的额度,思考过程和图表 XML 共用。如果 AI 一直在思考却没有生成图表,请把它调大。留空则使用默认值。",
"panelVisibility": "大厅面板",
"panelVisibilityDescription": "选择在聊天大厅显示哪些面板。",
"showRecentChats": "最近聊天",

110
lib/output-token-limit.ts Normal file
View File

@@ -0,0 +1,110 @@
import { wrapLanguageModel } from "ai"
type WrappedModel = ReturnType<typeof wrapLanguageModel>
/**
* Default output budget for a chat turn.
*
* This has to cover thinking + prose + the tool call, because reasoning models
* spend it in that order. Measured on deepseek-v4-flash: refining an existing
* diagram burned 16000 tokens on thinking alone and the request ended with
* finishReason "length" before display_diagram was ever called (issue #924).
* 64000 leaves room for the plan and the XML in one turn.
*/
export const DEFAULT_MAX_OUTPUT_TOKENS = 64000
/** Ceiling for the user-supplied override, to catch typos like an extra zero. */
export const MAX_OUTPUT_TOKENS_LIMIT = 200000
/**
* A budget this large exceeds what some models accept. Providers reject it with a
* 400 that names the real limit, so we parse the number out and retry once
* instead of failing the turn.
*
* Formats seen in the wild:
* - Bedrock: "The maximum tokens you requested exceeds the model limit of 4096."
* - OpenRouter: "This endpoint's maximum context length is 64000 tokens. However,
* you requested about 64025 tokens (25 of text input, 64000 in the output)."
* Note this one is an input+output ceiling, so the input has to be subtracted.
* - Anthropic: "max_tokens: 200000 > 64000, which is the maximum allowed..."
* - OpenAI: "This model supports at most 16384 completion tokens"
*/
export function parseOutputTokenLimit(error: unknown): number | null {
const err = error as { message?: unknown; responseBody?: unknown }
const text = [
typeof err?.message === "string" ? err.message : "",
typeof err?.responseBody === "string" ? err.responseBody : "",
].join(" ")
if (!text) return null
// Combined input+output ceiling: subtract the input the provider counted,
// plus a small margin because its estimate is approximate.
const context = text.match(/maximum context length is (\d+)/i)
if (context) {
const input = text.match(/(\d+) of text input/i)
const budget =
Number(context[1]) - (input ? Number(input[1]) : 0) - 1024
return budget > 0 ? budget : null
}
const output =
text.match(/model limit of (\d+)/i) ||
text.match(/> (\d+), which is the maximum/i) ||
text.match(/at most (\d+) completion tokens/i) ||
text.match(/lower than (\d+)/i)
return output ? Number(output[1]) : null
}
/**
* Retry the stream once with a smaller budget when the provider rejects the
* requested one. Without this, raising the default breaks every model whose
* ceiling is below it (measured: bedrock claude-3-haiku 4096, nova-lite 10000,
* openrouter deepseek-r1 64000 shared with the input).
*/
export function withOutputTokenLimitFallback(
model: WrappedModel,
): WrappedModel {
return wrapLanguageModel({
model,
middleware: {
specificationVersion: "v3",
async wrapStream({ doStream, params, model: inner }) {
try {
return await doStream()
} catch (error) {
const limit = parseOutputTokenLimit(error)
const requested = params.maxOutputTokens
if (!limit || !requested || limit >= requested) throw error
console.warn(
`[maxOutputTokens] ${requested} rejected, retrying with ${limit}`,
)
return await inner.doStream({
...params,
maxOutputTokens: limit,
})
}
},
},
})
}
/**
* Resolve the output budget: user setting (sent as a header so it works in the
* desktop app too), then server env, then the default.
*/
export function resolveMaxOutputTokens(headerValue: string | null): number {
const fromHeader = Number(headerValue)
if (
Number.isInteger(fromHeader) &&
fromHeader > 0 &&
fromHeader <= MAX_OUTPUT_TOKENS_LIMIT
) {
return fromHeader
}
return Number(process.env.MAX_OUTPUT_TOKENS) || DEFAULT_MAX_OUTPUT_TOKENS
}

View File

@@ -31,6 +31,9 @@ export const STORAGE_KEYS = {
// Custom system message
customSystemMessage: "next-ai-draw-io-custom-system-message",
// Output token budget per turn (empty = server default)
maxOutputTokens: "next-ai-draw-io-max-output-tokens",
// Panel visibility
showRecentChats: "next-ai-draw-io-show-recent-chats",
showMyTemplates: "next-ai-draw-io-show-my-templates",