feat(embedding): 接入阿里云多模态向量端点

This commit is contained in:
Entropy.Xu
2026-05-28 16:05:36 +08:00
parent 535039c29e
commit 0f6d4b9146
56 changed files with 1605 additions and 42 deletions
+81 -3
View File
@@ -32,7 +32,7 @@ Required fields:
| Field | Type | Notes |
| --- | --- | --- |
| `model` | string | Must name a model allowed for the API key and user. Blank strings are rejected. |
| `input` | string, string array, integer token array, or nested integer token arrays | Must be non empty. Empty strings, empty arrays, and nested arrays with empty token arrays are rejected. |
| `input` | string, string array, integer token array, nested integer token arrays, or multimodal object array | Must be non empty. Empty strings, empty arrays, empty token arrays, and empty multimodal objects are rejected. |
Optional fields that pass through the embedding conversion path when supported by the provider:
@@ -40,6 +40,7 @@ Optional fields that pass through the embedding conversion path when supported b
| --- | --- |
| `encoding_format` | Passed to OpenAI compatible providers. |
| `dimensions` | Passed to providers whose embedding request shape supports it. |
| `parameters` | Provider-specific embedding parameters. For Aliyun DashScope this maps to DashScope `parameters`; `dimensions` is emitted as `parameters.dimension` unless `parameters.dimension` is already set. |
| `user` | Passed to OpenAI compatible providers. |
| `task` | Passed to Jina and OpenAI compatible embedding requests. Jina defaults to `text-matching` when no task is supplied. |
@@ -61,7 +62,18 @@ Accepted `input` shapes:
{ "model": "text-embedding-3-small", "input": [[1, 2], [3, 4]] }
```
Use string or string array input when routing to Gemini or Doubao embedding providers. Token arrays are accepted by the OpenAI compatible public endpoint, but Gemini and Doubao provider request emitters require text input.
```json
{
"model": "qwen3-vl-embedding",
"input": [
{ "text": "white running shoes" },
{ "image": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/256_1.png" }
],
"parameters": { "enable_fusion": true }
}
```
Use string or string array input when routing to Gemini or Doubao embedding providers. Token arrays are accepted by the OpenAI compatible public endpoint, but Gemini, Doubao, and Aliyun provider request emitters require text or multimodal content input.
## Provider Format Mapping
@@ -73,6 +85,7 @@ Embedding routes can select only embedding provider API formats. Chat, responses
| `jina:embedding` | `/v1/embeddings` | OpenAI compatible payload with a Jina `task`. Defaults to `text-matching` if omitted. |
| `gemini:embedding` | `models/{model}:embedContent` | Single text input uses `content.parts[].text`. Multiple text inputs use `requests[].content.parts[].text`. |
| `doubao:embedding` | `/embeddings/multimodal` | Text input is emitted as `input` items like `{ "type": "text", "text": "..." }`. |
| `aliyun:multimodal_embedding` | `/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding` | Text and multimodal inputs are emitted as DashScope `input.contents`. Supports `text`, `image`, `video`, `multi_images`, `parameters.enable_fusion`, `parameters.res_level`, and `parameters.max_video_frames`. Alias: `dashscope:multimodal_embedding`. |
Custom provider endpoint paths are available when the endpoint is configured for an embedding API format. Gemini custom paths can use `{model}` and `{action}`. For `gemini:embedding`, `{action}` expands to `embedContent`.
@@ -81,7 +94,7 @@ Custom provider endpoint paths are available when the endpoint is configured for
To use embeddings through the gateway:
1. The global model should include embedding metadata, for example `supported_capabilities: ["embedding"]`, `config.model_type: "embedding"`, or `config.api_formats` with one of the embedding formats.
2. The provider model or mapping must expose an embedding API format, one of `openai:embedding`, `gemini:embedding`, `jina:embedding`, or `doubao:embedding`.
2. The provider model or mapping must expose an embedding API format, one of `openai:embedding`, `gemini:embedding`, `jina:embedding`, `doubao:embedding`, or `aliyun:multimodal_embedding`.
3. The user and API key must be allowed to access the model and the `openai:embedding` client API format.
4. Public and admin catalog responses expose `supports_embedding` so clients can display embedding capability separately from chat.
@@ -89,6 +102,71 @@ Billing fails closed for embedding global models. A model marked as embedding ca
No schema migration is needed for embedding metadata. Existing model capability, config, provider mapping, API format, and pricing fields carry the data.
## Aliyun Qwen3-VL Examples
Text request through Aether:
```bash
curl -sS "http://localhost:8084/v1/embeddings" \
-H "Authorization: Bearer sk-your-aether-key" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-vl-embedding",
"input": "white running shoes",
"dimensions": 1024
}'
```
Image and text fusion request:
```bash
curl -sS "http://localhost:8084/v1/embeddings" \
-H "Authorization: Bearer sk-your-aether-key" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-vl-embedding",
"input": [
{ "text": "white running shoes, lightweight and breathable" },
{ "image": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/256_1.png" }
],
"parameters": { "enable_fusion": true }
}'
```
Video request:
```bash
curl -sS "http://localhost:8084/v1/embeddings" \
-H "Authorization: Bearer sk-your-aether-key" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-vl-embedding",
"input": [
{ "video": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250107/lbcemt/new+video.mp4" }
],
"parameters": { "max_video_frames": 64 }
}'
```
Multi-image fusion request:
```bash
curl -sS "http://localhost:8084/v1/embeddings" \
-H "Authorization: Bearer sk-your-aether-key" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3-vl-embedding",
"input": [
{ "text": "product photos from multiple angles" },
{ "multi_images": [
"https://example.com/front.png",
"https://example.com/side.png"
] }
],
"parameters": { "enable_fusion": true }
}'
```
## Failure Behavior
The gateway validates deterministic request errors before local execution or provider transport.