mirror of
https://github.com/fawney19/Aether.git
synced 2026-09-09 20:50:20 +08:00
feat(embedding): 接入阿里云多模态向量端点
This commit is contained in:
+81
-3
@@ -32,7 +32,7 @@ Required fields:
|
||||
| Field | Type | Notes |
|
||||
| --- | --- | --- |
|
||||
| `model` | string | Must name a model allowed for the API key and user. Blank strings are rejected. |
|
||||
| `input` | string, string array, integer token array, or nested integer token arrays | Must be non empty. Empty strings, empty arrays, and nested arrays with empty token arrays are rejected. |
|
||||
| `input` | string, string array, integer token array, nested integer token arrays, or multimodal object array | Must be non empty. Empty strings, empty arrays, empty token arrays, and empty multimodal objects are rejected. |
|
||||
|
||||
Optional fields that pass through the embedding conversion path when supported by the provider:
|
||||
|
||||
@@ -40,6 +40,7 @@ Optional fields that pass through the embedding conversion path when supported b
|
||||
| --- | --- |
|
||||
| `encoding_format` | Passed to OpenAI compatible providers. |
|
||||
| `dimensions` | Passed to providers whose embedding request shape supports it. |
|
||||
| `parameters` | Provider-specific embedding parameters. For Aliyun DashScope this maps to DashScope `parameters`; `dimensions` is emitted as `parameters.dimension` unless `parameters.dimension` is already set. |
|
||||
| `user` | Passed to OpenAI compatible providers. |
|
||||
| `task` | Passed to Jina and OpenAI compatible embedding requests. Jina defaults to `text-matching` when no task is supplied. |
|
||||
|
||||
@@ -61,7 +62,18 @@ Accepted `input` shapes:
|
||||
{ "model": "text-embedding-3-small", "input": [[1, 2], [3, 4]] }
|
||||
```
|
||||
|
||||
Use string or string array input when routing to Gemini or Doubao embedding providers. Token arrays are accepted by the OpenAI compatible public endpoint, but Gemini and Doubao provider request emitters require text input.
|
||||
```json
|
||||
{
|
||||
"model": "qwen3-vl-embedding",
|
||||
"input": [
|
||||
{ "text": "white running shoes" },
|
||||
{ "image": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/256_1.png" }
|
||||
],
|
||||
"parameters": { "enable_fusion": true }
|
||||
}
|
||||
```
|
||||
|
||||
Use string or string array input when routing to Gemini or Doubao embedding providers. Token arrays are accepted by the OpenAI compatible public endpoint, but Gemini, Doubao, and Aliyun provider request emitters require text or multimodal content input.
|
||||
|
||||
## Provider Format Mapping
|
||||
|
||||
@@ -73,6 +85,7 @@ Embedding routes can select only embedding provider API formats. Chat, responses
|
||||
| `jina:embedding` | `/v1/embeddings` | OpenAI compatible payload with a Jina `task`. Defaults to `text-matching` if omitted. |
|
||||
| `gemini:embedding` | `models/{model}:embedContent` | Single text input uses `content.parts[].text`. Multiple text inputs use `requests[].content.parts[].text`. |
|
||||
| `doubao:embedding` | `/embeddings/multimodal` | Text input is emitted as `input` items like `{ "type": "text", "text": "..." }`. |
|
||||
| `aliyun:multimodal_embedding` | `/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding` | Text and multimodal inputs are emitted as DashScope `input.contents`. Supports `text`, `image`, `video`, `multi_images`, `parameters.enable_fusion`, `parameters.res_level`, and `parameters.max_video_frames`. Alias: `dashscope:multimodal_embedding`. |
|
||||
|
||||
Custom provider endpoint paths are available when the endpoint is configured for an embedding API format. Gemini custom paths can use `{model}` and `{action}`. For `gemini:embedding`, `{action}` expands to `embedContent`.
|
||||
|
||||
@@ -81,7 +94,7 @@ Custom provider endpoint paths are available when the endpoint is configured for
|
||||
To use embeddings through the gateway:
|
||||
|
||||
1. The global model should include embedding metadata, for example `supported_capabilities: ["embedding"]`, `config.model_type: "embedding"`, or `config.api_formats` with one of the embedding formats.
|
||||
2. The provider model or mapping must expose an embedding API format, one of `openai:embedding`, `gemini:embedding`, `jina:embedding`, or `doubao:embedding`.
|
||||
2. The provider model or mapping must expose an embedding API format, one of `openai:embedding`, `gemini:embedding`, `jina:embedding`, `doubao:embedding`, or `aliyun:multimodal_embedding`.
|
||||
3. The user and API key must be allowed to access the model and the `openai:embedding` client API format.
|
||||
4. Public and admin catalog responses expose `supports_embedding` so clients can display embedding capability separately from chat.
|
||||
|
||||
@@ -89,6 +102,71 @@ Billing fails closed for embedding global models. A model marked as embedding ca
|
||||
|
||||
No schema migration is needed for embedding metadata. Existing model capability, config, provider mapping, API format, and pricing fields carry the data.
|
||||
|
||||
## Aliyun Qwen3-VL Examples
|
||||
|
||||
Text request through Aether:
|
||||
|
||||
```bash
|
||||
curl -sS "http://localhost:8084/v1/embeddings" \
|
||||
-H "Authorization: Bearer sk-your-aether-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen3-vl-embedding",
|
||||
"input": "white running shoes",
|
||||
"dimensions": 1024
|
||||
}'
|
||||
```
|
||||
|
||||
Image and text fusion request:
|
||||
|
||||
```bash
|
||||
curl -sS "http://localhost:8084/v1/embeddings" \
|
||||
-H "Authorization: Bearer sk-your-aether-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen3-vl-embedding",
|
||||
"input": [
|
||||
{ "text": "white running shoes, lightweight and breathable" },
|
||||
{ "image": "https://dashscope.oss-cn-beijing.aliyuncs.com/images/256_1.png" }
|
||||
],
|
||||
"parameters": { "enable_fusion": true }
|
||||
}'
|
||||
```
|
||||
|
||||
Video request:
|
||||
|
||||
```bash
|
||||
curl -sS "http://localhost:8084/v1/embeddings" \
|
||||
-H "Authorization: Bearer sk-your-aether-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen3-vl-embedding",
|
||||
"input": [
|
||||
{ "video": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250107/lbcemt/new+video.mp4" }
|
||||
],
|
||||
"parameters": { "max_video_frames": 64 }
|
||||
}'
|
||||
```
|
||||
|
||||
Multi-image fusion request:
|
||||
|
||||
```bash
|
||||
curl -sS "http://localhost:8084/v1/embeddings" \
|
||||
-H "Authorization: Bearer sk-your-aether-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen3-vl-embedding",
|
||||
"input": [
|
||||
{ "text": "product photos from multiple angles" },
|
||||
{ "multi_images": [
|
||||
"https://example.com/front.png",
|
||||
"https://example.com/side.png"
|
||||
] }
|
||||
],
|
||||
"parameters": { "enable_fusion": true }
|
||||
}'
|
||||
```
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
The gateway validates deterministic request errors before local execution or provider transport.
|
||||
|
||||
Reference in New Issue
Block a user