Provider routing

Choose a provider

Put a provider object in the request body. To use one provider, set only to its exact provider ID. Use the ID from that model's endpoints in GET /v1/models, not its display name. A provider that does not serve the selected model cannot be added by a routing preference.

The Models filter groups both Amazon Bedrock routes under one cloud. Routing still uses the exact endpoint IDs: bedrock-global or bedrock-us-east-1. The display group name amazon-bedrock is not an API code. “Global” is an endpoint label, not a data-residency promise.

{
  "provider": { "only": ["bedrock-us-east-1"] }
}

To allow either route within one provider family, list both exact IDs. AI Hub still considers only routes that serve the requested model; a listed but unsupported route does not become eligible.

{
  "provider": { "only": ["bedrock-global", "bedrock-us-east-1"] }
}

These provider IDs cover text, image and video routes. Each model supports only a subset. Availability can change, so check that model's endpoints before pinning.

The deepseek/deepseek-v4-pro model below currently offers three of these IDs: Alibaba, Baidu and Tencent Cloud (TokenHub). This request uses only Alibaba Model Studio. If that endpoint is unavailable, AI Hub returns an error instead of trying another provider. Set TOKENHUB_API_KEY to your key before running an example.

import json
import os
import urllib.request

body = {
  "model": "deepseek/deepseek-v4-pro",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "provider": {
    "only": [
      "alibaba-model-studio"
    ]
  }
}
request = urllib.request.Request(
    "https://tokenhub-api.belive.sg/api/v1/chat/completions",
    data=json.dumps(body).encode("utf-8"),
    headers={
        "Authorization": f"Bearer {os.environ['TOKENHUB_API_KEY']}",
        "Content-Type": "application/json",
    },
)
with urllib.request.urlopen(request, timeout=60) as response:
    print(json.load(response))

const body = {
  "model": "deepseek/deepseek-v4-pro",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "provider": {
    "only": [
      "alibaba-model-studio"
    ]
  }
};
const response = await fetch("https://tokenhub-api.belive.sg/api/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: "Bearer " + process.env.TOKENHUB_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify(body),
});
if (!response.ok) throw new Error("Request failed: " + response.status);
console.log(await response.json());

curl --fail-with-body -sS 'https://tokenhub-api.belive.sg/api/v1/chat/completions' \
  -H "Authorization: Bearer $TOKENHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "deepseek/deepseek-v4-pro",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "provider": {
    "only": [
      "alibaba-model-studio"
    ]
  }
}'

Find provider IDs for any model

Change MODEL_ID below to the model you want. GET /v1/models lists each model's endpoints. Read the provider field of each endpoint; those exact values work in only, order and ignore for that model.

MODEL_ID='deepseek/deepseek-v4-pro'
curl --fail-with-body -sS 'https://tokenhub-api.belive.sg/api/v1/models' | \
  jq --arg model "$MODEL_ID" \
    '[.data[] | select(.id == $model) | .endpoints[].provider]'

[
  "alibaba-model-studio",
  "baidu-intl",
  "tencent-tokenhub"
]

For image models, read GET /v1/images/models; for video models, read GET /v1/videos/models. Both return the same data list with endpoints[].provider; the video list adds the supported_* fields. Some routes, such as tencent-mps, appear only in a media catalogue. A model with one endpoint has one provider, so routing preferences cannot change how it is served.

Provider options

Send provider in the body of POST /v1/chat/completions or POST /v1/embeddings. Every key is optional.

Each list names at most 16 providers. An absent provider and an empty {} mean the same thing: your workspace default applies, or AI Hub's usual order when there is none.

Prefer a provider, then fall back

With order alone, the named providers are tried first and every other provider stays available as a fallback.

{
  "provider": {
    "order": ["alibaba-model-studio", "baidu-intl"]
  }
}

Fall back to another model

models lists the models to try after model, in order. Provider preferences apply to each model separately.

{
  "model": "deepseek/deepseek-v4-pro",
  "models": ["qwen/qwen3.8-max", "moonshotai/kimi-k3"],
  "messages": [{ "role": "user", "content": "Hello" }]
}

Exclude a provider

{
  "provider": {
    "ignore": ["tencent-tokenhub"]
  }
}

Naming the same provider in only and ignore is 400 invalid_request.

Sort by price, latency or throughput

{
  "provider": {
    "sort": "latency"
  }
}

sort ranks only the providers that order did not name. "price" ranks by the estimated cost of the request, cheapest first. "latency" ranks by the median time from dispatch to the first token, lowest first. "throughput" ranks by the median output tokens per second after the first token, highest first.

Both medians come from streamed requests over the last five minutes, measured per provider of each model. A provider with fewer than three recent samples ranks after the measured ones. Ties, and providers without measurements, keep AI Hub's usual order. The model's details on the Models page show the current medians.

How preferences are applied

1. AI Hub starts from the providers currently available for the requested model and key. A preference can remove or reorder these providers. It can never add one.
2. ignore removes providers, then only removes every provider it does not name.
3. With require_parameters: true, a provider recorded as refusing a sampling field the request sends is removed. A field with no recorded result does not remove a provider.
4. The providers named in order go first, in that order. The rest follow, ranked by sort when it is set.
5. allow_fallbacks: false then keeps only the providers order named, or the first provider when there is no order.

With a models fallback list, the preferences apply to each model separately. allow_fallbacks: false limits the providers of each model; the next model in the list is still tried.

A provider id that serves none of the models has no effect in order or ignore. In only it allows nothing.

Model pins

A workspace admin can pin a model to one provider in the model's details on the Models page. Every member can see the pin. For that model, the pin acts as "provider": {"only": ["<provider>"]} on every chat completion and embedding sent with the workspace's API keys. A models fallback list still applies, with each model's own pin.

When the pinned provider cannot serve a request, because it is unavailable or an ignore list excludes it, the pin is skipped for that request and the usual order applies. A pin never causes a 503 by itself.

How the layers combine

Provider preferences can come from four layers: the request, then the preset, then the model pin, then the workspace default. A preset can carry provider preferences, and a workspace can set a default sort under Settings → Organisation & integrations → Routing defaults. AI Hub merges the layers key by key:

• order or only in a higher layer replaces the lower layers' list. A request's or preset's order or only replaces a model pin.
• sort alone keeps a model pin: the pinned provider still serves the request.
• The ignore lists from every layer are combined. A provider ignored by any layer is never used.
• allow_fallbacks and require_parameters in a higher layer replace the lower layers' value.

The request's entry in Activity names the layer that set its provider order, including a model pin.

When no provider matches

If the preferences exclude every provider of every model in the request, AI Hub answers 503 provider_unavailable before calling anyone. Nothing is billed. metadata.excluded names each provider and the preference that removed it. For example, qwen/qwen3.8-max with "provider": {"only": ["bitdeer-ai"]}, a provider that does not serve it:

{
  "error": {
    "code": 503,
    "type": "provider_unavailable",
    "message": "no provider matches the routing preferences",
    "metadata": {
      "reason": "no_provider_matches",
      "excluded": [
        { "provider": "alibaba-model-studio", "by": "only" },
        { "provider": "partner-cloud-international", "by": "only" }
      ]
    }
  }
}

This response has no Retry-After: the same request cannot succeed later. Change the preferences instead.

When require_parameters: true removed providers and none is left, the answer is 400 unsupported_parameter instead. metadata.unsupported_fields names the fields no remaining provider accepts. Remove those fields, or drop require_parameters to let AI Hub remove them per provider. A value that is not a boolean is 400 invalid_request.

{
  "error": {
    "code": 400,
    "type": "unsupported_parameter",
    "message": "no provider for this model supports: temperature (`provider.require_parameters` is true)",
    "metadata": { "unsupported_fields": ["temperature"] }
  }
}

Without require_parameters, the request succeeds and the X-TokenHub-Ignored-Fields response header names the removed fields. See Sampling fields a route refuses.

See which provider served

The X-Provider-Name response header names the provider that answered. The request's entry in Activity shows the same provider.

Media fallback and manual reruns

Images, video and named speech synthesis accept provider.order, provider.only and provider.allow_fallbacks. Auto routing or an ordered list may try another eligible provider only after a verified refusal before acceptance, such as capacity refusal. allow_fallbacks: false keeps the ordered providers, or only the first eligible provider when none is ordered. A single-provider only disables fallback. Accepted jobs, uncertain submissions, timeouts after dispatch, content-policy refusals and invalid requests are not automatically sent to another provider.

A failed media response may include metadata.charged (true, false or unknown), permitted public alternatives, and the actual failed_provider when known. Charge state describes customer settlement evidence, not an upstream provider's bill. The optional X-TokenHub-Generation-Attempt: not_sent header reports evidence that no attempt was sent; when it is absent, read the job or metadata.charged before retrying.

POST /api/v1/media/offers is a bearer-key-only, read-only operation. Send a route of image_generations, video_generations or audio_speech, the same original JSON in request, and an optional failed_provider. Speech offers require a named model. This operation does not submit work, reserve funds, call a provider or store the JSON. Original routing preferences do not exclude manual alternatives; all settings and current permissions still apply.

Each offer has a public provider, installed price_version_id, quote: {basis: 'estimate', amount_usd}, and advisory capacity: {free_slots, scope}. The amount estimates the full same-settings admission, including the applicable platform charge. Display Estimated US$…; actual usage may vary, never a guaranteed maximum. Null slots with unreported scope mean not reported, not zero. Slots and estimates are snapshots, not reservations.

After explicit confirmation, create a new logical request with the same settings, provider: {only: [chosen_provider]} and tokenhub_quote: {price_version_id, amount_usd} copied from the selected estimate. Replace an old quote and routing order. Video uses a fresh Idempotency-Key for this intentionally new request. Malformed or unpinned controls return 400; a changed price or full estimate returns 409 conflict with metadata.reason: quote_changed before admission. Load new offers and ask for confirmation again. This control detects estimate drift; it does not cap actual usage.

The console's media modes and failed Activity drawer offer Re-run on another provider only for permitted alternatives on supported failures. Activity requires you to enter the original JSON and an inference key; both stay in browser memory and clear when the dialog closes. Original JSON is not reconstructed or retained by Activity. Validation and content-policy errors do not offer reruns. A failed provider is disabled in the picker; historical unknown providers remain unnamed. Read the safe retry rulesbefore confirming another paid request.

Where routing preferences apply

Chat completions and embeddings honour provider. Image, video and named speech generation accept order and only; any other key returns 400 unsupported_parameter. They select only among the endpoints that serve the requested model; with one eligible endpoint they cannot change the route. Capability endpoints take no model, so they take no provider either.