Multimodal inputs
Send images, audio, PDFs and video to compatible models through Chat Completions.
A message's content can be a string or an array of typed parts. Put your instruction in a text part and the media in its own part. Supported formats and transports depend on the model and provider route.
- Image understandingRead pictures, diagrams and receipts.
- Audio inputsAsk a model about a recording.
- PDFsSend a PDF to a native file-capable route.
- Video understandingAnalyze a short clip or submit a long-video job.
Input understanding and generation
These guides describe media input to a model. Image generation, video generation and audio output create media. Capability APIs perform fixed tasks such as OCR or transcription. Media embeddings return vectors and use different part shapes and meters.
Protocol compatibility
| Protocol | Input boundary |
|---|---|
| Chat Completions | Image, audio, PDF and video on eligible routes |
| Responses | Text and URL/data image input |
| Messages | Text and image blocks; document, audio and video blocks are not supported |
| Embeddings | Inline image/audio/video on supported models; no PDF or remote media URLs |
Choose a model and route
Read GET /api/v1/models in your environment. architecture.input_modalities describes a model, while each endpoint's capabilities identifies the serving route: vision for images, audio_input, video_input or file_input. Check the Models page for available routes. The examples pin a declared route; use a different eligible route only with its own supported request form.
Limits, cost and failures
Base64 increases the request size. The synchronous chat body limit defaults to 20 MiB, and provider format, duration, page and context limits also apply. Keep inline samples small; an uploaded video uses a different path described below. A model's context limit is not a universal media-size limit.
Media input can contribute input tokens or a modality-specific meter. Read the selected route's price table and the resulting usage; do not estimate solely from text length. Key caps and model restrictions still apply.
Malformed input can return 400, an oversized body 413, and an unsuitable route 400 or a routing refusal. See Errors before retrying; after a timeout, reconcile before resubmitting.
Support and limits
Each protocol has its own media block: URL and data inputs and uploaded file IDs have their own schemas and size limits, and a model's page lists its input modalities. PDFs, audio and video are refused on routes that do not take them.
Gateway validation, defaults and response forms are in the operation index and the OpenAPI schemas.
Error codes and recovery are in Errors and recovery. Keep the request or job ID from each response: it identifies the call in Activity and for support.