DMR REST API

Once Model Runner is enabled, new API endpoints are available. You can use these endpoints to interact with a model programmatically. Docker Model Runner provides compatibility with OpenAI, Anthropic, and Ollama API formats.

Determine the base URL#

The base URL to interact with the endpoints depends on how you run Docker and which API format you're using.

{{< tabs >}} {{< tab name="Docker Desktop">}}

[!NOTE] TCP host access must be enabled. See Enable Docker Model Runner.

{{< /tab >}} {{< tab name="Docker Engine">}}

[!NOTE] The 172.17.0.1 interface may not be available by default to containers within a Compose project. In this case, add an extra_hosts directive to your Compose service YAML:

yaml extra_hosts: - "model-runner.docker.internal:host-gateway" Then you can access the Docker Model Runner APIs at http://model-runner.docker.internal:12434/

{{< /tab >}} {{}}

Base URLs for third-party tools#

When configuring third-party tools that expect OpenAI-compatible APIs, use these base URLs:

See IDE and tool integrations for specific configuration examples.

Supported APIs#

Docker Model Runner supports multiple API formats:

OpenAI-compatible API#

DMR implements the OpenAI API specification for maximum compatibility with existing tools and frameworks.

Endpoints#

[!NOTE] You can optionally include the engine name in the path: /engines/llama.cpp/v1/chat/completions. This is useful when running multiple inference engines.

Model name format#

When specifying a model in API requests, use the full model identifier including the namespace:

{
  "model": "ai/smollm2",
  "messages": [...]
}

Common model name formats: - Docker Hub models: ai/smollm2, ai/llama3.2, ai/qwen2.5-coder - Tagged versions: ai/smollm2:360M-Q4_K_M - Custom models: myorg/mymodel

Supported parameters#

The following OpenAI API parameters are supported:

Limitations and differences from OpenAI#

Be aware of these differences when using DMR's OpenAI-compatible API:

Anthropic-compatible API#

DMR provides Anthropic Messages API compatibility for tools and frameworks built for Claude.

Endpoints#

Supported parameters#

The following Anthropic API parameters are supported:

Example: Chat with Anthropic API#

curl http://localhost:12434/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

Example: Streaming response#

curl http://localhost:12434/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "max_tokens": 1024,
    "stream": true,
    "messages": [
      {"role": "user", "content": "Count from 1 to 10"}
    ]
  }'

Ollama-compatible API#

DMR also provides Ollama-compatible endpoints for tools and frameworks built for Ollama.

Endpoints#

Example: Chat with Ollama API#

curl http://localhost:12434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/smollm2",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'

Example: List models#

curl http://localhost:12434/api/tags

Image generation API (Diffusers)#

DMR supports image generation through the Diffusers backend, enabling you to generate images from text prompts using models like Stable Diffusion.

[!NOTE] The Diffusers backend requires an NVIDIA GPU with CUDA support and is only available on Linux (x86_64 and ARM64). See Inference engines for setup instructions.

Endpoint#

Supported parameters#

Response format#

The API returns a JSON response with the generated image encoded in base64:

{
  "data": [
    {
      "b64_json": "<base64-encoded-image-data>"
    }
  ]
}

Example: Generate an image#

curl -s -X POST http://localhost:12434/engines/diffusers/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stable-diffusion:Q4",
    "prompt": "A picture of a nice cat",
    "size": "512x512"
  }' | jq -r '.data[0].b64_json' | base64 -d > image.png

This command: 1. Sends a POST request to the Diffusers image generation endpoint 2. Specifies the model, prompt, and output image size 3. Extracts the base64-encoded image from the response using jq 4. Decodes the base64 data and saves it as image.png

DMR native endpoints#

These endpoints are specific to Docker Model Runner for model management:

REST API examples#

Request from within a container#

To call the chat/completions OpenAI endpoint from within another container using curl:

#!/bin/sh

curl http://model-runner.docker.internal/engines/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "ai/smollm2",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Please write 500 words about the fall of Rome."
            }
        ]
    }'

Request from the host using TCP#

To call the chat/completions OpenAI endpoint from the host via TCP:

1. Enable the host-side TCP support from the Docker Desktop GUI, or via the Docker Desktop CLI. For example: docker desktop enable model-runner --tcp <port>.

If you are running on Windows, also enable GPU-backed inference. See Enable Docker Model Runner.

1. Interact with it as documented in the previous section using localhost and the correct port.

#!/bin/sh

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "model": "ai/smollm2",
      "messages": [
          {
              "role": "system",
              "content": "You are a helpful assistant."
          },
          {
              "role": "user",
              "content": "Please write 500 words about the fall of Rome."
          }
      ]
  }'

Request from the host using a Unix socket#

To call the chat/completions OpenAI endpoint through the Docker socket from the host using curl:

#!/bin/sh

curl --unix-socket $HOME/.docker/run/docker.sock \
    localhost/exp/vDD4.40/engines/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "ai/smollm2",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Please write 500 words about the fall of Rome."
            }
        ]
    }'

Streaming responses#

To receive streaming responses, set stream: true:

curl http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "model": "ai/smollm2",
      "stream": true,
      "messages": [
          {"role": "user", "content": "Count from 1 to 10"}
      ]
  }'

Using with OpenAI SDKs#

Python#

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:12434/engines/v1",
    api_key="not-needed"  # DMR doesn't require an API key
)

response = client.chat.completions.create(
    model="ai/smollm2",
    messages=[
        {"role": "user", "content": "Hello!"}
    ]
)

print(response.choices[0].message.content)

Node.js#

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:12434/engines/v1',
  apiKey: 'not-needed',
});

const response = await client.chat.completions.create({
  model: 'ai/smollm2',
  messages: [{ role: 'user', content: 'Hello!' }],
});

console.log(response.choices[0].message.content);

What's next#

• IDE and tool integrations - Configure Cline, Continue, Cursor, and other tools
• Configuration options - Adjust context size and runtime parameters
• Inference engines - Learn about llama.cpp, vLLM, and Diffusers options