Use Docker Model Runner with Compose Bridge

Compose Bridge supports model-aware deployments. It can deploy and configure Docker Model Runner, a lightweight service that hosts and serves machine LLMs.

This reduces manual setup for LLM-enabled services and keeps deployments consistent across Docker Desktop and Kubernetes environments.

If you have a models top-level element in your compose.yaml file, Compose Bridge:

• Automatically injects environment variables for each model’s endpoint and name.
• Configures model endpoints differently for Docker Desktop vs Kubernetes.
• Optionally deploys Docker Model Runner in Kubernetes when enabled in Helm values

Configure model runner settings#

When deploying using generated Helm Charts, you can control the model runner configuration through Helm values.

# Model Runner settings
modelRunner:
    # Set to false for Docker Desktop (uses host instance)
    # Set to true for standalone Kubernetes clusters
    enabled: false
    # Endpoint used when enabled=false (Docker Desktop)
    hostEndpoint: "http://host.docker.internal:12434/engines/v1/"
    # Deployment settings when enabled=true
    image: "docker/model-runner:latest"
    imagePullPolicy: "IfNotPresent"
    # GPU support
    gpu:
        enabled: false
        vendor: "nvidia" # nvidia or amd
        count: 1
    # Node scheduling (uncomment and customize as needed)
    # nodeSelector:
    #   accelerator: nvidia-tesla-t4
    # tolerations: []
    # affinity: {}

    # Security context
    securityContext:
        allowPrivilegeEscalation: false
    # Environment variables (uncomment and add as needed)
    # env:
    #   DMR_ORIGINS: "http://localhost:31246"
    resources:
        limits:
            cpu: "1000m"
            memory: "2Gi"
        requests:
            cpu: "100m"
            memory: "256Mi"
    # Storage for models
    storage:
        size: "100Gi"
        storageClass: "" # Empty uses default storage class
    # Models to pre-pull
    models:
        - ai/qwen2.5:latest
        - ai/mxbai-embed-large

Deploying a model runner#

Docker Desktop#

When modelRunner.enabled is false, Compose Bridge configures your workloads to connect to Docker Model Runner running on the host:

http://host.docker.internal:12434/engines/v1/

The endpoint is automatically injected into your service containers.

Kubernetes#

When modelRunner.enabled is true, Compose Bridge uses the generated manifests to deploy Docker Model Runner in your cluster, including:

• Deployment: Runs the docker-model-runner container
• Service: Exposes port 80 (maps to container port 12434)
• PersistentVolumeClaim: Stores model files

The modelRunner.enabled setting also determines the number of replicas for the in-cluster model-runner-deployment:

• When true, the Kubernetes deployment replica count is set to 1, and Docker Model Runner is deployed in the cluster.
• When false, the Kubernetes deployment replica count is 0 (no in-cluster Model Runner). On Docker Desktop, workloads still connect to the host Model Runner as described above.