You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -18,11 +21,12 @@ Compose lets you define AI models as core components of your application, so you
18
21
## Prerequisites
19
22
20
23
- Docker Compose v2.38 or later
21
-
- A platform that supports Compose models such as Docker Model Runner or compatible cloud providers
24
+
- A platform that supports Compose models such as Docker Model Runner (DMR) or compatible cloud providers.
25
+
If you are using DMR, see the [requirements](/manuals/ai/model-runner/_index.md#requirements).
22
26
23
27
## What are Compose models?
24
28
25
-
Compose `models` are a standardized way to define AI model dependencies in your application. By using the []`models` top-level element](/reference/compose-file/models.md) in your Compose file, you can:
29
+
Compose `models` are a standardized way to define AI model dependencies in your application. By using the [`models` top-level element](/reference/compose-file/models.md) in your Compose file, you can:
26
30
27
31
- Declare which AI models your application needs
28
32
- Specify model configurations and requirements
@@ -66,8 +70,15 @@ models:
66
70
Common configuration options include:
67
71
- `model` (required): The OCI artifact identifier for the model. This is what Compose pulls and runs via the model runner.
68
72
- `context_size`: Defines the maximum token context size for the model.
73
+
74
+
> [!NOTE]
75
+
> Each model has its own maximum context size. When increasing the context length,
76
+
> consider your hardware constraints. In general, try to keep context size
77
+
> as small as feasible for your specific needs.
78
+
69
79
- `runtime_flags`: A list of raw command-line flags passed to the inference engine when the model is started.
70
-
- Platform-specific options may also be available via extensions attributes `x-*`
80
+
For example, if you use llama.cpp, you can pass any of [the available parameters](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
81
+
- Platform-specific options may also be available via extension attributes `x-*`
71
82
72
83
## Service model binding
73
84
@@ -131,25 +142,58 @@ One of the key benefits of using Compose models is portability across different
131
142
132
143
### Docker Model Runner
133
144
134
-
When Docker Model Runner is enabled:
145
+
When [Docker Model Runner is enabled](/manuals/ai/model-runner/_index.md):
135
146
136
147
```yaml
137
148
services:
138
149
chat-app:
139
150
image: my-chat-app
140
151
models:
141
-
- llm
152
+
llm:
153
+
endpoint_var: AI_MODEL_URL
154
+
model_var: AI_MODEL_NAME
142
155
143
156
models:
144
157
llm:
145
158
model: ai/smollm2
159
+
context_size: 4096
160
+
runtime_flags:
161
+
- "--no-prefill-assistant"
146
162
```
147
163
148
164
Docker Model Runner will:
149
165
- Pull and run the specified model locally
150
166
- Provide endpoint URLs for accessing the model
151
167
- Inject environment variables into the service
152
168
169
+
#### Alternative configuration with provider services
170
+
171
+
> [!TIP]
172
+
>
173
+
> This approach is deprecated. Use the [`models` top-level element](#basic-model-definition) instead.
174
+
175
+
You can also use the `provider` service type, which allows you to declare platform capabilities required by your application.
176
+
For AI models, you can use the `model` type to declare model dependencies.
177
+
178
+
To define a model provider:
179
+
180
+
```yaml
181
+
services:
182
+
chat:
183
+
image: my-chat-app
184
+
depends_on:
185
+
- ai_runner
186
+
187
+
ai_runner:
188
+
provider:
189
+
type: model
190
+
options:
191
+
model: ai/smollm2
192
+
context-size: 1024
193
+
runtime-flags: "--no-prefill-assistant"
194
+
```
195
+
196
+
153
197
### Cloud providers
154
198
155
199
The same Compose file can run on cloud providers that support Compose models:
Copy file name to clipboardExpand all lines: content/manuals/desktop/release-notes.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -240,7 +240,7 @@ For more frequently asked questions, see the [FAQs](/manuals/desktop/troubleshoo
240
240
- Docker Model Runner is now available on x86 Windows machines with NVIDIA GPUs.
241
241
- You can now [push models](/manuals/ai/model-runner.md#push-a-model-to-docker-hub) to Docker Hub with Docker Model Runner.
242
242
- Added support for Docker Model Runner's model management and chat interface in Docker Desktop for Mac and Windows (on hardware supporting Docker Model Runner). Users can now view, interact with, and manage local AI models through a new dedicated interface.
243
-
-[Docker Compose](/manuals/ai/compose/model-runner.md) and Testcontainers [Java](https://java.testcontainers.org/modules/docker_model_runner/) and [Go](https://golang.testcontainers.org/modules/dockermodelrunner/) now support Docker Model Runner.
243
+
-[Docker Compose](/manuals/ai/compose/models-and-compose.md) and Testcontainers [Java](https://java.testcontainers.org/modules/docker_model_runner/) and [Go](https://golang.testcontainers.org/modules/dockermodelrunner/) now support Docker Model Runner.
244
244
- Introducing Docker Desktop in the [Microsoft App Store](https://apps.microsoft.com/detail/xp8cbj40xlbwkx?hl=en-GB&gl=GB).
0 commit comments