Skip to content

Commit d03b5df

Browse files
committed
Add documentation for instance group kind of type 'KIND_MODEL'
1 parent ead0e23 commit d03b5df

File tree

1 file changed

+10
-0
lines changed

1 file changed

+10
-0
lines changed

README.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,16 @@ complex execution modes and dynamic shapes. If not specified, all are enabled by
206206

207207
`ENABLE_TENSOR_FUSER`
208208

209+
### Support
210+
211+
Starting from the 23.06 release, the PyTorch backend supports an instance group
212+
kind of type
213+
[`KIND_MODEL`](https://github.com/triton-inference-server/common/blob/r23.05/protobuf/model_config.proto#L174-L181)
214+
where the backend will not choose the GPU device for the model. Instead, it
215+
will respect the device(s) used in the model and use it as is when the type of
216+
the instance group is set to `KIND_MODEL` in the model config file. This is
217+
useful when the model is using multiple GPUs internally.
218+
209219
### Important Notes
210220

211221
* The execution of PyTorch model on GPU is asynchronous in nature. See

0 commit comments

Comments
 (0)