Skip to content

Add documentation for instance group kind of type 'KIND_MODEL' #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 13, 2023
Merged

Add documentation for instance group kind of type 'KIND_MODEL' #110

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6a28cdd
Add documentation for instance group kind of type 'KIND_MODEL'
krishung5 Jun 5, 2023
c6a79eb
Address comment
krishung5 Jun 6, 2023
df7d782
Address comment
krishung5 Jun 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,28 @@ complex execution modes and dynamic shapes. If not specified, all are enabled by

`ENABLE_TENSOR_FUSER`

### Support

#### Model Instance Group Kind

The PyTorch backend supports the following kinds of
[Model Instance Groups](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups)
where the input tensors are placed as follows:

* `KIND_GPU`: Inputs are prepared on the GPU device associated with the model
instance.

* `KIND_CPU`: Inputs are prepared on the CPU.

* `KIND_MODEL`: Inputs are prepared on the CPU. When loading the model, the
backend does not choose the GPU device for the model; instead, it respects the
device(s) specified in the model and uses them as they are during inference.
This is useful when the model internally utilizes multiple GPUs, as demonstrated
in this
[example model](https://github.com/triton-inference-server/server/blob/main/qa/L0_libtorch_instance_group_kind_model/gen_models.py).
If no device is specified in the model, the backend uses the first available
GPU device. This feature is available starting in the 23.06 release.

### Important Notes

* The execution of PyTorch model on GPU is asynchronous in nature. See
Expand Down