Add documentation for instance group kind of type 'KIND_MODEL' #110
Conversation
README.md
Outdated
| [`KIND_MODEL`](https://github.com/triton-inference-server/common/blob/r23.05/protobuf/model_config.proto#L174-L181). | ||
| In this case, the inputs reside on the CPU. The backend does not choose the GPU | ||
| device for the model; instead, it respects the device(s) specified in the model | ||
| and uses them as they are when the instance group kind is set to `KIND_MODEL` |
There was a problem hiding this comment.
Nit: I think this section could be shortened, which could add clarity. It feels like a bit of a run on. I think something like the previous sections could be good, like "Inputs are located on the devices specified by the model. This feature is available starting in the 23.06 release."
Also, a couple of questions:
- Is there a default case when the model does not specify a device? Does this fail? If not, wondering if the default value is worth adding here.
- Is there a test model that we could link to? It's okay if not. If we do already have it, linking to that as an example could be useful. It would show users how to specify devices in a PyTorch model.
There was a problem hiding this comment.
Modified the section, let me know if there is anything unclear!
Is there a default case when the model does not specify a device? Does this fail? If not, wondering if the default value is worth adding here.
By default the first available GPU device will be used. Added this to the documentation.
Is there a test model that we could link to? It's okay if not. If we do already have it, linking to that as an example could be useful. It would show users how to specify devices in a PyTorch model.
Added the link to the testing model. (The link will be valid once the server PR merged)
* Add documentation for instance group kind of type 'KIND_MODEL' * Address comment * Address comment
* Add documentation for instance group kind of type 'KIND_MODEL' * Address comment * Address comment
The documentation for #107.