Skip to content

Commit b1bd8af

Browse files
committed
Add comment to mention the possible timestamp issue
1 parent 1bab812 commit b1bd8af

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

src/libtorch.cc

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -660,7 +660,14 @@ ModelInstanceState::ModelInstanceState(
660660

661661
if (Kind() == TRITONSERVER_INSTANCEGROUPKIND_MODEL) {
662662
#ifdef TRITON_ENABLE_GPU
663-
// Create a CUDA stream for every availble device.
663+
// Since we cannot determine the exact devices used by the model, we create
664+
// a CUDA stream for every available device to ensure proper synchronization
665+
// of CUDA streams. This approach may have implications when a timestamp is
666+
// captured on a device that is not used by the model. Currently, this issue
667+
// is addressed by synchronizing the CUDA streams before recording
668+
// timestamps to prevent timestamp skewing. However, in the future, any
669+
// modifications to the CUDA stream synchronization logic should be handled
670+
// with caution.
664671
for (int i = 0; i < torch::cuda::device_count(); i++) {
665672
cudaStream_t stream;
666673
THROW_IF_BACKEND_INSTANCE_ERROR(

0 commit comments

Comments
 (0)