Skip to content

Commit 65e8cb4

Browse files
committed
Add comment to mention the possible timestamp issue
1 parent 67abc75 commit 65e8cb4

File tree

1 file changed

+8
-1
lines changed

1 file changed

+8
-1
lines changed

src/libtorch.cc

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -661,7 +661,14 @@ ModelInstanceState::ModelInstanceState(
661661

662662
if (Kind() == TRITONSERVER_INSTANCEGROUPKIND_MODEL) {
663663
#ifdef TRITON_ENABLE_GPU
664-
// Create a CUDA stream for every availble device.
664+
// Since we cannot determine the exact devices used by the model, we create
665+
// a CUDA stream for every available device to ensure proper synchronization
666+
// of CUDA streams. This approach may have implications when a timestamp is
667+
// captured on a device that is not used by the model. Currently, this issue
668+
// is addressed by synchronizing the CUDA streams before recording
669+
// timestamps to prevent timestamp skewing. However, in the future, any
670+
// modifications to the CUDA stream synchronization logic should be handled
671+
// with caution.
665672
for (int i = 0; i < torch::cuda::device_count(); i++) {
666673
cudaStream_t stream;
667674
THROW_IF_BACKEND_INSTANCE_ERROR(

0 commit comments

Comments
 (0)