You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: prototype_source/flight_recorder_tutorial.rst
+4-4Lines changed: 4 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -48,8 +48,6 @@ Enabling Flight Recorder
48
48
------------------------
49
49
There are two required environment variables to get the initial version of Flight Recorder working.
50
50
51
-
- ``TORCH_NCCL_DEBUG_INFO_TEMP_FILE``: Setting the path where the flight recorder will be dumped with file prefix. One file per
52
-
rank. The default value is ``/tmp/nccl_trace_rank_``.
53
51
- ``TORCH_NCCL_TRACE_BUFFER_SIZE = (0, N)``: Setting ``N`` to a positive number enables collection.
54
52
``N`` represents the number of entries that will be kept internally in a circular buffer.
55
53
We recommended to set this value at *2000*.
@@ -58,6 +56,8 @@ There are two required environment variables to get the initial version of Fligh
58
56
59
57
**Optional settings:**
60
58
59
+
- ``TORCH_NCCL_DEBUG_INFO_TEMP_FILE``: Setting the path where the flight recorder will be dumped with file prefix. One file per
60
+
rank. The default value is ``/tmp/nccl_trace_rank_``.
61
61
- ``TORCH_NCCL_TRACE_CPP_STACK = (true, false)``: Setting this to true enables C++ stack traces to be captured in Flight Recorder.
62
62
C++ stack traces can be useful in providing the exact code path from a PyTorch Python call down to the primitive
63
63
C++ implementation. Also see ``TORCH_SYMBOLIZE_MODE`` in additional settings.
@@ -74,7 +74,7 @@ Additional Settings
74
74
``fast`` is a new experimental mode that is shown to be much faster than the traditional ``addr2line``.
75
75
Use this setting in conjunction with ``TORCH_NCCL_TRACE_CPP_STACK`` to collect C++ traces in the Flight Recorder data.
76
76
- If you prefer not to have the flight recorder data dumped into the local disk but rather onto your own storage, you can define your own writer class.
77
-
This class should inherit from class ``::c10d::DebugInfoWriter`` and then register the new writer using ``::c10d::DebugInfoWriter::registerWriter``
77
+
This class should inherit from class ``::c10d::DebugInfoWriter`` `(code) <https://github.com/pytorch/pytorch/blob/release/2.5/torch/csrc/distributed/c10d/NCCLUtils.hpp#L237>`__ and then register the new writer using ``::c10d::DebugInfoWriter::registerWriter``
78
78
before we initiate PyTorch distributed.
79
79
80
80
Retrieving Flight Recorder Data via an API
@@ -189,7 +189,7 @@ command directly:
189
189
Currently, we support two modes for the analyzer script. The first mode allows the script to apply some heuristics to the parsed flight
190
190
recorder dumps to generate a report identifying potential culprits for the timeout. The second mode is simply outputs the raw dumps.
191
191
By default, the script prints flight recoder dumps for all ranks and all ``ProcessGroups``(PGs). This can be narrowed down to certain
192
-
ranks and PGs. An example command is:
192
+
ranks and PGs using the *--selected-ranks* argument. An example command is:
193
193
194
194
Caveat: tabulate module is needed, so you might need pip install it first.
0 commit comments