Skip to content

Commit 683f9f7

Browse files
committed
Update FR tutorial to include file path, writer and print usage
1 parent df285cf commit 683f9f7

File tree

1 file changed

+26
-2
lines changed

1 file changed

+26
-2
lines changed

prototype_source/flight_recorder_tutorial.rst

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,8 @@ Enabling Flight Recorder
4848
------------------------
4949
There are two required environment variables to get the initial version of Flight Recorder working.
5050

51+
- ``TORCH_NCCL_DEBUG_INFO_TEMP_FILE``: Setting the path where the flight recorder will be dumped with file prefix. The dump is one
52+
file per rank. The default value is ``/tmp/nccl_trace_rank_``.
5153
- ``TORCH_NCCL_TRACE_BUFFER_SIZE = (0, N)``: Setting ``N`` to a positive number enables collection.
5254
``N`` represents the number of entries that will be kept internally in a circular buffer.
5355
We recommended to set this value at *2000*.
@@ -71,6 +73,9 @@ Additional Settings
7173

7274
``fast`` is a new experimental mode that is shown to be much faster than the traditional ``addr2line``.
7375
Use this setting in conjunction with ``TORCH_NCCL_TRACE_CPP_STACK`` to collect C++ traces in the Flight Recorder data.
76+
- If you don't want the flight recorder to be dumped into the local disk but instead onto your own storage, users can define your own writer class
77+
which inherits from class ``::c10d::DebugInfoWriter`` and then register the new writer using ``::c10d::DebugInfoWriter::registerWriter`` before
78+
we initiate c10d distributed.
7479

7580
Retrieving Flight Recorder Data via an API
7681
------------------------------------------
@@ -169,9 +174,28 @@ To run the convenience script, follow these steps:
169174

170175
2. To run the script, use this command:
171176

172-
.. code:: python
177+
.. code:: shell
178+
179+
python fr_trace.py <dump dir containing trace files> [-o <output file>]
180+
181+
Or if you install PyTorch nightly build or build from scratch (with ``USE_DISTRIBUTED=1``), you can directly use the following command:
182+
183+
.. code:: shell
184+
185+
torchfrtrace <dump dir containing trace files> [-o <output file>]
186+
187+
188+
For now, we support two modes for the analyzer script, one is to let the script to apply some heuristics to the parsed flight recorder
189+
dumps to generate a report on potential culprit for the timeout/hang; the other one is to just print out raw dumps. For the latter, by
190+
default the script prints for all ranks and all ProcessGroups(PGs), and this can be narrowed down to certain ranks and PGs. Example
191+
command is:
192+
193+
Caveat: tabulate module is needed, so you might need pip install it first.
194+
195+
.. code:: shell
173196
174-
python fr_trace.py -d <dump dir containing trace files> [-o <output file>]
197+
python fr_trace.py <dump dir containing trace files> -j [--selected-ranks i j k ...]
198+
torchfrtrace <dump dir containing trace files> -j [--selected-ranks i j k ...]
175199
176200
Conclusion
177201
----------

0 commit comments

Comments
 (0)