Skip to content

Commit 5bfd160

Browse files
committed
adding as a prototype
1 parent 5c60fe0 commit 5bfd160

File tree

4 files changed

+61
-0
lines changed

4 files changed

+61
-0
lines changed
874 KB
Loading
Loading
Loading
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
.. _torchinductor-provenance:
2+
3+
TorchInductor and AOTInductor Provenance Tracking
4+
=================================================
5+
6+
**Author:** `Shangdi Yu <https://github.com/yushangdi>`__ and `Rachel Guo <https://github.com/YUNQIUGUO>`__
7+
8+
This section describes how to use the provenance tracking feature for TorchInductor and AOTInductor in ``tlparse``.
9+
Some example screenshots of the provenance tracking tool are shown below.
10+
The tool visualizes the mapping between nodes in the input graph (panel 1), the post grad graph (panel 2), and the Inductor generated code (panel 3).
11+
12+
.. raw:: html
13+
14+
<style>
15+
.yellow {background-color:#FFFF00;}
16+
</style>
17+
18+
.. role:: yellow
19+
20+
The **bolded** lines represent nodes/kernels covered by the current provenance tracing functionality.
21+
We currently cover triton kernels, cpp kernels, and combo kernels.
22+
The :yellow:`yellow highlighting` shows the provenance of the nodes/kernels.
23+
24+
25+
Example screenshot of the provenance tracking tool for TorchInductor:
26+
.. image:: ../_static/img/inductor_provenance/provenance_jit_inductor.png
27+
28+
Example screenshot of the provenance tracking tool for AOTInductor:
29+
.. image:: ../_static/img/inductor_provenance/provenance_aot_inductor.png
30+
31+
32+
Get Started
33+
~~~~~~~~~~~
34+
35+
36+
37+
- Use the following flags when running your program to produce necessary artifacts: ``TORCH_TRACE=~/my_trace_log_dir TORCH_LOGS="+inductor" TORCH_COMPILE_DEBUG=1``
38+
39+
- These flags will produce a log file in ``~/my_trace_log_dir``. The log file will be used by tlparse to generate the provenance tracking highlighter.
40+
41+
42+
- Then run ``tlparse`` on the log with ``--inductor-provenance`` flag. For example, ``tlparse log_file_name.log --inductor-provenance``.
43+
44+
- See a demo video at https://github.com/pytorch/tlparse/pull/93.
45+
- Even if you don't add the --inductor-provenance flag, you should be able to see the mapping in json format in the ``inductor_provenance_tracking_node_mappings_<number>.json`` file in the ``index.html`` tlparse output.
46+
- Please run ``tlpare`` directly on the log file. It might not work if you run "tlparse parse <folder_name> --inductor-provenance".
47+
- The ``tlparse`` artifacts used by the provenance tracking highlighter are: inductor_pre_grad_graph.txt, inductor_post_grad_graph.txt, inductor_aot_wrapper_code.txt, inductor_output_code,txt inductor_provenance_tracking_node_mappings.json.
48+
49+
50+
After running ``tlparse <file_name> --inductor-provenance``, you should see an additional "Provenance Tracking" section in the tlparse output. Clicking into the link(s) to access the provenance tracking tool.
51+
52+
.. image:: ../_static/img/inductor_provenance/index.png
53+
54+
55+
More about tlparse
56+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
57+
58+
``tlparse`` is a tool written in Rust, and can be installed by ``cargo install tlparse``.
59+
60+
- Link to the tlparse GitHub repo: https://github.com/pytorch/tlparse
61+
- Learn more about ``tlparse`` at :ref:`torch.compiler_troubleshooting`

0 commit comments

Comments
 (0)