|
| 1 | +.. _torchinductor-provenance: |
| 2 | + |
| 3 | +TorchInductor and AOTInductor Provenance Tracking |
| 4 | +================================================= |
| 5 | + |
| 6 | +**Author:** `Shangdi Yu <https://github.com/yushangdi>`__ and `Rachel Guo <https://github.com/YUNQIUGUO>`__ |
| 7 | + |
| 8 | +This section describes how to use the provenance tracking feature for TorchInductor and AOTInductor in ``tlparse``. |
| 9 | +Some example screenshots of the provenance tracking tool are shown below. |
| 10 | +The tool visualizes the mapping between nodes in the input graph (panel 1), the post grad graph (panel 2), and the Inductor generated code (panel 3). |
| 11 | + |
| 12 | +.. raw:: html |
| 13 | + |
| 14 | + <style> |
| 15 | + .yellow {background-color:#FFFF00;} |
| 16 | + </style> |
| 17 | + |
| 18 | +.. role:: yellow |
| 19 | + |
| 20 | +The **bolded** lines represent nodes/kernels covered by the current provenance tracing functionality. |
| 21 | +We currently cover triton kernels, cpp kernels, and combo kernels. |
| 22 | +The :yellow:`yellow highlighting` shows the provenance of the nodes/kernels. |
| 23 | + |
| 24 | + |
| 25 | +Example screenshot of the provenance tracking tool for TorchInductor: |
| 26 | + .. image:: ../_static/img/inductor_provenance/provenance_jit_inductor.png |
| 27 | + |
| 28 | +Example screenshot of the provenance tracking tool for AOTInductor: |
| 29 | + .. image:: ../_static/img/inductor_provenance/provenance_aot_inductor.png |
| 30 | + |
| 31 | + |
| 32 | +Get Started |
| 33 | +~~~~~~~~~~~ |
| 34 | + |
| 35 | + |
| 36 | + |
| 37 | +- Use the following flags when running your program to produce necessary artifacts: ``TORCH_TRACE=~/my_trace_log_dir TORCH_LOGS="+inductor" TORCH_COMPILE_DEBUG=1`` |
| 38 | + |
| 39 | + - These flags will produce a log file in ``~/my_trace_log_dir``. The log file will be used by tlparse to generate the provenance tracking highlighter. |
| 40 | + |
| 41 | + |
| 42 | +- Then run ``tlparse`` on the log with ``--inductor-provenance`` flag. For example, ``tlparse log_file_name.log --inductor-provenance``. |
| 43 | + |
| 44 | + - See a demo video at https://github.com/pytorch/tlparse/pull/93. |
| 45 | + - Even if you don't add the --inductor-provenance flag, you should be able to see the mapping in json format in the ``inductor_provenance_tracking_node_mappings_<number>.json`` file in the ``index.html`` tlparse output. |
| 46 | + - Please run ``tlpare`` directly on the log file. It might not work if you run "tlparse parse <folder_name> --inductor-provenance". |
| 47 | + - The ``tlparse`` artifacts used by the provenance tracking highlighter are: inductor_pre_grad_graph.txt, inductor_post_grad_graph.txt, inductor_aot_wrapper_code.txt, inductor_output_code,txt inductor_provenance_tracking_node_mappings.json. |
| 48 | + |
| 49 | + |
| 50 | +After running ``tlparse <file_name> --inductor-provenance``, you should see an additional "Provenance Tracking" section in the tlparse output. Clicking into the link(s) to access the provenance tracking tool. |
| 51 | + |
| 52 | + .. image:: ../_static/img/inductor_provenance/index.png |
| 53 | + |
| 54 | + |
| 55 | +More about tlparse |
| 56 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 57 | + |
| 58 | +``tlparse`` is a tool written in Rust, and can be installed by ``cargo install tlparse``. |
| 59 | + |
| 60 | +- Link to the tlparse GitHub repo: https://github.com/pytorch/tlparse |
| 61 | +- Learn more about ``tlparse`` at :ref:`torch.compiler_troubleshooting` |
0 commit comments