Skip to content

Commit 011d42b

Browse files
Olivia-liufacebook-github-bot
authored andcommitted
New URL for the Model Debugging page (#5817)
Summary: Pull Request resolved: #5817 This diff is to rename the "sdk-debugging" documentation page to "model-debugging". Old URL: https://pytorch.org/executorch/main/sdk-debugging.html New URL: https://pytorch.org/executorch/main/model-debugging.html Design doc: https://docs.google.com/document/d/1l6DYTq9Kq6VrPohruRFP-qScZDj01W_g4zlKyvqKGF4/edit?usp=sharing <- This is where the new file name come from. Reviewed By: dbort Differential Revision: D63738950 fbshipit-source-id: 9060881ec03132d772966a8695cd2d82252ca819
1 parent 9ff3351 commit 011d42b

File tree

3 files changed

+84
-81
lines changed

3 files changed

+84
-81
lines changed

docs/source/index.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,7 +205,7 @@ Topics in this section will help you get started with ExecuTorch.
205205
etrecord
206206
sdk-etdump
207207
sdk-profiling
208-
sdk-debugging
208+
model-debugging
209209
sdk-inspector
210210
memory-planning-inspection
211211
delegate-debugging

docs/source/model-debugging.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Debugging Models in ExecuTorch
2+
3+
With the ExecuTorch Developer Tools, users can debug their models for numerical inaccurcies and extract model outputs from their device to do quality analysis (such as Signal-to-Noise, Mean square error etc.).
4+
5+
Currently, ExecuTorch supports the following debugging flows:
6+
- Extraction of model level outputs via ETDump.
7+
- Extraction of intermediate outputs (outside of delegates) via ETDump:
8+
- Linking of these intermediate outputs back to the eager model python code.
9+
10+
11+
## Steps to debug a model in ExecuTorch
12+
13+
### Runtime
14+
For a real example reflecting the steps below, please refer to [example_runner.cpp](https://github.com/pytorch/executorch/blob/main/examples/devtools/example_runner/example_runner.cpp).
15+
16+
1. [Optional] Generate an [ETRecord](./etrecord.rst) while exporting your model. When provided, this enables users to link profiling information back to the eager model source code (with stack traces and module hierarchy).
17+
2. Integrate [ETDump generation](./sdk-etdump.md) into the runtime and set the debugging level by configuring the `ETDumpGen` object. Then, provide an additional buffer to which intermediate outputs and program outputs will be written. Currently we support two levels of debugging:
18+
- Program level outputs
19+
```C++
20+
Span<uint8_t> buffer((uint8_t*)debug_buffer, debug_buffer_size);
21+
etdump_gen.set_debug_buffer(buffer);
22+
etdump_gen.set_event_tracer_debug_level(
23+
EventTracerDebugLogLevel::kProgramOutputs);
24+
```
25+
26+
- Intermediate outputs of executed (non-delegated) operations (will include the program level outputs too)
27+
```C++
28+
Span<uint8_t> buffer((uint8_t*)debug_buffer, debug_buffer_size);
29+
etdump_gen.set_debug_buffer(buffer);
30+
etdump_gen.set_event_tracer_debug_level(
31+
EventTracerDebugLogLevel::kIntermediateOutputs);
32+
```
33+
3. Build the runtime with the pre-processor flag that enables tracking of debug events. Instructions are in the [ETDump documentation](./sdk-etdump.md).
34+
4. Run your model and dump out the ETDump buffer as described [here](./sdk-etdump.md). (Do so similarly for the debug buffer if configured above)
35+
36+
37+
### Accessing the debug outputs post run using the Inspector API's
38+
Once a model has been run, using the generated ETDump and debug buffers, users can leverage the [Inspector API's](./sdk-inspector.rst) to inspect these debug outputs.
39+
40+
```python
41+
from executorch.devtools import Inspector
42+
43+
# Create an Inspector instance with etdump and the debug buffer.
44+
inspector = Inspector(etdump_path=etdump_path,
45+
buffer_path = buffer_path,
46+
# etrecord is optional, if provided it'll link back
47+
# the runtime events to the eager model python source code.
48+
etrecord = etrecord_path)
49+
50+
# Accessing program outputs is as simple as this:
51+
for event_block in inspector.event_blocks:
52+
if event_block.name == "Execute":
53+
print(event_blocks.run_output)
54+
55+
# Accessing intermediate outputs from each event (an event here is essentially an instruction that executed in the runtime).
56+
for event_block in inspector.event_blocks:
57+
if event_block.name == "Execute":
58+
for event in event_block.events:
59+
print(event.debug_data)
60+
# If an ETRecord was provided by the user during Inspector initialization, users
61+
# can print the stacktraces and module hierarchy of these events.
62+
print(event.stack_traces)
63+
print(event.module_hierarchy)
64+
```
65+
66+
We've also provided a simple set of utilities that let users perform quality analysis of their model outputs with respect to a set of reference outputs (possibly from the eager mode model).
67+
68+
69+
```python
70+
from executorch.devtools.inspector import compare_results
71+
72+
# Run a simple quality analysis between the model outputs sourced from the
73+
# runtime and a set of reference outputs.
74+
#
75+
# Setting plot to True will result in the quality metrics being graphed
76+
# and displayed (when run from a notebook) and will be written out to the
77+
# filesystem. A dictionary will always be returned which will contain the
78+
# results.
79+
for event_block in inspector.event_blocks:
80+
if event_block.name == "Execute":
81+
compare_results(event_blocks.run_output, ref_outputs, plot = True)
82+
```

docs/source/sdk-debugging.md

Lines changed: 1 addition & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,82 +1,3 @@
11
# Debugging Models in ExecuTorch
22

3-
With the ExecuTorch Developer Tools, users can debug their models for numerical inaccurcies and extract model outputs from their device to do quality analysis (such as Signal-to-Noise, Mean square error etc.).
4-
5-
Currently, ExecuTorch supports the following debugging flows:
6-
- Extraction of model level outputs via ETDump.
7-
- Extraction of intermediate outputs (outside of delegates) via ETDump:
8-
- Linking of these intermediate outputs back to the eager model python code.
9-
10-
11-
## Steps to debug a model in ExecuTorch
12-
13-
### Runtime
14-
For a real example reflecting the steps below, please refer to [example_runner.cpp](https://github.com/pytorch/executorch/blob/main/examples/devtools/example_runner/example_runner.cpp).
15-
16-
1. [Optional] Generate an [ETRecord](./etrecord.rst) while exporting your model. When provided, this enables users to link profiling information back to the eager model source code (with stack traces and module hierarchy).
17-
2. Integrate [ETDump generation](./sdk-etdump.md) into the runtime and set the debugging level by configuring the `ETDumpGen` object. Then, provide an additional buffer to which intermediate outputs and program outputs will be written. Currently we support two levels of debugging:
18-
- Program level outputs
19-
```C++
20-
Span<uint8_t> buffer((uint8_t*)debug_buffer, debug_buffer_size);
21-
etdump_gen.set_debug_buffer(buffer);
22-
etdump_gen.set_event_tracer_debug_level(
23-
EventTracerDebugLogLevel::kProgramOutputs);
24-
```
25-
26-
- Intermediate outputs of executed (non-delegated) operations (will include the program level outputs too)
27-
```C++
28-
Span<uint8_t> buffer((uint8_t*)debug_buffer, debug_buffer_size);
29-
etdump_gen.set_debug_buffer(buffer);
30-
etdump_gen.set_event_tracer_debug_level(
31-
EventTracerDebugLogLevel::kIntermediateOutputs);
32-
```
33-
3. Build the runtime with the pre-processor flag that enables tracking of debug events. Instructions are in the [ETDump documentation](./sdk-etdump.md).
34-
4. Run your model and dump out the ETDump buffer as described [here](./sdk-etdump.md). (Do so similarly for the debug buffer if configured above)
35-
36-
37-
### Accessing the debug outputs post run using the Inspector API's
38-
Once a model has been run, using the generated ETDump and debug buffers, users can leverage the [Inspector API's](./sdk-inspector.rst) to inspect these debug outputs.
39-
40-
```python
41-
from executorch.devtools import Inspector
42-
43-
# Create an Inspector instance with etdump and the debug buffer.
44-
inspector = Inspector(etdump_path=etdump_path,
45-
buffer_path = buffer_path,
46-
# etrecord is optional, if provided it'll link back
47-
# the runtime events to the eager model python source code.
48-
etrecord = etrecord_path)
49-
50-
# Accessing program outputs is as simple as this:
51-
for event_block in inspector.event_blocks:
52-
if event_block.name == "Execute":
53-
print(event_blocks.run_output)
54-
55-
# Accessing intermediate outputs from each event (an event here is essentially an instruction that executed in the runtime).
56-
for event_block in inspector.event_blocks:
57-
if event_block.name == "Execute":
58-
for event in event_block.events:
59-
print(event.debug_data)
60-
# If an ETRecord was provided by the user during Inspector initialization, users
61-
# can print the stacktraces and module hierarchy of these events.
62-
print(event.stack_traces)
63-
print(event.module_hierarchy)
64-
```
65-
66-
We've also provided a simple set of utilities that let users perform quality analysis of their model outputs with respect to a set of reference outputs (possibly from the eager mode model).
67-
68-
69-
```python
70-
from executorch.devtools.inspector import compare_results
71-
72-
# Run a simple quality analysis between the model outputs sourced from the
73-
# runtime and a set of reference outputs.
74-
#
75-
# Setting plot to True will result in the quality metrics being graphed
76-
# and displayed (when run from a notebook) and will be written out to the
77-
# filesystem. A dictionary will always be returned which will contain the
78-
# results.
79-
for event_block in inspector.event_blocks:
80-
if event_block.name == "Execute":
81-
compare_results(event_blocks.run_output, ref_outputs, plot = True)
82-
```
3+
Please update your link to <https://pytorch.org/executorch/main/model-debugging.html>. This URL will be deleted after v0.4.0.

0 commit comments

Comments
 (0)