Adding CMake and OSS Buck build details for ETDump (#779)

Summary:
Adding documentation on how to generate ETDump with CMake and Buck.

Pull Request resolved: #779

Reviewed By: Jack-Khuu

Differential Revision: D50127466

fbshipit-source-id: d698144514d61e5101a730bac72769882cbe1783

Author: tarun292

docs/source/index.rst

@@ -155,11 +155,11 @@ Topics in this section will help you get started with ExecuTorch.
    :hidden:
 
    sdk-overview
+   sdk-bundled-io
    sdk-etrecord
    sdk-etdump
    sdk-profiling
    sdk-inspector
-   sdk-bundled-io
    sdk-delegate-integration
 
 .. toctree::

docs/source/sdk-etdump.md

@@ -3,7 +3,7 @@
 ETDump (ExecuTorch Dump) is one of the core components of the ExecuTorch SDK experience. It is the mechanism through which all forms of profiling and debugging data is extracted from the runtime. Users can't parse ETDump directly; instead, they should pass it into the Inspector API, which deserializes the data, offering interfaces for flexible analysis and debugging.
 
 
-## Generating an ETDump:
+## Generating an ETDump
 
 Generating an ETDump is a relatively straight forward process. Users can follow the steps detailed below to integrate it into their application that uses ExecuTorch.
 
@@ -20,7 +20,7 @@ Result<Method> method =
       program->load_method(method_name, &memory_manager, &etdump_gen);
 ```
 
-3. ***Dump out the ETDump buffer*** - after the inference iterations have been completed, users can dump out the ETDump buffer. If users are on a device which has a file-system, they could just write it out to the fileystem. For more constrained embedded devices, users will have to extract the ETDump buffer from the device through a mechanism that best suits them (e.g. UART, JTAG etc.)
+3. ***Dump Out the ETDump Buffer*** - after the inference iterations have been completed, users can dump out the ETDump buffer. If users are on a device which has a file-system, they could just write it out to the fileystem. For more constrained embedded devices, users will have to extract the ETDump buffer from the device through a mechanism that best suits them (e.g. UART, JTAG etc.)
 
 ```C++
 etdump_result result = etdump_gen.get_etdump_data();
@@ -34,28 +34,31 @@ if (result.buf != nullptr && result.size > 0) {
   }
 ```
 
-4. ***Compile*** your binary with the flags that enable events to be traced and logged into ETDump inside the ExecuTorch runtime. The pre-processor flag that controls this is `ET_EVENT_TRACER_ENABLED`.
+4. ***Compile*** your binary with the `ET_EVENT_TRACER_ENABLED` flag to enable events to be traced and logged into ETDump inside the ExecuTorch runtime.
 
-    i). ***CMake***
+    i). ***Buck***
 
-    In CMake users can add this to their compile flags:
+    In Buck, users simply depend on the etdump target which is:
     ```
-    -DET_EVENT_TRACER_ENABLED
+    //executorch/sdk/etdump:etdump_flatcc
+    ```
+    When compiling their binary through Buck, users can pass in this buck config to enable the pre-processor flag. For example, when compiling `sdk_example_runner` to enable ETDump generation, users compile using the following command:
+    ```
+    buck2 build -c executorch.event_tracer_enabled=true examples/sdk/sdk_example_runner:sdk_example_runner
     ```
 
-    ii). ***Buck***
+    ii). ***CMake***
 
-    In Buck users can simply depend on the etdump target which is:
+    In CMake, users add this to their compile flags:
     ```
-    //executorch/sdk/etdump:etdump_flatcc
+    -DET_EVENT_TRACER_ENABLED
     ```
-    When compiling their binary through Buck, users can pass in this buck config which will enable this pre-processor flag:
+
+    This flag needs to be added to the ExecuTorch library and any operator library that the users are compiling into their binary. For reference, users can take a look at `examples/sdk/CMakeLists.txt`. The lines of of interest are:
     ```
-    buck build -c executorch.event_tracer_enabled=true your_binary_target
+    target_compile_options(executorch PUBLIC -DET_EVENT_TRACER_ENABLED)
+    target_compile_options(portable_ops_lib PUBLIC -DET_EVENT_TRACER_ENABLED)
     ```
+## Using an ETDump
 
-    TODO : Point to sample runner in examples here.
-
-## Using an ETDump:
-
-1. Pass this ETDump into the [Inspector API](./sdk-inspector.rst) for access to this data and to do post-run analysis on this data.
+1. Pass this ETDump into the [Inspector API](./sdk-inspector.rst) to access this data and  do post-run analysis.

docs/source/sdk-overview.md

@@ -8,10 +8,10 @@ All the components of the SDK have been designed from the ground up with deep in
 
 The ExecuTorch SDK supports the following features:
 
+- **BundledProgram** is a utility tool for exporting the model bundled with a sample set of (representative) inputs and expected outputs, so that during runtime users can validate that the actual output is in fact the same as the expected output.
 - **Profiling** models with operator level breakdown of performance stats
     - Linking back operator performance stats to source code and module hierarchy
     - Model loading and execution time
-- **BundledProgram** is a utility tool for exporting the model bundled with a sample set of (representative) inputs and expected outputs, so that during runtime users can validate that the actual output is in fact the same as the expected output.
 - **Delegate Integration** - Surfacing performance details from delegate backends
     - Link back delegate operator execution to the nodes they represent in the edge dialect graph (and subsequently linking back to source code and module hierarchy)
 - **Debugging** (Intermediate outputs and output quality analysis) - Coming soon

docs/source/sdk-profiling.md

@@ -1,11 +1,11 @@
 # Profiling Models in ExecuTorch
 
 Profiling in ExecuTorch gives users access to these runtime metrics:
-- Model loading time.
-- Operator level execution time
-- Delegate execution time
+- Model Load Time.
+- Operator Level Execution Time.
+- Delegate Execution Time.
   - If the delegate that the user is calling into has been integrated with the [SDK](./sdk-delegate-integration.md), then users will also be able to access delegated operator execution time.
-- End-to-end inference execution time
+- End-to-end Inference Execution Time.
 
 One uniqe aspect of ExecuTorch Profiling is the ability to link every runtime executed operator back to the exact line of python code from which this operator originated. This capability enables users to easily identify hotspots in their model, source them back to the exact line of Python code, and optimize if chosen to.
 
@@ -20,4 +20,4 @@ We provide access to all the profiling data via the Python [Inspector API](./sdk
     - Through the Inspector API, users can do a wide range of analysis varying from printing out performance details to doing more finer granular calculation on module level.
 
 
-Please refer to the [SDK tutorial](link to SDK tutorial) for a step-by-step walkthrough of the above process on a sample model.
+Please refer to the [SDK tutorial](./tutorials/sdk-integration-tutorial.rst) for a step-by-step walkthrough of the above process on a sample model.

docs/source/tutorials_source/sdk-integration-tutorial.py

@@ -102,13 +102,23 @@
 # ---------------
 #
 # Next step is to generate an ``ETDump``. ``ETDump`` contains runtime results
-# from executing the model. To generate, simply pass the ExecuTorch model
-# to the ``executor_runner``::
+# from executing the model. To generate, users have two options:
 #
-#   buck2 run executorch/examples/portable/scripts:export -- -m mv2
-#   buck2 run @mode/opt -c executorch.event_tracer_enabled=true executorch/sdk/runners:executor_runner -- --model_path mv2.pte
+# **Option 1:**
 #
-# TODO: Add Instructions for CMake, when landed
+# Use Buck::
+#
+#  python3 -m examples.sdk.scripts.export_bundled_program -m mv2
+#  buck2_oss run -c executorch.event_tracer_enabled=true examples/sdk/sdk_example_runner:sdk_example_runner -- --bundled_program_path mv2_bundled.bp
+#
+# **Option 2:**
+#
+# Use CMake::
+#   cd executorch
+#   rm -rf cmake-out && mkdir cmake-out && cd cmake-out && cmake -DBUCK2=buck2_oss -DEXECUTORCH_BUILD_SDK=1 -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=1 ..
+#   cd ..
+#   cmake --build cmake-out -j8 -t sdk_example_runner
+#   ./cmake-out/examples/sdk/sdk_example_runner --bundled_program_path mv2_bundled.bp
 
 ######################################################################
 # Creating an Inspector

examples/sdk/sdk_example_runner/sdk_example_runner.cpp

@@ -40,6 +40,7 @@ DEFINE_string(
     bundled_program_path,
     "model_bundled.bp",
     "Model serialized in flatbuffer format.");
+
 DEFINE_string(
     prof_result_path,
     "prof_result.bin",
@@ -56,6 +57,11 @@ DEFINE_string(
     "etdump.etdp",
     "If etdump generation is enabled an etdump will be written out to this path");
 
+DEFINE_bool(
+    output_verification,
+    false,
+    "Comapre the model output to the reference outputs present in the BundledProgram.");
+
 using namespace torch::executor;
 using torch::executor::util::FileDataLoader;
 
@@ -255,21 +261,23 @@ int main(int argc, char** argv) {
     free(result.buf);
   }
 
-  // Verify the outputs.
-  status = torch::executor::util::VerifyResultWithBundledExpectedOutput(
-      *method,
-      file_data->data(),
-      &bundled_input_allocator,
-      method_name,
-      FLAGS_testset_idx,
-      1e-5, // rtol
-      1e-8 // atol
-  );
-  ET_CHECK_MSG(
-      status == Error::Ok,
-      "Bundle verification failed with status 0x%" PRIx32,
-      status);
-  ET_LOG(Info, "Model verified successfully.");
+  if (FLAGS_output_verification) {
+    // Verify the outputs.
+    status = torch::executor::util::VerifyResultWithBundledExpectedOutput(
+        *method,
+        file_data->data(),
+        &bundled_input_allocator,
+        method_name,
+        FLAGS_testset_idx,
+        1e-4, // rtol
+        1e-8 // atol
+    );
+    ET_CHECK_MSG(
+        status == Error::Ok,
+        "Bundle verification failed with status 0x%" PRIx32,
+        status);
+    ET_LOG(Info, "Model verified successfully.");
+  }
 
   return 0;
 }