|
| 1 | +DEPRECATED: This document is moving to //executorch/docs/source/sdk-bundled-io.md |
| 2 | + |
| 3 | +# Bundled Program |
| 4 | + |
| 5 | +## Introduction |
| 6 | +Bundled Program is a wrapper around the core ExecuTorch program designed to help users wrapping test cases and other related info with the models they deploy. Bundled Program is not necessarily a core part of the program and not needed for its execution but is more necessary for various other use-cases, especially for model correctness evaluation such as e2e testing during model bring-up etc. |
| 7 | + |
| 8 | +Overall procedure can be broken into two stages, and in each stage we are supporting: |
| 9 | +* **Emit stage**: Bundling test I/O cases as well as other useful info in key-value pairs along with the ExecuTorch program. |
| 10 | +* **Runtime stage**: Accessing, executing and verifying the bundled test cases during runtime. |
| 11 | + |
| 12 | +## Emit stage |
| 13 | + |
| 14 | + This stage mainly focuses on the creation of a BundledProgram, and dump it out to the disk as a flatbuffer file. Please refer to Bento notebook [N2744997](https://www.internalfb.com/intern/anp/view/?id=2744997) for details on how to create a bundled program. |
| 15 | + |
| 16 | +## Runtime Stage |
| 17 | +This stage mainly focuses on executing the model with the bundled inputs and and comparing the model's output with the bundled expected output. We provide multiple APIs to handle the key parts of it. |
| 18 | + |
| 19 | +### Get executorch program ptr from BundledProgram buffer |
| 20 | +We need the pointer to executorch program to do the execution. To unify the process of loading and executing BundledProgram and Program flatbuffer, we create an API: |
| 21 | + ```c++ |
| 22 | + |
| 23 | +/** |
| 24 | + * Finds the serialized ExecuTorch program data in the provided file data. |
| 25 | + * |
| 26 | + * The returned buffer is appropriate for constructing a |
| 27 | + * torch::executor::Program. |
| 28 | + * |
| 29 | + * Calling this is only necessary if the file could be a bundled program. If the |
| 30 | + * file will only contain an unwrapped ExecuTorch program, callers can construct |
| 31 | + * torch::executor::Program with file_data directly. |
| 32 | + * |
| 33 | + * @param[in] file_data The contents of an ExecuTorch program or bundled program |
| 34 | + * file. |
| 35 | + * @param[in] file_data_len The length of file_data, in bytes. |
| 36 | + * @param[out] out_program_data The serialized Program data, if found. |
| 37 | + * @param[out] out_program_data_len The length of out_program_data, in bytes. |
| 38 | + * |
| 39 | + * @returns Error::Ok if the program was found, and |
| 40 | + * out_program_data/out_program_data_len point to the data. Other values |
| 41 | + * on failure. |
| 42 | + */ |
| 43 | +Error GetProgramData( |
| 44 | + void* file_data, |
| 45 | + size_t file_data_len, |
| 46 | + const void** out_program_data, |
| 47 | + size_t* out_program_data_len); |
| 48 | +``` |
| 49 | +
|
| 50 | +Here's an example of how to use the GetProgramData API: |
| 51 | +```c++ |
| 52 | + std::shared_ptr<char> buff_ptr; |
| 53 | + size_t buff_len; |
| 54 | +
|
| 55 | +// FILE_PATH here can be either BundledProgram or Program flatbuffer file. |
| 56 | + Error status = torch::executor::util::read_file_content( |
| 57 | + FILE_PATH, &buff_ptr, &buff_len); |
| 58 | + ET_CHECK_MSG( |
| 59 | + status == Error::Ok, |
| 60 | + "read_file_content() failed with status 0x%" PRIx32, |
| 61 | + status); |
| 62 | +
|
| 63 | + uint32_t prof_tok = EXECUTORCH_BEGIN_PROF("de-serialize model"); |
| 64 | +
|
| 65 | + const void* program_ptr; |
| 66 | + size_t program_len; |
| 67 | + status = torch::executor::bundled_program::GetProgramData( |
| 68 | + buff_ptr.get(), buff_len, &program_ptr, &program_len); |
| 69 | + ET_CHECK_MSG( |
| 70 | + status == Error::Ok, |
| 71 | + "GetProgramData() failed with status 0x%" PRIx32, |
| 72 | + status); |
| 73 | +``` |
| 74 | + |
| 75 | +### Load bundled input to ExecutionPlan |
| 76 | +To execute the program on the bundled input, we need to load the bundled input into the ExecutionPlan. Here we provided an API called `torch::executor::bundled_program::LoadBundledInput`: |
| 77 | + |
| 78 | +```c++ |
| 79 | + |
| 80 | +/** |
| 81 | + * Load testset_idx-th bundled input of method_idx-th Method test in |
| 82 | + * bundled_program_ptr to given Method. |
| 83 | + * |
| 84 | + * @param[in] method The Method to verify. |
| 85 | + * @param[in] bundled_program_ptr The bundled program contains expected output. |
| 86 | + * @param[in] method_name The name of the Method being verified. |
| 87 | + * @param[in] testset_idx The index of input needs to be set into given Method. |
| 88 | + * |
| 89 | + * @returns Return Error::Ok if load successfully, or the error happens during |
| 90 | + * execution. |
| 91 | + */ |
| 92 | +__ET_NODISCARD Error LoadBundledInput( |
| 93 | + Method& method, |
| 94 | + serialized_bundled_program* bundled_program_ptr, |
| 95 | + MemoryAllocator* memory_allocator, |
| 96 | + const char* method_name, |
| 97 | + size_t testset_idx); |
| 98 | +``` |
| 99 | +
|
| 100 | +### Verify the plan's output. |
| 101 | +We call `torch::executor::bundled_program::VerifyResultWithBundledExpectedOutput` to verify the method's output with bundled expected outputs. Here's the details of this API: |
| 102 | +
|
| 103 | +```c++ |
| 104 | +/** |
| 105 | + * Compare the Method's output with testset_idx-th bundled expected |
| 106 | + * output in method_idx-th Method test. |
| 107 | + * |
| 108 | + * @param[in] method The Method to extract outputs from. |
| 109 | + * @param[in] bundled_program_ptr The bundled program contains expected output. |
| 110 | + * @param[in] method_name The name of the Method being verified. |
| 111 | + * @param[in] testset_idx The index of expected output needs to be compared. |
| 112 | + * @param[in] rtol Relative tolerance used for data comparsion. |
| 113 | + * @param[in] atol Absolute tolerance used for data comparsion. |
| 114 | + * |
| 115 | + * @returns Return Error::Ok if two outputs match, or the error happens during |
| 116 | + * execution. |
| 117 | + */ |
| 118 | +__ET_NODISCARD Error VerifyResultWithBundledExpectedOutput( |
| 119 | + Method& method, |
| 120 | + serialized_bundled_program* bundled_program_ptr, |
| 121 | + MemoryAllocator* memory_allocator, |
| 122 | + const char* method_name, |
| 123 | + size_t testset_idx, |
| 124 | + double rtol = 1e-5, |
| 125 | + double atol = 1e-8); |
| 126 | +
|
| 127 | +``` |
| 128 | + |
| 129 | +### Example |
| 130 | + |
| 131 | +Here we provide an example about how to run the bundled program step by step. Most of the code are borrowed from "fbcode/executorch/sdk/fb/runners/executor_runner.cpp" and please review that file if you need more info and context: |
| 132 | + |
| 133 | +```c++ |
| 134 | + // method_name is the name for the method we want to test |
| 135 | + // memory_manager is the executor::MemoryManager variable for executor memory allocation. |
| 136 | + // program is the executorch program. |
| 137 | + Result<Method> method = program->load_method(method_name, &memory_manager); |
| 138 | + EXECUTORCH_END_PROF(prof_tok); |
| 139 | + ET_CHECK_MSG( |
| 140 | + method.ok(), |
| 141 | + "load_method() failed with status 0x%" PRIx32, |
| 142 | + method.error()); |
| 143 | + |
| 144 | + // Load testset_idx-th input in the buffer to plan |
| 145 | + status = torch::executor::bundled_program::LoadBundledInput( |
| 146 | + *method, |
| 147 | + program_data.bundled_program_data(), |
| 148 | + &bundled_input_allocator, |
| 149 | + method_name, |
| 150 | + FLAGS_testset_idx); |
| 151 | + ET_CHECK_MSG( |
| 152 | + status == Error::Ok, |
| 153 | + "LoadBundledInput failed with status 0x%" PRIx32, |
| 154 | + status); |
| 155 | + |
| 156 | + // Execute the plan |
| 157 | + status = method->execute(); |
| 158 | + ET_CHECK_MSG( |
| 159 | + status == Error::Ok, |
| 160 | + "method->execute() failed with status 0x%" PRIx32, |
| 161 | + status); |
| 162 | + |
| 163 | + // Verify the result. |
| 164 | + status = torch::executor::bundled_program::VerifyResultWithBundledExpectedOutput( |
| 165 | + *method, |
| 166 | + program_data.bundled_program_data(), |
| 167 | + &bundled_input_allocator, |
| 168 | + method_name, |
| 169 | + FLAGS_testset_idx, |
| 170 | + FLAGS_rtol, |
| 171 | + FLAGS_atol); |
| 172 | + ET_CHECK_MSG( |
| 173 | + status == Error::Ok, |
| 174 | + "Bundle verification failed with status 0x%" PRIx32, |
| 175 | + status); |
| 176 | + |
| 177 | +``` |
0 commit comments