|
1 |
| -# OpSet definition |
| 1 | +# Definition of the Core ATen Operator Set |
2 | 2 |
|
3 |
| -TBA |
| 3 | +This page provides the description and background of the Core ATen Operator Set (opset). This page is recommended reading for those developing a new kernel library or delegate for ExecuTorch. It is also recommended that one is familiar with [`torch.export`](https://pytorch.org/docs/2.1/export.html) as a prerequisite; in particular, the concepts of torch FX graphs, operator decomposition, and functionalization. |
| 4 | + |
| 5 | +The list of operators that have been identified as a Core ATen operator can be found on the [IRs page of the PyTorch documentation website](https://pytorch.org/docs/stable/ir.html). |
| 6 | + |
| 7 | +## What is an Operator Set? |
| 8 | + |
| 9 | +`torch.export` performs a full graph capture on a given PyTorch program, producing a graph IR that describes the computation performed by the program. An operator (i.e. an operation performed on a Tensor) is the basic unit of computation in the graph, often corresponding to a unique node in the graph IR. The primary source of operators is the [ATen library](https://pytorch.org/cppdocs/#aten); outside of ATen operators, developers can also define their own operators (i.e. custom operators). |
| 10 | + |
| 11 | +An “ATen operator set” or “ATen opset” is the set of ATen operators that can be produced when a PyTorch program is captured. The ATen opset produced by a program export is primarily influenced by two processes performed by `torch.export`: |
| 12 | + |
| 13 | +* **Functionalization**: the graph IR must only contain functional operators; non-functional operators in a program must therefore be replaced with equivalent functional variants (i.e. variants that do not mutate or alias inputs) |
| 14 | +* **Operator Decomposition**: A decomposition table may be supplied to `torch.export`; ATen operators listed in the decomposition table will be replaced with equivalent sequences of other ATen operators |
| 15 | + * For instance, replacing `aten.hardsigmoid` with `aten.clamp(aten.clamp(self + 3, min=0), max=6) / 6` |
| 16 | + * A “default” decomposition table is used during `torch.export`, referred to as the Core ATen Decomposition Table |
| 17 | + |
| 18 | +## The Core ATen Operator Set |
| 19 | + |
| 20 | +If a PyTorch program is exported without any decomposition table, then the resultant graph IR will contain the “full” or “functional” ATen opset, which contains only functional ATen operators. |
| 21 | + |
| 22 | +If a PyTorch program is exported with the default decomposition table (i.e. the Core ATen Decomposition Table) then the resulting graph IR will contain the “Core ATen” opset. This opset will be smaller than the functional ATen opset, as many operators will be decomposed. This is referenced in the [`torch.export` documentation](https://pytorch.org/docs/2.1/export.html): |
| 23 | + |
| 24 | +> **Defined Operator Set:** The graph produced contains only a small defined [Core ATen IR opset](https://pytorch.org/docs/stable/ir.html) and registered custom operators. |
| 25 | +
|
| 26 | +The Core ATen Operator Set can be understood as the set of ATen operators that will not be decomposed by `torch.export` when using the default decomposition table. A core ATen operator will therefore be preserved in the resultant IR. The set of core operators is obtained by selecting representative functional variants of operators that cannot be easily re-expressed by other ATen operators through decomposition. |
| 27 | + |
| 28 | +The key motivation behind the Core ATen Operator Set is to reduce the number of operators that need to be handled by PyTorch backends and compilers once a model is exported. Not only are there a great number of operators defined in the ATen library, but new operators may be added, or the schema of existing operators may change. Without defining a core opset, backends built on top of the IR produced by `torch.export` would have to deal with both a large operator surface, as well as an opset that is constantly in flux. The Core ATen Operator Set addresses this by defining a much smaller, more manageable set of operators that was developed with stability in mind. |
| 29 | + |
| 30 | +## Development of the Core ATen Operator Set |
| 31 | + |
| 32 | +Although Executorch uses the Core ATen Operator Set, it is not specific to ExecuTorch. One of the primary design goals of the Core ATen Operator Set is that it should be as generic as possible; that the vast majority of use-cases will not want to decompose the operators contained within it. By extension, the decompositions contained in the Core ATen Decomposition Table should be useful to the vast majority of use-cases. |
| 33 | + |
| 34 | +Another key consideration was to keep the Operator Set as minimal as possible, but not at the expense of imposing decompositions that would have a profound negative impact on performance or developer experience. |
| 35 | + |
| 36 | +The Core ATen Operator Set was developed by reviewing a list of ATen operators created by surveying models in public GitHub repositories in addition to well-known open source models. The purpose of the surveying process was to obtain a reduced list of ATen operators that is a proxy of which ATen operators are used the most, so that the most commonly used operators may be reviewed first. |
| 37 | + |
| 38 | +The decision of whether each operator should be a core operator or be decomposed by the Core ATen Decomposition Table was determined by: |
| 39 | + |
| 40 | +1. Examining potential decompositions of the operators; the decomposition should be a relatively straightforward re-expression of the operator using other ATen operators. |
| 41 | + * The decomposition shouldn’t look like an outright implementation of the operator. |
| 42 | + * The decomposition shouldn't vary based on run-time characteristics of the input. |
| 43 | + * We also consider if decomposing the operator will impact the precision, numerical validity or memory layout of the output. |
| 44 | +2. Thinking about whether developers would want to preserve the operator in the graph for performance or other reasons. |
| 45 | + * For instance, perhaps an operator can be decomposed but it can map to a single hardware instruction on most platforms, in which case it would be preferable to promote it to a core operator. |
| 46 | + |
| 47 | +## Future Work |
| 48 | + |
| 49 | +Until every ATen operator has been reviewed and evaluated, the Core ATen Operator Set cannot be considered fully complete. However, this is a monumental task, and there is a long tail of operators that are not often used. This is why an approach was taken where models were surveyed to determine which ops were the most commonly used; this allows “higher impact” operators to be prioritized. |
| 50 | + |
| 51 | +Nonetheless, there are still many operators which have not been evaluated; that is, a decision has not yet been made whether it should be a core operator, or be decomposed during the export process. The plan is to continue evaluating additional operators as the need arises; the PyTorch community may propose additional core operators or additional core decompositions through opening a GitHub issue or by [commenting on this post on the PyTorch Forums](https://dev-discuss.pytorch.org/t/defining-the-core-aten-opset/1464). |
0 commit comments