[SYCL][E2E][Docs] Update test-mode documentation
#16875
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,10 @@ | |
| * [Marking tests as expected to fail](#marking-tests-as-expected-to-fail) | ||
| * [Marking tests as unsupported](#marking-tests-as-unsupported) | ||
| * [SYCL core header file](#sycl-core-header-file) | ||
| * [Compiling and executing tests on separate systems](#compiling-and-executing-tests-on-separate-systems) | ||
| * [Run only mode](#run-only-mode) | ||
| * [Build only mode](#build-only-mode) | ||
| * [Common Issues with separate build and run](#common-issues-with-separate-build-and-run) | ||
|
|
||
| ## Overview | ||
|
|
||
|
|
@@ -393,109 +397,117 @@ machines. By default the `test-mode` parameter is set to `full`, indicating that | |
| both stages will run. This parameter can be set to `build-only`, or `run-only`, | ||
| to only run the compilation stage, or the execution stage respectively. | ||
|
|
||
| #### Run only mode | ||
|
|
||
| Pass: `--param test-mode=run-only` | ||
|
|
||
| In this mode, tests will not be compiled, they will only run. To do this only | ||
| the `RUN:` lines that contain a "run" expansion will be executed (`%{run}`, | ||
| `%{run-unfiltered-devices}`, or `%{run-aux}`). Since tests are not compiled in | ||
| this mode, for any test to pass the test binaries should already be in the | ||
| `test_exec_root` directory, either by having ran `full` or `build-only` modes | ||
| previously on the system, or having transferred the test binaries into that | ||
| directory. To mark a test as expected to fail at run-time the `XFAIL` | ||
| expression should use runtime features, such as `run-mode` or device-specific | ||
| features. | ||
|
|
||
| `%{run-aux}` is an empty expansion and executes a line as is, without | ||
| expanding for each selected device and without using the `run_launcher`. | ||
|
|
||
| #### Build only mode | ||
|
|
||
| Pass: `--param test-mode=build-only` | ||
|
|
||
| This mode can be used to compile all test binaries that can be built on the | ||
| system. To do this `REQUIRES`, and `UNSUPPORTED` statements are handled | ||
| differently to accommodate for the fact that in `build-only` mode we do not | ||
| have any devices, and as a result no device-specific features. Instead of | ||
| considering these features as missing, we assign a third "unknown" value to | ||
| them. When evaluating an expression it will result in an unknown value if its | ||
| result could be changed by setting the unknown features to either true or | ||
| false. i.e., `false || unknown = unknown` but `true || unknown = true`. If an | ||
| expression's final value is unknown we consider it to have met the | ||
| requirements. The list of device-agnostic features that are not considered | ||
| unknown in `build-only` is found in the `E2EExpr.py`. | ||
|
|
||
| The triples to compile in this mode are set via the `sycl_build_targets` lit | ||
| parameter. Valid build targets are: `spir`,`nvidia`, `amd`, `native_cpu`. | ||
| These correspond to `spir64`, `nvptx64-nvidia-cuda`, `amdgcn-amd-amdhsa`, and | ||
| `native_cpu` triples respectively. Each build target should be separated with | ||
| a semicolon. This parameter is set to `all` by default, which enables | ||
| autodetection for the available build targets. A test can be marked as | ||
| requiring, or not supporting a particular triple via the `target-*` features. | ||
| Build targets are selected if they are able to pass the test's requirements | ||
| independent of the availability of other build targets. This is done to avoid | ||
| having to deal with a boolean satisfiability problem. For example, | ||
| `REQUIRES: target-spir && target-nvidia` will always be marked as unsupported | ||
| since it requires multiple targets simultaneously. Instead we can use | ||
| `any-target-is-*` features in this case, to check if a target is available in | ||
| the current lit configuration. | ||
|
|
||
| When executing the test in `build-only`, all `RUN:` lines that do not have a | ||
| run expansion will execute. | ||
|
|
||
| The `build-mode` feature is added when in this mode. | ||
|
|
||
| Some examples of `REQUIRES`/`UNSUPPORTED` in build-only: | ||
| If `linux` and `zstd` are available, and `sycl_build_targets` is set to | ||
| `spir;amd` | ||
| * `REQUIRES: linux && zstd`: This would be supported, this is treated normally | ||
| since both features are device-agnostic. | ||
| * `REQUIRES: linux && sg-32`: Despite the `sg-32` feature not being available, | ||
| this would be supported. Since the `sg-32` is a device-specific feature it is | ||
| evaluated as unknown in this expression. | ||
| * `REQUIRES: windows && sg-32`: This would be unsupported. `sg-32` would be | ||
| evaluated as unknown, and `windows` would evaluate as false. The fact that we | ||
| have an unknown value does not affect the end result, since the result of an | ||
| `&&` expression where one sub-expression is false is always false. | ||
| * `REQUIRES: windows || sg-32`: this would be supported. Here because the | ||
| result of the `||` expression would change if we considered `sg-32` to be | ||
| either true or false the overall expression evaluates to unknown. | ||
| * `UNSUPPORTED: !sg-32`: this would be supported. `sg-32` is evaluated as | ||
| unknown, and the negation of unknown is also unknown. | ||
| * `REQUIRES: target-spir`: This will be supported, and only the `spir64` | ||
| triple will be selected. | ||
| * `REQUIRES: target-spir && target-amd`: This will not be supported. When | ||
| checking if `target-spir` should be selected, `target-amd` will not be an | ||
| available feature, and vice versa when checking `target-amd`. | ||
| * `REQUIRES: target-spir && any-target-is-amd`: This will be supported, but | ||
| only the `spir64` triple will be selected. | ||
|
|
||
| #### Common Issues with separate build and run | ||
|
There was a problem hiding this comment. It is maybe unfair to ask, since this part of the doc is pre-existing. But we have this instruction There was a problem hiding this comment. Ok, I added an extra sentence with an example. Also if you wanted more real examples the changes in #15631, are exactly what this is describing. |
||
|
|
||
| A number of extra considerations need to be taken to write tests that are able | ||
| to be compiled and executed on separate machines. | ||
|
|
||
| * Tests that build and execute multiple binaries need to be written such that | ||
| the output of each compilation has a different name. This way no files are | ||
| overwritten, and all the necessary binaries can be transferred to the running | ||
| system. For example, instead of setting the output of all compilation steps to | ||
| a file named `%t.out`, we can number them `%t1.out`, `%t2.out`, and so on. | ||
|
|
||
| * Two scenarios need to be considered for tests that expectedly fail: | ||
| * Tests that are expected to fail on compilation, and thus also during | ||
| execution, need to be marked as `XFAIL` for a device-agnostic feature, or | ||
| with `XFAIL: *`. This is due to the fact that there are no devices in | ||
| `build-only` mode. For example if a test cannot compile for a triple, then it | ||
| should be marked as `XFAIL` for the corresponding build target feature, rather | ||
| than a backend feature. | ||
| * If the expected failure occurs during run-time we will need to mark the test | ||
| with `XFAIL` with an expression dependent on runtime features. If it is | ||
| expected to fail for any device at run-time we can use `XFAIL: run-mode`, | ||
| This must be done because otherwise the test would compile and pass on | ||
| `build-only` mode and be reported as an `XPASS`. | ||
|
|
||
| * To separate compilation and execution of tests, `RUN:` lines are filtered in | ||
| `build-only` and `run-only` mode based on the presence of "run" expansions. | ||
| * Any line that is meant to execute the test binary should be marked with | ||
| `%{run}` or `%{run-unfiltered-devices}` so that it is not ran in `build-only`, | ||
| and the `run_launcher` substitution is properly employed. | ||
| * The `%{run-aux}` expansion can be used if a `RUN:` line that does not | ||
| execute a test binary needs to be ran in `run-only`. | ||
|
|
||
| * CPU AOT compilation will target the ISA of the host CPU, thus compiling | ||
| these tests on a different system will lead to failures if the build system and | ||
| run system support different ISAs. To accommodate this, these compilations | ||
| should be delayed to the "run" stage by using the `%{run-aux}` markup. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
including
{run-aux}?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, the point of
%{run-aux}is to be able to switch over a line from running at the build stage, to running at the run stage.