|
1 |
| -# `compiletest` |
| 1 | +# Using `compiletest` commands to control test execution |
2 | 2 |
|
3 | 3 | ## Introduction
|
4 | 4 |
|
5 |
| -`compiletest` is the main test harness of the Rust test suite. It allows |
| 5 | +`compiletest` is the main test harness of the Rust test suite. It allows |
6 | 6 | test authors to organize large numbers of tests (the Rust compiler has many
|
7 | 7 | thousands), efficient test execution (parallel execution is supported), and
|
8 | 8 | allows the test author to configure behavior and expected results of both
|
9 | 9 | individual and groups of tests.
|
10 | 10 |
|
11 |
| -`compiletest` tests may check test code for success, for failure or in some |
12 |
| -cases, even failure to compile. Tests are typically organized as a Rust source |
13 |
| -file with annotations in comments before and/or within the test code, which |
14 |
| -serve to direct `compiletest` on if or how to run the test, what behavior to |
15 |
| -expect, and more. If you are unfamiliar with the compiler testing framework, |
16 |
| -see [this chapter](./tests/intro.html) for additional background. |
| 11 | +`compiletest` tests may check test code for success, for runtime failure, or for |
| 12 | +compile-time failure. Tests are typically organized as a Rust source file with |
| 13 | +annotations in comments before and/or within the test code, which serve to |
| 14 | +direct `compiletest` on if or how to run the test, what behavior to expect, |
| 15 | +and more. If you are unfamiliar with the compiler testing framework, |
| 16 | +see [this chapter](./tests/intro.md) for additional background. |
17 | 17 |
|
18 | 18 | The tests themselves are typically (but not always) organized into
|
19 |
| -"suites" – for example, `run-fail`, |
20 |
| -a folder holding tests that should compile successfully, |
21 |
| -but return a failure (non-zero status), `compile-fail`, a folder holding tests |
22 |
| -that should fail to compile, and many more. The various suites are defined in |
23 |
| -[src/tools/compiletest/src/common.rs][common] in the `pub enum Mode` |
24 |
| -declaration. And a very good introduction to the different suites of compiler |
25 |
| -tests along with details about them can be found in [Adding new |
26 |
| -tests](./tests/adding.html). |
| 19 | +"suites" – for example, `run-fail`, a folder holding tests that should compile |
| 20 | +successfully, but return a failure (non-zero status) at runtime, `compile-fail`, |
| 21 | +a folder holding tests that should fail to compile, and many more. The various |
| 22 | +suites are defined in [`src/tools/compiletest/src/common.rs`] in the |
| 23 | +`pub enum Mode` declaration. And a good introduction to the different |
| 24 | +suites of compiler tests along with details about them can be found in |
| 25 | +[Adding new tests](./tests/adding.md). |
27 | 26 |
|
28 | 27 | ## Adding a new test file
|
29 | 28 |
|
30 | 29 | Briefly, simply create your new test in the appropriate location under
|
31 |
| -[src/test][test]. No registration of test files is necessary as `compiletest` |
32 |
| -will scan the [src/test][test] subfolder recursively, and will execute any Rust |
33 |
| -source files it finds as tests. See [`Adding new tests`](./tests/adding.html) |
34 |
| -for a complete guide on how to adding new tests. |
| 30 | +[`src/test`]. No registration of test files is necessary as `compiletest` |
| 31 | +will scan the [`src/test`] subfolder recursively, and will execute any |
| 32 | +Rust source files it finds as tests. |
| 33 | +See [Adding new tests](./tests/adding.md) for a complete guide on how to add |
| 34 | +new tests. |
35 | 35 |
|
36 | 36 | ## Header Commands
|
37 | 37 |
|
38 | 38 | Source file annotations which appear in comments near the top of the source
|
39 |
| -file *before* any test code are known as header commands. These commands can |
| 39 | +file *before* any test code are known as header commands. These commands can |
40 | 40 | instruct `compiletest` to ignore this test, set expectations on whether it is
|
41 | 41 | expected to succeed at compiling, or what the test's return code is expected to
|
42 |
| -be. Header commands (and their inline counterparts, Error Info commands) are |
43 |
| -described more fully |
44 |
| -[here](./tests/adding.html#header-commands-configuring-rustc). |
| 42 | +be. Header commands and inline `//~ ERROR` commands are described more fully |
| 43 | +[here](./tests/adding.md#header-commands-configuring-rustc). |
45 | 44 |
|
46 | 45 | ### Adding a new header command
|
47 | 46 |
|
48 | 47 | Header commands are defined in the `TestProps` struct in
|
49 |
| -[src/tools/compiletest/src/header.rs][header]. At a high level, there are |
| 48 | +[`src/tools/compiletest/src/header.rs`]. At a high level, there are |
50 | 49 | dozens of test properties defined here, all set to default values in the
|
51 | 50 | `TestProp` struct's `impl` block. Any test can override this default value by
|
52 | 51 | specifying the property in question as header command as a comment (`//`) in
|
@@ -77,48 +76,47 @@ fn main() -> Result<(), Box<Error>> {
|
77 | 76 | #### Adding a new header command property
|
78 | 77 |
|
79 | 78 | One would add a new header command if there is a need to define some test
|
80 |
| -property or behavior on an individual, test-by-test basis. A header command |
| 79 | +property or behavior on an individual, test-by-test basis. A header command |
81 | 80 | property serves as the header command's backing store (holds the command's
|
82 | 81 | current value) at runtime.
|
83 | 82 |
|
84 | 83 | To add a new header command property:
|
85 |
| - 1. Look for the `pub struct TestProps` declaration in |
86 |
| - [src/tools/compiletest/src/header.rs][header] and add the new public |
87 |
| - property to the end of the declaration. |
88 |
| - 2. Look for the `impl TestProps` implementation block immediately following |
89 |
| - the struct declaration and initialize the new property to its default |
90 |
| - value. |
| 84 | + |
| 85 | + 1. Look for the `pub struct TestProps` declaration in |
| 86 | + [`src/tools/compiletest/src/header.rs`] and add the new public property to |
| 87 | + the end of the declaration. |
| 88 | + 2. Look for the `impl TestProps` implementation block immediately following |
| 89 | + the struct declaration and initialize the new property to its default |
| 90 | + value. |
91 | 91 |
|
92 | 92 | #### Adding a new header command parser
|
93 | 93 |
|
94 | 94 | When `compiletest` encounters a test file, it parses the file a line at a time
|
95 | 95 | by calling every parser defined in the `Config` struct's implementation block,
|
96 |
| -also in [src/tools/compiletest/src/header.rs][header] (note the `Config` |
97 |
| -struct's declaration block is found in |
98 |
| -[src/tools/compiletest/src/common.rs][common]. `TestProps`'s `load_from()` |
99 |
| -method will try passing the current line of text to each parser, which, in turn |
100 |
| -typically checks to see if the line begins with a particular commented (`//`) |
101 |
| -header command such as `// must-compile-successfully` or `// failure-status`. |
102 |
| -Whitespace after the comment marker is optional. |
| 96 | +also in [`src/tools/compiletest/src/header.rs`][] (note that the `Config` |
| 97 | +struct's declaration block is found in [`src/tools/compiletest/src/common.rs`]). |
| 98 | +`TestProps`'s `load_from()` method will try passing the current line of text to |
| 99 | +each parser, which, in turn typically checks to see if the line begins with a |
| 100 | +particular commented (`//`) header command such as `// must-compile-successfully` |
| 101 | +or `// failure-status`. Whitespace after the comment marker is optional. |
103 | 102 |
|
104 | 103 | Parsers will override a given header command property's default value merely by
|
105 | 104 | being specified in the test file as a header command or by having a parameter
|
106 | 105 | value specified in the test file, depending on the header command.
|
107 | 106 |
|
108 | 107 | Parsers defined in `impl Config` are typically named `parse_<header_command>`
|
109 | 108 | (note kebab-case `<header-command>` transformed to snake-case
|
110 |
| -`<header_command>`). `impl Config` also defines several 'low-level' parsers |
| 109 | +`<header_command>`). `impl Config` also defines several 'low-level' parsers |
111 | 110 | which make it simple to parse common patterns like simple presence or not
|
112 | 111 | (`parse_name_directive()`), header-command:parameter(s)
|
113 | 112 | (`parse_name_value_directive()`), optional parsing only if a particular `cfg`
|
114 |
| -attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers |
| 113 | +attribute is defined (`has_cfg_prefix()`) and many more. The low-level parsers |
115 | 114 | are found near the end of the `impl Config` block; be sure to look through them
|
116 | 115 | and their associated parsers immediately above to see how they are used to
|
117 | 116 | avoid writing additional parsing code unnecessarily.
|
118 | 117 |
|
119 | 118 | As a concrete example, here is the implementation for the
|
120 |
| -`parse_failure_status()` parser, in |
121 |
| -[src/tools/compiletest/src/header.rs][header]: |
| 119 | +`parse_failure_status()` parser, in [`src/tools/compiletest/src/header.rs`]: |
122 | 120 |
|
123 | 121 | ```diff
|
124 | 122 | @@ -232,6 +232,7 @@ pub struct TestProps {
|
@@ -163,18 +161,17 @@ As a concrete example, here is the implementation for the
|
163 | 161 | ## Implementing the behavior change
|
164 | 162 |
|
165 | 163 | When a test invokes a particular header command, it is expected that some
|
166 |
| -behavior will change as a result. What behavior, obviously, will depend on the |
167 |
| -purpose of the header command. In the case of `failure-status`, the behavior |
| 164 | +behavior will change as a result. What behavior, obviously, will depend on the |
| 165 | +purpose of the header command. In the case of `failure-status`, the behavior |
168 | 166 | that changes is that `compiletest` expects the failure code defined by the
|
169 | 167 | header command invoked in the test, rather than the default value.
|
170 | 168 |
|
171 | 169 | Although specific to `failure-status` (as every header command will have a
|
172 | 170 | different implementation in order to invoke behavior change) perhaps it is
|
173 | 171 | helpful to see the behavior change implementation of one case, simply as an
|
174 |
| -example. To implement `failure-status`, the `check_correct_failure_status()` |
| 172 | +example. To implement `failure-status`, the `check_correct_failure_status()` |
175 | 173 | function found in the `TestCx` implementation block, located in
|
176 |
| -[src/tools/compiletest/src/runtest.rs](https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs), |
177 |
| -was modified as per below: |
| 174 | +[`src/tools/compiletest/src/runtest.rs`], was modified as per below: |
178 | 175 |
|
179 | 176 | ```diff
|
180 | 177 | @@ -295,11 +295,14 @@ impl<'test> TestCx<'test> {
|
@@ -213,13 +210,14 @@ was modified as per below:
|
213 | 210 | }
|
214 | 211 | ```
|
215 | 212 | Note the use of `self.props.failure_status` to access the header command
|
216 |
| -property. In tests which do not specify the failure status header command, |
| 213 | +property. In tests which do not specify the failure status header command, |
217 | 214 | `self.props.failure_status` will evaluate to the default value of 101 at the
|
218 |
| -time of this writing. But for a test which specifies a header command of, for |
| 215 | +time of this writing. But for a test which specifies a header command of, for |
219 | 216 | example, `// failure-status: 1`, `self.props.failure_status` will evaluate to
|
220 | 217 | 1, as `parse_failure_status()` will have overridden the `TestProps` default
|
221 | 218 | value, for that test specifically.
|
222 | 219 |
|
223 |
| -[test]: https://github.com/rust-lang/rust/tree/master/src/test |
224 |
| -[header]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs |
225 |
| -[common]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs |
| 220 | +[`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test |
| 221 | +[`src/tools/compiletest/src/header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs |
| 222 | +[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs |
| 223 | +[`src/tools/compiletest/src/runtest.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs |
0 commit comments