Skip to content

Commit 2190a79

Browse files
Apply more feedback
1 parent 7d67f88 commit 2190a79

File tree

1 file changed

+77
-44
lines changed

1 file changed

+77
-44
lines changed

posts/2021-05-07-caught-red-handed.md

Lines changed: 77 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,11 @@ team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
66
release: true
77
---
88

9-
The Rust team has prepared a new release, 1.52.1, working around a bug
10-
introduced in 1.52.0. We recommend all Rust users, including those currently
11-
using stable versions prior to 1.52.0 upgrade to 1.52.1.
9+
The Rust team has prepared a new release, 1.52.1, working around a bug in
10+
incremental compilation which was made into a compiler error in 1.52.0. We
11+
recommend all Rust users, including those currently using stable versions prior
12+
to 1.52.0, upgrade to 1.52.1 or disable incremental compilation. Guidance on how
13+
to do so is available below.
1214

1315
If you have a previous version of Rust installed via rustup, getting Rust
1416
1.52.1 is as easy as:
@@ -22,44 +24,51 @@ from the appropriate page on our website.
2224

2325
[install]: https://www.rust-lang.org/install.html
2426

25-
# What's in 1.52.1 stable
26-
27-
This point release contains a single change: it disables incremental
28-
compilation. Read on for more details as to why, and the next steps the Rust
29-
project is pursuing on this issue.
30-
3127
# Summary
3228

33-
The Rust teams are always excited to report on new features offered with each release. Sometimes, however, an important change that is not yet "fully baked" gets accidentally included in a release, and we need to issue a point release.
34-
35-
There was an instance of this in the most recent release, 1.52.0, which added a new bit of internal-consistency checking, called "incremental compilation hash verification" (abbreviated `verify-ich`). This check is also called an "unstable fingerprint" check, because the diagnostic it currently prints look [like this](https://github.com/rust-lang/rust/issues/84336):
36-
37-
```
38-
thread 'rustc' panicked at 'assertion failed: `(left == right)`
39-
left: `Some(Fingerprint(4565771098143344972, 7869445775526300234))`,
40-
right: `Some(Fingerprint(14934403843752251060, 623484215826468126))`: found unstable fingerprints for <massive text describing rustc internals elided>
41-
42-
error: internal compiler error: unexpected panic
29+
This release works around broken builds on 1.52.0, which are caused by newly
30+
added verification. The bugs this verification detects are present in all Rust
31+
versions, and can trigger miscompilations in incremental builds, so downgrading
32+
to a prior stable version is not a fix.
4333

44-
note: the compiler unexpectedly panicked. this is a bug.
45-
```
34+
Users are encouraged to upgrade to 1.52.1 or disable incremental in their local
35+
environment if on a prior versions: please see the [what you should do][part3]
36+
section for details on how to do so.
4637

47-
This internal-consistency check, as stated in the diagnostic, yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, though, the ICE is revealing a bug that 1.) predates the 1.52.0 release and 2.) could result in miscompilation if it had not been caught by `verify-ich`.
48-
49-
In other words: If you are seeing the above Internal Compiler Error, you may be tempted to respond by reverting to the 1.51 release. It is important to note that a downgrade is *not* the best response to this problem.
38+
Incremental compilation is off by default for release builds, so few
39+
production builds should be affected (only users who may have opted in).
5040

5141
This post is going to:
5242

43+
1. Explain [what the errors look like][part0],
5344
1. Explain [what the check does][part1], at a high level,
5445
2. Explain [how the check is presenting itself][part2] in the Rust 1.52.0 release,
5546
3. Tell you [what you should do][part3] if you see an unstable fingerprint on your project,
5647
4. Describe our plans for [how the Rust project will address][part4] the problems discussed here.
5748

49+
[part0]: #what-does-the-error-look-like
5850
[part1]: #what-are-fingerprints-why-are-we-checking-them
5951
[part2]: #how-does-this-show-up
6052
[part3]: #what-should-a-rust-programmer-do-in-response
6153
[part4]: #what-is-the-rust-project-going-to-do-to-fix-this
6254

55+
## What does the error look like?
56+
57+
The error message looks something like this, with the key piece being the "found
58+
unstable fingerprints" text.
59+
60+
```
61+
thread 'rustc' panicked at 'assertion failed: `(left == right)`
62+
left: `Some(Fingerprint(4565771098143344972, 7869445775526300234))`,
63+
right: `Some(Fingerprint(14934403843752251060, 623484215826468126))`: found unstable fingerprints for <massive text describing rustc internals elided>
64+
65+
error: internal compiler error: unexpected panic
66+
67+
note: the compiler unexpectedly panicked. this is a bug.
68+
```
69+
70+
This is the error caused by the internal consistency check, and as stated in the diagnostic, it yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, the ICE is revealing a bug that predates the 1.52.0 release and could result in miscompilation if it had not been caught by `verify-ich`.
71+
6372
## What are fingerprints? Why are we checking them?
6473

6574
The Rust compiler has support for "incremental compilation", which has been described in a [2016 blog post][]. When incremental compilation is turned on, the compiler breaks the input source into pieces, and tracks how those input pieces influence the final build product. Then, when the inputs change, it detects this and reuses artifacts from previous builds, striving to expend effort solely on building the parts that need to respond to the changes to the input source code.
@@ -70,14 +79,18 @@ Fingerprints are part of our architecture for detecting when inputs change. More
7079

7180
[rustc-dev-guide-fingerprints]: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#checking-query-results-for-changes-hashstable-and-fingerprints
7281

73-
The `verify-ich` check is a safeguard asserting internal inconsistency of the fingerprints.
74-
The compiler stores fingerprints for both cached and uncached values.
75-
Every time we compute an uncached value, we double-check that its newly computed fingerprint against the finger print stored in the cache.
76-
There are multiple ways that a fingerprint mismatch could arise, but they all represent bugs within the Rust compiler itself.
82+
The `verify-ich` check is a safeguard asserting internal consistency of the
83+
fingerprints. Sometimes the compiler is forced to rerun a query, and expects
84+
that the output is the same as from a prior incremental compilation session. The
85+
newly enabled verification checks that the value is indeed as expected, rather
86+
than assuming so. In some cases, due to bugs in the compiler's implementation,
87+
this was not actually the case.
7788

78-
## History of deployment
89+
## History
7990

80-
We [initially added][pr-45867] `verify-ich` as a tool to use when developing rustc itself, back in 2017; it was solely provided via an unstable `-Z` flag, only available to nightly and development builds.
91+
We [initially added][pr-45867] `verify-ich` as a tool to use when developing
92+
rustc itself, back in 2017. It was solely provided via an unstable `-Z` flag,
93+
only available to nightly and development builds.
8194

8295
More recently, in March, we encountered a [miscompilation][issue-82920] that led us to [turn on `verify-ich` by default][pr-83007]. The Rust compiler team decided it was better to catch fingerprint problems and abort compilation, rather than allow for potential miscompilations (and subsequent misbehavior) to sneak into Rust programmer's binaries.
8396

@@ -87,11 +100,13 @@ More recently, in March, we encountered a [miscompilation][issue-82920] that led
87100

88101
When we first turned on `verify-ich` by default, there was a steady stream of
89102
issues filed by users of the nightly (and beta) toolchains, and steady progress
90-
has been made on identifying fixes, a number of which have already landed. This
91-
last week, we noted incorrectly that the error would be shipping to stable
92-
next cycle in 1.53.0, and we started [making plans][issue-84970] to improve the
103+
has been made on identifying fixes, a number of which have already landed.
104+
105+
In the past week, we had started [making plans][issue-84970] to improve the
93106
user-experience, so that the diagnostic issued by the check would do a better
94-
job of telling the programmer what to do in response.
107+
job of telling the programmer what to do in response. Unfortunately, this was
108+
done under the assumption that the new verification would ship in 1.53, not
109+
1.52.
95110

96111
[issue-84970]: https://github.com/rust-lang/rust/issues/84970
97112

@@ -110,20 +125,27 @@ Essentially, for some crates, certain sequences of edit-compile cycles will caus
110125
Another recent example looks [like this](https://github.com/rust-lang/rust/issues/85039):
111126

112127
```
113-
thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(<massive text describing rustc internals elided>)', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5
128+
thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(<massive text describing rustc internals elided>)', /rustc/.../compiler/rustc_query_system/src/query/plumbing.rs:593:5
114129
```
115130

116131
They all arise from inconsistencies when comparing the incremental-compilation cache stored on disk against the values computed during a current `rustc` invocation, which means they all arise from using incremental compilation.
117132

118-
There are three ways that you may have incremental compilation turned on: You may have set the [environment variable][env-vars] `CARGO_INCREMENTAL=1`, or you may have enabled the `build.incremental` [setting in your Cargo.toml][cargo-toml], or you may be building with the `dev` or `test` [profiles][], which default to having incremental compilation enabled.
133+
There are several ways that you may have incremental compilation turned on:
134+
135+
1. You may be building with the `dev` or `test` [profiles][] which default to having incremental compilation enabled.
136+
2. You may have set the [environment variable][env-vars] `CARGO_INCREMENTAL=1`
137+
3. You may have enabled the `build.incremental` [setting in your Cargo config][cargo-config]
138+
4. You may have enabled the `incremental` [setting in your Cargo.toml][cargo-toml] for a given profile
119139

120140
[env-vars]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-reads
121-
[cargo-toml]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
141+
[cargo-config]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
142+
[cargo-toml]: https://doc.rust-lang.org/cargo/reference/profiles.html#incremental
122143
[profiles]: https://doc.rust-lang.org/cargo/reference/profiles.html
123144

124-
Incremental is disabled by default for the release profile, which should mean
125-
that unless you have enabled it yourself, these errors do not affect release
126-
builds, as they are only present in incremental builds.
145+
If your project has not adjusted the defaults, then when running `cargo build
146+
--release` or otherwise in the `release` profile configuration incremental is
147+
disabled on all Rust versions, and these issues should not affect your release
148+
builds.
127149

128150
## What should a Rust programmer do in response
129151

@@ -144,9 +166,18 @@ We do *not* recommend that users of 1.52.0 downgrade to an earlier version of Ru
144166
If a user is willing to deal with the incremental verification ICE's, and wishes
145167
to opt back into the 1.52.0 behavior, they may set `RUSTC_FORCE_INCREMENTAL` to
146168
`1` in their environment. The Rust compiler will then respect the
147-
`-Cincremental` option passed by Cargo, and things will work as before. Note
148-
that this flag does not enable incremental if it has not already been separately
149-
enabled (whether by Cargo or otherwise).
169+
`-Cincremental` option passed by Cargo, and things will work as before, though
170+
with the added verification. Note that this flag does not enable incremental if
171+
it has not already been separately enabled (whether by Cargo or otherwise).
172+
173+
If you are currently using a toolchain prior to 1.52.0, and wish to continue
174+
doing so, we recommend that you disable incremental compilation to avoid hitting
175+
silent miscompilations.
176+
177+
On all Rust builds since incremental has landed, it has been a major
178+
improvement to compile times for many users, and has only improved over time. We
179+
acknowledge that the workarounds presented here and recommendations are painful,
180+
and will be working hard to ensure the situation is as temporary as possible.
150181

151182
## What is the Rust project going to do to fix this
152183

@@ -156,7 +187,9 @@ We have issued 1.52.1 today which:
156187

157188
* Disables incremental compilation in the Rust compiler (unless asked for by a
158189
new environment variable, `RUSTC_FORCE_INCREMENTAL=1`).
159-
* Improves diagnostic output for the new verification if incremental compilation is enabled, indicating how to work around the bugs.
190+
* Improves diagnostic output for the new verification if incremental compilation is enabled,
191+
indicating how to work around the bugs by purging incremental state or
192+
disabling incremental.
160193

161194
This is intended to be a mitigation that helps the majority of Rust users have
162195
an upgrade path to a safe Rust compiler which does not have the risk of

0 commit comments

Comments
 (0)