Auto merge of #142581 - Kobzol:bootstrap-std-method, r=jieyouxu

Enforce in bootstrap that build must have stage at least 1

This PR is a step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917. It's very hard or me to make self-contained changes to bootstrap at this moment, so this PR kind of does several things:

1) (first two commits) Try to reduce the usage of `Std::new` in bootstrap, and replace it with a `Builder::std` method (similar to `Builder::compiler`). This is mostly to remove executions of the `Std` step for stage 0, which doesn't make a lot of sense; I'd like to ideally have the invariant that when a step is invoked, it actually builds or does something. Eventually, I'd like for everything to go through `Builder::std`. (Note: I'm not totally married to this idea, if you don't like it, we can remove it from this PR. I mostly did it right now to remove stage 0 std steps from snapshot tests, which shouldn't be there, but we can also filter them out in a different way)
2) Make sure that when you pass `x build compiler`, only the `Assemble` root level step will be invoked, and not the `Rustc` step. Before, both were invoked, which actually ran `Rustc` twice, once with all `crates` filled, and once with no crates (but both actually represent the same situation). Since the `Rustc::make_run` step actually requests a compile that is one stage below it, this actually made `build compiler --stage 0` work, which we don't want to have anymore.
3) Enforce a bootstrap-global invariant that all `build` commands are always on stage `>=1`. If you try to `build` anything on stage 0, it will print a warning and exit bootstrap. This follows the intuition from the new staging rules after the stage redesign; artifacts that are "stage 0" come outside of bootstrap, and we can't really build something for which we don't have source (although we can still test it, but that's for another PR).

Now the logic for build should be quite simple. For pretty much everything except for `Std`, you first use the stage0 compiler to build stage 1. Then you can build a stage 2 <something> using the previously built stage 1 (and then you can continue to stage 3 etc.). And that's it. The default build stage for everything is 1 (modulo download-ci-rustc, but that's a separate can of worms).

The snapshot test infra isn't super nice at the moment, as one of next steps I want to create some simple Builder pattern that will allow us to configure the bootstrap invocations in a more "forward-compatible" way (e.g. now it's not possible to modify the config passed to `configure_with_args`).

There are some things not yet fully resolved for build stage 0:
1) Cargo is still a `ModeRustc` tool, even though it doesn't really have to be, it is buildable with the stage0 compiler
2) bootstrap tools (`opt-dist`, `build-manifest` etc.) are still called stage0 tools, and in the bootstrap output it says something like "stage 0 rustc builds stage 0 opt-dist". Which is a bit weird, but functionally there's no difference, it's just a slightly inconsistent output. We still haven't decided if we should make these tools ignore staging altogether (which is IMO the right choice) or if we want to allow building stage 1/2/3/... bootstrap tools.

r? `@jieyouxu`

try-job: x86_64-rust-for-linux

Author: bors

src/bootstrap/defaults/bootstrap.library.toml

@@ -1,7 +1,6 @@
 # These defaults are meant for contributors to the standard library and documentation.
 [build]
 bench-stage = 1
-build-stage = 1
 check-stage = 1
 test-stage = 1
 

src/bootstrap/src/core/build_steps/check.rs

@@ -1,6 +1,5 @@
 //! Implementation of compiling the compiler and standard library, in "check"-based modes.
 
-use crate::core::build_steps::compile;
 use crate::core::build_steps::compile::{
     add_to_sysroot, run_cargo, rustc_cargo, rustc_cargo_env, std_cargo, std_crates_for_run_make,
 };
@@ -87,7 +86,7 @@ impl Step for Std {
             }
 
             // Reuse the stage0 libstd
-            builder.ensure(compile::Std::new(compiler, target));
+            builder.std(compiler, target);
             return;
         }
 
@@ -221,8 +220,8 @@ impl Step for Rustc {
             // the sysroot for the compiler to find. Otherwise, we're going to
             // fail when building crates that need to generate code (e.g., build
             // scripts and their dependencies).
-            builder.ensure(crate::core::build_steps::compile::Std::new(compiler, compiler.host));
-            builder.ensure(crate::core::build_steps::compile::Std::new(compiler, target));
+            builder.std(compiler, compiler.host);
+            builder.std(compiler, target);
         } else {
             builder.ensure(Std::new(target));
         }

src/bootstrap/src/core/build_steps/clippy.rs

@@ -1,8 +1,8 @@
 //! Implementation of running clippy on the compiler, standard library and various tools.
 
+use super::check;
 use super::compile::{run_cargo, rustc_cargo, std_cargo};
 use super::tool::{SourceType, prepare_tool_cargo};
-use super::{check, compile};
 use crate::builder::{Builder, ShouldRun};
 use crate::core::build_steps::compile::std_crates_for_run_make;
 use crate::core::builder;
@@ -212,8 +212,8 @@ impl Step for Rustc {
                 // the sysroot for the compiler to find. Otherwise, we're going to
                 // fail when building crates that need to generate code (e.g., build
                 // scripts and their dependencies).
-                builder.ensure(compile::Std::new(compiler, compiler.host));
-                builder.ensure(compile::Std::new(compiler, target));
+                builder.std(compiler, compiler.host);
+                builder.std(compiler, target);
             } else {
                 builder.ensure(check::Std::new(target));
             }

src/bootstrap/src/core/build_steps/compile.rs

@@ -211,7 +211,7 @@ impl Step for Std {
         {
             trace!(?compiler_to_use, ?compiler, "compiler != compiler_to_use, uplifting library");
 
-            builder.ensure(Std::new(compiler_to_use, target));
+            builder.std(compiler_to_use, target);
             let msg = if compiler_to_use.host == target {
                 format!(
                     "Uplifting library (stage{} -> stage{})",
@@ -688,7 +688,7 @@ pub fn std_cargo(builder: &Builder<'_>, target: TargetSelection, stage: u32, car
 }
 
 #[derive(Debug, Clone, PartialEq, Eq, Hash)]
-struct StdLink {
+pub struct StdLink {
     pub compiler: Compiler,
     pub target_compiler: Compiler,
     pub target: TargetSelection,
@@ -699,7 +699,7 @@ struct StdLink {
 }
 
 impl StdLink {
-    fn from_std(std: Std, host_compiler: Compiler) -> Self {
+    pub fn from_std(std: Std, host_compiler: Compiler) -> Self {
         Self {
             compiler: host_compiler,
             target_compiler: std.compiler,
@@ -1020,6 +1020,12 @@ impl Step for Rustc {
     }
 
     fn make_run(run: RunConfig<'_>) {
+        // If only `compiler` was passed, do not run this step.
+        // Instead the `Assemble` step will take care of compiling Rustc.
+        if run.builder.paths == vec![PathBuf::from("compiler")] {
+            return;
+        }
+
         let crates = run.cargo_crates_in_set();
         run.builder.ensure(Rustc {
             build_compiler: run
@@ -1065,7 +1071,7 @@ impl Step for Rustc {
 
         // Build a standard library for `target` using the `build_compiler`.
         // This will be the standard library that the rustc which we build *links to*.
-        builder.ensure(Std::new(build_compiler, target));
+        builder.std(build_compiler, target);
 
         if builder.config.keep_stage.contains(&build_compiler.stage) {
             trace!(stage = build_compiler.stage, "`keep-stage` requested");
@@ -1106,10 +1112,10 @@ impl Step for Rustc {
         // build scripts and proc macros.
         // If we are not cross-compiling, the Std build above will be the same one as the one we
         // prepare here.
-        builder.ensure(Std::new(
+        builder.std(
             builder.compiler(self.build_compiler.stage, builder.config.host_target),
             builder.config.host_target,
-        ));
+        );
 
         let mut cargo = builder::Cargo::new(
             builder,
@@ -2077,15 +2083,15 @@ impl Step for Assemble {
         if builder.download_rustc() {
             trace!("`download-rustc` requested, reusing CI compiler for stage > 0");
 
-            builder.ensure(Std::new(target_compiler, target_compiler.host));
+            builder.std(target_compiler, target_compiler.host);
             let sysroot =
                 builder.ensure(Sysroot { compiler: target_compiler, force_recompile: false });
             // Ensure that `libLLVM.so` ends up in the newly created target directory,
             // so that tools using `rustc_private` can use it.
             dist::maybe_install_llvm_target(builder, target_compiler.host, &sysroot);
             // Lower stages use `ci-rustc-sysroot`, not stageN
             if target_compiler.stage == builder.top_stage {
-                builder.info(&format!("Creating a sysroot for stage{stage} compiler (use `rustup toolchain link 'name' build/host/stage{stage}`)", stage=target_compiler.stage));
+                builder.info(&format!("Creating a sysroot for stage{stage} compiler (use `rustup toolchain link 'name' build/host/stage{stage}`)", stage = target_compiler.stage));
             }
 
             let mut precompiled_compiler = target_compiler;

src/bootstrap/src/core/build_steps/dist.rs

@@ -23,7 +23,7 @@ use crate::core::build_steps::doc::DocumentationFormat;
 use crate::core::build_steps::tool::{self, Tool};
 use crate::core::build_steps::vendor::{VENDOR_DIR, Vendor};
 use crate::core::build_steps::{compile, llvm};
-use crate::core::builder::{Builder, Kind, RunConfig, ShouldRun, Step};
+use crate::core::builder::{Builder, Kind, RunConfig, ShouldRun, Step, StepMetadata};
 use crate::core::config::TargetSelection;
 use crate::utils::build_stamp::{self, BuildStamp};
 use crate::utils::channel::{self, Info};
@@ -84,6 +84,10 @@ impl Step for Docs {
         tarball.add_file(builder.src.join("src/doc/robots.txt"), dest, FileType::Regular);
         Some(tarball.generate())
     }
+
+    fn metadata(&self) -> Option<StepMetadata> {
+        Some(StepMetadata::dist("docs", self.host))
+    }
 }
 
 #[derive(Debug, PartialOrd, Ord, Clone, Hash, PartialEq, Eq)]
@@ -354,6 +358,10 @@ impl Step for Mingw {
 
         Some(tarball.generate())
     }
+
+    fn metadata(&self) -> Option<StepMetadata> {
+        Some(StepMetadata::dist("mingw", self.host))
+    }
 }
 
 #[derive(Debug, PartialOrd, Ord, Clone, Hash, PartialEq, Eq)]
@@ -540,6 +548,10 @@ impl Step for Rustc {
             }
         }
     }
+
+    fn metadata(&self) -> Option<StepMetadata> {
+        Some(StepMetadata::dist("rustc", self.compiler.host))
+    }
 }
 
 #[derive(Debug, Clone, Hash, PartialEq, Eq)]
@@ -711,7 +723,7 @@ impl Step for Std {
             return None;
         }
 
-        builder.ensure(compile::Std::new(compiler, target));
+        builder.std(compiler, target);
 
         let mut tarball = Tarball::new(builder, "rust-std", &target.triple);
         tarball.include_target_in_component_name(true);
@@ -723,6 +735,10 @@ impl Step for Std {
 
         Some(tarball.generate())
     }
+
+    fn metadata(&self) -> Option<StepMetadata> {
+        Some(StepMetadata::dist("std", self.target).built_by(self.compiler))
+    }
 }
 
 /// Tarball containing the compiler that gets downloaded and used by
@@ -1002,6 +1018,10 @@ impl Step for Src {
 
         tarball.generate()
     }
+
+    fn metadata(&self) -> Option<StepMetadata> {
+        Some(StepMetadata::dist("src", TargetSelection::default()))
+    }
 }
 
 #[derive(Debug, PartialOrd, Ord, Clone, Hash, PartialEq, Eq)]

src/bootstrap/src/core/build_steps/doc.rs

@@ -14,7 +14,8 @@ use std::{env, fs, mem};
 use crate::core::build_steps::compile;
 use crate::core::build_steps::tool::{self, SourceType, Tool, prepare_tool_cargo};
 use crate::core::builder::{
-    self, Alias, Builder, Compiler, Kind, RunConfig, ShouldRun, Step, crate_description,
+    self, Alias, Builder, Compiler, Kind, RunConfig, ShouldRun, Step, StepMetadata,
+    crate_description,
 };
 use crate::core::config::{Config, TargetSelection};
 use crate::helpers::{submodule_path_of, symlink_dir, t, up_to_date};
@@ -662,6 +663,10 @@ impl Step for Std {
             }
         }
     }
+
+    fn metadata(&self) -> Option<StepMetadata> {
+        Some(StepMetadata::doc("std", self.target).stage(self.stage))
+    }
 }
 
 /// Name of the crates that are visible to consumers of the standard library.
@@ -804,7 +809,7 @@ impl Step for Rustc {
         // Build the standard library, so that proc-macros can use it.
         // (Normally, only the metadata would be necessary, but proc-macros are special since they run at compile-time.)
         let compiler = builder.compiler(stage, builder.config.host_target);
-        builder.ensure(compile::Std::new(compiler, builder.config.host_target));
+        builder.std(compiler, builder.config.host_target);
 
         let _guard = builder.msg_sysroot_tool(
             Kind::Doc,
@@ -947,7 +952,7 @@ macro_rules! tool_doc {
                 t!(fs::create_dir_all(&out));
 
                 let compiler = builder.compiler(stage, builder.config.host_target);
-                builder.ensure(compile::Std::new(compiler, target));
+                builder.std(compiler, target);
 
                 if true $(&& $rustc_tool)? {
                     // Build rustc docs so that we generate relative links.
@@ -1195,7 +1200,7 @@ impl Step for RustcBook {
         let rustc = builder.rustc(self.compiler);
         // The tool runs `rustc` for extracting output examples, so it needs a
         // functional sysroot.
-        builder.ensure(compile::Std::new(self.compiler, self.target));
+        builder.std(self.compiler, self.target);
         let mut cmd = builder.tool_cmd(Tool::LintDocs);
         cmd.arg("--src");
         cmd.arg(builder.src.join("compiler"));
@@ -1272,7 +1277,7 @@ impl Step for Reference {
 
         // This is needed for generating links to the standard library using
         // the mdbook-spec plugin.
-        builder.ensure(compile::Std::new(self.compiler, builder.config.host_target));
+        builder.std(self.compiler, builder.config.host_target);
 
         // Run rustbook/mdbook to generate the HTML pages.
         builder.ensure(RustbookSrc {

src/bootstrap/src/core/build_steps/perf.rs

@@ -1,7 +1,7 @@
 use std::env::consts::EXE_EXTENSION;
 use std::fmt::{Display, Formatter};
 
-use crate::core::build_steps::compile::{Std, Sysroot};
+use crate::core::build_steps::compile::Sysroot;
 use crate::core::build_steps::tool::{RustcPerf, Rustdoc};
 use crate::core::builder::Builder;
 use crate::core::config::DebuginfoLevel;
@@ -152,7 +152,7 @@ Consider setting `rust.debuginfo-level = 1` in `bootstrap.toml`."#);
     }
 
     let compiler = builder.compiler(builder.top_stage, builder.config.host_target);
-    builder.ensure(Std::new(compiler, builder.config.host_target));
+    builder.std(compiler, builder.config.host_target);
 
     if let Some(opts) = args.cmd.shared_opts()
         && opts.profiles.contains(&Profile::Doc)