Benchmark categorization

collector/benchmarks/README.md

@@ -7,20 +7,31 @@ The suite changes over time. Sometimes the code for a benchmark is updated, in
 which case a small suffix will be added (starting with "-2", then "-3", and so
 on.)
 
-## Real programs that are important
+There are two categories of benchmarks, **Primary** and **Secondary**.
+
+## Primary
 
 These are real programs that are important in some way, and worth tracking.
+They mostly consist of real-world crates.
 
 - **cargo**: The Rust package manager. An important program in the Rust
   ecosystem.
 - **clap-rs**: A command line argument parser. A crate used by many Rust
   programs.
 - **cranelift-codegen**: The largest crate from a code generator. Used by
   Firefox.
+- **diesel**: A type safe SQL query builder. Utilizes the type system to
+  ensure a lot of invariants. Stresses anything related to resolving
+  trait bounds, by having a lot of trait impls for a large number of different
+  types.
+- **encoding**: Character encoding support. Contains some large tables.
 - **futures**: A futures implementation. Used by many Rust programs.
 - **helloworld**: A trivial program. Gives a lower bound on compile time.
+- **html5ever**: An HTML parser. Stresses macro parsing code significantly.
 - **hyper-2**: A fairly large crate. Utilizes async/await, and used by
   many Rust programs.
+- **inflate**: An old implementation of the DEFLATE algorithm. Stresses the
+  compiler in certain ways.
 - **piston-image**: A modular game engine. An interesting Rust program.
 - **regex**: A regular expression parser. Used by many Rust programs.
 - **ripgrep**: A line-oriented search tool. A widely-used utility.
@@ -33,40 +44,20 @@ These are real programs that are important in some way, and worth tracking.
   ecosystem.
 - **tokio-webpush-simple**: A simple web server built with tokio. Uses futures
   a lot.
-- **webrender**: A web renderer. Used by Firefox and Servo.
-- **webrender-wrench**: WebRender's test bench. An executable pulling in large
-  dependencies.
-
-## Real programs that stress the compiler
-
-These are real programs that are known to stress the compiler in interesting
-ways.
-
-- **diesel**: A type save SQL query builder. Utilizes the type system to
-  ensure a lot of invariants. Stresses anything related to resolving
-  trait bounds, by having a lot of trait impls for a large number of different
-  types.
-- **encoding**: Character encoding support. Contains some large tables.
-- **html5ever**: An HTML parser. Stresses macro parsing code significantly.
-- **inflate**: An old implementation of the DEFLATE algorithm. Stresses the
-  compiler in certain ways.
-- **keccak**: A cryptography algorithm. Contains a very high number of locals
-  and basic blocks.
 - **ucd**: A Unicode crate. Contains large statics that
   [stress](https://github.com/rust-lang/rust/issues/53643) the borrow checker's
   implementation of NLL.
 - **unicode_normalization**: Unicode character composition and decomposition
   utilities. Uses huge `match` statements that stress the compiler in unusual
   ways.
-- **wg-grammar**: A parser generator.
-  [Stresses](https://github.com/rust-lang/rust/issues/58178) the borrow
-  checker's implementation of NLL.
+- **webrender**: A web renderer. Used by Firefox and Servo.
+- **webrender-wrench**: WebRender's test bench. An executable pulling in large
+  dependencies.
 
-## Artificial stress tests
+## Secondary
 
-These are artificial programs that stress one particular aspect of the
-compiler. Some are entirely artificial, and some are extracted from real
-programs.
+These are either artificial programs or real crates that stress one particular aspect of the
+compiler in interesting ways.
 
 - **await-call-tree**: A tree of async fns that await each other, creating a
   large type composed of many repeated `impl Future` types. Such types caused
@@ -85,13 +76,17 @@ programs.
 - **deep-vector**: A test containing a single large vector of zeroes, which
   caused [poor performance](https://github.com/rust-lang/rust/issues/20936) in
   the past.
+- **derive**: A large amount of simple structs with a `#[derive]` attribute for common built-in traits such as Copy and Debug.
+- **externs**: A large amount of extern functions has caused [slowdowns in the past](https://github.com/rust-lang/rust/pull/78448).
 - **issue-46449**: A small program that caused [poor
   performance](https://github.com/rust-lang/rust/issues/46449) in the past.
 - **issue-58319**: A small program that caused [poor
   performance](https://github.com/rust-lang/rust/issues/58319) in the past.
 - **issue-88862**: A MCVE of a program that had a
   [severe performance regression](https://github.com/rust-lang/rust/issues/88862)
   when trying to normalize large opaque types with late-bound regions.
+- **keccak**: A cryptography algorithm. Contains a very high number of locals
+  and basic blocks.
 - **many-assoc-items**: Contains a struct with many associated items, which
   caused [quadratic behavior](https://github.com/rust-lang/rust/issues/68957)
   in the past.
@@ -103,7 +98,7 @@ programs.
 - **projection-caching**: A small program that causes extremely, deeply nested
   types which stress the trait system's projection cache. Removing that cache
   resulted in hours long compilations for some programs using futures,
-  actix-web and other libraries with similiarly nested type combinators.
+  actix-web and other libraries with similarly nested type combinators.
 - **regression-31157**: A small program that caused a [large performance
   regression](https://github.com/rust-lang/rust/issues/31157) from the past.
 - **token-stream-stress**: Constructs a long token stream much like the `quote`
@@ -122,5 +117,6 @@ programs.
 - **wf-projection-stress-65510**: A stress test which showcases [quadratic
   behavior](https://github.com/rust-lang/rust/issues/65510) (in the number of
   associated type bounds).
-- **externs**: A large amount of extern functions has caused [slowdowns in the past](https://github.com/rust-lang/rust/pull/78448).
-- **derive**: A large amount of simple structs with a `#[derive]` attribute for common built-in traits such as Copy and Debug.
+- **wg-grammar**: A parser generator.
+  [Stresses](https://github.com/rust-lang/rust/issues/58178) the borrow
+  checker's implementation of NLL.

collector/benchmarks/await-call-tree/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/cargo/perf-config.json

@@ -1,5 +1,6 @@
 {
     "cargo_opts": "--lib",
     "runs": 1,
-    "touch_file": "src/cargo/lib.rs"
+    "touch_file": "src/cargo/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/clap-rs/perf-config.json

@@ -0,0 +1,3 @@
+{
+  "category": "primary"
+}

collector/benchmarks/coercions/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/cranelift-codegen/perf-config.json

@@ -1,4 +1,5 @@
 {
     "cargo_toml": "cranelift-codegen/Cargo.toml",
-    "touch_file": "cranelift-codegen/src/lib.rs"
+    "touch_file": "cranelift-codegen/src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/ctfe-stress-4/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/deep-vector/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/deeply-nested-async/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/deeply-nested-closures/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/deeply-nested/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/derive/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/diesel/perf-config.json

@@ -1,4 +1,5 @@
 {
     "cargo_toml": "diesel/Cargo.toml",	
-    "touch_file": "diesel/src/lib.rs"
+    "touch_file": "diesel/src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/encoding/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/externs/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/futures/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "supports_stable": true
+    "supports_stable": true,
+    "category": "primary"
 }

collector/benchmarks/helloworld/perf-config.json

@@ -0,0 +1,3 @@
+{
+  "category": "primary"
+}

collector/benchmarks/html5ever/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/hyper-2/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/inflate/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "supports_stable": true
+    "supports_stable": true,
+    "category": "primary"
 }

collector/benchmarks/issue-46449/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/issue-58319/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/issue-88862/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/keccak/perf-config.json

@@ -0,0 +1,3 @@
+{
+  "category": "secondary"
+}

collector/benchmarks/many-assoc-items/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/match-stress-enum/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/match-stress-exhaustive_patterns/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": false,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "secondary"
 }

collector/benchmarks/piston-image/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "runs": 1
+    "runs": 1,
+    "category": "primary"
 }

collector/benchmarks/projection-caching/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/regex/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/regression-31157/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/ripgrep/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "touch_file": "src/main.rs"
+    "touch_file": "src/main.rs",
+    "category": "primary"
 }

collector/benchmarks/serde/perf-config.json

@@ -1,4 +1,5 @@
 {
     "cargo_toml": "serde/Cargo.toml",
-    "touch_file": "serde/src/lib.rs"
+    "touch_file": "serde/src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/stm32f4/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "cargo_opts": "--features=stm32f405"
+    "cargo_opts": "--features=stm32f405",
+    "category": "primary"
 }

collector/benchmarks/style-servo/perf-config.json

@@ -4,5 +4,6 @@
     "cargo_toml": "components/style/Cargo.toml",
     "runs": 1,
     "supports_stable": true,
-    "touch_file": "components/style/lib.rs"
+    "touch_file": "components/style/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/syn/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/token-stream-stress/perf-config.json

@@ -1,4 +1,5 @@
 {
     "cargo_opts": "--bin token-stream-stress-bin",
-    "touch_file": "src/main.rs"
+    "touch_file": "src/main.rs",
+    "category": "secondary"
 }

collector/benchmarks/tokio-webpush-simple/perf-config.json

@@ -1,4 +1,5 @@
 {
     "supports_stable": true,
-    "touch_file": "src/main.rs"
+    "touch_file": "src/main.rs",
+    "category": "primary"
 }

collector/benchmarks/tuple-stress/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "runs": 1
+    "runs": 1,
+    "category": "secondary"
 }

collector/benchmarks/ucd/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "runs": 1
+    "runs": 1,
+    "category": "primary"
 }

collector/benchmarks/unicode_normalization/perf-config.json

@@ -0,0 +1,3 @@
+{
+  "category": "primary"
+}

collector/benchmarks/unify-linearly/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/unused-warnings/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/webrender-wrench/perf-config.json

@@ -1,5 +1,6 @@
 {
     "cargo_toml": "wrench/Cargo.toml",
     "runs": 1,
-    "touch_file": "wrench/src/main.rs"
+    "touch_file": "wrench/src/main.rs",
+    "category": "primary"
 }

collector/benchmarks/webrender/perf-config.json

@@ -1,5 +1,6 @@
 {
     "cargo_toml": "webrender/Cargo.toml",
     "runs": 1,
-    "touch_file": "webrender/src/lib.rs"
+    "touch_file": "webrender/src/lib.rs",
+    "category": "primary"
 }

collector/benchmarks/wf-projection-stress-65510/perf-config.json

@@ -0,0 +1,3 @@
+{
+    "category": "secondary"
+}

collector/benchmarks/wg-grammar/perf-config.json

@@ -1,3 +1,4 @@
 {
-    "touch_file": "src/lib.rs"
+    "touch_file": "src/lib.rs",
+    "category": "secondary"
 }