Update glossary to add the notion of runtime benchmarks

Author: Kobzol

database/schema.md

@@ -65,6 +65,8 @@ id          name        date        type
 
 A "collection" of benchmarks tied only differing by the statistic collected.
 
+This corresponds to a [`test result`](../docs/glossary.md#testing).
+
 This is a way to collect statistics together signifying that they belong to the same logical benchmark run.
 
 Currently, the collection also marks the git sha of the currently running collector binary.
@@ -125,6 +127,8 @@ of a crate, profile, scenario and the metric being collected.
 * cache (aka `scenario`): describes how much of the incremental cache is full. An empty incremental cache means that the compiler must do a full build.
 * statistic (aka `metric`): the type of metric being collected
 
+This corresponds to a [`statistic description`](../docs/glossary.md).
+
 There is a separate table for this collection to avoid duplicating crates, prfiles, scenarios etc.
 many times in the `pstat` table.
 

docs/glossary.md

@@ -2,30 +2,40 @@
 
 The following is a glossary of domain specific terminology. Although benchmarks are a seemingly simple domain, they have a surprising amount of complexity. It is therefore useful to ensure that the vocabulary used to describe the domain is consistent and precise to avoid confusion. 
 
-## Basic terms
+## Common terms
+
+* **metric**: a name of a quantifiable metric being measured (e.g., instruction count).
+* **artifact**: a specific rustc binary labeled by some identifier tag (usually a commit sha or some sort of human readable id like "1.51.0" or "test").
+
+## Compile-time benchmark terms
 
 * **benchmark**: the source of a crate which will be used to benchmark rustc. For example, ["hello world"](https://github.com/rust-lang/rustc-perf/tree/master/collector/compile-benchmarks/helloworld).
 * **profile**: a [cargo profile](https://doc.rust-lang.org/cargo/reference/profiles.html). Note: the database uses "opt" whereas cargo uses "release". 
 * **scenario**: The scenario under which a user is compiling their code. Currently, this is the incremental cache state and an optional change in the source since last compilation (e.g., full incremental cache and a `println!` statement is added).  
-* **metric**: a name of a quantifiable metric being measured (e.g., instruction count)
-* **artifact**: a specific rustc binary labeled by some identifier tag (usually a commit sha or some sort of human readable id like "1.51.0" or "test")
 * **category**: a high-level group of benchmarks. Currently, there are three categories, primary (mostly real-world crates), secondary (mostly stress tests), and stable (old real-world crates, only used for the dashboard).
 
-## Benchmarks
+### Types of compile-time benchmarks
+
+* **stress test benchmark**: a benchmark that is specifically designed to stress a certain part of the compiler. For example, [projection-caching](https://github.com/rust-lang/rustc-perf/tree/master/collector/compile-benchmarks/projection-caching) stresses the compiler's projection caching mechanisms. Corresponds to the `secondary` category.
+* **real world benchmark**: a benchmark based on a real world crate. These are typically copied as-is from crates.io. For example, [serde](https://github.com/rust-lang/rustc-perf/tree/master/collector/compile-benchmarks/serde-1.0.136) is a popular crate and the benchmark has not been altered from a release of serde on crates.io. Corresponds to the `primary` or `stable` categories.
+
+## Runtime benchmark terms
 
-* **stress test benchmark**: a benchmark that is specifically designed to stress a certain part of the compiler. For example, [projection-caching](https://github.com/rust-lang/rustc-perf/tree/master/collector/compile-benchmarks/projection-caching) stresses the compiler's projection caching mechanisms.
-* **real world benchmark**: a benchmark based on a real world crate. These are typically copied as-is from crates.io. For example, [serde](https://github.com/rust-lang/rustc-perf/tree/master/collector/compile-benchmarks/serde-1.0.136) is a popular crate and the benchmark has not been altered from a release of serde on crates.io. 
+* **benchmark**: a function compiled by rustc, which function will be benchmarked.
+* **benchmark group**: a crate that contains a set of runtime benchmarks.
 
-## Testing 
+## Testing
 
-* **test case**: a combination of a benchmark, a profile, and a scenario.
-* **test**: the act of running an artifact under a test case. Each test result is composed of many iterations.
-* **test iteration**: a single iteration that makes up a test. Note: we currently normally run 2 test iterations for each test. 
-* **test result**: the result of the collection of all statistics from running a test. Currently, the minimum value of a statistic from all the test iterations is used.
-* **statistic**: a single value of a metric in a test result
+* **test case**: a combination of parameters that describe the measurement of a single (compile-time or runtime) benchmark - a single `test`
+    - For compile-time benchmarks, it is a combination of a benchmark, a profile, and a scenario.
+    - For runtime benchmarks, it is currently only the benchmark name.
+* **test**: the act of running an artifact under a test case. Each test is composed of many iterations.
+* **test iteration**: a single iteration that makes up a test. Note: we currently normally run 3 test iterations for each test. 
+* **test result**: the result of the collection of all statistics from running a test. Currently, the minimum value of a statistic from all the test iterations is used for analysis calculations and the website.
+* **statistic**: a single measured value of a metric in a test result
 * **statistic description**: the combination of a metric and a test case which describes a statistic.
 * **statistic series**: statistics for the same statistic description over time.
-* **run**: a collection of test results for all currently available test cases run on a given artifact. 
+* **run**: a set of tests for all currently available test cases measured on a given artifact. 
 
 ## Analysis