Skip to content

Clarify the overall result in the GitHub comment. #1415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 22, 2022

Conversation

nnethercote
Copy link
Contributor

Specifics:

  • Adds a big "Overall result" heading at the top of the comment, which
    includes "ACTION NEEDED" when there are regressions.
  • Moves the next steps and "if you disagree..." text to the top of the
    comment, just below the overall result, before the tables of
    measurements. This means all the instructions that people need to read
    are in one place.
  • Removes the short summaries in front of each table, because (a) they just
    duplicates what the table says, and (b) saying "regressions found" for
    cycles and max-rss is a bit misleading because it doesn't contribute
    to the overall conclusion. Replaces them with a sentence indicating if
    the measurements contributed to the overall result and why.
  • Changes Direction::join to a non-static method, it's nicer that way.

@nnethercote nnethercote requested a review from rylev August 19, 2022 02:20
@nnethercote
Copy link
Contributor Author

Here is a before and after comparison of a real comment.


Finished benchmarking commit (b998821e4c51c44a9ebee395c91323c374236bbb): comparison url.

Instruction count

  • Primary benchmarks: ❌ relevant regressions found
  • Secondary benchmarks: ❌ relevant regression found
mean1 max count2
Regressions ❌
(primary)
0.3% 0.9% 6
Regressions ❌
(secondary)
0.2% 0.2% 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 0.3% 0.9% 6

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results
  • Primary benchmarks: ❌ relevant regression found
  • Secondary benchmarks: mixed results
mean1 max count2
Regressions ❌
(primary)
4.3% 4.3% 1
Regressions ❌
(secondary)
7.3% 7.3% 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-3.7% -4.2% 3
All ❌✅ (primary) 4.3% 4.3% 1

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance


Finished benchmarking commit (b998821e4c51c44a9ebee395c91323c374236bbb): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean1 max count2
Regressions ❌
(primary)
0.3% 0.9% 6
Regressions ❌
(secondary)
0.2% 0.2% 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 0.3% 0.9% 6

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean1 max count2
Regressions ❌
(primary)
4.3% 4.3% 1
Regressions ❌
(secondary)
7.3% 7.3% 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-3.7% -4.2% 3
All ❌✅ (primary) 4.3% 4.3% 1

Footnotes

  1. the arithmetic mean of the percent change 2 3 4

  2. number of relevant changes 2 3 4

@rust-lang rust-lang deleted a comment from rustbot Aug 19, 2022
Specifics:
- Adds a big "Overall result" heading at the top of the comment, which
  includes "ACTION NEEDED" when there are regressions.
- Moves the next steps and "if you disagree..." text to the top of the
  comment, just below the overall result, before the tables of
  measurements. This means all the instructions that people need to read
  are in one place.
- Removes the short summaries in front of each table, because (a) they just
  duplicates what the table says, and (b) saying "regressions found" for
  cycles and max-rss is a bit misleading because it doesn't contribute
  to the overall conclusion. Replaces them with a sentence indicating if
  the measurements contributed to the overall result and why.
- Changes `Direction::join` to a non-static method, it's nicer that way.
@nnethercote nnethercote force-pushed the clarify-overall-result branch from 7b34b1e to 9716b16 Compare August 19, 2022 03:54
Copy link
Contributor

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I like that the actions to do and especially the overall is are now clearly visible at the beginning.

@nnethercote
Copy link
Contributor Author

I have addressed the review comments. I also added " - no action needed" to the heading in the case where no action is needed, for a clearer contrast to the " - ACTION NEEDED" case.

@nnethercote nnethercote force-pushed the clarify-overall-result branch from b84ae45 to 2bd9b6a Compare August 22, 2022 02:36
@nnethercote nnethercote merged commit 870445d into rust-lang:master Aug 22, 2022
@nnethercote nnethercote deleted the clarify-overall-result branch August 22, 2022 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants