[benchmark] Baseline test

[palimondo]

This PR adds new a -check-added option to the run_smoke_bench.py that finds the added benchmarks and runs an equivalent of Benchmark_Driver check on them. It can be run together with -verbose option to output more details during the benchmark analysis.

It outputs the BenchmarkDoctor's report with Warnings and Errors, if the new benchmarks violate the best practices. BenchmarkDoctor currently always returns exit code 0. In the future, we might change this to depend on the presence of ERRORs in the report, if deemed useful, so that violations of good baseline could not be committed to the tree.

It depends on BenchmarkDriver and compare_perf_tests.py being in the same directory.

[palimondo]

@eeckstein Please review 🙏

Can you integrate this into @swift-ci?

[eeckstein]

Can you include that in run_smoke_bench?
Then it would run automatically for each benchmark run.
I don't know how long it takes to check new benchmarks, but if it only checks new benchmarks, then the overhead time is not so important.
Integrating it into a regular CI test is not really doable because it requires two benchmark builds.

[palimondo]

I could? But than it would be invoked for each of the optimization levels, right? It should run only once, as it validates only the -O mode. I though it would be invoked from @swift-ci benchmark after the run_smoke_bench.py... Am I misunderstanding how run_smoke_bench.py get's invoked by CI?

[palimondo]

Or do you mean I should add it as another option to the run_smoke_bench? That could work… but it would still require some kind of modification to the CI, to invoke it. Is that what you meant?
(And yes, it only checks the added benchmarks. It takes about 10s per benchmark.)

[eeckstein]

Yes, just add an option and we can change the ci script to pass that option

[palimondo]

@eeckstein I've integrated the check_added_bench.py into run_smoke_bench.py as -check-added option and edited this PR's description to match. Please review.

[eeckstein]

How do you derive the list of added benchmarks? Is it getting the list by invoking Benchmark_O --list?

[palimondo]

Yes. BenchmarkDriver.__init__ calls _get_tests, which does Benchmark_O --list and applies the filters (regex), if any are present.

[eeckstein]

@swift-ci benchmark

[swift-ci]

Build comment file:

No performance and code size changes

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

---

[eeckstein]

@swift-ci smoke test

[eeckstein]

@swift-ci smoke test

[eeckstein]

@swift-ci smoke test

[eeckstein]

@swift-ci smoke test