Skip to content

[benchmark] Fix quantile estimation type #19637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 1, 2018

Conversation

palimondo
Copy link
Contributor

@palimondo palimondo commented Oct 1, 2018

The correct quantile estimation type for printing all measurements in the summary report while quantile == num-samples - 1 is R-1, SAS-3. It's the inverse of empirical distribution function.

This enables printing all measured values in the one line of summary stats when executing Benchmark_O with --quantile parameter set to one less than --num-samples, without the need to resort to use of much heavier output in --verbose mode.

(The mistake was detected while porting the benchmark analysis to R-language. The previously chosen type R-3 behaves differently than I thought initially.)

References:

The correct quantile estimation type for printing all measurements in the summary report while `quantile == num-samples - 1` is R-1, SAS-3.  It's the inverse of empirical distribution function.

References:
* https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample
* discussion in swiftlang#19097 (comment)
@palimondo
Copy link
Contributor Author

palimondo commented Oct 1, 2018

@eeckstein Please review 🙏
(… also please run a full test, since that's the one that executes the benchmarks integration tests)

@eeckstein
Copy link
Contributor

@swift-ci test

@eeckstein
Copy link
Contributor

@swift-ci benchmark

@eeckstein eeckstein self-requested a review October 1, 2018 19:27
Copy link
Contributor

@eeckstein eeckstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@swift-ci
Copy link
Contributor

swift-ci commented Oct 1, 2018

Build comment file:

No performance and code size changes

How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein).

Hardware Overview
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

@eeckstein eeckstein merged commit 24dd963 into swiftlang:master Oct 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants