Skip to content

[benchmark] Disable some new Set tests with overlong runtime #24725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 12, 2019

Conversation

palimondo
Copy link
Contributor

@palimondo palimondo commented May 12, 2019

PRs #23690 and #24156 have improved the performance coverage of Set type. The workloads in #24156 have been tune in with #21300 in mind. This means that some larger multipliers were selected assuming the presence of optimizations from #23100. This PR temporarily disables following test where it currently results in runtimes over 30 ms (these are guaranteed to include a context switch and are therefore more susceptible to be noisy and case false performance change):

  • Set.isStrictSubset.Int100
  • Set.isStrictSubset.Seq.Empty.Int
  • Set.isStrictSuperset.Seq.Int.Empty

Temporarily disable benchmarks that run for too long without optimizations pending in set-on-fire branch.
@palimondo
Copy link
Contributor Author

@swift-ci please benchmark

@palimondo
Copy link
Contributor Author

@swift-ci please smoke test and merge

@swift-ci
Copy link
Contributor

Performance: -O

TEST OLD NEW DELTA RATIO
Regression
Set.isStrictSubset.Int.Empty 49 53 +8.2% 0.92x (?)
Removed
Set.isStrictSubset.Int100 42225 42363 42272
Set.isStrictSubset.Seq.Empty.Int 31865 31951 31911
Set.isStrictSuperset.Seq.Int.Empty 31860 31883 31873

Performance: -Osize

TEST OLD NEW DELTA RATIO
Improvement
Set.isDisjoint.Seq.Int.Empty 55 51 -7.3% 1.08x (?)
PrefixWhileCountableRangeLazy 15 14 -6.7% 1.07x (?)
Removed
Set.isStrictSubset.Int100 39676 39822 39763
Set.isStrictSubset.Seq.Empty.Int 32752 32891 32816
Set.isStrictSuperset.Seq.Int.Empty 33253 33387 33329

Performance: -Onone

TEST MIN MAX MEAN MAX_RSS
Removed
Set.isStrictSubset.Int100 103900 104122 104036
Set.isStrictSubset.Seq.Empty.Int 536153 565882 555163
Set.isStrictSuperset.Seq.Int.Empty 572162 573096 572531
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview
  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

@swift-ci swift-ci merged commit c7593ce into master May 12, 2019
@compnerd compnerd deleted the skip-bench-pending-set-on-fire branch May 31, 2025 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants