Skip to content

[test] Add Range Overlaps Benchmark #23294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

PatrickPijnappel
Copy link
Contributor

Separate PR for benchmark from #23293, to merge first so that PR can compare its performance to master. Let me know if there is a better way to achieve this. I've run the benchmark locally, would just like the CI to verify.

@airspeedswift
Copy link
Member

@swift-ci please benchmark

@airspeedswift
Copy link
Member

@swift-ci please smoke test

@swift-ci
Copy link
Contributor

Performance: -O

TEST MIN MAX MEAN MAX_RSS
Added
ClosedRangeOverlapsClosedRange 4184 4572 4416
RangeOverlapsClosedRange 4462 4567 4514
RangeOverlapsRange 5583 5635 5615

Performance: -Osize

TEST OLD NEW DELTA RATIO
Improvement
ObjectiveCBridgeStubFromNSDateRef 4330 4000 -7.6% 1.08x (?)
Added
ClosedRangeOverlapsClosedRange 4049 4196 4130
RangeOverlapsClosedRange 4361 4445 4394
RangeOverlapsRange 5051 5110 5072

Performance: -Onone

TEST OLD NEW DELTA RATIO
Improvement
ObjectiveCBridgeStubFromNSDateRef 4650 4310 -7.3% 1.08x (?)
Added
ClosedRangeOverlapsClosedRange 878831 879563 879220
RangeOverlapsClosedRange 935580 939357 937335
RangeOverlapsRange 917752 922438 919511
Benchmark Check Report
⚠️ RangeOverlapsRange execution took at least 5519 μs.
Decrease the workload of RangeOverlapsRange by a factor of 8 (10), to be less than 1000 μs.
⚠️Ⓜ️ RangeOverlapsRange has very wide range of memory used between independent, repeated measurements.
RangeOverlapsRange mem_pages [i1, i2]: min=[5, 5] 𝚫=0 R=[0, 24]
⚠️ RangeOverlapsClosedRange execution took at least 4447 μs.
Decrease the workload of RangeOverlapsClosedRange by a factor of 8 (10), to be less than 1000 μs.
⚠️Ⓜ️ RangeOverlapsClosedRange has very wide range of memory used between independent, repeated measurements.
RangeOverlapsClosedRange mem_pages [i1, i2]: min=[6, 6] 𝚫=0 R=[49, 0]
⚠️🔤 ClosedRangeOverlapsClosedRange name is composed of 5 words.
Split ClosedRangeOverlapsClosedRange name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⛔️⏱ ClosedRangeOverlapsClosedRange has setup overhead of 746 μs (16.4%).
Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.
⚠️ ClosedRangeOverlapsClosedRange execution took at least 3797 μs (excluding the setup overhead).
Decrease the workload of ClosedRangeOverlapsClosedRange by a factor of 4 (10), to be less than 1000 μs.
⚠️Ⓜ️ ClosedRangeOverlapsClosedRange has very wide range of memory used between independent, repeated measurements.
ClosedRangeOverlapsClosedRange mem_pages [i1, i2]: min=[5, 5] 𝚫=0 R=[48, 48]
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

@airspeedswift
Copy link
Member

Hi @PatrickPijnappel – can you tweak the number of iterations a little to keep the benchmark runs performant?

@PatrickPijnappel
Copy link
Contributor Author

@airspeedswift Thanks! Updated. Also split up the set up to fix the other warnings

@PatrickPijnappel
Copy link
Contributor Author

@swift-ci Please benchmark

@swift-ci
Copy link
Contributor

Build failed before running benchmark.

@PatrickPijnappel PatrickPijnappel force-pushed the range-overlaps-performance branch from 7f416bf to 37b2522 Compare March 18, 2019 02:56
@PatrickPijnappel PatrickPijnappel force-pushed the range-overlaps-performance branch from 37b2522 to 143f253 Compare March 18, 2019 03:39
@PatrickPijnappel
Copy link
Contributor Author

@swift-ci Please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

TEST OLD NEW DELTA RATIO
Improvement
CharacterLiteralsLarge 108 97 -10.2% 1.11x (?)
SortLettersInPlace 560 513 -8.4% 1.09x (?)
Added
ClosedRangeOverlapsClosedRange 145 146 145
RangeOverlapsClosedRange 128 129 128
RangeOverlapsRange 159 162 161

Performance: -Osize

TEST MIN MAX MEAN MAX_RSS
Added
ClosedRangeOverlapsClosedRange 144 146 145
RangeOverlapsClosedRange 137 141 138
RangeOverlapsRange 150 153 151

Performance: -Onone

TEST MIN MAX MEAN MAX_RSS
Added
ClosedRangeOverlapsClosedRange 26343 26464 26400
RangeOverlapsClosedRange 28349 28427 28392
RangeOverlapsRange 27886 28262 28074
Benchmark Check Report
⚠️🔤 ClosedRangeOverlapsClosedRange name is composed of 5 words.
Split ClosedRangeOverlapsClosedRange name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⛔️⏱ ClosedRangeOverlapsClosedRange has setup overhead of 22 μs (16.1%).
Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

@PatrickPijnappel
Copy link
Contributor Author

@swift-ci Please smoke test

@airspeedswift airspeedswift merged commit 9250e9f into swiftlang:master Mar 18, 2019
@PatrickPijnappel PatrickPijnappel deleted the range-overlaps-performance branch March 18, 2019 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants