Skip to content

[benchmark] Add ChaCha20-based performance benchmark #27699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 29, 2019
Merged

[benchmark] Add ChaCha20-based performance benchmark #27699

merged 1 commit into from
Oct 29, 2019

Conversation

Lukasa
Copy link
Contributor

@Lukasa Lukasa commented Oct 15, 2019

This patch adds a benchmark to the Swift benchmark suite based on the
ChaCha20 encryption algorithm.

As Swift evolves it is important that it tackles more and more features
and possible use cases. One of these great use-cases is low-level CPU
intensive code, and cryptographic algorithms are a really important
test-bed.

This benchmark therefore provides a real-world test case for Swift's
optimiser. My ideal outcome here is that Swift should be able to perform
as well at this benchmark as a naive equivalent C implementation.

@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please smoke test

@Lukasa Lukasa force-pushed the cb-chacha-benchmark branch from e2c9ebf to 59806c8 Compare October 15, 2019 20:12
@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please smoke test

@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

Regression OLD NEW DELTA RATIO
ObjectiveCBridgeStubToNSStringRef 72 80 +11.1% 0.90x (?)
DataAccessBytesMedium 45 49 +8.9% 0.92x (?)
 
Improvement OLD NEW DELTA RATIO
Breadcrumbs.IdxToUTF16Range.longASCII 29 27 -6.9% 1.07x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 0 0 0 —

Code size: -O

Performance: -Osize

Regression OLD NEW DELTA RATIO
ObjectiveCBridgeStubNSDateRefAccess 174 196 +12.6% 0.89x (?)
 
Improvement OLD NEW DELTA RATIO
Dictionary4 211 168 -20.4% 1.26x
DataSubscriptMedium 42 39 -7.1% 1.08x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 0 0 0 —

Code size: -Osize

Performance: -Onone

Regression OLD NEW DELTA RATIO
ArrayAppendOptionals 620 1330 +114.5% 0.47x (?)
 
Improvement OLD NEW DELTA RATIO
TypeFlood 157 143 -8.9% 1.10x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 164 176 171 —

Code size: -swiftlibs

✅ Benchmark Check Report
⛔️⏱ ChaCha execution took 0 μs.
Ensure the workload of ChaCha has a properly measurable size (runtime > 20 μs) and is not eliminated by the compiler (use blackHole function if necessary).
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

@Lukasa Lukasa force-pushed the cb-chacha-benchmark branch from 59806c8 to 1be1eb5 Compare October 15, 2019 22:35
@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

Regression OLD NEW DELTA RATIO
SuffixAnyCollection 12 13 +8.3% 0.92x (?)
SubstringFromLongStringGeneric 12 13 +8.3% 0.92x (?)
 
Improvement OLD NEW DELTA RATIO
ObjectiveCBridgeStubFromNSStringRef 109 93 -14.7% 1.17x (?)
Set.subtracting.Empty.Box 9 8 -11.1% 1.12x (?)
Breadcrumbs.IdxToUTF16Range.longASCII 29 27 -6.9% 1.07x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 3 3 3 —

Code size: -O

Performance: -Osize

Regression OLD NEW DELTA RATIO
ArrayAppendGenericStructs 610 1510 +147.5% 0.40x (?)
ObjectiveCBridgeStubNSDateRefAccess 174 196 +12.6% 0.89x (?)
 
Improvement OLD NEW DELTA RATIO
Dictionary4 210 168 -20.0% 1.25x
DataSubscriptMedium 43 39 -9.3% 1.10x (?)
PrefixCountableRangeLazy 14 13 -7.1% 1.08x (?)
DropFirstArray 14 13 -7.1% 1.08x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 3 3 3 —

Code size: -Osize

Performance: -Onone

Improvement OLD NEW DELTA RATIO
ArrayAppendGenericStructs 1530 650 -57.5% 2.35x (?)
ObjectiveCBridgeStubToNSDateRef 2700 2280 -15.6% 1.18x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 1101 1355 1215 —

Code size: -swiftlibs

✅ Benchmark Check Report
⚠️⏱ ChaCha execution took 3 μs.
Increase the workload of ChaCha to be more than 20 μs.
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

@Lukasa Lukasa force-pushed the cb-chacha-benchmark branch from 1be1eb5 to 6d22009 Compare October 15, 2019 22:59
@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

Regression OLD NEW DELTA RATIO
SuffixAnyCollection 12 13 +8.3% 0.92x (?)
 
Improvement OLD NEW DELTA RATIO
ArrayAppendGenericStructs 910 640 -29.7% 1.42x (?)
ObjectiveCBridgeStubFromNSStringRef 106 93 -12.3% 1.14x (?)
Breadcrumbs.IdxToUTF16Range.longASCII 29 27 -6.9% 1.07x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 28 28 28 —

Code size: -O

Performance: -Osize

Regression OLD NEW DELTA RATIO
SuffixCountableRange 4 5 +25.0% 0.80x (?)
DropLastCountableRangeLazy 5 6 +20.0% 0.83x (?)
ObjectiveCBridgeStubNSDateRefAccess 174 196 +12.6% 0.89x (?)
DropWhileAnySeqCntRangeLazy 158 174 +10.1% 0.91x (?)
ObjectiveCBridgeStubNSDataAppend 1510 1630 +7.9% 0.93x (?)
 
Improvement OLD NEW DELTA RATIO
Dictionary4 210 168 -20.0% 1.25x (?)
ArrayAppendGenericStructs 640 590 -7.8% 1.08x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 35 35 35 —

Code size: -Osize

Performance: -Onone

Added MIN MAX MEAN MAX_RSS
ChaCha 10959 11073 11009 —

Code size: -swiftlibs

✅ Benchmark Check Report
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

@Lukasa
[benchmark] Add ChaCha20-based performance benchmark
b43bfa8 
This patch adds a benchmark to the Swift benchmark suite based on the
ChaCha20 encryption algorithm.

As Swift evolves it is important that it tackles more and more features
and possible use cases. One of these great use-cases is low-level CPU
intensive code, and cryptographic algorithms are a really important
test-bed.

This benchmark therefore provides a real-world test case for Swift's
optimiser. My ideal outcome here is that Swift should be able to perform
as well at this benchmark as a naive equivalent C implementation.
@Lukasa Lukasa force-pushed the cb-chacha-benchmark branch from 6d22009 to b43bfa8 Compare October 15, 2019 23:33
@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please smoke test

@Lukasa
Copy link
Contributor Author

Lukasa commented Oct 15, 2019

@swift-ci please benchmark

@swift-ci
Copy link
Contributor

Performance: -O

Regression OLD NEW DELTA RATIO
FlattenListLoop 2765 3219 +16.4% 0.86x (?)
DropWhileAnySeqCntRange 43 49 +14.0% 0.88x (?)
FlattenListFlatMap 5662 6118 +8.1% 0.93x (?)
 
Improvement OLD NEW DELTA RATIO
Breadcrumbs.IdxToUTF16Range.longASCII 29 27 -6.9% 1.07x (?)
 
Added MIN MAX MEAN MAX_RSS
ChaCha 84 84 84 —

Code size: -O

Performance: -Osize

Regression OLD NEW DELTA RATIO
ObjectiveCBridgeStubNSDateRefAccess 174 196 +12.6% 0.89x (?)
ObjectiveCBridgeStubToArrayOfNSString2 2640 2840 +7.6% 0.93x (?)
 
Improvement OLD NEW DELTA RATIO
Dictionary4 210 168 -20.0% 1.25x
 
Added MIN MAX MEAN MAX_RSS
ChaCha 112 115 114 —

Code size: -Osize

Performance: -Onone

Added MIN MAX MEAN MAX_RSS
ChaCha 32694 32762 32731 —

Code size: -swiftlibs

✅ Benchmark Check Report
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

@gottesmm
Copy link
Contributor

LGTM

@gottesmm gottesmm merged commit 9565300 into swiftlang:master Oct 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Lukasa @swift-ci @gottesmm