[benchmark][NFC] Use Swift naming conventions #39336

lorentey · 2021-09-16T05:10:03Z

Update benchmarks to bring them in line with basic Swift naming conventions.

Hopefully this will make working with benchmarks a little bit less scary for new contributors.

Stop capitalizing function, variable, and parameter names.
Unify/simplify benchmark registration a bit. Get rid of custom property names for each module; have each benchmark simply expose a public benchmarks property.
Rename file names where the resulting module names clash with standard type/protocol names. (We don't have fully qualified names yet.)

Capitalizing function names is against Swift naming conventions.

lorentey · 2021-09-16T05:10:26Z

lorentey · 2021-09-17T00:52:30Z

Regression OLD NEW DELTA RATIO

Fibonacci 145 4386 +2924.8% 0.03x

Ackermann 299 4382 +1365.5% 0.07x

What.

The code size changes I think I understand -- these tests are now exporting arrays instead of just standalone BenchmarkInfo values. The change in main.o demonstrates the danger in overusing @inline(__always).

…hmarks It just produces unnecessary code sign churn.

lorentey · 2021-09-17T01:54:32Z

@swift-ci test

lorentey · 2021-09-17T01:54:40Z

@swift-ci benchmark

swift-ci · 2021-09-17T03:12:26Z

Performance (x86_64): -O

Regression	OLD	NEW	DELTA	RATIO
Fibonacci	97	2276	+2246.4%	0.04x
Ackermann	211	1270	+501.9%	0.17x
DictionaryOfAnyHashableStrings_insert	2184	3794	+73.7%	0.58x
DataCreateEmpty	80	100	+25.0%	0.80x
ConvertFloatingPoint.MockFloat64ToDouble	14	17	+21.4%	0.82x (?)
OpenClose	26	30	+15.4%	0.87x
ReversedArray2	95	109	+14.7%	0.87x (?)
ProtocolDispatch	196	219	+11.7%	0.89x (?)

Improvement	OLD	NEW	DELTA	RATIO
FlattenListLoop	1562	998	-36.1%	1.57x (?)
FlattenListFlatMap	4312	3191	-26.0%	1.35x (?)
ObjectiveCBridgeStubDateAccess	152	130	-14.5%	1.17x (?)
CreateObjects	10	9	-10.0%	1.11x (?)

Code size: -O

Regression	OLD	NEW	DELTA	RATIO
Fibonacci.o	1553	1969	+26.8%	0.79x
Ackermann.o	1828	1956	+7.0%	0.93x
CreateObjects.o	897	959	+6.9%	0.94x
TestsUtils.o	25435	26336	+3.5%	0.97x
MonteCarloPi.o	1498	1514	+1.1%	0.99x

Improvement	OLD	NEW	DELTA	RATIO
main.o	52761	12525	-76.3%	4.21x
BinaryFloatingPointProperties.o	5104	4867	-4.6%	1.05x
StringInterpolation.o	6698	6412	-4.3%	1.04x
FlattenList.o	3994	3896	-2.5%	1.03x
CharacterProperties.o	20883	20447	-2.1%	1.02x
ArraySubscript.o	2382	2350	-1.3%	1.01x
StringComparison.o	37304	36824	-1.3%	1.01x

Performance (x86_64): -Osize

Regression	OLD	NEW	DELTA	RATIO
Fibonacci	113	2603	+2203.5%	0.04x
Ackermann	155	562	+262.6%	0.28x
StrToInt	1010	1100	+8.9%	0.92x (?)
SIMDRandomMask.Int8x16	339	369	+8.8%	0.92x (?)

Improvement	OLD	NEW	DELTA	RATIO
StaticArray	2	1	-50.0%	2.00x
CreateObjects	10	8	-20.0%	1.25x (?)
ConvertFloatingPoint.MockFloat64Exactly	6	5	-16.7%	1.20x
OpenClose	30	26	-13.3%	1.15x (?)
RandomInt8LCG	459	403	-12.2%	1.14x
DataCountMedium	19	17	-10.5%	1.12x (?)

Code size: -Osize

Regression	OLD	NEW	DELTA	RATIO
Fibonacci.o	1392	1773	+27.4%	0.79x
CreateObjects.o	793	892	+12.5%	0.89x
Ackermann.o	1662	1784	+7.3%	0.93x
TestsUtils.o	19338	20067	+3.8%	0.96x
MonteCarloPi.o	1319	1348	+2.2%	0.98x

Improvement	OLD	NEW	DELTA	RATIO
main.o	48064	9045	-81.2%	5.31x
BinaryFloatingPointProperties.o	5111	4912	-3.9%	1.04x
StringInterpolation.o	6473	6264	-3.2%	1.03x
CharacterProperties.o	20229	19868	-1.8%	1.02x
StringComparison.o	33197	32627	-1.7%	1.02x
FlattenList.o	3846	3788	-1.5%	1.02x

Performance (x86_64): -Onone

Regression	OLD	NEW	DELTA	RATIO
Fibonacci	420	9324	+2120.0%	0.05x
Ackermann	1175	6243	+431.3%	0.19x

Code size: -swiftlibs

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: 6-Core Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

lorentey · 2021-09-17T07:45:24Z

@swift-ci benchmark

swift-ci · 2021-09-17T09:02:48Z

Performance (x86_64): -O

Regression	OLD	NEW	DELTA	RATIO
Fibonacci	145	3499	+2313.1%	0.04x
Ackermann	299	1799	+501.7%	0.17x
UTF8Decode_InitFromBytes_ascii	282	310	+9.9%	0.91x (?)

Improvement	OLD	NEW	DELTA	RATIO
RandomShuffleLCG2	480	448	-6.7%	1.07x (?)

Code size: -O

Regression	OLD	NEW	DELTA	RATIO
Fibonacci.o	1553	1969	+26.8%	0.79x
Ackermann.o	1828	1956	+7.0%	0.93x
CreateObjects.o	897	959	+6.9%	0.94x
TestsUtils.o	25451	26352	+3.5%	0.97x
MonteCarloPi.o	1498	1514	+1.1%	0.99x

Improvement	OLD	NEW	DELTA	RATIO
main.o	52761	12525	-76.3%	4.21x
BinaryFloatingPointProperties.o	5104	4867	-4.6%	1.05x
StringInterpolation.o	6698	6412	-4.3%	1.04x
FlattenList.o	3994	3896	-2.5%	1.03x
CharacterProperties.o	20883	20447	-2.1%	1.02x
ArraySubscript.o	2382	2350	-1.3%	1.01x
StringComparison.o	37304	36824	-1.3%	1.01x

Performance (x86_64): -Osize

Regression	OLD	NEW	DELTA	RATIO
Fibonacci	201	4453	+2115.4%	0.05x
Ackermann	251	1668	+564.5%	0.15x
ObjectiveCBridgeStubDateAccess	257	285	+10.9%	0.90x (?)
String.data.Medium	96	105	+9.4%	0.91x (?)

Improvement	OLD	NEW	DELTA	RATIO
DataAppendSequence	10300	9600	-6.8%	1.07x (?)

Code size: -Osize

Regression	OLD	NEW	DELTA	RATIO
Fibonacci.o	1392	1773	+27.4%	0.79x
CreateObjects.o	793	892	+12.5%	0.89x
Ackermann.o	1662	1784	+7.3%	0.93x
TestsUtils.o	19362	20091	+3.8%	0.96x
MonteCarloPi.o	1319	1348	+2.2%	0.98x

Improvement	OLD	NEW	DELTA	RATIO
main.o	48064	9045	-81.2%	5.31x
BinaryFloatingPointProperties.o	5111	4912	-3.9%	1.04x
StringInterpolation.o	6473	6264	-3.2%	1.03x
CharacterProperties.o	20229	19868	-1.8%	1.02x
StringComparison.o	33197	32627	-1.7%	1.02x
FlattenList.o	3846	3788	-1.5%	1.02x

Performance (x86_64): -Onone

Regression	OLD	NEW	DELTA	RATIO
Fibonacci	731	16913	+2213.7%	0.04x
Ackermann	2494	14732	+490.7%	0.17x
StringToDataLargeUnicode	6300	7100	+12.7%	0.89x (?)
DictionaryBridgeToObjC_Access	1136	1266	+11.4%	0.90x (?)

Code size: -swiftlibs

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

lorentey · 2021-09-17T19:53:43Z

Weird; it looks like the compiler decided to unroll the loop over n in both Fibonacci & Ackermann. Not sure why that'd happen after these renames, or why that'd slow these down so much; perhaps my baselines are outdated.

lorentey · 2021-09-21T00:16:31Z

My local benchmark results for Fibonacci and Ackermann are the opposite of the results here. I'm going to chalk these results up to some pre-existing fragility rather than a consequence of these renames. ¯\_(ツ)_/¯

Merging.

lorentey added 5 commits September 15, 2021 22:08

[benchmark] CheckResults → check

203dc55

Capitalizing function names is against Swift naming conventions.

[benchmark] Random(), SRand() → LFSR.init(), .next()

bd9a220

[benchmark] False() → getFalse()

958733c

[benchmark] Stop capitalizing function and variable names

8910b75

[benchmark] Simplify benchmark registration

8944591

lorentey mentioned this pull request Sep 16, 2021

[DO NOT MERGE][benchmark] Modernize benchmark definitions #39331

Draft

lorentey added 2 commits September 16, 2021 12:12

[benchmark] Regenerate gyb

526527e

[benchmark] Fixup expected output in MirrorTest after module rename

ee34b02

This comment has been minimized.

Sign in to view

lorentey added 2 commits September 16, 2021 18:00

[benchmark] Fix silly thinkos in Ackermann & Fibonacci

110d123

[benchmark] Don't create array instance in modules with solitary benc…

758c52b

…hmarks It just produces unnecessary code sign churn.

lorentey merged commit 8304e6c into swiftlang:main Sep 21, 2021

lorentey deleted the decapitate-benchmarks branch September 21, 2021 00:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[benchmark][NFC] Use Swift naming conventions #39336

[benchmark][NFC] Use Swift naming conventions #39336

Uh oh!

lorentey commented Sep 16, 2021

Uh oh!

lorentey commented Sep 16, 2021

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lorentey commented Sep 17, 2021 •

edited

Loading

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

swift-ci commented Sep 17, 2021

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

swift-ci commented Sep 17, 2021

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

lorentey commented Sep 21, 2021

Uh oh!

Uh oh!

[benchmark][NFC] Use Swift naming conventions #39336

[benchmark][NFC] Use Swift naming conventions #39336

Uh oh!

Conversation

lorentey commented Sep 16, 2021

Uh oh!

lorentey commented Sep 16, 2021

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lorentey commented Sep 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

swift-ci commented Sep 17, 2021

Performance (x86_64): -O

Code size: -O

Performance (x86_64): -Osize

Code size: -Osize

Performance (x86_64): -Onone

Code size: -swiftlibs

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

swift-ci commented Sep 17, 2021

Performance (x86_64): -O

Code size: -O

Performance (x86_64): -Osize

Code size: -Osize

Performance (x86_64): -Onone

Code size: -swiftlibs

Uh oh!

lorentey commented Sep 17, 2021

Uh oh!

lorentey commented Sep 21, 2021

Uh oh!

Uh oh!

lorentey commented Sep 17, 2021 •

edited

Loading