[benchmark] Add StringSplitting and lines benchmarks #36160

milseman · 2021-02-25T15:50:48Z

Add a new benchmark module StringSplitting for split-like benchmarking.

Add lineSink benchmarks, which separates Unicode content by lines and feeds
Strings into a sink.

milseman · 2021-02-25T15:50:58Z

@swift-ci please smoke test

milseman · 2021-02-25T15:51:06Z

@swift-ci please benchmark

swift-ci · 2021-02-25T16:41:57Z

Build failed before running benchmark.

milseman · 2021-02-25T17:47:38Z

@swift-ci please smoke test macOS platform

milseman · 2021-02-25T18:11:04Z

@swift-ci please smoke test macOS platform

milseman · 2021-02-25T18:11:39Z

@swift-ci please benchmark

swift-ci · 2021-02-25T18:28:33Z

Build failed before running benchmark.

milseman · 2021-02-25T20:02:13Z

@swift-ci please smoke test macOS platform

milseman · 2021-02-25T21:33:41Z

@swift-ci please benchmark

benchmark/single-source/StringSplitting.swift

swift-ci · 2021-02-25T22:30:22Z

Build failed before running benchmark.

Add a new benchmark module StringSplitting for split-like benchmarking. Add lineSink benchmarks, which separates Unicode content by lines and feeds Strings into a sink.

milseman · 2021-03-06T00:37:19Z

@swift-ci please smoke test

milseman · 2021-03-06T00:37:28Z

@swift-ci please benchmark

swift-ci · 2021-03-06T02:45:51Z

Performance: -O

Regression	OLD	NEW	DELTA	RATIO
FlattenListFlatMap	3914	4261	+8.9%	0.92x (?)
ObjectiveCBridgeFromNSArrayAnyObjectForced	4340	4680	+7.8%	0.93x (?)

Improvement	OLD	NEW	DELTA	RATIO
DictionaryKeysContainsNative	29	26	-10.3%	1.12x (?)

Added	MIN	MAX	MEAN	MAX_RSS
StringSplitting_LineSink_bytes_alpha	1	1	1	—
StringSplitting_LineSink_bytes_complex	197	197	197	—
StringSplitting_LineSink_characters_alpha	2	2	2	—
StringSplitting_LineSink_characters_complex	645	647	646	—
StringSplitting_LineSink_scalars_alpha	1	1	1	—
StringSplitting_LineSink_scalars_complex	197	197	197	—

Code size: -O

Improvement	OLD	NEW	DELTA	RATIO
main.o	53161	52553	-1.1%	1.01x

Performance: -Osize

Regression	OLD	NEW	DELTA	RATIO
UTF8Decode_InitFromData_ascii_as_ascii	636	706	+11.0%	0.90x (?)
CharacterLiteralsLarge	100	111	+11.0%	0.90x (?)

Improvement	OLD	NEW	DELTA	RATIO
DictionaryKeysContainsNative	28	26	-7.1%	1.08x (?)
DictionaryKeysContainsCocoa	29	27	-6.9%	1.07x (?)

Added	MIN	MAX	MEAN	MAX_RSS
StringSplitting_LineSink_bytes_alpha	1	1	1	—
StringSplitting_LineSink_bytes_complex	200	200	200	—
StringSplitting_LineSink_characters_alpha	2	2	2	—
StringSplitting_LineSink_characters_complex	642	643	642	—
StringSplitting_LineSink_scalars_alpha	1	1	1	—
StringSplitting_LineSink_scalars_complex	200	203	201	—

Code size: -Osize

Performance: -Onone

Regression	OLD	NEW	DELTA	RATIO
UTF8Decode_InitFromData_ascii_as_ascii	753	842	+11.8%	0.89x (?)

Added	MIN	MAX	MEAN	MAX_RSS
StringSplitting_LineSink_bytes_alpha	5	5	5	—
StringSplitting_LineSink_bytes_complex	874	874	874	—
StringSplitting_LineSink_characters_alpha	3	3	3	—
StringSplitting_LineSink_characters_complex	738	739	738	—
StringSplitting_LineSink_scalars_alpha	6	6	6	—
StringSplitting_LineSink_scalars_complex	873	874	873	—

Code size: -swiftlibs

✅	Benchmark Check Report
⛔️🔤	`StringSplitting_LineSink_scalars_complex` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`StringSplitting_LineSink_characters_alpha` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`StringSplitting_LineSink_characters_alpha` name is 41 characters long. _{Benchmark name should not be longer than 40 characters.}
⚠️⏱	`StringSplitting_LineSink_characters_alpha` execution took 2 μs. _{Increase the workload of StringSplitting_LineSink_characters_alpha to be more than 20 μs.}
⛔️🔤	`StringSplitting_LineSink_scalars_alpha` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⚠️⏱	`StringSplitting_LineSink_scalars_alpha` execution took 1 μs. _{Increase the workload of StringSplitting_LineSink_scalars_alpha to be more than 20 μs.}
⛔️🔤	`StringSplitting_LineSink_bytes_complex` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`StringSplitting_LineSink_bytes_alpha` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⚠️⏱	`StringSplitting_LineSink_bytes_alpha` execution took 1 μs. _{Increase the workload of StringSplitting_LineSink_bytes_alpha to be more than 20 μs.}
⛔️🔤	`StringSplitting_LineSink_characters_complex` name doesn`t conform to benchmark naming convention. _{See http://bit.ly/BenchmarkNaming}
⛔️🔤	`StringSplitting_LineSink_characters_complex` name is 43 characters long. _{Benchmark name should not be longer than 40 characters.}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

milseman · 2021-03-06T16:05:38Z

@swift-ci please benchmark

milseman · 2021-03-06T16:05:50Z

@swift-ci please smoke test

swift-ci · 2021-03-06T16:44:18Z

Performance: -O

Improvement	OLD	NEW	DELTA	RATIO
NSError	188	157	-16.5%	1.20x (?)
String.data.Medium	127	118	-7.1%	1.08x (?)

Added	MIN	MAX	MEAN	MAX_RSS
LineSink.bytes.alpha	63	63	63	—
LineSink.bytes.complex	198	199	198	—
LineSink.characters.alpha	115	115	115	—
LineSink.characters.complex	639	639	639	—
LineSink.scalars.alpha	76	79	77	—
LineSink.scalars.complex	198	199	198	—

Code size: -O

Improvement	OLD	NEW	DELTA	RATIO
main.o	53161	52553	-1.1%	1.01x

Performance: -Osize

Regression	OLD	NEW	DELTA	RATIO
NSStringConversion.UTF8	988	1132	+14.6%	0.87x (?)
CharacterLiteralsLarge	100	111	+11.0%	0.90x (?)

Added	MIN	MAX	MEAN	MAX_RSS
LineSink.bytes.alpha	61	61	61	—
LineSink.bytes.complex	196	196	196	—
LineSink.characters.alpha	118	120	119	—
LineSink.characters.complex	649	663	654	—
LineSink.scalars.alpha	84	84	84	—
LineSink.scalars.complex	197	211	204	—

Code size: -Osize

Performance: -Onone

Regression	OLD	NEW	DELTA	RATIO
DataAppendDataSmallToSmall	5460	5980	+9.5%	0.91x (?)

Added	MIN	MAX	MEAN	MAX_RSS
LineSink.bytes.alpha	274	274	274	—
LineSink.bytes.complex	865	866	866	—
LineSink.characters.alpha	155	156	155	—
LineSink.characters.complex	732	741	736	—
LineSink.scalars.alpha	338	339	338	—
LineSink.scalars.complex	866	867	866	—

Code size: -swiftlibs

✅	Benchmark Check Report

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

milseman · 2021-03-06T18:25:19Z

@swift-ci please smoke test

milseman · 2021-03-06T21:13:00Z

@swift-ci please smoke test macOS platform

milseman · 2021-03-07T15:14:06Z

@swift-ci please smoke test macOS platform

milseman requested a review from Catfish-Man February 25, 2021 15:51

benrimmington reviewed Feb 25, 2021

View reviewed changes

benchmark/single-source/StringSplitting.swift Show resolved Hide resolved

benchmark/single-source/StringSplitting.swift Show resolved Hide resolved

benchmark/single-source/StringSplitting.swift Outdated Show resolved Hide resolved

milseman added 2 commits March 4, 2021 17:20

[benchmark] Add StringSplitting and lines benchmarks

73af20f

Add a new benchmark module StringSplitting for split-like benchmarking. Add lineSink benchmarks, which separates Unicode content by lines and feeds Strings into a sink.

[benchmark] Cleanup and bug fix

4612a5b

milseman force-pushed the string_hew branch from b7fca2b to 4612a5b Compare March 6, 2021 00:36

[benchmark] Conform to naming and timing conventions

4fe6873

milseman merged commit fe667da into swiftlang:main Mar 7, 2021

milseman deleted the string_hew branch March 7, 2021 17:14

[benchmark] Add StringSplitting and lines benchmarks #36160

[benchmark] Add StringSplitting and lines benchmarks #36160

Uh oh!

Conversation

milseman commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

swift-ci commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

swift-ci commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

milseman commented Feb 25, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

swift-ci commented Feb 25, 2021

Uh oh!

milseman commented Mar 6, 2021

Uh oh!

milseman commented Mar 6, 2021

Uh oh!

swift-ci commented Mar 6, 2021

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Code size: -swiftlibs

Uh oh!

milseman commented Mar 6, 2021

Uh oh!

milseman commented Mar 6, 2021

Uh oh!

swift-ci commented Mar 6, 2021

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Code size: -swiftlibs

Uh oh!

milseman commented Mar 6, 2021

Uh oh!

milseman commented Mar 6, 2021

Uh oh!

milseman commented Mar 7, 2021

Uh oh!

Uh oh!