[benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes #21717

palimondo · 2019-01-08T20:57:18Z

The new Set.Empty tests introduced in #20631 have too small workloads with runtimes near or at 0 μs. This needs to be fixed by adjusting the loop multipliers. In order to prevent this kind of error when writing new benchmarks, this PR extends BenchmarkDoctor with a check for lower runtime bound.

For really small runtimes < 20 μs this method of setup overhead detection doesn’t work. Even 1μs change in 20μs runtime is 5%. Just return no overhead.

Warn about runtimes under 20 μs and flag 0 μs runtimes as errors.

Most of these recently added benchmarks have too low loop multiplier that results in near zero or zero measured runtime. Since fixing this will change the runtimes, it is also an opportunity to properly apply the new naming convention.

Let’s also test Empty sets as the right-hand side parameter.

palimondo · 2019-01-08T20:59:51Z

@swift-ci please benchmark

swift-ci · 2019-01-08T21:45:32Z

Build comment file:

Performance: -O

TEST	MIN	MAX	MEAN	MAX_RSS
Added
Set.isDisjoint.Box0	699	700	700	—
Set.isDisjoint.Box0.Empty	2	2	2	—
Set.isDisjoint.Box25	5	5	5	—
Set.isDisjoint.Empty.Box0	1	1	1	—
Set.isDisjoint.Empty.Int0	1	1	1	—
Set.isDisjoint.Int0	296	298	297	—
Set.isDisjoint.Int0.Empty	1	1	1	—
Set.isDisjoint.Int100	1	1	1	—
Set.isDisjoint.Int25	2	2	2	—
Set.isDisjoint.Int50	2	2	2	—
Set.isSubset.Empty.Int0	220	222	221	—
Set.isSubset.Int0.Empty	111	113	112	—
Set.subtracting.Box0.Empty	0	0	0	—
Set.subtracting.Empty.Box0	0	0	0	—
Set.subtracting.Empty.Int0	0	0	0	—
Set.subtracting.Int0.Empty	0	0	0	—
Removed
Set.Empty.IsDisjointBox0	1	1	1	—
Set.Empty.IsDisjointInt0	1	1	1	—
Set.Empty.IsSubsetInt0	220	221	220	—
Set.Empty.SubtractingBox0	0	0	0	—
Set.Empty.SubtractingInt0	0	0	0	—
SetIsDisjointBox0	693	694	693	—
SetIsDisjointBox25	5	5	5	—
SetIsDisjointInt0	297	299	298	—
SetIsDisjointInt100	1	1	1	—
SetIsDisjointInt25	2	2	2	—
SetIsDisjointInt50	2	2	2	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	71591	75047	+4.8%	0.95x

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
DataAppendDataLargeToLarge	37800	51200	+35.4%	0.74x (?)
Added
Set.isDisjoint.Box0	846	847	846	—
Set.isDisjoint.Box0.Empty	2	2	2	—
Set.isDisjoint.Box25	7	7	7	—
Set.isDisjoint.Empty.Box0	1	1	1	—
Set.isDisjoint.Empty.Int0	1	1	1	—
Set.isDisjoint.Int0	306	308	307	—
Set.isDisjoint.Int0.Empty	2	2	2	—
Set.isDisjoint.Int100	3	3	3	—
Set.isDisjoint.Int25	3	3	3	—
Set.isDisjoint.Int50	3	3	3	—
Set.isSubset.Empty.Int0	227	231	228	—
Set.isSubset.Int0.Empty	118	120	119	—
Set.subtracting.Box0.Empty	0	0	0	—
Set.subtracting.Empty.Box0	0	0	0	—
Set.subtracting.Empty.Int0	0	0	0	—
Set.subtracting.Int0.Empty	0	0	0	—
Removed
Set.Empty.IsDisjointBox0	1	1	1	—
Set.Empty.IsDisjointInt0	1	1	1	—
Set.Empty.IsSubsetInt0	227	230	228	—
Set.Empty.SubtractingBox0	0	0	0	—
Set.Empty.SubtractingInt0	0	0	0	—
SetIsDisjointBox0	850	851	850	—
SetIsDisjointBox25	7	7	7	—
SetIsDisjointInt0	303	310	305	—
SetIsDisjointInt100	3	3	3	—
SetIsDisjointInt25	3	3	3	—
SetIsDisjointInt50	3	3	3	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	61335	64551	+5.2%	0.95x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Regression
DataToStringSmall	4800	5450	+13.5%	0.88x (?)
Improvement
CharacterPropertiesFetch	6290	5860	-6.8%	1.07x (?)
Added
Set.isDisjoint.Box0	3135	3256	3177	—
Set.isDisjoint.Box0.Empty	8	8	8	—
Set.isDisjoint.Box25	26	26	26	—
Set.isDisjoint.Empty.Box0	5	5	5	—
Set.isDisjoint.Empty.Int0	5	5	5	—
Set.isDisjoint.Int0	1337	1407	1361	—
Set.isDisjoint.Int0.Empty	7	8	7	—
Set.isDisjoint.Int100	10	10	10	—
Set.isDisjoint.Int25	13	14	13	—
Set.isDisjoint.Int50	13	13	13	—
Set.isSubset.Empty.Int0	501	502	501	—
Set.isSubset.Int0.Empty	372	373	372	—
Set.subtracting.Box0.Empty	1	1	1	—
Set.subtracting.Empty.Box0	1	1	1	—
Set.subtracting.Empty.Int0	1	1	1	—
Set.subtracting.Int0.Empty	1	1	1	—
Removed
Set.Empty.IsDisjointBox0	5	5	5	—
Set.Empty.IsDisjointInt0	5	5	5	—
Set.Empty.IsSubsetInt0	500	503	501	—
Set.Empty.SubtractingBox0	1	1	1	—
Set.Empty.SubtractingInt0	1	1	1	—
SetIsDisjointBox0	3137	3204	3160	—
SetIsDisjointBox25	25	26	25	—
SetIsDisjointInt0	1336	1508	1395	—
SetIsDisjointInt100	10	10	10	—
SetIsDisjointInt25	13	14	13	—
SetIsDisjointInt50	13	13	13	—

✅	Benchmark Check Report
⚠️⏱	`Set.isDisjoint.Box25` execution took 5 μs. _{Increase the workload of Set.isDisjoint.Box25 to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Int25` execution took 2 μs. _{Increase the workload of Set.isDisjoint.Int25 to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Int50` execution took 2 μs. _{Increase the workload of Set.isDisjoint.Int50 to be more than 20 μs.}
⚠️⏱	`Set.subtracting.Empty.Int0` execution took 3 μs. _{Increase the workload of Set.subtracting.Empty.Int0 to be more than 20 μs.}
⚠️⏱	`Set.subtracting.Int0.Empty` execution took 3 μs. _{Increase the workload of Set.subtracting.Int0.Empty to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Empty.Box0` execution took 1 μs. _{Increase the workload of Set.isDisjoint.Empty.Box0 to be more than 20 μs.}
⚠️⏱	`Set.subtracting.Empty.Box0` execution took 2 μs. _{Increase the workload of Set.subtracting.Empty.Box0 to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Int0.Empty` execution took 1 μs. _{Increase the workload of Set.isDisjoint.Int0.Empty to be more than 20 μs.}
⚠️⏱	`Set.subtracting.Box0.Empty` execution took 2 μs. _{Increase the workload of Set.subtracting.Box0.Empty to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Int100` execution took 1 μs. _{Increase the workload of Set.isDisjoint.Int100 to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Box0.Empty` execution took 2 μs. _{Increase the workload of Set.isDisjoint.Box0.Empty to be more than 20 μs.}
⚠️⏱	`Set.isDisjoint.Empty.Int0` execution took 1 μs. _{Increase the workload of Set.isDisjoint.Empty.Int0 to be more than 20 μs.}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

--------------

Increase the multipliers to get reliably measurable runtimes.

palimondo · 2019-01-09T11:23:30Z

@swift-ci please benchmark

palimondo · 2019-01-09T11:37:12Z

@lorentey @eeckstein Please review 🙏

I'm unsure if adding swapped Set.*.Empty variants makes sense (illustrated with my local results):

TEST	MIN	Q1	MED	Q3	MAX
Set.isDisjoint.Box0.Empty	826	832	837	844	869
Set.isDisjoint.Empty.Box0	446	453	454	458	465
Set.isDisjoint.Empty.Int0	436	448	452	452	460
Set.isDisjoint.Int0.Empty	450	450	454	458	480
Set.isSubset.Empty.Int0	801	810	813	817	832
Set.isSubset.Int0.Empty	425	425	425	437	449
Set.subtracting.Box0.Empty	157	157	157	158	160
Set.subtracting.Empty.Box0	76	76	76	77	77
Set.subtracting.Empty.Int0	216	216	216	219	223
Set.subtracting.Int0.Empty	282	282	282	285	291

Do you want me to keep it?

benchmark/scripts/Benchmark_Driver

Clarified limit of accuracy in setup overhead detection.

palimondo · 2019-01-09T17:01:49Z

@swift-ci smoke test

lorentey

Looks good overall, but I have a minor naming concern and apparently there is a serious performance issue between raw integers and boxes.

benchmark/single-source/SetTests.swift

palimondo · 2019-01-10T17:04:38Z

@swift-ci please benchmark

palimondo · 2019-01-10T17:05:58Z

@swift-ci please smoke test and merge

swift-ci · 2019-01-10T17:48:43Z

Build comment file:

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
ObjectiveCBridgeStubToNSDate2	1258	1359	+8.0%	0.93x (?)
Added
Set.isDisjoint.Box.Empty	225	225	225	—
Set.isDisjoint.Box0	688	689	689	—
Set.isDisjoint.Box25	513	522	516	—
Set.isDisjoint.Empty.Box	117	120	118	—
Set.isDisjoint.Empty.Int	111	114	112	—
Set.isDisjoint.Int.Empty	120	122	121	—
Set.isDisjoint.Int0	296	298	297	—
Set.isDisjoint.Int100	164	167	165	—
Set.isDisjoint.Int25	238	243	240	—
Set.isDisjoint.Int50	236	239	237	—
Set.isSubset.Empty.Int	220	224	221	—
Set.isSubset.Int.Empty	111	114	113	—
Set.subtracting.Box.Empty	41	42	41	—
Set.subtracting.Empty.Box	21	23	22	—
Set.subtracting.Empty.Int	59	60	59	—
Set.subtracting.Int.Empty	81	83	82	—
Removed
Set.Empty.IsDisjointBox0	1	1	1	—
Set.Empty.IsDisjointInt0	1	1	1	—
Set.Empty.IsSubsetInt0	220	221	220	—
Set.Empty.SubtractingBox0	0	0	0	—
Set.Empty.SubtractingInt0	0	1	0	—
SetIsDisjointBox0	691	691	691	—
SetIsDisjointBox25	5	5	5	—
SetIsDisjointInt0	299	301	300	—
SetIsDisjointInt100	1	1	1	—
SetIsDisjointInt25	2	2	2	—
SetIsDisjointInt50	2	2	2	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	71591	75463	+5.4%	0.95x

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataAppendDataLargeToLarge	51000	37000	-27.5%	1.38x (?)
Added
Set.isDisjoint.Box.Empty	242	242	242	—
Set.isDisjoint.Box0	845	845	845	—
Set.isDisjoint.Box25	719	719	719	—
Set.isDisjoint.Empty.Box	121	123	122	—
Set.isDisjoint.Empty.Int	118	119	118	—
Set.isDisjoint.Int.Empty	232	236	235	—
Set.isDisjoint.Int0	303	306	304	—
Set.isDisjoint.Int100	312	312	312	—
Set.isDisjoint.Int25	356	356	356	—
Set.isDisjoint.Int50	347	349	348	—
Set.isSubset.Empty.Int	227	231	228	—
Set.isSubset.Int.Empty	118	120	119	—
Set.subtracting.Box.Empty	41	41	41	—
Set.subtracting.Empty.Box	21	21	21	—
Set.subtracting.Empty.Int	60	67	64	—
Set.subtracting.Int.Empty	82	83	82	—
Removed
Set.Empty.IsDisjointBox0	1	1	1	—
Set.Empty.IsDisjointInt0	1	1	1	—
Set.Empty.IsSubsetInt0	227	230	228	—
Set.Empty.SubtractingBox0	0	0	0	—
Set.Empty.SubtractingInt0	0	0	0	—
SetIsDisjointBox0	850	850	850	—
SetIsDisjointBox25	7	7	7	—
SetIsDisjointInt0	305	307	306	—
SetIsDisjointInt100	2	2	2	—
SetIsDisjointInt25	3	3	3	—
SetIsDisjointInt50	3	3	3	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	61335	64263	+4.8%	0.95x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Improvement
CharIndexing_punctuated_unicodeScalars_Backwards	114760	106640	-7.1%	1.08x (?)
Added
Set.isDisjoint.Box.Empty	795	795	795	—
Set.isDisjoint.Box0	3343	3347	3345	—
Set.isDisjoint.Box25	2846	2871	2855	—
Set.isDisjoint.Empty.Box	554	566	560	—
Set.isDisjoint.Empty.Int	541	545	543	—
Set.isDisjoint.Int.Empty	778	784	782	—
Set.isDisjoint.Int0	1338	1396	1357	—
Set.isDisjoint.Int100	997	998	997	—
Set.isDisjoint.Int25	1343	1389	1358	—
Set.isDisjoint.Int50	1307	1308	1308	—
Set.isSubset.Empty.Int	517	518	517	—
Set.isSubset.Int.Empty	390	390	390	—
Set.subtracting.Box.Empty	171	174	173	—
Set.subtracting.Empty.Box	122	125	124	—
Set.subtracting.Empty.Int	139	142	140	—
Set.subtracting.Int.Empty	189	191	190	—
Removed
Set.Empty.IsDisjointBox0	5	5	5	—
Set.Empty.IsDisjointInt0	5	5	5	—
Set.Empty.IsSubsetInt0	516	516	516	—
Set.Empty.SubtractingBox0	1	1	1	—
Set.Empty.SubtractingInt0	1	1	1	—
SetIsDisjointBox0	3146	3227	3195	—
SetIsDisjointBox25	26	26	26	—
SetIsDisjointInt0	1339	1401	1360	—
SetIsDisjointInt100	10	10	10	—
SetIsDisjointInt25	13	13	13	—
SetIsDisjointInt50	13	13	13	—

✅	Benchmark Check Report
⛔️⏱	`Set.subtracting.Int.Empty` has setup overhead of 6 μs (7.4%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

--------------

palimondo · 2019-01-10T18:58:18Z

@swift-ci please smoke test linux

palimondo added 4 commits January 8, 2019 19:15

[benchmark] Limit setup overhead detection (>20)

8a8a3ad

For really small runtimes < 20 μs this method of setup overhead detection doesn’t work. Even 1μs change in 20μs runtime is 5%. Just return no overhead.

[benchmark] BenchmarkDoctor: Lower runtime limit

2096151

Warn about runtimes under 20 μs and flag 0 μs runtimes as errors.

[benchmark] Adjust new tests to Naming Convention

84580f8

Most of these recently added benchmarks have too low loop multiplier that results in near zero or zero measured runtime. Since fixing this will change the runtimes, it is also an opportunity to properly apply the new naming convention.

[benchmark] Add swapped Set.*.Empty variants

c011bd7

Let’s also test Empty sets as the right-hand side parameter.

palimondo mentioned this pull request Jan 8, 2019

[stdlib] Resolving some FIXME comments on Set type. #20631

Merged

[benchmark] Adjust loop multipliers Set.*

b4d8172

Increase the multipliers to get reliably measurable runtimes.

palimondo requested review from lorentey and eeckstein January 9, 2019 11:23

palimondo changed the title ~~[WIP][benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes~~ [benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes Jan 9, 2019

This comment has been minimized.

Sign in to view

LucianoPAlmeida reviewed Jan 9, 2019

View reviewed changes

benchmark/scripts/Benchmark_Driver Outdated Show resolved Hide resolved

[benchmark] Limit of Accuracy in Setup Overhead

2c27149

Clarified limit of accuracy in setup overhead detection.

lorentey approved these changes Jan 10, 2019

View reviewed changes

benchmark/single-source/SetTests.swift Outdated Show resolved Hide resolved

benchmark/single-source/SetTests.swift Show resolved Hide resolved

[benchmark] Remove 0s from Empty names.

b4f1f47

palimondo merged commit 5cc751c into swiftlang:master Jan 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes #21717

[benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes #21717

Uh oh!

palimondo commented Jan 8, 2019 •

edited

Loading

Uh oh!

palimondo commented Jan 8, 2019

Uh oh!

swift-ci commented Jan 8, 2019

Uh oh!

palimondo commented Jan 9, 2019

Uh oh!

palimondo commented Jan 9, 2019 •

edited

Loading

Uh oh!

This comment has been minimized.

Uh oh!

palimondo commented Jan 9, 2019

Uh oh!

lorentey left a comment

Uh oh!

Uh oh!

Uh oh!

palimondo commented Jan 10, 2019

Uh oh!

palimondo commented Jan 10, 2019

Uh oh!

swift-ci commented Jan 10, 2019

Uh oh!

palimondo commented Jan 10, 2019

Uh oh!

Uh oh!

[benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes #21717

[benchmark] BenchmarkDoctor: Lower runtime bound + Set.Empty fixes #21717

Uh oh!

Conversation

palimondo commented Jan 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

palimondo commented Jan 8, 2019

Uh oh!

swift-ci commented Jan 8, 2019

Build comment file:

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Uh oh!

palimondo commented Jan 9, 2019

Uh oh!

palimondo commented Jan 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

palimondo commented Jan 9, 2019

Uh oh!

lorentey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

palimondo commented Jan 10, 2019

Uh oh!

palimondo commented Jan 10, 2019

Uh oh!

swift-ci commented Jan 10, 2019

Build comment file:

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Uh oh!

palimondo commented Jan 10, 2019

Uh oh!

Uh oh!

palimondo commented Jan 8, 2019 •

edited

Loading

palimondo commented Jan 9, 2019 •

edited

Loading