add benchmark of set isStrictSubset #23690

Gumichocopengin8 · 2019-03-30T20:36:30Z

Add isStrictSubset benchmark of Set because benchmark of isStrictSubset is not written yet.

SetIsStrictSubsetBox25 and SetIsStrictSubsetBox25 could not compile it, so I commented them out.

lorentey · 2019-04-01T21:25:52Z

@swift-ci benchmark

lorentey

Thanks, these will be handy. The changes look good, but please add the boxed variants, too!

benchmark/single-source/SetTests.swift

lorentey · 2019-04-01T22:03:58Z

@swift-ci benchmark

swift-ci · 2019-04-01T22:52:00Z

Build failed before running benchmark.

Gumichocopengin8 · 2019-04-02T01:37:24Z

@swift-ci benchmark

lorentey · 2019-04-02T02:29:42Z

@swift-ci benchmark

lorentey · 2019-04-02T02:31:57Z

@Gumichocopengin8 swift-ci only listens to people with commit access -- but we're happy to trigger tests as needed!

lorentey · 2019-04-02T02:32:08Z

@swift-ci please smoke test

lorentey · 2019-04-02T02:35:41Z

Looks good!

The bot will analyze benchmarks and it will complain if some tests take too long; if that's the case, those need to be fixed before we merge this.

Gumichocopengin8 · 2019-04-02T04:06:36Z

@lorentey
Thank you! I'm happy to contribute to swift. I am going to contribute more.
I think it is a good start for GSoC.

lorentey · 2019-04-08T18:55:06Z

@swift-ci benchmark

Gumichocopengin8 · 2019-04-09T02:33:09Z

Do I still need to fix something to run benchmark without error?

lorentey · 2019-04-09T22:08:58Z

Hm; it looks like it failed while running the original benchmarks. Let's try again!

@swift-ci please benchmark

Gumichocopengin8 · 2019-04-10T17:59:43Z

I don't know why it failed.
Does it sometimes happen?

lorentey · 2019-04-10T20:35:31Z

It may have something to do with the new benchmarks -- benchmarking does not fail for other PRs. I'll take a look in a local build.

Gumichocopengin8 · 2019-04-10T22:52:35Z

Thank you.
I'll work on other issues.

lorentey · 2019-04-10T22:59:48Z

There is indeed an issue with SetIsStrictSubsetInt100:

...
637,SetIsStrictSubsetBox0,1,583,583,583,0,583
638,SetIsStrictSubsetBox25,1,297,297,297,0,297
639,SetIsStrictSubsetInt0,1,1040,1040,1040,0,1040
Incorrect result in run_SetIsStrictSubsetInt(_:_:_:_:), [...]/swift/benchmark/single-source/SetTests.swift:385

@Gumichocopengin8, could you please fix the expected value and double check there aren't any other issues by running the benchmarks on your system?

Gumichocopengin8 · 2019-04-11T00:58:05Z

OK!
I'll fix it and double check the benchmark on my Mac.

Gumichocopengin8 · 2019-04-11T14:39:55Z

I fixed the error and triple checked.
It must work well now :)

lorentey · 2019-04-11T15:42:12Z

@swift-ci benchmark

lorentey · 2019-04-11T15:42:28Z

@swift-ci smoke test

swift-ci · 2019-04-11T16:11:21Z

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
ObjectiveCBridgeStubFromNSDateRef	4060	4370	+7.6%	0.93x (?)
Improvement
CharacterLiteralsLarge	108	97	-10.2%	1.11x
Added
Set.isStrictSubset.Empty.Int	170	172	171	—
Set.isStrictSubset.Int.Empty	90	92	91	—
SetIsStrictSubsetBox0	381	381	381	—
SetIsStrictSubsetBox25	164	168	165	—
SetIsStrictSubsetInt0	265	267	266	—
SetIsStrictSubsetInt100	599	633	614	—
SetIsStrictSubsetInt25	78	80	79	—
SetIsStrictSubsetInt50	154	156	155	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	72118	79574	+10.3%	0.91x

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
CharacterLiteralsLarge	100	111	+11.0%	0.90x (?)
Added
Set.isStrictSubset.Empty.Int	180	183	181	—
Set.isStrictSubset.Int.Empty	100	101	100	—
SetIsStrictSubsetBox0	375	375	375	—
SetIsStrictSubsetBox25	255	259	256	—
SetIsStrictSubsetInt0	268	271	269	—
SetIsStrictSubsetInt100	590	592	591	—
SetIsStrictSubsetInt25	75	76	75	—
SetIsStrictSubsetInt50	148	150	149	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	64230	71398	+11.2%	0.90x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Regression
DataAppendDataLargeToLarge	37200	51000	+37.1%	0.73x (?)
Added
Set.isStrictSubset.Empty.Int	576	577	576	—
Set.isStrictSubset.Int.Empty	367	368	367	—
SetIsStrictSubsetBox0	934	935	935	—
SetIsStrictSubsetBox25	858	859	858	—
SetIsStrictSubsetInt0	622	632	626	—
SetIsStrictSubsetInt100	1500	1560	1526	—
SetIsStrictSubsetInt25	248	257	254	—
SetIsStrictSubsetInt50	466	467	467	—

✅	Benchmark Check Report
⚠️🔤	`SetIsStrictSubsetInt50` name is composed of 5 words. _{Split SetIsStrictSubsetInt50 name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️🔤	`SetIsStrictSubsetInt25` name is composed of 5 words. _{Split SetIsStrictSubsetInt25 name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️🔤	`SetIsStrictSubsetBox25` name is composed of 5 words. _{Split SetIsStrictSubsetBox25 name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️🔤	`SetIsStrictSubsetInt0` name is composed of 5 words. _{Split SetIsStrictSubsetInt0 name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️🔤	`SetIsStrictSubsetBox0` name is composed of 5 words. _{Split SetIsStrictSubsetBox0 name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️🔤	`SetIsStrictSubsetInt100` name is composed of 5 words. _{Split SetIsStrictSubsetInt100 name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

Gumichocopengin8 · 2019-04-11T17:15:50Z

As Benchmark Check Report mentions, benchmark names should follow the convention.
I will change the names. Just add . between chunks .
Example: SetIntersectionBox25 -> Set.intersection.Box25

lorentey

We shouldn't rename the existing benchmarks.

lorentey · 2019-04-15T21:56:48Z

benchmark/single-source/SetTests.swift

@@ -68,32 +68,32 @@ public let SetTests = [
    tags: [.validation, .api, .Set],
    setUpFunction: { blackHole([setAB, setE]) }),
  BenchmarkInfo(
-    name: "SetIsSubsetInt0",


Unfortunately we shouldn't change the names of existing benchmarks here -- we're using these to track performance across compiler releases.

It is a good idea to proper names for the new benchmarks, though.

Thank you for your advice.
I fixed it.

lorentey · 2019-04-16T01:36:08Z

Looks good, thanks @Gumichocopengin8!

lorentey · 2019-04-16T01:36:23Z

@swift-ci benchmark

lorentey · 2019-04-16T01:36:32Z

@swift-ci smoke test

swift-ci · 2019-04-16T02:07:17Z

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
CharacterLiteralsLarge	97	108	+11.3%	0.90x
Improvement
FlattenListLoop	4709	4335	-7.9%	1.09x (?)
Array2D	8112	7520	-7.3%	1.08x
RemoveWhereSwapInts	70	65	-7.1%	1.08x
MapReduce	426	397	-6.8%	1.07x (?)
FlattenListFlatMap	7007	6539	-6.7%	1.07x (?)
MapReduceAnyCollection	426	398	-6.6%	1.07x (?)
Added
Set.isStrictSubset.Box0	381	382	382	—
Set.isStrictSubset.Box25	165	169	166	—
Set.isStrictSubset.Empty.Int	172	176	173	—
Set.isStrictSubset.Int.Empty	95	101	98	—
Set.isStrictSubset.Int0	261	263	262	—
Set.isStrictSubset.Int100	600	601	600	—
Set.isStrictSubset.Int25	77	78	77	—
Set.isStrictSubset.Int50	153	155	154	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	72118	80038	+11.0%	0.90x

Performance: -Osize

TEST	MIN	MAX	MEAN	MAX_RSS
Added
Set.isStrictSubset.Box0	375	375	375	—
Set.isStrictSubset.Box25	258	262	259	—
Set.isStrictSubset.Empty.Int	181	185	182	—
Set.isStrictSubset.Int.Empty	101	107	103	—
Set.isStrictSubset.Int0	268	276	272	—
Set.isStrictSubset.Int100	595	597	596	—
Set.isStrictSubset.Int25	76	79	77	—
Set.isStrictSubset.Int50	148	152	149	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	64230	71446	+11.2%	0.90x

Performance: -Onone

TEST	MIN	MAX	MEAN	MAX_RSS
Added
Set.isStrictSubset.Box0	865	865	865	—
Set.isStrictSubset.Box25	861	881	868	—
Set.isStrictSubset.Empty.Int	585	586	585	—
Set.isStrictSubset.Int.Empty	380	415	392	—
Set.isStrictSubset.Int0	628	633	631	—
Set.isStrictSubset.Int100	1499	1546	1515	—
Set.isStrictSubset.Int25	248	251	249	—
Set.isStrictSubset.Int50	472	472	472	—

✅	Benchmark Check Report

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

Gumichocopengin8 · 2019-04-16T16:01:48Z

Need to run test again??

lorentey · 2019-04-16T18:03:59Z

Apparently so...

@swift-ci please smoke test macOS platform

add benchmark of set isStrictSubset

db45db3

lorentey requested changes Apr 1, 2019

View reviewed changes

benchmark/single-source/SetTests.swift Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

fix SetIsStrictSubsetBox

dc35738

lorentey approved these changes Apr 2, 2019

View reviewed changes

Gumichocopengin8 added 2 commits April 10, 2019 21:18

Merge remote-tracking branch 'origin' into add-set-benchmark

8918934

bug fixes

310bbaf

change benchmark names to follow naming convention

35cab86

lorentey requested changes Apr 15, 2019

View reviewed changes

fix benchmark names

699827d

lorentey approved these changes Apr 16, 2019

View reviewed changes

lorentey merged commit 09da304 into swiftlang:master Apr 17, 2019

This was referenced May 7, 2019

[benchmark] Variants for Set Sequence Methods #24156

Merged

[benchmark] Disable some new Set tests with overlong runtime #24725

Merged

Gumichocopengin8 deleted the add-set-benchmark branch October 6, 2020 01:55

add benchmark of set isStrictSubset #23690

add benchmark of set isStrictSubset #23690

Uh oh!

Conversation

Gumichocopengin8 commented Mar 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorentey commented Apr 1, 2019

Uh oh!

lorentey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

lorentey commented Apr 1, 2019

Uh oh!

swift-ci commented Apr 1, 2019

Uh oh!

Gumichocopengin8 commented Apr 2, 2019

Uh oh!

lorentey commented Apr 2, 2019

Uh oh!

lorentey commented Apr 2, 2019

Uh oh!

lorentey commented Apr 2, 2019

Uh oh!

lorentey commented Apr 2, 2019

Uh oh!

Gumichocopengin8 commented Apr 2, 2019

Uh oh!

lorentey commented Apr 8, 2019

Uh oh!

Gumichocopengin8 commented Apr 9, 2019

Uh oh!

lorentey commented Apr 9, 2019

Uh oh!

Gumichocopengin8 commented Apr 10, 2019

Uh oh!

lorentey commented Apr 10, 2019

Uh oh!

Gumichocopengin8 commented Apr 10, 2019

Uh oh!

lorentey commented Apr 10, 2019

Uh oh!

Gumichocopengin8 commented Apr 11, 2019

Uh oh!

Gumichocopengin8 commented Apr 11, 2019

Uh oh!

lorentey commented Apr 11, 2019

Uh oh!

lorentey commented Apr 11, 2019

Uh oh!

swift-ci commented Apr 11, 2019

Performance: -O

Code size: -O

Performance: -Osize

Code size: -Osize

Performance: -Onone

Uh oh!

Gumichocopengin8 commented Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lorentey left a comment

Choose a reason for hiding this comment

Uh oh!

lorentey Apr 15, 2019

Choose a reason for hiding this comment

Uh oh!

Gumichocopengin8 Apr 15, 2019

Choose a reason for hiding this comment

Uh oh!

lorentey commented Apr 16, 2019

Uh oh!

lorentey commented Apr 16, 2019

Uh oh!

lorentey commented Apr 16, 2019

Uh oh!

swift-ci commented Apr 16, 2019

Performance: -O

Code size: -O

Gumichocopengin8 commented Mar 30, 2019 •

edited

Loading

Gumichocopengin8 commented Apr 11, 2019 •

edited

Loading

Gumichocopengin8 commented Apr 16, 2019 •

edited

Loading