[interop] cxx vector benchmark - reenable without SwiftPM support #61565

hyp · 2022-10-13T16:55:17Z

No description provided.

hyp · 2022-10-13T16:55:24Z

@swift-ci please test

hyp · 2022-10-20T00:22:47Z

@swift-ci please test linux platform

hyp · 2022-10-20T00:22:54Z

@swift-ci please build macOS toolchain

hyp · 2022-10-20T06:14:47Z

@swift-ci please build toolchain

hyp · 2022-10-20T20:47:55Z

@swift-ci please test linux platform

hyp · 2022-10-20T20:48:06Z

@swift-ci please build toolchain

…workaround module serialization issue

hyp · 2022-10-20T23:15:03Z

@swift-ci please test linux platform

hyp · 2022-10-20T23:15:10Z

@swift-ci please build toolchain

hyp · 2022-10-21T15:08:17Z

@swift-ci please test

hyp · 2022-10-21T15:08:24Z

@swift-ci please benchmark

hyp · 2022-10-21T17:49:12Z

@swift-ci please benchmark

hyp · 2022-11-01T20:34:26Z

@swift-ci please benchmark

hyp · 2022-11-02T21:49:24Z

@swift-ci please benchmark

hyp · 2022-11-02T21:49:29Z

@swift-ci please test

hyp · 2022-11-02T21:49:36Z

@swift-ci please build toolchain

hyp · 2022-11-03T00:21:40Z

Results:

Performance (x86_64): -O

Regression	OLD	NEW	DELTA	RATIO
FlattenListLoop	976	1615	+65.5%	0.60x (?)
LineSink.characters.alpha	67	84	+25.4%	0.80x (?)
Dictionary3	121	134	+10.7%	0.90x (?)
NSError	101	109	+7.9%	0.93x (?)

Added	MIN	MAX	MEAN	MAX_RSS
CxxVecU32.sum.Cxx.rangedForLoop	11	11	11	—
CxxVecU32.sum.Swift.forInLoop	24	24	24	—
CxxVecU32.sum.Swift.iteratorLoop	11	11	11	—
CxxVecU32.sum.Swift.reduce	24	24	24	—
CxxVecU32.sum.Swift.subscriptLoop	11	11	11	—

Code size: -O

Regression	OLD	NEW	DELTA	RATIO
CxxVectorSum.o	78	10256	+13048.6%	0.01x

Performance (x86_64): -Osize

Regression	OLD	NEW	DELTA	RATIO
FlattenListFlatMap	2452	3879	+58.2%	0.63x (?)
Dictionary4	159	201	+26.4%	0.79x (?)
ArrayInClass	195	215	+10.3%	0.91x (?)
PrefixAnySeqCRangeIter	13	14	+7.7%	0.93x (?)
PrefixWhileAnySequence	185	199	+7.6%	0.93x (?)

Added	MIN	MAX	MEAN	MAX_RSS
CxxVecU32.sum.Cxx.rangedForLoop	81	81	81	—
CxxVecU32.sum.Swift.forInLoop	62	63	62	—
CxxVecU32.sum.Swift.iteratorLoop	81	81	81	—
CxxVecU32.sum.Swift.reduce	61	62	61	—
CxxVecU32.sum.Swift.subscriptLoop	81	81	81	—

Code size: -Osize

Regression	OLD	NEW	DELTA	RATIO
CxxVectorSum.o	70	8618	+12211.3%	0.01x

Performance (x86_64): -Onone

Regression	OLD	NEW	DELTA	RATIO
ObjectiveCBridgeStubNSDateRefAccess	3175	3555	+12.0%	0.89x (?)

Added	MIN	MAX	MEAN	MAX_RSS
CxxVecU32.sum.Cxx.rangedForLoop	1584	1614	1595	—
CxxVecU32.sum.Swift.forInLoop	7760	8695	8184	—
CxxVecU32.sum.Swift.iteratorLoop	1890	2163	2021	—
CxxVecU32.sum.Swift.reduce	9473	9708	9554	—
CxxVecU32.sum.Swift.subscriptLoop	1783	1894	1820	—

Code size: -swiftlibs

✅	Benchmark Check Report
⚠️🔤	`CxxVecU32.sum.Swift.forInLoop` name is composed of 5 words. _{Split CxxVecU32.sum.Swift.forInLoop name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⛔️⏱	`CxxVecU32.sum.Swift.forInLoop` has setup overhead of 6 μs (23.1%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}
⚠️🔤	`CxxVecU32.sum.Cxx.rangedForLoop` name is composed of 5 words. _{Split CxxVecU32.sum.Cxx.rangedForLoop name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️⏱	`CxxVecU32.sum.Cxx.rangedForLoop` execution took 10 μs. _{Increase the workload of CxxVecU32.sum.Cxx.rangedForLoop to be more than 20 μs.}
⚠️⏱	`CxxVecU32.sum.Swift.iteratorLoop` execution took 11 μs. _{Increase the workload of CxxVecU32.sum.Swift.iteratorLoop to be more than 20 μs.}
⚠️⏱	`CxxVecU32.sum.Swift.subscriptLoop` execution took 11 μs. _{Increase the workload of CxxVecU32.sum.Swift.subscriptLoop to be more than 20 μs.}
⛔️⏱	`CxxVecU32.sum.Swift.reduce` has setup overhead of 6 μs (23.1%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}
⚠️Ⓜ️	`CxxVecU32.sum.Swift.reduce` has very wide range of memory used between independent, repeated measurements. _{CxxVecU32.sum.Swift.reduce mem_pages [i1, i2]: min=[55, 55] 𝚫=0 R=[0, 57]}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: 6-Core Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 32 GB

hyp · 2022-11-03T14:56:30Z

@swift-ci please test macOS platform

hyp · 2022-11-03T17:40:48Z

@swift-ci please test

hyp · 2022-11-03T17:40:57Z

@swift-ci please benchmark

hyp · 2022-11-03T17:41:06Z

@swift-ci please build toolchain

…uffer

hyp · 2022-11-03T18:56:37Z

@swift-ci please benchmark

hyp · 2022-11-03T18:56:46Z

@swift-ci please test

hyp · 2022-11-03T18:56:54Z

@swift-ci please build toolchain

hyp · 2022-11-03T20:11:23Z

Results:

Performance (x86_64): -O

Regression	OLD	NEW	DELTA	RATIO
LineSink.characters.alpha	67	85	+26.9%	0.79x
FlattenListLoop	1128	1222	+8.3%	0.92x (?)

Improvement	OLD	NEW	DELTA	RATIO
FlattenListFlatMap	4553	3931	-13.7%	1.16x (?)

Added	MIN	MAX	MEAN	MAX_RSS
CxxVecU32.sum.Cxx.rangedForLoop	9	9	9	—
CxxVecU32.sum.Swift.forInLoop	21	21	21	—
CxxVecU32.sum.Swift.iteratorLoop	12	13	13	—
CxxVecU32.sum.Swift.reduce	25	33	28	—
CxxVecU32.sum.Swift.subscriptLoop	10	13	12	—

Code size: -O

Regression	OLD	NEW	DELTA	RATIO
CxxVectorSum.o	78	10256	+13048.6%	0.01x

Performance (x86_64): -Osize

Regression	OLD	NEW	DELTA	RATIO
FlattenListLoop	975	1380	+41.5%	0.71x (?)
Dictionary4	157	203	+29.3%	0.77x (?)
StringWalk	1240	1360	+9.7%	0.91x (?)
PrefixWhileAnySequence	185	199	+7.6%	0.93x (?)
PrefixWhileSequence	185	199	+7.6%	0.93x (?)

Improvement	OLD	NEW	DELTA	RATIO
PrefixAnySeqCntRange	16	14	-12.5%	1.14x (?)

Added	MIN	MAX	MEAN	MAX_RSS
CxxVecU32.sum.Cxx.rangedForLoop	76	78	77	—
CxxVecU32.sum.Swift.forInLoop	58	59	58	—
CxxVecU32.sum.Swift.iteratorLoop	76	77	76	—
CxxVecU32.sum.Swift.reduce	60	61	60	—
CxxVecU32.sum.Swift.subscriptLoop	76	76	76	—

Code size: -Osize

Regression	OLD	NEW	DELTA	RATIO
CxxVectorSum.o	70	8618	+12211.3%	0.01x

Performance (x86_64): -Onone

Added	MIN	MAX	MEAN	MAX_RSS
CxxVecU32.sum.Cxx.rangedForLoop	1697	1702	1699	—
CxxVecU32.sum.Swift.forInLoop	7612	8061	7876	—
CxxVecU32.sum.Swift.iteratorLoop	1777	1832	1808	—
CxxVecU32.sum.Swift.reduce	9524	9753	9612	—
CxxVecU32.sum.Swift.subscriptLoop	1774	1813	1790	—

Code size: -swiftlibs

✅	Benchmark Check Report
⚠️⏱	`CxxVecU32.sum.Swift.subscriptLoop` execution took 11 μs. _{Increase the workload of CxxVecU32.sum.Swift.subscriptLoop to be more than 20 μs.}
⚠️🔤	`CxxVecU32.sum.Swift.forInLoop` name is composed of 5 words. _{Split CxxVecU32.sum.Swift.forInLoop name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⛔️⏱	`CxxVecU32.sum.Swift.forInLoop` has setup overhead of 2 μs (8.0%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}
⚠️⏱	`CxxVecU32.sum.Swift.iteratorLoop` execution took 10 μs. _{Increase the workload of CxxVecU32.sum.Swift.iteratorLoop to be more than 20 μs.}
⚠️🔤	`CxxVecU32.sum.Cxx.rangedForLoop` name is composed of 5 words. _{Split CxxVecU32.sum.Cxx.rangedForLoop name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming}
⚠️⏱	`CxxVecU32.sum.Cxx.rangedForLoop` execution took 10 μs. _{Increase the workload of CxxVecU32.sum.Cxx.rangedForLoop to be more than 20 μs.}
⛔️⏱	`CxxVecU32.sum.Swift.reduce` has setup overhead of 4 μs (15.4%). _{Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.}

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: 6-Core Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 32 GB

hyp · 2022-11-04T14:43:36Z

@swift-ci please test windows platform

hyp added the c++ interop Feature: Interoperability with C++ label Oct 13, 2022

hyp marked this pull request as ready for review October 13, 2022 16:55

hyp force-pushed the eng/benchmark-relinux branch from 99694b0 to 63691a1 Compare October 20, 2022 00:22

hyp force-pushed the eng/benchmark-relinux branch from 63691a1 to b54434f Compare October 20, 2022 20:47

[interop] cxx vector benchmark - reenable it only outside SwiftPM to …

029ec7a

…workaround module serialization issue

hyp force-pushed the eng/benchmark-relinux branch from b54434f to 029ec7a Compare October 20, 2022 23:14

hyp changed the title ~~[interop] cxx vector benchmark - reenable linux after #61547 got fixed~~ [interop] cxx vector benchmark - reenable without SwiftPM support Oct 21, 2022

hyp force-pushed the eng/benchmark-relinux branch from d335ffd to d09f2c9 Compare November 3, 2022 17:40

[benchmark] markdown report handler - write encoded message to byte b…

cd63444

…uffer

hyp force-pushed the eng/benchmark-relinux branch from d09f2c9 to cd63444 Compare November 3, 2022 18:55

hyp merged commit 4e36e21 into swiftlang:main Nov 4, 2022

[interop] cxx vector benchmark - reenable without SwiftPM support #61565

[interop] cxx vector benchmark - reenable without SwiftPM support #61565

Uh oh!

Conversation

hyp commented Oct 13, 2022

Uh oh!

hyp commented Oct 13, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 20, 2022

Uh oh!

hyp commented Oct 21, 2022

Uh oh!

hyp commented Oct 21, 2022

Uh oh!

hyp commented Oct 21, 2022

Uh oh!

hyp commented Nov 1, 2022

Uh oh!

hyp commented Nov 2, 2022

Uh oh!

hyp commented Nov 2, 2022

Uh oh!

hyp commented Nov 2, 2022

Uh oh!

hyp commented Nov 3, 2022

Performance (x86_64): -O

Code size: -O

Performance (x86_64): -Osize

Code size: -Osize

Performance (x86_64): -Onone

Code size: -swiftlibs

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Uh oh!

hyp commented Nov 3, 2022

Performance (x86_64): -O

Code size: -O

Performance (x86_64): -Osize

Code size: -Osize

Performance (x86_64): -Onone

Code size: -swiftlibs

Uh oh!

hyp commented Nov 4, 2022

Uh oh!

Uh oh!