[AutoDiff] differentiation benchmarks

[marcrasi]

Adds some differentiation benchmarks.

I have tagged them .regression because I don't yet have any specific codegen/optimization operations in mind to test. When we start looking more closely at the performance of differentiation and identifying specific operations that are important for the performance, we can add .validation benchmarks.

@eeckstein could you check if I've used the framework correctly? (Or redirect me to someone else who can review this?)

[marcrasi]

@swift-ci Please benchmark

[marcrasi]

@swift-ci Please benchmark

[marcrasi]

@swift-ci Please benchmark

[marcrasi]

@swift-ci Please benchmark

[marcrasi]

Oh no, this is failing with a non-informative error: 17:20:52 subprocess.CalledProcessError: Command '['/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/benchmark_pr/bin/Benchmark_O', '--list', '--delim=\t']' returned non-zero exit status -6.

I have only tested this on Linux. Maybe it breaks on macos for some reason. I'll build it on my mac and see if I can get it to work on that.

[rxwei]

Nice, I think missing this caused my rpath errors as well!

[marcrasi]

I found a fix that may get the differentiation benchmark working on macos. But I'm a bit unsure about it, so I'll open a separate PR to discuss the fix. I've converted this PR to a draft in the meantime.

[marcrasi]

@swift-ci please benchmark

[marcrasi]

@swift-ci please benchmark

[swift-ci]

Performance: -O

Regression	OLD	NEW	DELTA	RATIO
DataCreateEmpty	80	100	+25.0%	0.80x (?)
ReversedArray2	95	110	+15.8%	0.86x (?)
CharacterLiteralsLarge	63	71	+12.7%	0.89x (?)
CharacterLiteralsSmall	198	213	+7.6%	0.93x (?)
Improvement	OLD	NEW	DELTA	RATIO
RemoveWhereMoveInts	19	17	-10.5%	1.12x (?)
ArrayPlusEqualFiveElementCollection	4847	4403	-9.2%	1.10x (?)
SuffixSequence	305	278	-8.9%	1.10x (?)
ArraySetElement	284	262	-7.7%	1.08x (?)
PrefixWhileSequence	177	165	-6.8%	1.07x (?)
MapReduceSequence	397	371	-6.5%	1.07x (?)
Added	MIN	MAX	MEAN	MAX_RSS
DifferentiationArraySum	1295	1546	1380	—
DifferentiationIdentity	288	289	289	—
DifferentiationSquare	493	503	496	—

Code size: -O

Regression	OLD	NEW	DELTA	RATIO
TestsUtils.o	28824	29348	+1.8%	0.98x
Improvement	OLD	NEW	DELTA	RATIO
DriverUtils.o	137593	131449	-4.5%	1.05x

Performance: -Osize

Regression	OLD	NEW	DELTA	RATIO
NopDeinit	9000	10100	+12.2%	0.89x
ObjectiveCBridgeStubToNSDateRef	2120	2280	+7.5%	0.93x (?)
Added	MIN	MAX	MEAN	MAX_RSS
DifferentiationArraySum	1305	1391	1338	—
DifferentiationIdentity	287	300	294	—
DifferentiationSquare	489	502	495	—

Code size: -Osize

Performance: -Onone

Added	MIN	MAX	MEAN	MAX_RSS
DifferentiationArraySum	1323	1475	1375	—
DifferentiationIdentity	314	319	316	—
DifferentiationSquare	514	530	520	—

Code size: -swiftlibs

✅	Benchmark Check Report
⚠️⏱	DifferentiationArraySum execution took at least 1217 μs. Decrease the workload of DifferentiationArraySum by a factor of 2 (10), to be less than 1000 μs.

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: 6-Core Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

[marcrasi]

Note to self: Add benchmarks for differentiableMap and differentiableReduce before undraftifying this, because they showed up as big bottlenecks in some real code recently.

[shahmishal]

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

• How to change the base branch: (Link)
• More detail about the branch update: (Link)