[benchmark] Fix: Set.*.Box use identity everywhere

[palimondo]

Consistently apply the identity in Box variants of benchmarks for subtracting and intersection to guard against future sufficiently smart compiler that could hoist the constant expression out of the loop. This pattern is used in all other variants and was explained by @lorentey in #18928 (comment).

[palimondo]

@swift-ci please benchmark

[palimondo]

This issue was discovered during tuning of benchmark workloads in #24156.
In my local tests with #21300 applied, these changes were almost undetectable, so I'm running a benchmark on CI to see if we can handle the fix simply like this, or if this would require some legacyFactor/legacyName trickery to maintain the continuity of long term tracking. cc @eeckstein

[swift-ci]

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
Set.subtracting.Empty.Box	9	30	+233.3%	0.30x
Set.subtracting.Box.Empty	24	47	+95.8%	0.51x
Improvement
ObjectiveCBridgeStubToArrayOfNSString2	2720	2520	-7.4%	1.08x (?)

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
Set.subtracting.Empty.Box	9	29	+222.2%	0.31x
Set.subtracting.Box.Empty	24	47	+95.8%	0.51x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Regression
Set.subtracting.Empty.Box	68	81	+19.1%	0.84x (?)
Set.subtracting.Box.Empty	100	112	+12.0%	0.89x (?)

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

[palimondo]

OK, this matches my local results, because the negligible performance changes in SetIntersectionBox0, SetIntersectionBox25, SetSubtractingBox0, SetSubtractingBox25 flew under the radar.

The I'm pretty sure there will be legitimate performance changes in Set.subtracting.Empty.Box and Set.subtracting.Box.Empty when #21300 lands, so I suggest we land this fix at the same time and avoid the need to come up with new name for these benchmarks. What do you think @eeckstein?

[eeckstein]

Can you add a legacyFactor for this change?
If #21300 improves performance we want to see that in the results. If both changes land together we cannot tell how much the #21300 improved performance.

[palimondo]

This fix increases the runtime and because legacyFactor is currently an integer multiplier, it can only inflate the reported time and cannot be used to work around this as is.

I've triggered the CI benchmark in #21300 to be sure, but without fixing this, the runtime dips too low in these two cases (Set.subtracting.Empty.Box, Set.subtracting.Box.Empty) and we would have to adjust the workload multipliers anyway.

[palimondo]

Erm, I misspoke, those two cases regress according to benchmark report (0.15x & 0.60x), but these are marginal cases with 1000x workload multipliers — so this means subtracting the empty set takes 158 ns and 15 ns depending on which side is empty.

[palimondo]

With this fix, these will be even a bit higher (by ratio 0.30x and 0.51x), because the loop will have to deal with identity, but I really wouldn't stress that case at all. As I said, this fix as well as #21300 change the runtime of those two test (both increase), so if we land around the same time, we can pretend this minor regression is all caused by #21300 and we would not need to burn the good benchmark names just to fix this identity omission. Would you agree, @eeckstein?

[palimondo]

@swift-ci please smoke test

[palimondo]

@eeckstein @lorentey Assuming #21300 is ready to be merged, I propose we coordinate the landing of these 3 PRs in following order:

1. [benchmark] Variants for Set Sequence Methods #24156 [benchmark] Variants for Set Sequence Methods
2. [benchmark] Fix: Set.*.Box use identity everywhere #24570 [benchmark] Fix: Set.*.Box use identity everywhere (this one)
3. [stdlib] Optimize some Set operations #21300 [stdlib] Optimize some Set operations

Edit: Karoy informed me that branch is not yet ready to merge, so I'll let this PR wait until it will be...

[shahmishal]

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

• How to change the base branch: (Link)
• More detail about the branch update: (Link)