[benchmark] Variants for Set Sequence Methods

[Gumichocopengin8]

Added benchmarks for Set.
To support this: #24156 (review)

Some Set operations have generic variants that take a sequence -- e.g., Set.isSuperset<S: Sequence>(of: S). These are currently not covered by benchmarks at all, even though we're planning to improve them soon! I'd very much welcome benchmarks that measure those.

[Gumichocopengin8]

CC: @lorentey
Could you please review this?

[lorentey]

Hm, I'm not sure we need these benchmarks.

• The non-generic isSuperset/isStrictSuperset are trivial wrappers around isSubset/isStrictSubset, and I don't expect they will ever change. (Or rather, they will remain mostly in sync with isSubset when they do.)

• The mutation tests don't seem too useful in their current form, since they include a COW copy on every iteration, and I expect that will often swamp the actual operation.

Some Set operations have generic variants that take a sequence -- e.g., Set.isSuperset<S: Sequence>(of: S). These are currently not covered by benchmarks at all, even though we're planning to improve them soon! I'd very much welcome benchmarks that measure those.

[lorentey]

@swift-ci benchmark

[Gumichocopengin8]

Thank you for reviewing.
I also wondered if isSuperset/isStrictSuperset and mutation functions are not necessary because they are almost the same as isSubset/isStrictSubset. Thanks for the confirmation. I'll revert them.

Then, I'll implement benchmark of Set.isSuperset<S: Sequence>(of: S) and more.

Improving set is amazing. I wish I could have skills like you...

[Gumichocopengin8]

@lorentey
To implement Set.isSuperset<S: Sequence>(of: S), should I use array or sequence?
I'm going to use array instead of sequence because sequence doesn't support reserveCapacity.
Also, document uses array too.
If there's problem to use array, please give me your advice.

[Gumichocopengin8]

@lorentey
Could you please review this?
I didn't implement filter function because this function works differently from others.
Could you give me tips to implement filter benchmark? <- I added filter benchmark.

[Gumichocopengin8]

@lorentey @palimondo @airspeedswift Could you please review this when you have time?

[palimondo]

Could you please use a shortened Seq to designate the sequence variants? I.e. Set.isStrictSuperset.Seq.Box25. (Just for consistency with a planned future rename of other Sequence benchmarks that will be using Seq to refer to the type, as illustrated in the examples under section Use groups and variants in the Naming Convention).

I don't see what are the Filter benchmark variants doing here at all.

It seems to me there are still missing benchmarks for 2 generic methods that take a Sequence parameter:

• union
• symmetricDifference

I'm assuming @lorentey wants to add these too...

[Gumichocopengin8]

@palimondo Thank you for reviewing. I fixed code.
Could you please review it again?

[palimondo]

Oh, OK, I understand now that this set of new benchmarks should be tailored specifically for the pending improvements in #21300. Then it makes sense not to include symmetricDifference and union families because they are not affected by those changes, even though a generic Sequence consuming variant exists in the Set API. Sorry @Gumichocopengin8, could you please remove them again and edit the PR description to clearly document the purpose of these benchmarks?

As for the filter family, I still think it would make sense to take inspiration from DictionaryCopy and do the four 16k–28k variants for each of Int50 (filter matches only even numbers) and Int100 (filter matches all) and leave out the other variants.

So we’d have Set.filter.Int100.16k (.20k, .24k, .28k) with “match all” filter and Set.filter.Int50.16k (etc.) with “match even” filter.

What do you think @lorentey?

[Gumichocopengin8]

@palimondo
I fixed filter benchmark, and delete symmetricDifference and union .
I also added filter.Box100 and filter.Box50.
I believe box is significant in this case.
Could you review this?

[palimondo]

@swift-ci please benchmark

[Gumichocopengin8]

I got warnings.
Should I fix them all?
If so, how can you usually fix them?

[palimondo]

I got warnings. Should I fix them all?

Not yet. The memory warnings are safe to ignore in these cases. But we need to discuss the surprises…

In principle, it is not a showstopper when some benchmarks are over the recommended runtime threshold of 1000 μs when there is pending improvement which will bring the runtime down. Going over 10 000 μs (10 ms) is much more serious issue, because this crosses the scheduler quantum on mac os — this means that every sample is guaranteed to include measurement error from a context switch which leads to very noisy results with sporadic false changes in our reports.

We should aim for these benchmarks to run in reasonable times after #21300 lands, buts some of these results are disturbingly high. So high that performance improvements @lorentey mentions in #21300 will not get the runtimes of these benchmark to the ranges we need for stable measurements. I'm guessing that the sequence iterators are not specialized? This needs more investigation. It might be issue with how the benchmarks are written or there might be much more fundamental issue with resilient API after ABI stability. I'll try to profile these locally. Edit: These were just copy/paste errors: wrong multipliers.

I don't really see the need to add filter.Box variants, as they do not add new information on top of what filter.Int does (they are all just a constant slowdown of 0.36x), but that is ultimately @lorentey's call.

[palimondo]

I still have a few minor points about Int100 comparisons and a nitpick about variable naming in filter case.

[palimondo]

The reason for those abnormally high runtimes are incorrect multipliers. In this benchmark family, we are using 5000 only for Int0, Box0 and Empty to inflate the workload for the case where the operation is almost nothing — to make it observable. All other cases use 50.

This works for set-to-set comparisons in both directions, because their size is known. The size of a Sequence is unknown before it is fully consumed. So we need to handle the set-to-sequence Int0 and Box0 variants depending on their runtime. 50 is better multiplier constant for those cases.

Edit: I forgot to comment on these, which should also use 50:

• Set.isSubset.Seq.[Int0, Box0]
• Set.isStrictSubset.Seq.[Int0, Box0]
• Set.isStrictSuperset.Seq.[Int0, Box0]

[palimondo]

These two runtime outliers are interesting:

• Set.isStrictSubset.Seq.Empty.Int
• Set.isStrictSuperset.Seq.Int.Empty

because they indicate we might benefit from special casing the empty set in the _NativeSet overrides. @lorentey?

[palimondo]

@Gumichocopengin8 Tip: You can apply the patch from PR 21300 locally to verify the final runtimes will fall, if possible into the 20–1000 μs range (adjusting for your machine's relative performance vis-a-vis the Mac Pros that are running the CI here).

[Gumichocopengin8]

@palimondo
I fixed and adjusted benchmark for PR 21300.

Edit: I forgot to comment on these, which should also use 50:
Set.isSubset.Seq.[Int0, Box0]
Set.isStrictSubset.Seq.[Int0, Box0]
Set.isStrictSuperset.Seq.[Int0, Box0]

Basically, I followed your instruction above. However, I didn't apply it sometimes in order to adjust benchmark for PR 21300.

[palimondo]

@Gumichocopengin8 We've overlooked one more workload multiplier. Could you please fix it and force-push again?

[palimondo]

@eeckstein, @lorentey How soon can we be landing the #21300? This additional performance coverage is tuned (in terms of workload scaling) for those improvements. Otherwise we'll have 4 benchmarks with really suboptimal runtimes:

A recent addition (21 days ago in #23690) Set.isStrictSubset.Int100 shows up as regression, because we are fixing up the workload multiplier to account for the new short circuiting optimization.

Similarly, these rely on the newly optimized handling and currently run at over 30k μs:

• Set.isStrictSuperset.Seq.Int0 (similar short-circuiting optimization as mentioned above)
• Set.isStrictSuperset.Seq.Int.Empty
• Set.isStrictSubset.Seq.Empty.Int

Are you OK with landing it this way now?

[Gumichocopengin8]

@palimondo
Do we need to force push commit? I heard we can squash commits when we merge them.

[palimondo]

@Gumichocopengin8 Hmm… I see there is an option to do that in the dropdown next to Merge pull request button 💡. I've never used that before. Theoretically that might have worked... but I don't know who would be credited as author then and according to this description this feature must be allowed for the repository (and I don't know if/how that's configured for apple/swift).

Just to clarify, it is not necessary to squash commits into single one for a PR to be merged here. It is in fact beneficial to have many small and semantically single-themed commits if you are doing non-trivial multi-step changes such as refactoring, to ease the reviewer's job. But it is good practice to clean up the fits and starts (all the reverted dead ends etc.) and not pollute the history with unnecessary distractions. So thank you again for your patience with me, you did excellent work here!

I wanted to wait for a green-light from @lorentey and @eeckstein (I'm assuming many Apple devs are quite busy preparing for WWDC these days), but I guess I could merge this myself and ask for forgiveness if they'd disagree with my judgement. But I have noticed something we should change before accepting this. I'll do a re-review later today.

[palimondo]

I think we should stack the deck a bit more in favor of #21300 to showcase how its optimizations don't have to consume the entire Sequence because they provide early exits as soon as all the required conditions are met. I did one more review pass, comparing the original Set.Set benchmarks with the the newly added Set.Sequence. You have faithfully translated the arguments for isSubset and isStrictSubset 👍. But to better showcase the optimizations, we should be comparing against the next-one-up arrays (they have same size and same intersection, but their set difference is at the end of the sequence, instead of the beginning):

• Int25 - use setB with arrayBC (instead of arrayAB); analogously for Box25
• Int50 - use setY with arrayYZ (instead of arrayXY).

This is similar argument swapping like @lorentey requested in his review before.
(We will not be changing the arguments on Set.Set variants to match these, because it would cause unnecessary performance changes in our long-term tracking.)

Generally we should always have the same size of sequence (400) on the right side for all of these Seq variants. So for the isSuperset.Seq and isStrictSuperset.Seq we also need to swap the arguments and expected results around a bit, again, to allow for early exit from the optimizations.

[palimondo]

@Gumichocopengin8 Will you find a time to do the requested modifications today? If not, I can make them for you, if you want... (I have the commit ready, just let me know.)

[Gumichocopengin8]

@palimondo I fixed them.

[palimondo]

I don’t see the fixes on Box25 and Int50 variants mentioned in the comments.

[Gumichocopengin8]

@palimondo oh sorry. Let me fix them tonight. I don't have time now...

[palimondo]

No problem. Here you go.

[palimondo]

@swift-ci please benchmark

[palimondo]

@swift-ci please benchmark

[palimondo]

@swift-ci please smoke test

[palimondo]

@swift-ci please smoke test os x platform

[Gumichocopengin8]

@palimondo Thanks!

[palimondo]

Regarding the concern about the 4 long running benchmarks from above, I asked @lorentey via email if there was something else, beside the missing benchmarks which this PR provides, that is holding the merge of those optimizations. His response:

#21300 is not ready to merge yet; it needs to be updated to ensure ABI compatibility (it currently lacks availability declarations and runtime checks).
While the enhancements are indeed very nice, it is not super urgent to get this landed. We can cherry pick the isStrictSuperset changes though; the O(n) → O(1) speedup is a trivial change.

[swift-ci]

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
Set.isStrictSubset.Int100	416	41991	+9994.0%	0.01x
SortLettersInPlace	379	417	+10.0%	0.91x (?)
Added
Set.filter.Int100.16k	658	659	658	—
Set.filter.Int100.20k	740	751	744	—
Set.filter.Int100.24k	838	855	844	—
Set.filter.Int100.28k	1242	1404	1296	—
Set.filter.Int50.16k	404	409	406	—
Set.filter.Int50.20k	468	468	468	—
Set.filter.Int50.24k	536	537	537	—
Set.filter.Int50.28k	741	742	742	—
Set.intersection.Seq.Box0	270	277	272	—
Set.intersection.Seq.Box25	359	359	359	—
Set.intersection.Seq.Int0	112	115	113	—
Set.intersection.Seq.Int100	288	303	294	—
Set.intersection.Seq.Int25	163	163	163	—
Set.intersection.Seq.Int50	203	210	205	—
Set.isDisjoint.Seq.Box.Empty	70	75	73	—
Set.isDisjoint.Seq.Box0	481	481	481	—
Set.isDisjoint.Seq.Box25	357	357	357	—
Set.isDisjoint.Seq.Empty.Box	88	88	88	—
Set.isDisjoint.Seq.Empty.Int	92	92	92	—
Set.isDisjoint.Seq.Int.Empty	51	52	52	—
Set.isDisjoint.Seq.Int0	165	173	168	—
Set.isDisjoint.Seq.Int100	153	153	153	—
Set.isDisjoint.Seq.Int25	121	121	121	—
Set.isDisjoint.Seq.Int50	83	83	83	—
Set.isStrictSubset.Seq.Box0	863	863	863	—
Set.isStrictSubset.Seq.Box25	975	977	976	—
Set.isStrictSubset.Seq.Empty.Int	33651	34041	33788	—
Set.isStrictSubset.Seq.Int.Empty	121	121	121	—
Set.isStrictSubset.Seq.Int0	346	346	346	—
Set.isStrictSubset.Seq.Int100	764	765	764	—
Set.isStrictSubset.Seq.Int25	394	395	394	—
Set.isStrictSubset.Seq.Int50	447	447	447	—
Set.isStrictSuperset.Seq.Box0	8601	8675	8638	—
Set.isStrictSuperset.Seq.Box25	860	860	860	—
Set.isStrictSuperset.Seq.Empty.Int	167	168	167	—
Set.isStrictSuperset.Seq.Int.Empty	33595	33983	33777	—
Set.isStrictSuperset.Seq.Int0	3364	3436	3388	—
Set.isStrictSuperset.Seq.Int100	765	766	765	—
Set.isStrictSuperset.Seq.Int25	341	342	342	—
Set.isStrictSuperset.Seq.Int50	340	340	340	—
Set.isSubset.Seq.Box0	858	859	859	—
Set.isSubset.Seq.Box25	973	973	973	—
Set.isSubset.Seq.Empty.Int	88	88	88	—
Set.isSubset.Seq.Int.Empty	123	124	124	—
Set.isSubset.Seq.Int0	345	346	346	—
Set.isSubset.Seq.Int100	556	557	557	—
Set.isSubset.Seq.Int25	394	402	397	—
Set.isSubset.Seq.Int50	447	449	448	—
Set.isSuperset.Seq.Box0	243	248	245	—
Set.isSuperset.Seq.Box25	117	117	117	—
Set.isSuperset.Seq.Empty.Int	49	50	49	—
Set.isSuperset.Seq.Int.Empty	94	94	94	—
Set.isSuperset.Seq.Int0	126	126	126	—
Set.isSuperset.Seq.Int100	175	175	175	—
Set.isSuperset.Seq.Int25	42	42	42	—
Set.isSuperset.Seq.Int50	85	85	85	—
Set.subtracting.Seq.Box.Empty	216	216	216	—
Set.subtracting.Seq.Box0	538	539	539	—
Set.subtracting.Seq.Box25	948	949	948	—
Set.subtracting.Seq.Empty.Box	143	144	143	—
Set.subtracting.Seq.Empty.Int	143	143	143	—
Set.subtracting.Seq.Int.Empty	187	204	193	—
Set.subtracting.Seq.Int0	179	179	179	—
Set.subtracting.Seq.Int100	505	505	505	—
Set.subtracting.Seq.Int25	288	288	288	—
Set.subtracting.Seq.Int50	362	368	364	—

Code size: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	80038	152958	+91.1%	0.52x

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
Set.isStrictSubset.Int100	394	39285	+9870.8%	0.01x
Improvement
DropWhileSequence	16	13	-18.7%	1.23x (?)
ArrayAppendStrings	4120	3710	-10.0%	1.11x (?)
Added
Set.filter.Int100.16k	678	679	679	—
Set.filter.Int100.20k	761	761	761	—
Set.filter.Int100.24k	866	885	873	—
Set.filter.Int100.28k	1289	1468	1349	—
Set.filter.Int50.16k	424	430	426	—
Set.filter.Int50.20k	485	494	488	—
Set.filter.Int50.24k	558	559	559	—
Set.filter.Int50.28k	772	784	777	—
Set.intersection.Seq.Box0	367	367	367	—
Set.intersection.Seq.Box25	480	480	480	—
Set.intersection.Seq.Int0	107	107	107	—
Set.intersection.Seq.Int100	290	291	290	—
Set.intersection.Seq.Int25	158	159	159	—
Set.intersection.Seq.Int50	202	206	203	—
Set.isDisjoint.Seq.Box.Empty	91	94	93	—
Set.isDisjoint.Seq.Box0	473	474	474	—
Set.isDisjoint.Seq.Box25	354	354	354	—
Set.isDisjoint.Seq.Empty.Box	101	101	101	—
Set.isDisjoint.Seq.Empty.Int	98	98	98	—
Set.isDisjoint.Seq.Int.Empty	59	60	60	—
Set.isDisjoint.Seq.Int0	171	172	172	—
Set.isDisjoint.Seq.Int100	134	134	134	—
Set.isDisjoint.Seq.Int25	126	126	126	—
Set.isDisjoint.Seq.Int50	81	82	81	—
Set.isStrictSubset.Seq.Box0	1233	1300	1255	—
Set.isStrictSubset.Seq.Box25	1427	1466	1440	—
Set.isStrictSubset.Seq.Empty.Int	32711	32750	32725	—
Set.isStrictSubset.Seq.Int.Empty	127	130	129	—
Set.isStrictSubset.Seq.Int0	332	333	332	—
Set.isStrictSubset.Seq.Int100	725	726	726	—
Set.isStrictSubset.Seq.Int25	379	380	380	—
Set.isStrictSubset.Seq.Int50	433	434	434	—
Set.isStrictSuperset.Seq.Box0	12315	12357	12329	—
Set.isStrictSuperset.Seq.Box25	1236	1264	1246	—
Set.isStrictSuperset.Seq.Empty.Int	179	180	180	—
Set.isStrictSuperset.Seq.Int.Empty	32657	32830	32732	—
Set.isStrictSuperset.Seq.Int0	3271	3321	3288	—
Set.isStrictSuperset.Seq.Int100	724	724	724	—
Set.isStrictSuperset.Seq.Int25	330	330	330	—
Set.isStrictSuperset.Seq.Int50	333	334	334	—
Set.isSubset.Seq.Box0	1232	1269	1244	—
Set.isSubset.Seq.Box25	1426	1463	1439	—
Set.isSubset.Seq.Empty.Int	93	93	93	—
Set.isSubset.Seq.Int.Empty	129	131	130	—
Set.isSubset.Seq.Int0	332	332	332	—
Set.isSubset.Seq.Int100	522	523	522	—
Set.isSubset.Seq.Int25	378	378	378	—
Set.isSubset.Seq.Int50	435	435	435	—
Set.isSuperset.Seq.Box0	254	255	255	—
Set.isSuperset.Seq.Box25	118	118	118	—
Set.isSuperset.Seq.Empty.Int	53	56	54	—
Set.isSuperset.Seq.Int.Empty	100	100	100	—
Set.isSuperset.Seq.Int0	131	132	132	—
Set.isSuperset.Seq.Int100	179	179	179	—
Set.isSuperset.Seq.Int25	42	42	42	—
Set.isSuperset.Seq.Int50	86	86	86	—
Set.subtracting.Seq.Box.Empty	224	224	224	—
Set.subtracting.Seq.Box0	646	646	646	—
Set.subtracting.Seq.Box25	1117	1117	1117	—
Set.subtracting.Seq.Empty.Box	143	144	143	—
Set.subtracting.Seq.Empty.Int	141	142	141	—
Set.subtracting.Seq.Int.Empty	192	192	192	—
Set.subtracting.Seq.Int0	181	181	181	—
Set.subtracting.Seq.Int100	517	517	517	—
Set.subtracting.Seq.Int25	289	289	289	—
Set.subtracting.Seq.Int50	368	369	368	—

Code size: -Osize

TEST	OLD	NEW	DELTA	RATIO
Regression
SetTests.o	71446	126038	+76.4%	0.57x

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Regression
Set.isStrictSubset.Int100	1037	104041	+9932.9%	0.01x
Added
Set.filter.Int100.16k	2057	2198	2145	—
Set.filter.Int100.20k	2438	2466	2448	—
Set.filter.Int100.24k	2821	2845	2830	—
Set.filter.Int100.28k	3705	3898	3811	—
Set.filter.Int50.16k	1209	1238	1220	—
Set.filter.Int50.20k	1442	1480	1457	—
Set.filter.Int50.24k	1701	1703	1702	—
Set.filter.Int50.28k	2196	2201	2198	—
Set.intersection.Seq.Box0	1840	1848	1843	—
Set.intersection.Seq.Box25	2157	2193	2169	—
Set.intersection.Seq.Int0	1202	1220	1208	—
Set.intersection.Seq.Int100	1670	1675	1672	—
Set.intersection.Seq.Int25	1336	1393	1355	—
Set.intersection.Seq.Int50	1442	1479	1457	—
Set.isDisjoint.Seq.Box.Empty	464	465	464	—
Set.isDisjoint.Seq.Box0	5994	6119	6073	—
Set.isDisjoint.Seq.Box25	4362	4401	4376	—
Set.isDisjoint.Seq.Empty.Box	263	264	264	—
Set.isDisjoint.Seq.Empty.Int	244	244	244	—
Set.isDisjoint.Seq.Int.Empty	433	433	433	—
Set.isDisjoint.Seq.Int0	4602	4634	4613	—
Set.isDisjoint.Seq.Int100	1462	1514	1480	—
Set.isDisjoint.Seq.Int25	3409	3458	3440	—
Set.isDisjoint.Seq.Int50	2292	2294	2293	—
Set.isStrictSubset.Seq.Box0	7182	7293	7256	—
Set.isStrictSubset.Seq.Box25	7666	7739	7707	—
Set.isStrictSubset.Seq.Empty.Int	531979	532910	532427	—
Set.isStrictSubset.Seq.Int.Empty	797	797	797	—
Set.isStrictSubset.Seq.Int0	5320	5372	5349	—
Set.isStrictSubset.Seq.Int100	6363	6448	6402	—
Set.isStrictSubset.Seq.Int25	5478	5530	5508	—
Set.isStrictSubset.Seq.Int50	5658	5659	5659	—
Set.isStrictSuperset.Seq.Box0	71376	71537	71471	—
Set.isStrictSuperset.Seq.Box25	7198	7269	7222	—
Set.isStrictSuperset.Seq.Empty.Int	936	953	942	—
Set.isStrictSuperset.Seq.Int.Empty	531800	532083	531980	—
Set.isStrictSuperset.Seq.Int0	53227	53581	53347	—
Set.isStrictSuperset.Seq.Int100	6395	6447	6413	—
Set.isStrictSuperset.Seq.Int25	5325	5481	5378	—
Set.isStrictSuperset.Seq.Int50	5368	5432	5398	—
Set.isSubset.Seq.Box0	7242	7358	7319	—
Set.isSubset.Seq.Box25	7673	7767	7716	—
Set.isSubset.Seq.Empty.Int	183	184	183	—
Set.isSubset.Seq.Int.Empty	797	818	805	—
Set.isSubset.Seq.Int0	5325	5373	5354	—
Set.isSubset.Seq.Int100	5946	6014	5980	—
Set.isSubset.Seq.Int25	5481	5527	5508	—
Set.isSubset.Seq.Int50	5628	5662	5650	—
Set.isSuperset.Seq.Box0	1669	1706	1692	—
Set.isSuperset.Seq.Box25	1556	1602	1571	—
Set.isSuperset.Seq.Empty.Int	429	436	432	—
Set.isSuperset.Seq.Int.Empty	1365	1400	1377	—
Set.isSuperset.Seq.Int0	1443	1482	1456	—
Set.isSuperset.Seq.Int100	4430	4495	4452	—
Set.isSuperset.Seq.Int25	1122	1122	1122	—
Set.isSuperset.Seq.Int50	2217	2251	2228	—
Set.subtracting.Seq.Box.Empty	615	627	619	—
Set.subtracting.Seq.Box0	6288	6359	6334	—
Set.subtracting.Seq.Box25	7133	7154	7147	—
Set.subtracting.Seq.Empty.Box	380	384	381	—
Set.subtracting.Seq.Empty.Int	358	358	358	—
Set.subtracting.Seq.Int.Empty	593	593	593	—
Set.subtracting.Seq.Int0	4893	4952	4925	—
Set.subtracting.Seq.Int100	5801	5807	5804	—
Set.subtracting.Seq.Int25	5143	5199	5174	—
Set.subtracting.Seq.Int50	5324	5404	5361	—

✅	Benchmark Check Report
⚠️Ⓜ️	Set.filter.Int100.20k has very wide range of memory used between independent, repeated measurements. Set.filter.Int100.20k mem_pages [i1, i2]: min=[157, 172] 𝚫=15 R=[54, 39]
⚠️Ⓜ️	Set.filter.Int50.16k has very wide range of memory used between independent, repeated measurements. Set.filter.Int50.16k mem_pages [i1, i2]: min=[105, 105] 𝚫=0 R=[57, 40]
⚠️Ⓜ️	Set.filter.Int100.24k has very wide range of memory used between independent, repeated measurements. Set.filter.Int100.24k mem_pages [i1, i2]: min=[172, 172] 𝚫=0 R=[0, 42]
⚠️⏱	Set.isStrictSuperset.Seq.Int0 execution took at least 3166 μs. Decrease the workload of Set.isStrictSuperset.Seq.Int0 by a factor of 4 (10), to be less than 1000 μs.
⚠️Ⓜ️	Set.filter.Int50.24k has very wide range of memory used between independent, repeated measurements. Set.filter.Int50.24k mem_pages [i1, i2]: min=[90, 89] 𝚫=1 R=[62, 55]
⛔️⏱	Set.isStrictSuperset.Seq.Int.Empty execution took at least 31701 μs. Decrease the workload of Set.isStrictSuperset.Seq.Int.Empty by a factor of 32 (100), to be less than 1000 μs.
⚠️Ⓜ️	Set.filter.Int50.20k has very wide range of memory used between independent, repeated measurements. Set.filter.Int50.20k mem_pages [i1, i2]: min=[105, 105] 𝚫=0 R=[39, 0]
⚠️⏱	Set.filter.Int100.28k execution took at least 1231 μs. Decrease the workload of Set.filter.Int100.28k by a factor of 2 (10), to be less than 1000 μs.
⚠️Ⓜ️	Set.filter.Int100.28k has very wide range of memory used between independent, repeated measurements. Set.filter.Int100.28k mem_pages [i1, i2]: min=[368, 368] 𝚫=0 R=[39, 0]
⚠️Ⓜ️	Set.filter.Int100.16k has very wide range of memory used between independent, repeated measurements. Set.filter.Int100.16k mem_pages [i1, i2]: min=[172, 156] 𝚫=16 R=[40, 135]
⚠️Ⓜ️	Set.isStrictSubset.Seq.Box0 has very wide range of memory used between independent, repeated measurements. Set.isStrictSubset.Seq.Box0 mem_pages [i1, i2]: min=[22, 22] 𝚫=0 R=[0, 28]
⛔️⏱	Set.isStrictSubset.Seq.Empty.Int execution took at least 31747 μs. Decrease the workload of Set.isStrictSubset.Seq.Empty.Int by a factor of 32 (100), to be less than 1000 μs.
⚠️Ⓜ️	Set.isStrictSubset.Seq.Empty.Int has very wide range of memory used between independent, repeated measurements. Set.isStrictSubset.Seq.Empty.Int mem_pages [i1, i2]: min=[14, 14] 𝚫=0 R=[11, 18]
⚠️Ⓜ️	Set.filter.Int50.28k has very wide range of memory used between independent, repeated measurements. Set.filter.Int50.28k mem_pages [i1, i2]: min=[236, 220] 𝚫=16 R=[39, 95]
⚠️⏱	Set.isStrictSuperset.Seq.Box0 execution took at least 8544 μs. Decrease the workload of Set.isStrictSuperset.Seq.Box0 by a factor of 16 (10), to be less than 1000 μs.

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac mini
  Model Identifier: Macmini8,1
  Processor Name: Intel Core i7
  Processor Speed: 3.2 GHz
  Number of Processors: 1
  Total Number of Cores: 6
  L2 Cache (per Core): 256 KB
  L3 Cache: 12 MB
  Memory: 64 GB

[palimondo]

I will merge this and run the benchmarks on #21300 to validate we have all multipliers set up correctly to get 20—1000 μs runtimes. Then I'll temporarily disable the following benchmarks in #24725 until set-on-fire's ready to merge:

• Set.isStrictSubset.Int100
• Set.isStrictSubset.Seq.Empty.Int
• Set.isStrictSuperset.Seq.Int.Empty

@Gumichocopengin8 Thank you for your great work!