Add tagged union benchmarks

[adriangb]

No description provided.

[codspeed-hq]

CodSpeed Performance Report

Merging #656 tagged-union-benches (9cdd52f) will not alter performances.

Summary

🔥 0 improvements
❌ 8 regressions
✅ 115 untouched benchmarks

🆕 2 new benchmarks
⁉️ 0 dropped benchmarks

Benchmarks breakdown

	Benchmark	main	tagged-union-benches	Change
🆕	test_tagged_union_int_keys_python	N/A	38.7 µs	N/A
🆕	test_tagged_union_int_keys_json	N/A	47.5 µs	N/A
❌	test_core_model_py	48.6 µs	55.5 µs	-14.22%
❌	test_core_model_py_extra	57.8 µs	65.1 µs	-12.60%
❌	test_core_model_json	44.9 µs	52.2 µs	-16.11%
❌	test_core_model_json_extra	54.8 µs	62.1 µs	-13.25%
❌	test_model_exclude_unset_false	52.9 µs	59.9 µs	-13.09%
❌	test_datetime	46.1 µs	53 µs	-15.00%
❌	test_to_string_format	33.1 µs	39.9 µs	-20.60%
❌	test_to_string_direct	25.8 µs	32.6 µs	-26.47%

[adriangb]

No idea why codspeed claims there were performance regressions, I only added benchmarks?

[adriangb]

@art049 any ideas? I also saw a bunch of what I believe is noise in #655

[art049]

Hey @adriangb , @samuelcolvin

I had a thorough look at the execution profiles. It seems the variability comes from mimalloc-rust (calling mimalloc directly). In the base run of this branch, the benches weren't calling the _mi_clock_now function while it seems they are calling it after this PR.

https://github.com/microsoft/mimalloc/blob/8d6a9df7521181afc276d94b3a6ef2a9dd60bd06/src/stats.c#L420-L428

This seems to have something to do with the order of the benches. Maybe some allocations are done and reused in later benchmarks? Do you think it makes sense?

Normally, the diff of the sub-costs should be visible in the UI when we will display the callgraph(it's coming very soon for Python but will require Cpython 3.12).

[adriangb]

Thank you for investigating!

We do have some GILOnceCell things in Rust that are things that live from one benchmark to another. Samuel do you think that may be the issue? Any suggestions as to what to do for #655 if that's the case?

Arthur, if it is something like that I maybe we could run a "warmup" round on all of the benchmarks first (i.e. run each one a single time) before doing the full run on each one. Note that I don't mean running a warmup for A then running A then a warmup for B then B, I mean warmup A, warmup B then run A run B.

[dmontagu]

Seems weird to me that, if it was explained by OnceCell etc., that adding benchmarks would make others slower. I would think it would just make them faster, but maybe if it reordered them it could explain something?

[adriangb]

Agreed, also:

1. I don't think there were any of those involved in these benchmarks.
2. We always execute the code right before benchmarking so those sorts of things should already be cached/created.

So @art049 I don't see how that could happen. Maybe @samuelcolvin knows something, I know he did some tweaking of allocators.

[adriangb]

I can confirm that the issue is from the allocator. I tested test_core_str from #655 and using the default allocator there is no performance difference. And in fact the default allocator performs a couple percent better than MiMalloc (for this benchmark at least). IMO it seems to me like we over optimized by messing with the allocator and are paying a price at least in terms of engineering time now. I suspect this is also what was going on with #602 (cc @davidhewitt on that because we spent some time looking at the compiled code to see if we could figure out what was happening).

[adriangb]

@art049 we're interested in maybe running benchmarks both with and without MiMalloc. Is there any way to do two runs and combine with with CodSpeed?

[art049]

Hey @adriangb, it's not possible out of the box at the moment. But we're looking forward to adding bench groups quite soon and then we'll be able to compare multiple runs with different feature sets.