[libc++] Reduce the compilation time required by SIMD tests

[ldionne]

Testing all the SIMD widths exhaustively is nice in theory, however in practice it leads to extremely slow tests. Given that

1. our testing resources are finite and actually pretty costly
2. we have thousands of other tests we also need to run
3. the value of executing these SIMD tests for absolutely all supported SIMD widths is fairly small compared to cherry-picking a few relevant widths

I think it makes a lot of sense to reduce the exhaustiveness of these tests. I'm getting a ~4x speedup for the worst offender (reference_assignment.pass.cpp) after this patch.

I'd also like to make this a reminder to anyone seeing this PR that tests impact everyone's productivity. Slow unit tests contribute to making the CI slower as a whole, and that has a direct impact on everyone's ability to iterate quickly during PRs. Even though we have a pretty robust CI setup in place, we should remember that it doesn't come for free and should strive to keep our tests at a good bang for the buck ratio.

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

Changes

Testing all the SIMD widths exhaustively is nice in theory, however in practice it leads to extremely slow tests. Given that

1. our testing resources are finite and actually pretty costly
2. we have thousands of other tests we also need to run
3. the value of executing these SIMD tests for absolutely all supported SIMD widths is fairly small compared to cherry-picking a few relevant widths

I think it makes a lot of sense to reduce the exhaustiveness of these tests. I'm getting a ~4x speedup for the worst offender (reference_assignment.pass.cpp) after this patch.

I'd also like to make this a reminder to anyone seeing this PR that tests impact everyone's productivity. Slow unit tests contribute to making the CI slower as a whole, and that has a direct impact on everyone's ability to iterate quickly during PRs. Even though we have a pretty robust CI setup in place, we should remember that it doesn't come for free and should strive to keep our tests at a good bang for the buck ratio.

---

Full diff: https://github.com/llvm/llvm-project/pull/72602.diff

1 Files Affected:

• (modified) libcxx/test/std/experimental/simd/test_utils.h (+2-2)

diff --git a/libcxx/test/std/experimental/simd/test_utils.h b/libcxx/test/std/experimental/simd/test_utils.h
index a478a43a87271cf..6954689d0876da7 100644
--- a/libcxx/test/std/experimental/simd/test_utils.h
+++ b/libcxx/test/std/experimental/simd/test_utils.h
@@ -28,7 +28,7 @@ struct TestAllSimdAbiFunctor {
 
   template <class T, std::size_t... Ns>
   void instantiate_with_n(std::index_sequence<Ns...>) {
-    (types::for_each(sized_abis<T, Ns + 1>{}, F<T, Ns + 1>{}), ...);
+    (types::for_each(sized_abis<T, Ns>{}, F<T, Ns>{}), ...);
   }
 
   template <class T>
@@ -36,7 +36,7 @@ struct TestAllSimdAbiFunctor {
     using abis = types::type_list<ex::simd_abi::scalar, ex::simd_abi::native<T>, ex::simd_abi::compatible<T>>;
     types::for_each(abis{}, F<T, 1>());
 
-    instantiate_with_n<T>(std::make_index_sequence<max_simd_size - 1>{});
+    instantiate_with_n<T>(std::index_sequence<1, 2, 8, 16, max_simd_size - 2, max_simd_size - 1, max_simd_size>{});
   }
 };
 

[ldionne]

CC @EricWF @philnik777 @joy2myself

FWIW I get the following times after:

$ libcxx-lit build/default -sv libcxx/test/std/experimental/simd/ --param std=c++20 --time-tests

Slowest Tests:
--------------------------------------------------------------------------
11.61s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.reference/reference_assignment.pass.cpp
2.66s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_ctor_default.pass.cpp
2.27s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_ctor_broadcast.pass.cpp
1.49s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_subscr.pass.cpp
1.30s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_ctor_generator.pass.cpp
1.24s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.reference/reference_value_type.pass.cpp
1.17s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_ctor_default.pass.cpp
0.90s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_subscr.pass.cpp
0.84s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/memory_alignment.pass.cpp
0.83s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_ctor_broadcast.pass.cpp
0.76s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_width.pass.cpp
0.72s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_abi_tag.pass.cpp
0.69s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_alias.pass.cpp
0.69s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.reference/reference_alias.pass.cpp
0.68s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/simd_size.pass.cpp
0.63s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_simd.pass.cpp
0.60s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_simd_mask.pass.cpp
0.57s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_alias.pass.cpp
0.57s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp
0.54s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_width.pass.cpp

And before:

$ libcxx-lit build/default -sv libcxx/test/std/experimental/simd/ --param std=c++20 --time-tests
Slowest Tests:
--------------------------------------------------------------------------
45.70s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.reference/reference_assignment.pass.cpp
9.15s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_ctor_default.pass.cpp
7.09s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_ctor_broadcast.pass.cpp
4.51s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_subscr.pass.cpp
3.64s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_ctor_generator.pass.cpp
3.52s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.reference/reference_value_type.pass.cpp
3.02s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_ctor_default.pass.cpp
1.75s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_subscr.pass.cpp
1.28s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_ctor_broadcast.pass.cpp
1.14s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/memory_alignment.pass.cpp
1.08s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.reference/reference_alias.pass.cpp
1.04s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_alias.pass.cpp
0.96s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.class/simd_width.pass.cpp
0.95s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_width.pass.cpp
0.88s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.mask.class/simd_mask_alias.pass.cpp
0.79s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp
0.76s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_simd.pass.cpp
0.73s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_abi_tag.pass.cpp
0.66s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/is_simd_mask.pass.cpp
0.62s: llvm-libc++-shared.cfg.in :: std/experimental/simd/simd.traits/simd_size.pass.cpp

reference_assignment.pass.cpp is still kinda beefy after the change so we should probably still look into improving that one specifically, but this is a first easy step.

[joy2myself]

LGTM. I will try to improve reference_assignment.pass.cpp later and keep an eye on the compilation time of subsequent patches. Thanks for the reminder!

[philnik777]

LGTM with comments addressed and green CI. (And great branch name BTW)

[ldionne]

No worries, thanks for working on SIMD! We love having a lot of test coverage, it's just that we sometimes lose track of how costly some tests can be :-)

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[vitalybuka]

Thanks!

BTW. There is outdated (~1 year)
https://github.com/llvm/llvm-zorg/blob/ff329ad2df7a5f20d3d250ea52766036b6b748e8/zorg/buildbot/builders/sanitizers/buildbot_functions.sh#L356

I will try them again and let you know if they still slow.

[philnik777]

No worries, thanks for working on SIMD! We love having a lot of test coverage, it's just that we sometimes lose track of how costly some tests can be :-)

Absolutely. I didn't realize how long the tests take until @vitalybuka pointed it out. It's really easy to miss the amount of instantiations with the generator tools we have now (they are way too nice to remove them over this little inconvenience though).

[EricWF]

Thanks for working on this.

This is a comment for the room, but there's a bigger problem with tests like this (than compile time).

That problem is complexity. Tests need to be correct upon inspection. There are no tests for our tests.

We need to be more discerning when writing tests.

[EricWF]

(they are way too nice to remove them over this little inconvenience though).

We may have to disagree on that point. But like any tool, it's how they're used that matters.
Some tools encourage or make easy bad testing practices.

That said, testing types are one of the trickier things we have to do here.