[libc++][CI] Replace LLDB test targets with libc++ test category

[Michael137]

We've been increasing the coverage of libc++ LLDB tests in the pre-merge CI (see #110570). Unfortunately the tests are spread across different targets. It would be great if we had a single target that libc++ maintainers could run.

We do this by passing the libc++ test-category as a parameter to LLDB's dotest testing framework. This will only run LLDB tests that have been marked as belonging to the libc++ category.

[llvmbot]

@llvm/pr-subscribers-libcxx

@llvm/pr-subscribers-lldb

Author: Michael Buch (Michael137)

Changes

We've been increasing the coverage of libc++ LLDB tests in the pre-merge CI (see #110570). Unfortunately the tests are spread across different targets. It would be great if we had a single target that libc++ maintainers could run. And we would just add additional tests to that list on the LLDB side as needed.

This patch adds such target.

I'm not sure if there's a more idiomatic way of doing this in CMake (or maybe lit? using the libc++ API test category?). But here's a first stab that does the right thing locally

---

Full diff: https://github.com/llvm/llvm-project/pull/110856.diff

1 Files Affected:

• (modified) lldb/test/CMakeLists.txt (+15)

diff --git a/lldb/test/CMakeLists.txt b/lldb/test/CMakeLists.txt
index 5ac474736eb63d..656f9a1727b316 100644
--- a/lldb/test/CMakeLists.txt
+++ b/lldb/test/CMakeLists.txt
@@ -267,6 +267,21 @@ add_lit_testsuite(check-lldb "Running lldb lit test suite"
     lldb-shell-test-deps
     lldb-unit-test-deps)
 
+# This target covers all targets that are tied to implementation details
+# of libc++, intended to be run by the libc++ pre-merge CI.
+add_lit_testsuite(check-lldb-libcxx-integration "Running lldb libc++ support test suite"
+  ${CMAKE_CURRENT_BINARY_DIR}
+  EXCLUDE_FROM_CHECK_ALL
+  DEPENDS
+    check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx
+    check-lldb-api-functionalities-data-formatter-data-formatter-stl-generic
+    check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx-simulators
+    check-lldb-api-commands-expression-import-std-module
+    check-lldb-api-lang-cpp-std-function-step-into-callable
+    check-lldb-api-lang-cpp-std-function-recognizer
+    check-lldb-api-lang-cpp-std-invoke-recognizer
+)
+
 if(LLDB_BUILT_STANDALONE)
   # This has to happen *AFTER* add_lit_testsuite.
   if (EXISTS ${LLVM_MAIN_SRC_DIR}/utils/llvm-lit)

[ldionne]

Libc++ changes LGTM. Thank you!

[Michael137]

Hmm actually currently doesn't quite do what I wanted..

[labath]

I'd probably use a test category for this (either the existing libc++ category, or if that doesn't fit your use case for some reason, then a new category created specifically for this purpose).

[Michael137]

I'd probably use a test category for this (either the existing libc++ category, or if that doesn't fit your use case for some reason, then a new category created specifically for this purpose).

Agreed, that would be preferable. Is the suggestion to run something like:

./bin/llvm-lit --param dotest-args='-G libc++' ../llvm-project/lldb/test/API/ 

Instead of a ninja target?

[labath]

If that works, then I guess the answer is "yes".

[ldionne]

Just for my understanding, is --category a Lit argument? I have never seen this before and I can't find documentation about it.

[Michael137]

Just for my understanding, is --category a Lit argument? I have never seen this before and I can't find documentation about it.

That's an option to LLDB's dotest.py testing framework (there's some documentation for it in https://lldb.llvm.org/resources/test.html, though doesn't explicitly mention test categories). We can mark tests in LLDB using categories. E.g., the following test is marked under the libc++ category :

llvm-project/lldb/test/API/commands/expression/import-std-module/basic/TestImportStdModule.py

Line 11 in 19c6958

@add_test_categories(["libc++"])

Lit will pass the --category flag to dotest, which then ensures we run only the tests that were decorated with @add_test_categories(["libc++"]) (or have libc++ in the categories file in the test directory).

[labath]

We don't want this, as one of the biggest advantages of these tests is that they don't require an actual libc++ library. Given that, it may not even be very useful to run them as a part of the libc++ suite, but if you really do want to run them here, then i think we ought to create a separate category for this.

[Michael137]

Yea that's fair. We still include headers in those tests to get type_traits etc.

But yea they shouldn't be libc++ dependent (at least not on the layout of the types).

[Michael137]

Hmm getting

Exit Code: 255

Command Output (stderr):
--
ERROR:root:"/home/runner/_work/llvm-project/llvm-project/build/bootstrapping-build/./bin/clang" is not a valid compiler executable; aborting...

In the stage3 bootstrapping-build, libcxx-runners-8-set bot. The clang executable doesn't seem to exist. Looks like we're trying to run the tests before the clang build happened?

[Michael137]

Yea i think we need to run make sure the lldb-api-test-deps target gets run.

[Michael137]

Hmm seems like we can't currently run dotest-param='--category libc++' on the API test suite because of the simulator tests:

 File "/home/runner/_work/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/test_result.py", line 183, in startTest
    self.hardMarkAsSkipped(test)
  File "/home/runner/_work/llvm-project/llvm-project/lldb/packages/Python/lldbsuite/test/test_result.py", line 162, in hardMarkAsSkipped
    getattr(test, test._testMethodName).__func__.__unittest_skip__ = True
AttributeError: 'functools.partial' object has no attribute '__func__'. Did you mean: '__doc__'?

Looks like the mechanism by which we mark tests skipped doesn't work for the partial methods we construct for the libcxx-simulators. Even unwrapping the functools.partial method in hardMarkAsSkipped (or adding a separate categories file), will silently run these tests. One option would be to add --filter-out=libcxx-simulators to the lit-invocation in this patch. @labath are you familiar enough with the unittest python library to know how to properly mark these simulator tests skipped?

[labath]

It looks like the simulator tests are too smart for their own good. #111353 ought to fix this.

[llvm-ci]

LLVM Buildbot has detected a new failure on builder sanitizer-aarch64-linux-fuzzer running on sanitizer-buildbot11 while building libcxx at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/159/builds/7709

Here is the relevant piece of the build log for the reference

Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py' (failure)
...
-- Building with -fPIC
-- LLVM host triple: aarch64-unknown-linux-gnu
-- LLVM default target triple: aarch64-unknown-linux-gnu
-- Using libunwind testing configuration: /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/libunwind/test/configs/llvm-libunwind-shared.cfg.in
-- Failed to locate sphinx-build executable (missing: SPHINX_EXECUTABLE) 
-- Using libc++abi testing configuration: /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/libcxxabi/test/configs/llvm-libc++abi-shared.cfg.in
-- Using libc++ testing configuration: /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/libcxx/test/configs/llvm-libc++-shared.cfg.in
-- ABI list file not generated for configuration aarch64-unknown-linux-gnu.libcxxabi.v1.stable.exceptions.nonew, `check-cxx-abilist` will not be available.
CMake Deprecation Warning at /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/cmake/Modules/CMakePolicy.cmake:6 (cmake_policy):
  The OLD behavior for policy CMP0114 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
  /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/compiler-rt/CMakeLists.txt:12 (include)


CMake Deprecation Warning at /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/cmake/Modules/CMakePolicy.cmake:11 (cmake_policy):
  The OLD behavior for policy CMP0116 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
  /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm-project/compiler-rt/CMakeLists.txt:12 (include)


-- Compiler-RT supported architectures: aarch64
-- Generated Sanitizer SUPPORTED_TOOLS list on "Linux" is "asan;lsan;hwasan;msan;tsan;ubsan"
-- sanitizer_common tests on "Linux" will run against "asan;lsan;hwasan;msan;tsan;ubsan"
-- Configuring done (1.2s)
-- Generating done (0.3s)
-- Build files have been written to: /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm_build0/runtimes/runtimes-bins
[47/50] Performing build step for 'runtimes'
[0/7] Performing build step for 'libcxx_fuzzer_aarch64'
ninja: no work to do.
[3/7] Building CXX object compiler-rt/lib/ubsan/CMakeFiles/RTUbsan_dynamic_version_script_dummy.aarch64.dir/dummy.cpp.o
[5/7] Linking CXX shared library /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm_build0/lib/clang/20/lib/aarch64-unknown-linux-gnu/libclang_rt.ubsan_standalone.so
[6/7] Linking CXX shared library /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm_build0/lib/clang/20/lib/aarch64-unknown-linux-gnu/libclang_rt.asan.so
[7/7] Linking CXX shared library /home/b/sanitizer-aarch64-linux-fuzzer/build/llvm_build0/lib/clang/20/lib/aarch64-unknown-linux-gnu/libclang_rt.hwasan.so
[48/50] No install step for 'runtimes'
[50/50] Completed 'runtimes'
9a78a4cf5f07a454421869b04cbc7de7  llvm_build0/bin/clang
@@@BUILD_STEP get fuzzer-test-suite @@@
fatal: unable to access 'https://github.com/google/fuzzer-test-suite.git/': SSL connection timeout
Step 8 (get fuzzer-test-suite) failure: get fuzzer-test-suite (failure)
@@@BUILD_STEP get fuzzer-test-suite @@@
fatal: unable to access 'https://github.com/google/fuzzer-test-suite.git/': SSL connection timeout
program finished with exit code 1
elapsedTime=347.292776