[libc++] Unify the benchmarks with the test suite (#101399)

Instead of building the benchmarks separately via CMake and running them
separately from the test suite, this patch merges the benchmarks into
the test suite and handles both uniformly.

As a result:
- It is now possible to run individual benchmarks like we run tests
  (e.g. using libcxx-lit), which is a huge quality-of-life improvement.

- The benchmarks will be run under exactly the same configuration as
  the rest of the tests, which is a nice simplification. This does
  mean that one has to be careful to enable the desired optimization
  flags when running benchmarks, but that is easy with e.g.
  `libcxx-lit <...> --param optimization=speed`.

- Benchmarks can use the same annotations as the rest of the test
  suite, such as `// UNSUPPORTED` & friends.

When running the tests via `check-cxx`, we only compile the benchmarks
because running them would be too time consuming. This introduces a bit
of complexity in the testing setup, and instead it would be better to
allow passing a --dry-run flag to GoogleBenchmark executables, which is
the topic of google/benchmark#1827.

I am not really satisfied with the layering violation of adding the
%{benchmark_flags} substitution to cmake-bridge, however I believe
this can be improved in the future.

Author: ldionne

.github/workflows/libcxx-build-and-test.yaml

@@ -159,9 +159,6 @@ jobs:
           'generic-no-rtti',
           'generic-optimized-speed',
           'generic-static',
-          # TODO Find a better place for the benchmark and bootstrapping builds to live. They're either very expensive
-          # or don't provide much value since the benchmark run results are too noise on the bots.
-          'benchmarks',
           'bootstrapping-build'
         ]
         machine: [ 'libcxx-runners-set' ]

libcxx/CMakeLists.txt

@@ -150,12 +150,19 @@ message(STATUS "Using libc++ testing configuration: ${LIBCXX_TEST_CONFIG}")
 set(LIBCXX_TEST_PARAMS "" CACHE STRING
     "A list of parameters to run the Lit test suite with.")
 
-# Benchmark options -----------------------------------------------------------
-option(LIBCXX_INCLUDE_BENCHMARKS "Build the libc++ benchmarks and their dependencies" ON)
-
-set(LIBCXX_BENCHMARK_TEST_ARGS_DEFAULT --benchmark_min_time=0.01)
-set(LIBCXX_BENCHMARK_TEST_ARGS "${LIBCXX_BENCHMARK_TEST_ARGS_DEFAULT}" CACHE STRING
-    "Arguments to pass when running the benchmarks using check-cxx-benchmarks")
+# TODO: Figure out how to build GoogleBenchmark on those platforms, and how to build when exceptions or RTTI is disabled
+if (WIN32 OR MINGW OR ANDROID OR ${CMAKE_SYSTEM_NAME} MATCHES "AIX"
+    OR NOT LIBCXX_ENABLE_LOCALIZATION
+    OR NOT LIBCXX_ENABLE_THREADS
+    OR NOT LIBCXX_ENABLE_FILESYSTEM
+    OR NOT LIBCXX_ENABLE_RANDOM_DEVICE
+    OR NOT LIBCXX_ENABLE_EXCEPTIONS
+    OR NOT LIBCXX_ENABLE_RTTI)
+  set(_include_benchmarks OFF)
+else()
+  set(_include_benchmarks ON)
+endif()
+option(LIBCXX_INCLUDE_BENCHMARKS "Build the libc++ benchmarks and their dependencies" ${_include_benchmarks})
 
 option(LIBCXX_INCLUDE_DOCS "Build the libc++ documentation." ${LLVM_INCLUDE_DOCS})
 set(LIBCXX_LIBDIR_SUFFIX "${LLVM_LIBDIR_SUFFIX}" CACHE STRING

libcxx/docs/TestingLibcxx.rst

@@ -392,6 +392,10 @@ Test Filenames`_ when determining the names for new test files.
        of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
        in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
 
+   * - ``FOO.bench.cpp``
+     - A benchmark test. These tests are linked against the GoogleBenchmark library and generally consist of micro-benchmarks of individual
+       components of the library.
+
 
 libc++-Specific Lit Features
 ----------------------------
@@ -438,43 +442,29 @@ Libc++ contains benchmark tests separately from the test of the test suite.
 The benchmarks are written using the `Google Benchmark`_ library, a copy of which
 is stored in the libc++ repository.
 
-For more information about using the Google Benchmark library see the
+For more information about using the Google Benchmark library, see the
 `official documentation <https://github.com/google/benchmark>`_.
 
-.. _`Google Benchmark`: https://github.com/google/benchmark
-
-Building Benchmarks
--------------------
-
-The benchmark tests are not built by default. The benchmarks can be built using
-the ``cxx-benchmarks`` target.
-
-An example build would look like:
+The benchmarks are located under ``libcxx/test/benchmarks``. Running a benchmark
+works in the same way as running a test. Both the benchmarks and the tests share
+the same configuration, so make sure to enable the relevant optimization level
+when running the benchmarks. For example,
 
 .. code-block:: bash
 
-  $ ninja -C build cxx-benchmarks
-
-This will build all of the benchmarks under ``<libcxx>/test/benchmarks`` to be
-built against the just-built libc++. The compiled tests are output into
-``build/libcxx/test/benchmarks``.
+  $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/benchmarks/string.bench.cpp --param optimization=speed
 
-Running Benchmarks
-------------------
+If you want to see where a benchmark is located (e.g. you want to store the executable
+for subsequent analysis), you can print that information by passing ``--show-all`` to
+``lit``. That will print the command-lines being executed, which includes the location
+of the executable created for that benchmark.
 
-The benchmarks must be run manually by the user. Currently there is no way
-to run them as part of the build.
-
-For example:
-
-.. code-block:: bash
-
-  $ cd build/libcxx/test/benchmarks
-  $ ./find.bench.out # Runs all the benchmarks
-  $ ./find.bench.out --benchmark_filter="bm_ranges_find<std::vector<char>>" # Only runs that specific benchmark
-
-For more information about running benchmarks see `Google Benchmark`_.
+Note that benchmarks are only dry-run when run via the ``check-cxx`` target since
+we only want to make sure they don't rot. Do not rely on the results of benchmarks
+run through ``check-cxx`` for anything, instead run the benchmarks manually using
+the instructions for running individual tests.
 
+.. _`Google Benchmark`: https://github.com/google/benchmark
 
 .. _testing-hardening-assertions:
 
@@ -518,4 +508,3 @@ A toy example:
 
 Note that error messages are only tested (matched) if the ``debug``
 hardening mode is used.
-

libcxx/docs/VendorDocumentation.rst

@@ -244,7 +244,8 @@ General purpose options
 
   **Default**: ``ON`` (or value of ``LLVM_INCLUDE_TESTS``)
 
-  Build the libc++ tests.
+  Build the libc++ test suite, which includes various types of tests like conformance
+  tests, vendor-specific tests and benchmarks.
 
 .. option:: LIBCXX_INCLUDE_BENCHMARKS:BOOL
 
@@ -253,15 +254,6 @@ General purpose options
   Build the libc++ benchmark tests and the Google Benchmark library needed
   to support them.
 
-.. option:: LIBCXX_BENCHMARK_TEST_ARGS:STRING
-
-  **Default**: ``--benchmark_min_time=0.01``
-
-  A semicolon list of arguments to pass when running the libc++ benchmarks using the
-  ``check-cxx-benchmarks`` rule. By default we run the benchmarks for a very short amount of time,
-  since the primary use of ``check-cxx-benchmarks`` is to get test and sanitizer coverage, not to
-  get accurate measurements.
-
 .. option:: LIBCXX_ASSERTION_HANDLER_FILE:PATH
 
   **Default**:: ``"${CMAKE_CURRENT_SOURCE_DIR}/vendor/llvm/default_assertion_handler.in"``

libcxx/test/CMakeLists.txt

@@ -1,10 +1,6 @@
 include(HandleLitArguments)
 add_subdirectory(tools)
 
-if (LIBCXX_INCLUDE_BENCHMARKS)
-  add_subdirectory(benchmarks)
-endif()
-
 # Install the library at a fake location so we can run the test suite against it.
 # This ensures that we run the test suite against a setup that matches what we ship
 # in production as closely as possible (in terms of file paths, rpaths, etc).
@@ -66,6 +62,14 @@ set(SERIALIZED_LIT_PARAMS "# Lit parameters serialized here for llvm-lit to pick
 
 serialize_lit_string_param(SERIALIZED_LIT_PARAMS compiler "${CMAKE_CXX_COMPILER}")
 
+if (LIBCXX_INCLUDE_BENCHMARKS)
+  add_subdirectory(benchmarks)
+  set(_libcxx_benchmark_mode "dry-run")
+else()
+  serialize_lit_string_param(SERIALIZED_LIT_PARAMS enable_benchmarks "no")
+  set(_libcxx_benchmark_mode "no")
+endif()
+
 if (NOT LIBCXX_ENABLE_EXCEPTIONS)
   serialize_lit_param(SERIALIZED_LIT_PARAMS enable_exceptions False)
 endif()
@@ -102,4 +106,5 @@ configure_lit_site_cfg(
 add_lit_testsuite(check-cxx
   "Running libcxx tests"
   ${CMAKE_CURRENT_BINARY_DIR}
+  PARAMS enable_benchmarks="${_libcxx_benchmark_mode}"
   DEPENDS cxx-test-depends)

libcxx/test/benchmarks/CMakeLists.txt

@@ -1,10 +1,8 @@
-include(ExternalProject)
-include(CheckCXXCompilerFlag)
-
 #==============================================================================
 # Build Google Benchmark
 #==============================================================================
 
+include(ExternalProject)
 set(BENCHMARK_COMPILE_FLAGS
     -Wno-unused-command-line-argument
     -nostdinc++
@@ -24,6 +22,12 @@ if (DEFINED LIBCXX_CXX_ABI_LIBRARY_PATH)
 endif()
 split_list(BENCHMARK_COMPILE_FLAGS)
 
+set(BENCHMARK_CXX_LIBRARIES)
+list(APPEND BENCHMARK_CXX_LIBRARIES c++)
+if (NOT LIBCXX_ENABLE_SHARED)
+  list(APPEND BENCHMARK_CXX_LIBRARIES c++abi)
+endif()
+
 ExternalProject_Add(google-benchmark
         EXCLUDE_FROM_ALL ON
         DEPENDS cxx cxx-headers
@@ -37,176 +41,7 @@ ExternalProject_Add(google-benchmark
           -DCMAKE_INSTALL_PREFIX:PATH=<INSTALL_DIR>
           -DCMAKE_CXX_FLAGS:STRING=${BENCHMARK_COMPILE_FLAGS}
           -DBENCHMARK_USE_LIBCXX:BOOL=ON
-          -DBENCHMARK_ENABLE_TESTING:BOOL=OFF)
-
-#==============================================================================
-# Benchmark tests configuration
-#==============================================================================
-add_custom_target(cxx-benchmarks)
-set(BENCHMARK_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR})
-set(BENCHMARK_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/google-benchmark)
-
-add_library(               cxx-benchmarks-flags INTERFACE)
-
-# TODO(cmake): remove. This is a workaround to prevent older versions of GCC
-# from failing the configure step because they don't support C++23.
-if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "13.0")
-  return()
-endif()
-#TODO(cmake): remove the `add_compile_options`. Currently we have to explicitly
-# pass the `std:c++latest` flag on Windows to work around an issue where
-# requesting `cxx_std_23` results in an error -- somehow CMake fails to
-# translate the `c++23` flag into `c++latest`, and the highest numbered C++
-# version that MSVC flags support is C++20.
-if (MSVC)
-  add_compile_options(/std:c++latest)
-# ibm-clang does not recognize the cxx_std_23 flag, so use this as a temporary
-# workaround on AIX as well.
-elseif (${CMAKE_SYSTEM_NAME} MATCHES "AIX")
-  add_compile_options(-std=c++23)
-else()
-  target_compile_features( cxx-benchmarks-flags INTERFACE cxx_std_23)
-endif()
-
-target_compile_options(cxx-benchmarks-flags INTERFACE -fsized-deallocation -nostdinc++
-                                                      ${SANITIZER_FLAGS} -Wno-user-defined-literals -Wno-suggest-override)
-target_include_directories(cxx-benchmarks-flags INTERFACE "${LIBCXX_GENERATED_INCLUDE_DIR}"
-                                                INTERFACE "${BENCHMARK_INSTALL_DIR}/include"
-                                                INTERFACE "${LIBCXX_SOURCE_DIR}/test/support")
-target_link_options(cxx-benchmarks-flags INTERFACE -lm -nostdlib++
-                                                   "-L${BENCHMARK_INSTALL_DIR}/lib" "-L${BENCHMARK_INSTALL_DIR}/lib64"
-                                                   ${SANITIZER_FLAGS})
+          -DBENCHMARK_ENABLE_TESTING:BOOL=OFF
+          -DBENCHMARK_CXX_LIBRARIES:STRING="${BENCHMARK_CXX_LIBRARIES}")
 
-set(libcxx_benchmark_targets)
-
-function(add_benchmark_test name source_file)
-  set(libcxx_target ${name}_libcxx)
-  list(APPEND libcxx_benchmark_targets ${libcxx_target})
-  add_executable(${libcxx_target} EXCLUDE_FROM_ALL ${source_file})
-  target_link_libraries(${libcxx_target} PRIVATE cxx-benchmarks-flags)
-  add_dependencies(${libcxx_target} cxx google-benchmark)
-  add_dependencies(cxx-benchmarks ${libcxx_target})
-  if (LIBCXX_ENABLE_SHARED)
-    target_link_libraries(${libcxx_target} PRIVATE cxx_shared)
-  else()
-    target_link_libraries(${libcxx_target} PRIVATE cxx_static)
-  endif()
-  target_link_libraries(${libcxx_target} PRIVATE cxx_experimental benchmark)
-  if (LLVM_USE_SANITIZER)
-    target_link_libraries(${libcxx_target} PRIVATE -ldl)
-  endif()
-  set_target_properties(${libcxx_target}
-    PROPERTIES
-          OUTPUT_NAME "${name}.bench.out"
-          RUNTIME_OUTPUT_DIRECTORY "${BENCHMARK_OUTPUT_DIR}"
-          CXX_EXTENSIONS NO)
-  cxx_link_system_libraries(${libcxx_target})
-endfunction()
-
-
-#==============================================================================
-# Register Benchmark tests
-#==============================================================================
-set(BENCHMARK_TESTS
-    algorithms.partition_point.bench.cpp
-    algorithms/count.bench.cpp
-    algorithms/equal.bench.cpp
-    algorithms/find.bench.cpp
-    algorithms/fill.bench.cpp
-    algorithms/for_each.bench.cpp
-    algorithms/lexicographical_compare.bench.cpp
-    algorithms/lower_bound.bench.cpp
-    algorithms/make_heap.bench.cpp
-    algorithms/make_heap_then_sort_heap.bench.cpp
-    algorithms/min.bench.cpp
-    algorithms/minmax.bench.cpp
-    algorithms/min_max_element.bench.cpp
-    algorithms/mismatch.bench.cpp
-    algorithms/pop_heap.bench.cpp
-    algorithms/pstl.stable_sort.bench.cpp
-    algorithms/push_heap.bench.cpp
-    algorithms/ranges_contains.bench.cpp
-    algorithms/ranges_ends_with.bench.cpp
-    algorithms/ranges_make_heap.bench.cpp
-    algorithms/ranges_make_heap_then_sort_heap.bench.cpp
-    algorithms/ranges_pop_heap.bench.cpp
-    algorithms/ranges_push_heap.bench.cpp
-    algorithms/ranges_sort.bench.cpp
-    algorithms/ranges_sort_heap.bench.cpp
-    algorithms/ranges_stable_sort.bench.cpp
-    algorithms/set_intersection.bench.cpp
-    algorithms/sort.bench.cpp
-    algorithms/sort_heap.bench.cpp
-    algorithms/stable_sort.bench.cpp
-    atomic_wait.bench.cpp
-    atomic_wait_vs_mutex_lock.bench.cpp
-    libcxxabi/dynamic_cast.bench.cpp
-    libcxxabi/dynamic_cast_old_stress.bench.cpp
-    allocation.bench.cpp
-    deque.bench.cpp
-    deque_iterator.bench.cpp
-    exception_ptr.bench.cpp
-    filesystem.bench.cpp
-    format/write_double_comparison.bench.cpp
-    format/write_int_comparison.bench.cpp
-    format/write_string_comparison.bench.cpp
-    format_to_n.bench.cpp
-    format_to.bench.cpp
-    format.bench.cpp
-    formatted_size.bench.cpp
-    formatter_float.bench.cpp
-    formatter_int.bench.cpp
-    function.bench.cpp
-    hash.bench.cpp
-    join_view.bench.cpp
-    lexicographical_compare_three_way.bench.cpp
-    map.bench.cpp
-    monotonic_buffer.bench.cpp
-    numeric/gcd.bench.cpp
-    ordered_set.bench.cpp
-    shared_mutex_vs_mutex.bench.cpp
-    stop_token.bench.cpp
-    std_format_spec_string_unicode.bench.cpp
-    std_format_spec_string_unicode_escape.bench.cpp
-    string.bench.cpp
-    stringstream.bench.cpp
-    system_error.bench.cpp
-    to_chars.bench.cpp
-    unordered_set_operations.bench.cpp
-    util_smartptr.bench.cpp
-    variant_visit_1.bench.cpp
-    variant_visit_2.bench.cpp
-    variant_visit_3.bench.cpp
-    vector_operations.bench.cpp
-    )
-
-foreach(test_path ${BENCHMARK_TESTS})
-  get_filename_component(test_file "${test_path}" NAME)
-  string(REPLACE ".bench.cpp" "" test_name "${test_file}")
-  if (NOT DEFINED ${test_name}_REPORTED)
-    message(STATUS "Adding Benchmark: ${test_file}")
-    # Only report the adding of the benchmark once.
-    set(${test_name}_REPORTED ON CACHE INTERNAL "")
-  endif()
-  add_benchmark_test(${test_name} ${test_path})
-endforeach()
-
-if (LIBCXX_INCLUDE_TESTS)
-  include(AddLLVM)
-
-  configure_lit_site_cfg(
-          ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py.in
-          ${CMAKE_CURRENT_BINARY_DIR}/lit.cfg.py)
-
-  configure_lit_site_cfg(
-          ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
-          ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py)
-
-  set(BENCHMARK_LIT_ARGS "--show-all --show-xfail --show-unsupported ${LIT_ARGS_DEFAULT}")
-
-  add_lit_target(check-cxx-benchmarks
-          "Running libcxx benchmarks tests"
-          ${CMAKE_CURRENT_BINARY_DIR}
-          DEPENDS cxx-benchmarks cxx-test-depends
-          ARGS ${BENCHMARK_LIT_ARGS})
-endif()
+add_dependencies(cxx-test-depends google-benchmark)

libcxx/test/benchmarks/algorithms/min.bench.cpp

@@ -78,6 +78,8 @@ BENCHMARK(BM_std_min<unsigned char>)->Apply(run_sizes);
 BENCHMARK(BM_std_min<unsigned short>)->Apply(run_sizes);
 BENCHMARK(BM_std_min<unsigned int>)->Apply(run_sizes);
 BENCHMARK(BM_std_min<unsigned long long>)->Apply(run_sizes);
+#ifndef TEST_HAS_NO_INT128
 BENCHMARK(BM_std_min<unsigned __int128>)->Apply(run_sizes);
+#endif
 
 BENCHMARK_MAIN();

libcxx/test/benchmarks/atomic_wait_vs_mutex_lock.bench.cpp

@@ -8,10 +8,6 @@
 
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 
-// To run this test, build libcxx and cxx-benchmarks targets
-// cd third-party/benchmark/tools
-// ./compare.py filters ../../../build/libcxx/benchmarks/atomic_wait_vs_mutex_lock.libcxx.out BM_atomic_wait BM_mutex
-
 #include <atomic>
 #include <mutex>
 #include <numeric>