Skip to content

[libc++] Check correctly ref-qualified __is_callable in algorithms #73451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 1, 2024

Conversation

changkhothuychung
Copy link
Contributor

@changkhothuychung changkhothuychung commented Nov 26, 2023

We were only checking that the comparator was rvalue callable,
when in reality the algorithms always call comparators as lvalues.
This patch also refactors the tests for callable requirements and
expands it to a few missing algorithms.

Fixes #69554

@changkhothuychung changkhothuychung requested a review from a team as a code owner November 26, 2023 17:13
@llvmbot llvmbot added the libc++ libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi. label Nov 26, 2023
@llvmbot
Copy link
Member

llvmbot commented Nov 26, 2023

@llvm/pr-subscribers-libcxx

Author: Nhat Nguyen (changkhothuychung)

Changes

Full diff: https://github.com/llvm/llvm-project/pull/73451.diff

8 Files Affected:

  • (modified) libcxx/include/__algorithm/equal_range.h (+3-2)
  • (modified) libcxx/include/__algorithm/includes.h (+3-2)
  • (modified) libcxx/include/__algorithm/is_permutation.h (+3-2)
  • (modified) libcxx/include/__algorithm/lower_bound.h (+3-2)
  • (modified) libcxx/include/__algorithm/min_element.h (+3-3)
  • (modified) libcxx/include/__algorithm/minmax_element.h (+3-2)
  • (modified) libcxx/include/__algorithm/partial_sort_copy.h (+3-2)
  • (added) libcxx/test/libcxx/random/random.uniform.real/test.cpp (+19)
diff --git a/libcxx/include/__algorithm/equal_range.h b/libcxx/include/__algorithm/equal_range.h
index dc1268a6ff110c7..9528635730f7515 100644
--- a/libcxx/include/__algorithm/equal_range.h
+++ b/libcxx/include/__algorithm/equal_range.h
@@ -60,8 +60,9 @@ __equal_range(_Iter __first, _Sent __last, const _Tp& __value, _Compare&& __comp
 template <class _ForwardIterator, class _Tp, class _Compare>
 _LIBCPP_NODISCARD_EXT _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 pair<_ForwardIterator, _ForwardIterator>
 equal_range(_ForwardIterator __first, _ForwardIterator __last, const _Tp& __value, _Compare __comp) {
-  static_assert(__is_callable<_Compare, decltype(*__first), const _Tp&>::value,
-                "The comparator has to be callable");
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
   static_assert(is_copy_constructible<_ForwardIterator>::value,
                 "Iterator has to be copy constructible");
   return std::__equal_range<_ClassicAlgPolicy>(
diff --git a/libcxx/include/__algorithm/includes.h b/libcxx/include/__algorithm/includes.h
index 88253e2653d27a7..9e20fe6174f337e 100644
--- a/libcxx/include/__algorithm/includes.h
+++ b/libcxx/include/__algorithm/includes.h
@@ -45,8 +45,9 @@ _LIBCPP_NODISCARD_EXT inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
     _InputIterator2 __first2,
     _InputIterator2 __last2,
     _Compare __comp) {
-  static_assert(__is_callable<_Compare, decltype(*__first1), decltype(*__first2)>::value,
-      "Comparator has to be callable");
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
 
   return std::__includes(
       std::move(__first1),
diff --git a/libcxx/include/__algorithm/is_permutation.h b/libcxx/include/__algorithm/is_permutation.h
index 105a0732283c902..17aaf12ceeda2be 100644
--- a/libcxx/include/__algorithm/is_permutation.h
+++ b/libcxx/include/__algorithm/is_permutation.h
@@ -191,8 +191,9 @@ template <class _ForwardIterator1, class _ForwardIterator2, class _BinaryPredica
 _LIBCPP_NODISCARD_EXT _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20 bool
 is_permutation(_ForwardIterator1 __first1, _ForwardIterator1 __last1, _ForwardIterator2 __first2,
                _BinaryPredicate __pred) {
-  static_assert(__is_callable<_BinaryPredicate, decltype(*__first1), decltype(*__first2)>::value,
-      "The predicate has to be callable");
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
 
   return std::__is_permutation<_ClassicAlgPolicy>(
       std::move(__first1), std::move(__last1), std::move(__first2), __pred);
diff --git a/libcxx/include/__algorithm/lower_bound.h b/libcxx/include/__algorithm/lower_bound.h
index 91c3bdaafd0cfda..399ebaa88e565de 100644
--- a/libcxx/include/__algorithm/lower_bound.h
+++ b/libcxx/include/__algorithm/lower_bound.h
@@ -49,8 +49,9 @@ _Iter __lower_bound(_Iter __first, _Sent __last, const _Type& __value, _Comp& __
 template <class _ForwardIterator, class _Tp, class _Compare>
 _LIBCPP_NODISCARD_EXT inline _LIBCPP_HIDE_FROM_ABI _LIBCPP_CONSTEXPR_SINCE_CXX20
 _ForwardIterator lower_bound(_ForwardIterator __first, _ForwardIterator __last, const _Tp& __value, _Compare __comp) {
-  static_assert(__is_callable<_Compare, decltype(*__first), const _Tp&>::value,
-                "The comparator has to be callable");
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
   auto __proj = std::__identity();
   return std::__lower_bound<_ClassicAlgPolicy>(__first, __last, __value, __comp, __proj);
 }
diff --git a/libcxx/include/__algorithm/min_element.h b/libcxx/include/__algorithm/min_element.h
index 45f3e85ef92d9fc..3e8f1a31bced402 100644
--- a/libcxx/include/__algorithm/min_element.h
+++ b/libcxx/include/__algorithm/min_element.h
@@ -54,9 +54,9 @@ min_element(_ForwardIterator __first, _ForwardIterator __last, _Compare __comp)
 {
   static_assert(__has_forward_iterator_category<_ForwardIterator>::value,
       "std::min_element requires a ForwardIterator");
-  static_assert(__is_callable<_Compare, decltype(*__first), decltype(*__first)>::value,
-              "The comparator has to be callable");
-
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
   return std::__min_element<__comp_ref_type<_Compare> >(std::move(__first), std::move(__last), __comp);
 }
 
diff --git a/libcxx/include/__algorithm/minmax_element.h b/libcxx/include/__algorithm/minmax_element.h
index 5bcaf8354d9ffb6..49b62b87c6b2602 100644
--- a/libcxx/include/__algorithm/minmax_element.h
+++ b/libcxx/include/__algorithm/minmax_element.h
@@ -85,8 +85,9 @@ pair<_ForwardIterator, _ForwardIterator>
 minmax_element(_ForwardIterator __first, _ForwardIterator __last, _Compare __comp) {
   static_assert(__has_forward_iterator_category<_ForwardIterator>::value,
                 "std::minmax_element requires a ForwardIterator");
-  static_assert(__is_callable<_Compare, decltype(*__first), decltype(*__first)>::value,
-                "The comparator has to be callable");
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
   auto __proj = __identity();
   return std::__minmax_element_impl(__first, __last, __comp, __proj);
 }
diff --git a/libcxx/include/__algorithm/partial_sort_copy.h b/libcxx/include/__algorithm/partial_sort_copy.h
index b9635c51d5fabef..a0b0525b4ee9e98 100644
--- a/libcxx/include/__algorithm/partial_sort_copy.h
+++ b/libcxx/include/__algorithm/partial_sort_copy.h
@@ -65,8 +65,9 @@ _RandomAccessIterator
 partial_sort_copy(_InputIterator __first, _InputIterator __last,
                   _RandomAccessIterator __result_first, _RandomAccessIterator __result_last, _Compare __comp)
 {
-  static_assert(__is_callable<_Compare, decltype(*__first), decltype(*__result_first)>::value,
-                "Comparator has to be callable");
+  static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
+  static_assert(
+      __is_callable<_Compare const&, decltype(*__first), const _Tp&>::value, "The comparator has to be const-callable");
 
   auto __result = std::__partial_sort_copy<_ClassicAlgPolicy>(__first, __last, __result_first, __result_last,
       static_cast<__comp_ref_type<_Compare> >(__comp), __identity(), __identity());
diff --git a/libcxx/test/libcxx/random/random.uniform.real/test.cpp b/libcxx/test/libcxx/random/random.uniform.real/test.cpp
new file mode 100644
index 000000000000000..bc84e362034ccd7
--- /dev/null
+++ b/libcxx/test/libcxx/random/random.uniform.real/test.cpp
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// <random>
+
+// template<_RealType = double>
+// class uniform_real_distribution;
+
+// result_type must be floating type, int type is unsupported
+
+#include <random>
+
+// expected-error@*:* {{static assertion failed due to requirement '__libcpp_random_is_valid_realtype<int>::value': RealType must be a supported floating-point type}}
+struct test_random : public std::uniform_real_distribution<int> {};

@changkhothuychung
Copy link
Contributor Author

@ldionne ldi
can you give me a test example so I can add for this PR? should it be a verify or pass test?

@changkhothuychung changkhothuychung changed the title <algorithm>: __is_callable checks whether the callable can be called with rvalue [libc++] <algorithm>: __is_callable checks whether the callable can be called with rvalue Nov 26, 2023
@@ -0,0 +1,19 @@
//===----------------------------------------------------------------------===//
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a .verify.cpp test. Please look at other .verify.cpp tests for inspiration, they're really not that hard to write.

To trigger the static_assert, you'll have to create a comparator that can't be called as a F const& and one that can't be called as a F&. You can do

struct ConstUncallable {
  void operator()(...) const& = delete;
  void operator()(...) & { implementation }
};

struct NonConstUncallable {
  void operator()(...) const&  { implementation }
  void operator()(...) & = delete;
};

With the proper signature as expected by each algorithm, of course.

@@ -0,0 +1,19 @@
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to add a test that a predicate like this works for all the algorithms:

struct RvalueRefUncallable {
  void operator()(...) && = delete;
  void operator()(...) { implementation }
};

Since that was the original bug report IIRC.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ldionne l
did you want to make the second operator const & ? I am getting errors, overloading either requires both functions to have ref-qualifiers or both lack.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a .pass file, can you take a look and let me know if they are good? if yes, I will add similar tests for the remaining algorithms.

Copy link
Member

@ldionne ldionne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, please add a description of the patch in the PR description!

Copy link

github-actions bot commented Dec 5, 2023

⚠️ C/C++ code formatter, clang-format found issues in your code. ⚠️

You can test this locally with the following command:
git-clang-format --diff 389146816711ef6d11e54726e8ea300bf9945dc3 c83bcc434b984e630402f54dad26bf6ad540a817 --extensions cpp,h -- libcxx/test/libcxx/algorithms/callable-requirements-rvalue.compile.pass.cpp libcxx/test/libcxx/algorithms/callable-requirements.verify.cpp libcxx/include/__algorithm/equal_range.h libcxx/include/__algorithm/includes.h libcxx/include/__algorithm/is_permutation.h libcxx/include/__algorithm/lower_bound.h libcxx/include/__algorithm/max_element.h libcxx/include/__algorithm/min_element.h libcxx/include/__algorithm/minmax.h libcxx/include/__algorithm/minmax_element.h libcxx/include/__algorithm/partial_sort_copy.h libcxx/include/__algorithm/search.h libcxx/include/__algorithm/search_n.h libcxx/include/__algorithm/upper_bound.h libcxx/src/regex.cpp libcxx/test/libcxx/algorithms/ranges_robust_against_copying_projections.pass.cpp libcxx/test/libcxx/algorithms/robust_against_cpp20_hostile_iterators.compile.pass.cpp libcxx/test/std/algorithms/alg.nonmodifying/alg.is_permutation/is_permutation_pred.pass.cpp libcxx/test/std/algorithms/alg.nonmodifying/alg.search/search_n_pred.pass.cpp libcxx/test/std/algorithms/alg.nonmodifying/alg.search/search_pred.pass.cpp libcxx/test/std/algorithms/alg.sorting/alg.merge/inplace_merge_comp.pass.cpp libcxx/test/std/algorithms/alg.sorting/alg.sort/stable.sort/stable_sort_comp.pass.cpp
View the diff from clang-format here.
diff --git a/libcxx/test/std/algorithms/alg.nonmodifying/alg.search/search_n_pred.pass.cpp b/libcxx/test/std/algorithms/alg.nonmodifying/alg.search/search_n_pred.pass.cpp
index 2fa3e9ffac..ccc948ae1f 100644
--- a/libcxx/test/std/algorithms/alg.nonmodifying/alg.search/search_n_pred.pass.cpp
+++ b/libcxx/test/std/algorithms/alg.nonmodifying/alg.search/search_n_pred.pass.cpp
@@ -32,14 +32,14 @@ TEST_CONSTEXPR bool test_constexpr() {
     }
 #endif
 
-struct count_equal {
-  static unsigned count;
-  template <class T>
-  bool operator()(const T& x, const T& y) const {
-    ++count;
-    return x == y;
-  }
-};
+    struct count_equal {
+      static unsigned count;
+      template <class T>
+      bool operator()(const T& x, const T& y) const {
+        ++count;
+        return x == y;
+      }
+    };
 
 unsigned count_equal::count = 0;
 

Copy link
Member

@EricWF EricWF left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the motivation behind static asserts like this. They have a number of downsides.

  1. They cost compile time and memory. The instantiations they perform live in the compilers AST until the end of the TU.
  2. They don't catch broken code. The broken code is already broken, this just diagnoses it differently.
  3. They break well formed code.

As evidenced here, when we get the needless checks wrong, they break valid code. Which is a lot worse than providing a slightly better diagnostic to ill-formed code.

I say we get rid of them entirely.

@changkhothuychung
Copy link
Contributor Author

I don't understand the motivation behind static asserts like this. They have a number of downsides.

  1. They cost compile time and memory. The instantiations they perform live in the compilers AST until the end of the TU.
  2. They don't catch broken code. The broken code is already broken, this just diagnoses it differently.
  3. They break well formed code.

As evidenced here, when we get the needless checks wrong, they break valid code. Which is a lot worse than providing a slightly better diagnostic to ill-formed code.

I say we get rid of them entirely.

@ldionne l
do you have any opinions on this? I will wait for your response before I continue.

@ldionne
Copy link
Member

ldionne commented Dec 11, 2023

do you have any opinions on this? I will wait for your response before I continue.

@philnik777 Do you remember why you added the __is_callable checks in the first place? The Standard isn't all that clear about what the requirements are for classic algorithms, so I think removing these checks would also be a viable path forward. I seem to remember there was a good reason for adding those, so I'm reluctant to do it without remembering the context.

@philnik777
Copy link
Contributor

do you have any opinions on this? I will wait for your response before I continue.

@philnik777 Do you remember why you added the __is_callable checks in the first place? The Standard isn't all that clear about what the requirements are for classic algorithms, so I think removing these checks would also be a viable path forward. I seem to remember there was a good reason for adding those, so I'm reluctant to do it without remembering the context.

We added them to disallow the use of functions to member pointers etc. to be passed to the classic algorithms.

@ldionne
Copy link
Member

ldionne commented Dec 11, 2023

@philnik777 Do you remember why you added the __is_callable checks in the first place? The Standard isn't all that clear about what the requirements are for classic algorithms, so I think removing these checks would also be a viable path forward. I seem to remember there was a good reason for adding those, so I'm reluctant to do it without remembering the context.

We added them to disallow the use of functions to member pointers etc. to be passed to the classic algorithms.

Oh yeah, that's it. Wouldn't that fail to compile already though?

@changkhothuychung
Copy link
Contributor Author

@ldionne it looks like there is a bug with the use of std::lower_bound(begin(collatenames), end(collatenames), s, use_strcmp()); in the file regex.cpp, because use_strcmp doesn't have a const reference operator overloading. Should we add one to that code?

@ldionne
Copy link
Member

ldionne commented Jan 8, 2024

@ldionne it looks like there is a bug with the use of std::lower_bound(begin(collatenames), end(collatenames), s, use_strcmp()); in the file regex.cpp, because use_strcmp doesn't have a const reference operator overloading. Should we add one to that code?

Yes, we should add a const-qualification to use_strcmp in our own code.

@EricWF Now that I think about it some more, I think the issue these static asserts are addressing is that without them, we'll end up using std::__invoke to call the functions, which means that passing a PMF as a function would appear to work with classic algorithms, when in reality it's non-standard. So these static assertions are guarding us against Hyrum's law since our underlying algorithm implementation would allow for PMFs to be passed.

We were only checking that the comparator was rvalue callable,
when in reality the algorithms always call comparators as lvalues.
This patch also refactors the tests for callable requirements and
expands it to a few missing algorithms.

Fixes llvm#69554
@ldionne ldionne force-pushed the is_callable_check branch from ce7bbe3 to c83bcc4 Compare August 1, 2024 14:30
@ldionne ldionne changed the title [libc++] <algorithm>: __is_callable checks whether the callable can be called with rvalue [libc++] Check correctly ref-qualified __is_callable in algorithms Aug 1, 2024
Copy link
Member

@ldionne ldionne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will merge once CI is green.

@ldionne ldionne merged commit 8d151f8 into llvm:main Aug 1, 2024
51 of 54 checks passed
@vitalybuka
Copy link
Collaborator

This one breaks multiple build bot

https://lab.llvm.org/buildbot/#/builders/66/builds/2419

/b/sanitizer-x86_64-linux/build/build_default/runtimes/runtimes-bins/compiler-rt/lib/sanitizer_common/symbolizer/RTSanitizerCommonSymbolizerInternal.i386/symbolizer/libcxx/include/c++/v1/__algorithm/upper_bound.h:54:17: error: static assertion failed due to requirement '__is_callable<(lambda at /b/sanitizer-x86_64-linux/build/llvm-project/llvm/lib/DebugInfo/CodeView/LazyRandomTypeCollection.cpp:174:33) &, const llvm::codeview::TypeIndexOffset &, const llvm::codeview::TypeIndex &>::value': The comparator has to be callable
   54 |   static_assert(__is_callable<_Compare&, decltype(*__first), const _Tp&>::value, "The comparator has to be callable");
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/b/sanitizer-x86_64-linux/build/llvm-project/llvm/include/llvm/ADT/STLExtras.h:1974:15: note: in instantiation of function template specialization 'std::upper_bound<llvm::FixedStreamArrayIterator<llvm::codeview::TypeIndexOffset>, llvm::codeview::TypeIndex, (lambda at /b/sanitizer-x86_64-linux/build/llvm-project/llvm/lib/DebugInfo/CodeView/LazyRandomTypeCollection.cpp:174:33)>' requested here
 1974 |   return std::upper_bound(adl_begin(Range), adl_end(Range),
      |               ^
1 error generated.
[191/638] Building CXX object lib/DebugInfo/CodeView/CMakeFiles/LLVMDebugInfoCodeView.dir/TypeTableCollection.cpp.o

@ldionne
Copy link
Member

ldionne commented Aug 1, 2024

I'll revert. I think that's a bug in the order of arguments I passed to __is_callable for upper_bound. That wasn't intended.

ldionne added a commit that referenced this pull request Aug 1, 2024
…ithms (#73451)"

This reverts commit 8d151f8, which
broke some build bots. I think that is caused by an invalid argument
order when checking __is_comparable in upper_bound.
ldionne added a commit to ldionne/llvm-project that referenced this pull request Aug 1, 2024
We were only checking that the comparator was rvalue callable,
when in reality the algorithms always call comparators as lvalues.
This patch also refactors the tests for callable requirements and
expands it to a few missing algorithms.

This is take 2 of llvm#73451, which was reverted because it broke some
CI bots.

Fixes llvm#69554
ldionne added a commit to ldionne/llvm-project that referenced this pull request Aug 2, 2024
We were only checking that the comparator was rvalue callable,
when in reality the algorithms always call comparators as lvalues.
This patch also refactors the tests for callable requirements and
expands it to a few missing algorithms.

This is take 2 of llvm#73451, which was reverted because it broke some
CI bots.

Fixes llvm#69554
ldionne added a commit that referenced this pull request Aug 5, 2024
…101553)

We were only checking that the comparator was rvalue callable,
when in reality the algorithms always call comparators as lvalues.
This patch also refactors the tests for callable requirements and
expands it to a few missing algorithms.

This is take 2 of #73451, which was reverted because it broke some
CI bots. The issue was that we checked __is_callable with arguments
in the wrong order inside std::upper_bound. This has now been fixed
and a test was added.

Fixes #69554
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libc++ libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

<algorithm>: __is_callable checks whether the callable can be called with rvalue
6 participants