@@ -5,6 +5,8 @@ Testing libc++
55.. contents ::
66 :local:
77
8+ .. _testing :
9+
810Getting Started
911===============
1012
@@ -121,7 +123,7 @@ modifying files on your local machine will also modify what the Docker container
121123This is useful for editing source files as you're testing your code in the Docker container.
122124
123125Writing Tests
124- -------------
126+ =============
125127
126128When writing tests for the libc++ test suite, you should follow a few guidelines.
127129This will ensure that your tests can run on a wide variety of hardware and under
@@ -143,6 +145,189 @@ few requirements to the test suite. Here's some stuff you should know:
143145 necessarily available on all devices we may want to run the tests on (even
144146 though supporting Python is probably trivial for the build-host).
145147
148+ Structure of the testing related directories
149+ --------------------------------------------
150+
151+ The tests of libc++ are stored in libc++'s testing related subdirectories:
152+
153+ - ``libcxx/test/support `` This directory contains several helper headers with
154+ generic parts for the tests. The most important header is ``test_macros.h ``.
155+ This file contains configuration information regarding the platform used.
156+ This is similar to the ``__config `` file in libc++'s ``include `` directory.
157+ Since libc++'s tests are used by other Standard libraries, tests should use
158+ the ``TEST_FOO `` macros instead of the ``_LIBCPP_FOO `` macros, which are
159+ specific to libc++.
160+ - ``libcxx/test/std `` This directory contains the tests that validate the library under
161+ test conforms to the C++ Standard. The paths and the names of the test match
162+ the section names in the C++ Standard. Note that the C++ Standard sometimes
163+ reorganises its structure, therefore some tests are at a location based on
164+ where they appeared historically in the standard. We try to strike a balance
165+ between keeping things at up-to-date locations and unnecessary churn.
166+ - ``libcxx/test/libcxx `` This directory contains the tests that validate libc++
167+ specific behavior and implementation details. For example, libc++ has
168+ "wrapped iterators" that perform bounds checks. Since those are specific to
169+ libc++ and not mandated by the Standard, tests for those are located under
170+ ``libcxx/test/libcxx ``. The structure of this directories follows the
171+ structure of ``libcxx/test/std ``.
172+
173+ Structure of a test
174+ -------------------
175+
176+ Some platforms where libc++ is tested have requirement on the signature of
177+ ``main `` and require ``main `` to explicitly return a value. Therefore the
178+ typical ``main `` function should look like:
179+
180+ .. code-block :: cpp
181+
182+ int main(int, char**) {
183+ ...
184+ return 0;
185+ }
186+
187+
188+ constexpr `` requirements. The typical way to test that,
189+ is to create a helper ``test `` function that returns a ``bool `` and use the
190+ following ``main `` function:
191+
192+ .. code-block :: cpp
193+
194+ constexpr bool test() {
195+ ...
196+ return true;
197+ }
198+
199+ int main(int, char**) {
200+ test()
201+ static_assert(test());
202+
203+ return 0;
204+ }
205+
206+ assert `` and ``static_assert `` for testing. There
207+ are a few helper macros and function that can be used to make it easier to
208+ write common tests.
209+
210+ libcxx/test/support/assert_macros.h
211+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
212+
213+ The header contains several macros with user specified log messages. This is
214+ useful when a normal assertion failure lacks the information to easily
215+ understand why the test has failed. This usually happens when the test is in a
216+ helper function. For example the ``std::format `` tests use a helper function
217+ for its validation. When the test fails it will give the line in the helper
218+ function with the condition ``out == expected `` failed. Without knowing what
219+ the value of ``format string ``, ``out `` and ``expected `` are it is not easy to
220+ understand why the test has failed. By logging these three values the point of
221+ failure can be found without resorting to a debugger.
222+
223+ Several of these macros are documented to take an ``ARG ``. This ``ARG ``:
224+
225+ - if it is a ``const char* `` or ``std::string `` its contents are written to
226+ the ``stderr ``,
227+ - otherwise it must be a callable that is invoked without any additional
228+ arguments and is expected to produce useful output to e.g. ``stderr ``.
229+
230+ This makes it possible to write additional information when a test fails,
231+ either by supplying a hard-coded string or generate it at runtime.
232+
233+ TEST_FAIL(ARG)
234+ ^^^^^^^^^^^^^^
235+
236+ This macro is an unconditional failure with a log message ``ARG ``. The main
237+ use-case is to fail when code is reached that should be unreachable.
238+
239+
240+ TEST_REQUIRE(CONDITION, ARG)
241+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
242+
243+ This macro requires its ``CONDITION `` to evaluate to ``true ``. If that fails it
244+ will fail the test with a log message ``ARG ``.
245+
246+
247+ TEST_LIBCPP_REQUIRE((CONDITION, ARG)
248+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
249+
250+ If the library under test is libc++ it behaves like ``TEST_REQUIRE ``, else it
251+ is a no-op. This makes it possible to test libc++ specific behaviour. For
252+ example testing whether the ``what() `` of an exception thrown matches libc++'s
253+ expectations. (Usually the Standard requires certain exceptions to be thrown,
254+ but not the contents of its ``what() `` message.)
255+
256+
257+ TEST_DOES_NOT_THROW(EXPR)
258+ ^^^^^^^^^^^^^^^^^^^^^^^^^
259+
260+ Validates execution of ``EXPR `` does not throw an exception.
261+
262+ TEST_THROWS_TYPE(TYPE, EXPR)
263+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
264+
265+ Validates the execution of ``EXPR `` throws an exception of the type ``TYPE ``.
266+
267+
268+ TEST_VALIDATE_EXCEPTION(TYPE, PRED, EXPR)
269+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
270+
271+ Validates the execution of ``EXPR `` throws an exception of the type ``TYPE ``
272+ which passes validation of ``PRED ``. Using this macro makes it easier to write
273+ tests using exceptions. The code to write a test manually would be:
274+
275+
276+ .. code-block :: cpp
277+
278+ void test_excption([[maybe_unused]] int arg) {
279+ #ifndef TEST_HAS_NO_EXCEPTIONS // do nothing when tests are disabled
280+ try {
281+ foo(arg);
282+ assert(false); // validates foo really throws
283+ } catch ([[maybe_unused]] const bar& e) {
284+ LIBCPP_ASSERT(e.what() == what);
285+ return;
286+ }
287+ assert(false); // validates bar was thrown
288+ #endif
289+ }
290+
291+
292+
293+ .. code-block :: cpp
294+
295+ void test_excption([[maybe_unused]] int arg) {
296+ TEST_VALIDATE_EXCEPTION(bar,
297+ [](const bar& e) {
298+ LIBCPP_ASSERT(e.what() == what);
299+ },
300+ foo(arg));
301+ }
302+
303+
304+
305+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
306+
307+ This file contains a helper macro ``TEST_WRITE_CONCATENATED `` to lazily
308+ concatenate its arguments to a ``std::string `` and write it to ``stderr ``. When
309+ the output can't be concatenated a default message will be written to
310+ ``stderr ``. This is useful for tests where the arguments use different
311+ character types like ``char `` and ``wchar_t ``, the latter can't simply be
312+ written to ``stderrr ``.
313+
314+ This macro is in a different header as ``assert_macros.h `` since it pulls in
315+ additional headers.
316+
317+ .. note: This macro can only be used in test using C++20 or newer. The macro
318+ was added at a time where most of lib++'s C++17 support was complete.
319+ Since it is not expected to add this to existing tests no effort was
320+ taken to make it work in earlier language versions.
321+
322+
323+
324+ ------------------
325+
326+ The function ``CxxStandardLibraryTest `` in the file
327+ ``libcxx/utils/libcxx/test/format.py `` has documentation about writing test. It
328+ explains the difference between the test named ``foo.pass.cpp `` and named
329+ ``foo.verify.cpp `` are.
330+
146331Benchmarks
147332==========
148333
0 commit comments