Allow CTest as a test driver #870
Conversation
Previously, a single test was defined for test-libmongoc. This change allows CTest to see the individual test cases contained within test-libmongoc. These can be selected and executed with the CTest CLI, including with some parallelism.
vector-of-bool
requested review from
rcsanchez97,
kevinAlbs,
chardan and
eramongodb
There was a problem hiding this comment.
There are two tests that fail to be registered with CTest (shows up in test-libmongoc --list-tests but not in ctest -N):
/bson/oid/after_fork
/Client/exhaust_cursor/after_reset
Total tests registered should total 2212, but currently total 2210 according to ctest -N. I could not figure out why these two tests are being dropped from examining the current set of changes.
vector-of-bool
force-pushed
the
ctest-awareness
branch
from
973ffbd to
03840ea
Compare
There was a problem hiding this comment.
I have not used ctest much. But it looks like an improvement over using test-libmongoc.
The -I option lets me specify an indexed range of tests to run. I have hacked test-libmongoc to support that when running a sequence of time consuming tests during development.
I like the consolidated output of skipped tests.
LGTM
There was a problem hiding this comment.
Other than a couple remaining questions, LGTM.
Output of attempts to run tests when none are registered (e.g. built Debug, attempt to run Release tests) can be rather noisy, but I prefer the noise over little/no feedback. Great work getting both single and multi-config generators to work properly. I hope this paves the way to a parallel-friendly test suite. 👍
Previously, a single CTest test was defined for test-libmongoc. This change allows CTest to see the individual test cases contained within test-libmongoc. These can be selected and executed with the CTest CLI, including with some parallelism.
Changes are only visible with CMake 3.10 or newer. Earlier CMake versions should be unaffected.
This changeset does not modify how the tests are executed in CI. In the future, if we wish to adopt CTest as a test runner, the CTest-emitted results could be transformed via simple scripting back into Evergreen test results.
CTest can be executed with parallelism with the
-j/--paralleloptions. Tests execute well in parallel except when they attempt to modify a shared resource. This can be fixed by attaching concurrency properties to tests as appropriate, or by ensuring that tests each receive and use a unique copy of the resource under contention, but such changes are beyond the scope of this changeset.These changes relate-to and partially-address CXX-2301 and CXX-2302, but do not complete them.
(The choice of
@@ctest-skipped@@as a skip-marker is arbitrary and insignificant.)