"dogfooding" caf and cafrun for tests

[zbeekman]

Should we be using caf to compile our tests and cafrun to execute them? This is often called "dogfooding" and can help further test our scripts and ensure their correctness, while also displaying confidence in them. I'm wondering how to go about this in CMake. I think this could be achieved via:

1. Splitting building the tests into an additional "check" target that is not included in "all" and depends on building the opencoarrays library target. This will split the building of the tests into a separate task so that if you don't want to run the tests, you won't have to build them, and emulate GNU/autotools "make check" behavior: build and run tests. This is not strictly required to accomplish "dogfooding"
2. Ensure that the caf and cafrun scripts fully resolve the compilers they wrap and pass necessary flags etc. and are available before install. RFC: Hard code full paths and more precisely specify linker/compiler flags & MPI flags in caf and cafrun scripts #268 We can use CMake's REALPATH capabilities for this.
3. Hardwiring the Fortran compiler in the test directories' CMakeLists.txt to be caf... this could be unstable/difficult. An alternative might be to look into externalproject_add ... here is some useful info: http://cmake.3232098.n2.nabble.com/ExternalProject-Add-without-download-of-local-sources-td6797293.html

[rouson]

Sounds good. It probably makes sense to seek the advice of CMake developer @brad.king. I'm not sure if the latter tag will work so I'll email him too.

[zbeekman]

Actually, the more I think about it the more I am drawn to add_custom_command() and add_custom_target. A difficulty may arise when treating the "fancy" performance/integration tests that link against the timer and FFT code... but it shouldn't be too tricky to figure out... I'm thinking that this is probably the way to go... after making some macros/functions to make life easier.

[bradking]

An alternative might be to look into externalproject_add
...
I am drawn to add_custom_command() and add_custom_target

Either of these approaches may work. An advantage of using ExternalProject_Add is that the "external" project can be a full CMake project. Its configuration and build are delayed until build time of the outer project, so one can use dependencies to make sure everything needed to use caf is ready before even configuring the test project. Combined with use of the export() command one could even have the external test project link to libraries from the main project cleanly.

[zbeekman]

Thanks for chiming in @bradking! It sounds like in the long run it might make the most sense to use ExternalProject_Add to ensure all of our export() machinery is working as expected, but in the short term (since tests aren't setup as external projects yet) perhaps add_custom_target and add_custom_command might be best?

@bradking said:

Combined with use of the export() command one could even have the external test project link to libraries from the main project cleanly.

@bradking: Do you know if this means that we need to export our build directory too if we want to do this before running make install in the main OpenCoarrays project?

[bradking]

@zbeekman yes, if an externally-built project is to test the libraries and such from the build tree then export() should be used to make the targets available for import into the external project.

[rouson]

@zbeekman this issue was prescient. I just submitted pull request #277 to correct for a fatal error in the caf script that was not being detected by the tests because the tests don't yet use the caf script.

[zbeekman]

🎩 ✨ 🐇 🔮

[zbeekman]

Well, actually if I was prescient, I would have made the changes to caf and cafrun when I changed CMakeLists.txt.... 😦

[zbeekman]

might make sense to handle this at the same time as #267

[zbeekman]

Note to self: See the generator expressions section of https://cmake.org/cmake/help/v3.8/command/add_test.html

[rouson]

@zbeekman Let's look at this together so we can implement it and close the issue?