Testing with fpm test

[awvwgk]

Since @milancurcic brought it up in #177 (comment), it might be worth to have a way to define unit tests within a test target, i.e. “export” tests from our testing framework as test targets to fpm test. I am also thinking of tests that will require external resources, we cannot really define in a unit test.

An example is the consistency test of the version number specified in the fpm.toml, meson.build or CMakeLists.txt and the internal one in a *_version module. In this case we would either like to get the correct fpm.toml as argument and parse it or have the build system pass the version string as command line argument. The unit test without arguments will usually be useless. The alternative would be to generate the *_version module with configure_file from a template, but this is nothing we can do with fpm (yet).

Other examples might be test targets requiring correctly set environment variables or test binaries that can run different tests depending on a test input file. The binary might even be an external program that takes an executable as input and provides it with input by some convention (i.e. STDIN) and expects a correct response (i.e. STDOUT or error code).

I am facing four different test scenarios with toml-f, for which I am relying on the unit test framework of meson and it does a pretty decent job to orchestrate those different scenarios. Only one of them would be supported by fpm right now. Here are the test definitions for reference: https://github.com/toml-f/toml-f/blob/master/test/meson.build.

[certik]

@awvwgk thanks for the idea. Can you give an example how it could look like from a user perspective for toml-f using fpm?

I would prefer if things can somehow work automatically so that users don't have to code a file like https://github.com/toml-f/toml-f/blob/cb04d91e55872f390210a63c3468f168c0da4cbd/test/meson.build, which seems quite long. Is the problem in different dependencies for tests?

[awvwgk]

Let's assume fpm would support a configure_file like syntax, a hypothetical transcription of my test definitions for toml-f with fpm could look like this:

# Check if the internal version number matches the package version number
[[test]]
name = "tftest-version"
main = "version.f90"
source-dir = "test"
[[test.target]]
args = ["@package-version@"]  # returns entry from package%version

# Check content of a local file (fpm.toml) for the version number
# Note: this test is actually redundant for fpm, if the first case works,
#       meson and CMake currently use this since fpm can't check itself
[[test]]
name = "tftest-fpm"
main = "fpm.f90"
source-dir = "test"
[[test.target]]
# given the current scope, the current source directory should be "test"
args = ["@fpm-current-source-dir@/../fpm.toml"]
# alternatively, if we have a package scope, this might work:
# args = ["@fpm-source-dir@/fpm.toml"]

# This is already valid with fpm
[[test]]
name = "tftester"
main = "tftest.f90"
source-dir = "test"

# Now the complicated part
# Decoder test, check if the deserialization of TOML documents produces the correct data structures
[[test]]
name = "toml2json"
main = "toml2json.f90"
source-dir = "test"
[[test.target]]
# requires an external program to perform testing
executable.find = "toml-test"
# referencing of the test executable is required
args = [
  "-testdir", "@fpm-source-dir@/subprojects/toml-test",
  "@fpm-current-binary-dir@/toml2json",
]
[[test.target]]
executable.find = "toml-test"
args = ["@fpm-current-binary-dir@/toml2json"]

# Encoder test, check if the serialization of data structures produces correct TOML documents
[[test]]
name = "json2toml"
main = "json2toml.f90"
source-dir = "test"
[[test.target]]
executable.find = "toml-test"
args = ["-encoder", "@fpm-current-binary-dir@/json2toml"]

Certainly, the test definitions won't get much shorted compared to the declarations in the meson.build of toml-f. I actually prefer to be explicit about the declarations.

For the sake of the discussion, we can ignore the second half of the tests for now, if they don't fit into fpm's scope of testing (I currently don't support them in CMake as well because they are a bit more involved and so far only meson is able to handle this test scenario gracefully).

But supporting the first two scenarios (tftest-version and tftest-fpm) in some way would be great.

[ivan-pi]

At least for the first scenario, I believe in Cargo it is done using environment variables. Quoting the relevant sections:

Cargo exposes these environment variables to your crate when it is compiled. Note that this applies for running binaries with cargo run and cargo test as well. To get the value of any of these variables in a Rust program, do this:

let version = env!("CARGO_PKG_VERSION");

version will now contain the value of CARGO_PKG_VERSION.

In rust env! is a macro which inspects an environment variable at compile time! Then you just need to assert the environment variable version (coming from the manifest file) is equal to the parameter in your Fortran API.

For the second version you could use an environment variable FPM_MANIFEST_DIR.

For the remaining cases I have the feeling it can be done with the environment variables:

• CARGO_BIN_EXE_<name> — The absolute path to a binary target's executable. This is only set when building an integration test or benchmark. This may be used with the env macro to find the executable to run for testing purposes. The <name> is the name of the binary target, exactly as-is. For example, CARGO_BIN_EXE_my-program for a binary named my-program. Binaries are automatically built when the test is built, unless the binary has required features that are not enabled.

[awvwgk]

The Rust macro processor is quite powerful, but we would have to emulate something like this with fypp or cpp.

Using environment variables at runtime would be the (easier) alternative, which again leads to the problem how we to specify those in the package manifest in case we need non-standard information.