source pre-processing prior to determining dependencies

[p-costa]

Hi all,

As far as I understand, pre-processor support by fpm is still under discussion (#78, #308, #425).

I have a use case where I build my source code together with modules from an external library. Some of these modules are not present in my src directory because they not needed -- the underlying dependencies within this library are masked by (cpp) pre-processing.

So, in short, I wanted to ask if you consider supporting pre-processing specific source files (similar to this #425 (comment)), prior to generating dependencies?

E.g.:

module mod_a
#ifdef _B
use mod_b, only: b
#endif
(...)

with the dependency of mod_b in mod_a not determined if mod_a is not compiled with -cpp -D_B?

Thanks!

[LKedward]

Thanks Pedro @p-costa, this is definitely something that needs discussing before we start implementing it.

The situation you describe is the main difficulty in implementing pre-processing in fpm.
If fpm could just pre-process after determining dependencies then it would be quite straightforward to implement.
Unfortunately pre-processing before determining dependencies requires a little more thought.

I'm glad you opened this issue because now I know that it is important to users for use-cases like this.
In your particular case, does it matter if the unused modules are compiled anyway even if not used? Currently fpm doesn't have any mechanism yet to selectively exclude certain library sources from compilation. So even if pre-processing happened before dependency checking, fpm would still compile all library modules (which may be a problem if you are using this mechanism to switch out a dependency like MPI for example).

[p-costa]

Thanks! I just wanted to raise awareness about this potential use case.

In your particular case, does it matter if the unused modules are compiled anyway even if not used? Currently fpm doesn't have any mechanism yet to selectively exclude certain library sources from compilation.

In my case it is not a major issue. It is just that I need to have 11 additional source files on my repo once it supports fpm, or modify the source code of the library. But perhaps there are other use cases where this could matter (e.g. less permissive license of some source files). Of course, the build process would also be more efficient.

(just for context -- I built the code CaNS using fpm, which is compiled with the source of the 2DECOMP&FFT library (with some additional changes w.r.t. the original 2DECOMP&FFT implementation), without using the Library's FFT API).

[awvwgk]

One more tricky case is the Fortran template library where the name of the modules is generated by a preprocessor statement.

module CAT3(ftlArray,FTL_TEMPLATE_TYPE_NAME,Module)

I also talked briefly with Bálint (@aradi) on supporting fypp in fpm and we arrived at the conclusion that this is currently not easily possible.

A notable reference could be ninja which has added a dynamic dependency feature explicitly for Fortran in ninja-build/ninja#1521, which is required in CMake for building Fortran projects (see cmake/cmake!3256) and for meson's recently introduced dependency scanner (mesonbuild/meson#8095).

[LKedward]

... we arrived at the conclusion that this is currently not easily possible.

To be clear, I am not aware of any fundamental reason why this cannot be done, it will just require some significant restructuring of fpm to insert a new pre-process phase to the workflow. Please correct me if I am wrong.

[awvwgk]

I am not aware of any fundamental reason why this cannot be done, it will just require some significant restructuring of fpm to insert a new pre-process phase to the workflow

I fully agree with this statement.

[urbanjost]

In a system I use you can specify a pre-processor command based on file suffix (including plain .f and .f90) and all files are preprocessed and/or copied to a second scratch directory and then the dependencies and build is based only on the expanded files.
That was more straight-forward. In addition, those expanded files are put into an ar(1) library as well and several other steps, but the reasons for that do not apply here. You have to make sure the preprocessor is available on the platforms you want to build on which raises other issues, but I would find fpm much easier to use if I could put a line in my manifest that specified the pre-processor rule for different suffixes. As it is, I maintain virtually every file as a *.FF file and have to use custom tools to pre-process those and place the output files in the fpm directories post-processed. Since fpm uses a cached checksum instead of the date I realized recently I actually do not have to only do that with changed files which is simpler than what I have been doing. Since fpm` also does not require a manually edited "make-file" that might be easier? I find it cleaner than having a .FF file creating a .f90 file in the same directory. There are also a set of "standard" environmental variables set so you know OSTYPE and COMPILER and BUILDTIME are always set, as the built-in pre-processor variables vary a lot between different programming environments.

[gnikit]

I am curious what's the status for this issue and in general preprocessor support #78 in fpm? Especially, with all the great work that has been going on as part of GSoC 2022.

At this point I would settle for a workaround on this.

[awvwgk]

For this one there is not really a hack available, but you can do #78 already by creating a wrapper script fypp-gfortran, which calls fypp first with only -D and -I arguments piping the output into gfortran which is compiling as usual.

We already use such a script for creating a single source version, adapting it for a preprocessor pass based on the file extension should be doable: https://github.com/fortran-lang/fpm/blob/main/ci/single-file-gfortran.sh.

[gnikit]

Thanks, I was hoping there was a way around this, since I wanted to add FPM in the teaching material for Imperial College. I guess I will put a pin on this and add FPM when this type of preprocessor support is available.

[awvwgk]

I was hoping there was a way around this, since I wanted to add FPM in the teaching material for Imperial College.

What is your use case not supported in fpm at the moment? Do you need preprocessed module names for your teaching? Preprocessor can be used already without problems as long as you do not generate module identifiers or use statements with macros.

[gnikit]

I was hoping there was a way around this, since I wanted to add FPM in the teaching material for Imperial College.

What is your use case not supported in fpm at the moment? Do you need preprocessed module names for your teaching? Preprocessor can be used already without problems as long as you do not generate module identifiers or use statements with macros.

So it's a few modules that are optional and contain code like this. PETSc is optional and we slowly build up on using external dependencies like PETSc

Module PETSc_Init_Mod
#ifdef PETSC
!!Initialize the PETSc Database
#include <petsc/finclude/petscsys.h>
  use petscsys
  Implicit None
contains
...
#endif
End Module PETSc_Init_Mod

[awvwgk]

So it's a few modules that are optional and contain code like this. PETSc is optional and we slowly build up on using external dependencies like PETSc

In this case you are dealing with a system installed module, which you can declare in external-modules and fpm will ignore it.

[gnikit]

So it's a few modules that are optional and contain code like this. PETSc is optional and we slowly build up on using external dependencies like PETSc

In this case you are dealing with a system installed module, which you can declare in external-modules and fpm will ignore it.

Is it still okay, if the module is missing? i.e. we PETSc is not installed yet?

[awvwgk]

We don't attempt to find the module if it is declared external to fpm, this is up to the compiler to resolve.