API for file system operations: directory manipulation

[aradi]

So, following the suggestion in #201, let's start to discuss file system access API. In order to keep it focused, I'd suggest to start with the directory related operations first. My proposal:

open_directory(dirname, dirdesc, status): Opens a directory and returns a descriptor

type(dirdesc_t): Type containing the data of an opened directory

dirdesc_t%get_next(): Returns the next entry in the open directory or "" if
there are no more entries.

make_directory(dirname, parents, status): Makes a directory. If parents is .true.
also parent directories are created in case they do not exist yet.

remove_directory(dirname, content, status): Removes a directory. If content is
.true. also the directory content is removed (recursive delete).

change_directory(dirname, status): Changes to the given directory

get_working_dir() -> char(:), allocatable: Returns the current working directory.

is_dir(fname) -> logical: GIve .true. if fname is a directory.

A few notes:

• The functionality above can be easily realized with a libc-interface (should work on all POSIX systems) and a bit of C-code. The big question is Windows, as I have no clue how to implement this functionality on WIndows.

• As for error handling we could have a status argument. I'd argue for having a derived type type(status_t) containg the result of the operation (OK/Error) and a string with the error message in case of error. Whether the status argument should be optional or not is open for discussions.

[certik]

We should survey Python, Rust, Julia, Matlab, etc. to see what API they use for this.

[arjenmarkus]

That looks like a good start :). I would like to add though: dirdesc_t%get_first() - to get the first entry or to restart the search. Should it be possible to specify a filter? As for Windows, as that is the main OS I work with, I might give that a try. While the function names differ, the functionality is largely the same. Wrt the type of the various arguments such as dirname, these would be simple strings? Then I suggest we use the forward slash as the directory separator - it is accepted by all the major OSs and anything else can be solved inside the implementation (notably the C bit). Regards, Arjen Op do 16 jul. 2020 om 23:01 schreef Bálint Aradi <notifications@github.com>:

…

So, following the suggestion in #201 <#201>, let's start to discuss file system access API. In order to keep it focused, I'd suggest to start with the directory related operations first. My proposal: open_directory(dirname, dirdesc, status): Opens a directory and returns a descriptor type(dirdesc_t): Type containing the data of an opened directory dirdesc_t%get_next(): Returns the next entry in the open directory or "" if there are no more entries. make_directory(dirname, parents, status): Makes a directory. If parents is .true. also parent directories are created in case they do not exist yet. remove_directory(dirname, content, status): Removes a directory. If content is .true. also the directory content is removed (recursive delete). change_directory(dirname, status): Changes to the given directory get_working_dir() -> char(:), allocatable: Returns the current working directory. is_dir(fname) -> logical: GIve .true. if fname is a directory. A few notes: - The functionality above can be easily realized with a libc-interface (should work on all POSIX systems) and a bit of C-code. The big question is Windows, as I have no clue how to implement this functionality on WIndows. - As for error handling we could have a status argument. I'd argue for having a derived type type(status_t) containg the result of the operation (OK/Error) and a string with the error message in case of error. Whether the status argument should be optional or not is open for discussions. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#220>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR3UW3EBFNYXV2PRHWTR35TDHANCNFSM4O45YWQA> .

[everythingfunctional]

I think we should start with just the file path bit. See the Rust library, or the Haskell library.

Also, I don't particularly like the manual iterator design for open_directory. I'd much rather have a function get_contents(directory) -> type(file_path), allocatable :: paths(:). To me, open_directory and dirdesc%get_next() smacks of object-oriented assembly, in an attempt to satisfy primitive obsession.

[milancurcic]

I'd much rather have a function get_contents(directory) -> type(file_path), allocatable :: paths(:)

👍 +1

Also, I see inconsistent naming, e.g. is_dir() and dirdesc_t, but change_directory and remove_directory. I like shorter names if they're common enough. In this case, I think they are. What do you think about:

• change_directory -> chdir
• make_directory -> mkdir
• remove_directory -> rmdir
• get_working_dir -> get_cwd or getcwd

Python API reference: https://docs.python.org/3/library/os.html

[milancurcic]

I think we should start with just the file path bit. See the Rust library, or the Haskell library.

Should this be a separate thread? I think in this one we discuss only directory stuff, plus I think most directory operations here are independent of file path.

[aradi]

@everythingfunctional The interface here is basically a 1-1 mapping of the libc API, which also uses an iterator. (Works in Rust also similarly) But sure, if we have a decent file path type, we could have a convenience funtion on top of it which returns an allocatable array instead.

@milancurcic Although I'd be very happy to go with Pythons naming convention, it would not be consistent with Fortrans recent naming strategy (e.g. execute_command_line instead of exec or even command_argument_count). For sake of adoption, I'd probably prefer the style which is more Fortran like, even if I am not big fan of it. We could use _dir everywhere to make it more consistent and somehwat shorter, though: create_dir (or make_dir), remove_dir, change_dir, etc...

[everythingfunctional]

I see. In that case the question becomes one of philosophy for stdlib. Does stdlib provide lots of low level stuff for people to build abstractions on top of, or does it aim to provide useful abstractions and ergonomic APIs?

[aradi]

@everythingfunctional I agree, it is a philosophical question. And given the traditional reluctance of Fortran to delve into the depths of the underlying operating system, a higher level solution would indeed be probably more appropriate.

If we return an array, we have to decide on the element type. I see 3 different options:

• characters of some fixed length (e.g. max path length size on the system, or similar)
• a string derived type (offering variable string sizes and maybe also some string capabilties)
• a path derived type (offering path manipulation and path-query capabilties).

I am working on Option 3 at the moment, but as discussed in #224, we should decide, whether and if yes how many derived types we want to impose on the users of stdlib...

[certik]

@everythingfunctional regarding the goal of stdlib, I consider it both:

• provide reusable low level functionality that gets the job done and people can use it to build more high level interfaces, or just use it directly; ideally these are just raw functions / subroutines that do not use derived types
• nicely designed high level interfaces (perhaps OO based) that use the low level blocks; this is designed to be ergonomic and can use derived types or other OO features

That way everybody gets what they want, both people who prefer more OO, as well as people who prefer more direct low level API.

[everythingfunctional]

In that case, I think the original proposal is perfectly suitable as low-level building blocks. It wouldn't be difficult to build a more ergonomic API using them.

[MarDiehl]

Since this is related to #201, I want to explain the rational why I prefer to rely on standard types and no type bound procedures for low level operations:

In Fortran, one cannot chain function calls (and array access), so an object oriented approach will result in something like:

type(t_path) :: filename
filename = getcwd()
call filename%join('my_file.ext)
print*, filename%islink()

while the functional/procedural way gives:

print*, islink(join(getcwd(),'my_file.ext'))

I acknowledge that this is not the most elaborate example, but in generally I think that a procedural approach is often more natural for Fortran.