Refactors the test suite

[willtebbutt]

Doesn't modify any of the tests beyond adding the occasional random number generator explicitly.

Now that we're having more contributions to the repo, it makes sense to formalise the structure of our tests. The one I'm proposing is widely used and I have always found it to work well.

edit: I appear to have lost two tests somewhere, but I'm not sure where 😬 I found them, and some extra ones!

[theogf]

Thanks for the cleaning! See just my comment on the include. It will avoid having to add each new file separately

[theogf]

Replace each list of include by

for f in readdir(folder)
   include(joinpath(folder,f))
end

as in https://github.com/JuliaGaussianProcesses/KernelFunctions.jl/pull/56/files#diff-7a2da5c915dbe72497c7dfda6c7fcb8b

[willtebbutt]

Ah, yeah, I specifically don't want to do that. See here

[theogf]

Looks like I stopped reading after point 5 😅

[devmotion]

Nice and clear test setup, in particular I like the detailed instructions and motivation for the approach!

Just wanted to add that I was a bit surprised about point 8 (all using statements in the runtests.jl file) since that means it's not possible anymore to run each test file separately. Additionally, I'm a fan of using SafeTestsets to avoid/limit the interactions between different test files. Maybe that could be useful here as well?

[willtebbutt]

Ah, interesting -- I had not come across SafeTestSets before. I can see how they would be helpful. What does the workflow look like if I want to have some utility functionality floating around? Say I have a file called test_utils located in test, which provides some utility functionality that I want to use in, say, test/kernels/exponential. Does putting everything inside its own module not complicate that procedure somewhat? (I'm imagining that you would include("test_utils.jl") directly in runtests.jl)

As regards testing individual files, my preferred workflow is simply to comment out tests that you don't want to run during development.

The benefit of having all of the usings in the same file is that it prevents you from accidentally depending on something using-ed in another test file which (which is really easy to do and happens all of the time), if you comment it out / don't run it, causes annoying errors.

[devmotion]

The benefit of having all of the usings in the same file is that it prevents you from accidentally depending on something using-ed in another test file which (which is really easy to do and happens all of the time), if you comment it out / don't run it, causes annoying errors.

That's exactly why I like SafeTestsets - you can keep the using statements in the individual files (so you can just run them, include them in the REPL, Juno etc.) if you wrap them inside a @safetestset statement.

Say I have a file called test_utils located in test, which provides some utility functionality that I want to use in, say, test/kernels/exponential. Does putting everything inside its own module not complicate that procedure somewhat? (I'm imagining that you would include("test_utils.jl") directly in runtests.jl)

If you want to be able to run each test file separately, you have to add include("test_utils.jl") to each test file that uses the utility functionality. If you don't care about this and rather load the utilities in runtests.jl, I guess you have to import that functionality in each test file (but I think that's a bit weird, I've never done this).

[willtebbutt]

@devmotion I think you've got some really good points. Would you be happy for us to merge this, and you to open another PR with the proposed approach? I would like to get this in so that I can push on some other stuff concurrently.

[devmotion]

Sure, go ahead!