Add warnings on function exported and optional callback

[ityonemo]

No description provided.

[josevalim]

Suggested change

	> Unloaded modules {: .warning }
	>
	> This function does *not* load the module in case
	> it is not loaded. In particular, `mix test` does not load modules
	> by default, so testing this function may cause irreproducible
	> results depending on if the target module was compiled due to
	> code changes.
	>
	> See `Code.ensure_loaded/1` for more
	> information.
	> ### Unloaded modules {: .warning }
	>
	> This function does *not* load the module in case it is not loaded
	> and Elixir lazily loads modules by default (except on releases).
	>
	> See `Code.ensure_loaded/1` for more information.

---

Copy and pasting from the other PR here:

We can definitely make the warning stand out, so this change is welcome, but keep in mind this is not specific to mix test. Modules are always lazy loaded by default, except on releases or when a specific flag is given.

so testing this function may cause irreproducible results depending on if the target module was compiled due to code changes.

FWIW, Elixir v1.19 no longer loads modules during compilation, so the behaviour will be more consistent on Elixir v1.19 as well, but still dependent if another test loaded the module first.

[ityonemo]

ok but i think calling out tests somehow is important because that's where expectations are most easily subverted, because module load state is global and might be dependent on what random order the tests get run in, or if you're isolating a single test. in prod its a nonissue and in dev you're likely to have an early stage that calls the module and hydrates it that doesnt trigger if youre running a single test or possibly races which can be frustrating.

[ityonemo]

I'll merge in a suggestion that keeps the generality but calls out tests

[josevalim]

Can you please provide an example where this is more likely to happen in tests than in dev?

it can also be an issue in prod if you are not using releases. So I am not sure we should call out tests in particular. It is an issue in all environments except releases.

[ityonemo]

I have a private repo where this was a problem, specifically due to testing a /support/...ex module (doesn't even exist in dev) that provides a struct and callback for an optimization algorithm (so I in this case I have a test struct that exists solely to provide the optional callback). If i call out just the test that tries the optional algorithm, the optimization does not trigger. But the optimization does trigger if I mix clean first. Of course this specific situation will be "fixed" in 1.19 but if I had another test that called the module on an non-optional callback function, then in full mix test it would be random (the test with the non-optional function races with the optional callback test) and if i called out the just the one test on the optional callback it would always fail.

This was several hours of debugging when there was absolutely nothing wrong at all.

[josevalim]

This example happened in test because that’s when your were running this code, but it could also completely happen in development in a Phoenix app, where it would or not depending if the code reloader ran or if a previous request loaded said module. It could even happen in prod if you were using mix.

I understand it was a frustrating experience but this is not a test exclusive behavior. :)

[ityonemo]

I'm not claiming that it's test exclusive. I'm claiming it's most likely to occur to people in test. This didn't happen to me in this case but test cycles can also get really long, if it has to go through CI, for example, the feedback loop on this could be minutes, and if we wind up in a situation where this sort of a test failure is random, it could get especially frustrating if the race has a much higher likelihood on CI than in dev.

[josevalim]

Thanks, I will merge it and update some notes locally.

[josevalim]

💚 💙 💜 💛 ❤️