Allow ccache to reuse results across build directories #1522
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR proposes adding environment variables instructing ccache to allow reuse of compilation results across different build directories. Verified by this patch (the unexpected task failures seem unrelated to this PR).
This is motivated by the realization that Evergreen tasks are typically executed under a working directory of the form
/data/mci/<hash>
, where<hash>
is used to avoid conflicts between tasks running on the given host. Unfortunately this is also preventing ccache from reusing compilation results as intended.Per ccache documentation under "Compiling in different directories":
The presence of
<hash>
in the absolute paths means every execution of a task will fail to reuse cached compilation results on the given host... or even re-execution of the same task on the same host, as<hash>
is apparently computed using a combination of task ID, execution number, and PID. A proper solution would probably also involve a remote storage backend (so that cached results can be reused across hosts as well), but I have not explored how to go about supporting such a setup yet. Instead, this PR applies the instructions given "to enable cache hits between different build directories":This PR applies both suggestions using the environment variables
CCACHE_BASEDIR
andCCACHE_NOHASHDIR
. These are only added to scripts that are expected to be executed on non-Windows-like distros (our Windows tasks don't appear to be using ccache anyways). The scope of the env vars are deliberately such that they (generally) only apply to our builds (positioned immediately before CMake configure commands run on the C Driver, which also identifies the directory to use asbase_dir
, and unset as necessary to avoid impacting unrelated builds).I've elected to use the path to the CMake source directory (as identified by the CMake configure command) as
base_dir
, since (I believe) paths to source files (including header files and include directories) are primarily what impact the ccache hash, and these should be consistent regardless of the location of the source directory to maximize cache hits. Incidentally, for many tasks in the C Driver, this is equivalent to the binary directory (meaning they are in-source builds, which we should probably change to be out-of-source builds at some point...).I do not expect these changes to lead to problems with cache reuse on Evergreen hosts. The hash still includes many toolchain and configuration details which in aggregate are unlikely to lead to undesirable conflicts. Concerning
base_dir
, ccache warns:This is probably not a concern for our EVG tasks, which are always(?) doing a clean build, thus even if dependency detection is flawed, so long as the required artifacts are still built, it should not be an issue. Similarly, concerning
hash_dir
:I do not think we will mind the
<hash>
in/data/mci/<hash>
being different in debug info so long as the relative paths to actual source and binary files remains consistent and understandable, which should be the case given the other information that are still included in the ccache hash (preprocessor output, preprocessor and compiler options, input source file, etc.).