Skip to content

[🍒][llvm][cas] Add validate-if-needed to recover from invalid data #10602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
May 1, 2025
Merged

[🍒][llvm][cas] Add validate-if-needed to recover from invalid data #10602

merged 8 commits into from
May 1, 2025

Conversation

benlangmuir
Copy link

AmrDeveloper and others added 8 commits May 1, 2025 10:39
@AmrDeveloper @benlangmuir
[LLVM][Support] Add new CreateFileError functions (llvm#125906)
8430421 
Add new CreateFileError functions to create a StringError with the
specified error code and prepend the file path to it

Needed for: llvm#125345

(cherry picked from commit 2464f4b)
@benlangmuir
[llvm][cas] Extend on-disk CAS validation to ActionCache
d2b9a1b 
Validate the ActionCache hash-mapped trie structure and sanity check the
resulting values. Unlike the CAS itself there is no direct way to check
the values are "correct", but at least we can check for invalid zero
offsets, which is what we would get if we dropped page writes or
truncated the file.

(cherry picked from commit 2966de4)
@benlangmuir
[llvm][cas] Fix build with LLVM_CAS_ENABLE_REMOTE_CACHE (ActionCache:…
9fd0c15 
…:validate)

(cherry picked from commit eb2d1ea)
@benlangmuir
[llvm][cas] Add validate-if-needed to recover from invalid data
a316c4e 
Introduce a new validate-if-needed API to the UnifiedOnDiskCache and
llvm-cas tool that triggers out-of-process validation of the CAS once
for every machine boot, and optionally recovers from invalid data by
marking it for garbage collection. This fixes a hole in the CAS data
coherence when a power loss or similar failure causes the OS to not
flush all of the pages in the mmaped on-disk CAS files.

The intent is that clients such as the clang scanning daemon or a build
system should trigger this validation at least once before using the
CAS.

rdar://123542312
(cherry picked from commit 17aa9cf)
@benlangmuir
[clang][cas] Pass through LLVM_CAS_LOG env var to the depscan daemon
46e43bb 
Ensure that the scanning daemon modifications to the CAS are captured
properly by the log.

(cherry picked from commit f6a775f)
@benlangmuir
[clang][cas] Adopt validate-if-needed in clang scanning daemon
87b9f40 
Use the new validate-if-needed functionality to ensure the clang
scanning daemon's CAS data is valid.

(cherry picked from commit 1101d73)
@benlangmuir
Updates for review
71f0197 
(cherry picked from commit cb7bc8c)
@benlangmuir
Handle EEXIST for Linux rename failure
fe19246 
Documentation says it can either be ENOTEMPTY (like Darwin) or EEXIST.
Also print the error.

(cherry picked from commit bb52d96)
@benlangmuir benlangmuir requested a review from a team as a code owner May 1, 2025 18:04
@benlangmuir
Copy link
Author

@swift-ci please test

@benlangmuir
Copy link
Author

@swift-ci please test llvm

@benlangmuir
Copy link
Author

@swift-ci please test macOS

@benlangmuir
Copy link
Author

The macOS failure was unrelated.

@mikeash mikeash merged commit 94374fe into swiftlang:swift/release/6.2 May 1, 2025
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akyrtzi akyrtzi akyrtzi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@benlangmuir @akyrtzi @mikeash @AmrDeveloper