Emit module trace atomically by using LLVM's LockFileManager

[artemcm]

Ensuring that creation of and writing to the module trace file happens atomically overall.

LLVM provides utilities to perform an atomic write-to-file:
https://github.com/llvm/llvm-project/blob/5de69e16ea9ab916401f4a8390fff91f18bbba2a/llvm/include/llvm/Support/FileUtilities.h#L111

However, we must support emitting module trace that appends to an existing file, which isn't currently supported by those utilities since they write-to-temporary and perform an atomic rename to achieve this atomicity.

Instead, use the same approach as the ModuleInterfaceBuilder and place a filesystem lock on the module trace output file in order to do the write.

Resolves rdar://76743462

[artemcm]

@swift-ci please test

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 71184f0b69e21eb91e690e3b436f3d83dce0442c

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 71184f0b69e21eb91e690e3b436f3d83dce0442c

[tschuett]

https://github.com/llvm/llvm-project/blob/5de69e16ea9ab916401f4a8390fff91f18bbba2a/llvm/include/llvm/Support/FileUtilities.h#L111?

[artemcm]

@swift-ci please test

[artemcm]

https://github.com/llvm/llvm-project/blob/5de69e16ea9ab916401f4a8390fff91f18bbba2a/llvm/include/llvm/Support/FileUtilities.h#L111?

Thanks @tschuett! I didn't know about this tool and with my original approach, this would have been much much better.

However, I later realized that we must be able to atomically append to an existing trace file (even testing for that in loaded_module_trace_append.swift), which we cannot do using these utilities. I opted for an approach similar to ModuleInterfaceBuilder that uses filesystem locks to achieve atomicity.

[artemcm]

@swift-ci please test

[xymus]

IIUC, we're clearing the lock here for the next invocation to work. Could we just clear the lock and try again right away instead? I'd expect that if other processes are currently waiting on this lock they'd trigger right away too.

[artemcm]

My thinking was to fail here because this timeout is 256 seconds and I wanted to avoid waiting forever if this times out. Is there not a risk that something goes wrong with this lock - we are not able to acquire it - and we hang forever, if we re-try here?

[xymus]

Yeah that makes sense, I see 2 possible problematic situations here.

Too many processes are competing for access to the lock file and each one is taking too long. We've seen this before creating long hangs but it's less likely here as the work is simple.

The other one would be if an interruption prevents one run from clearing the lock then we'd have the interrupted run that failed (as expected), the next run would fail on the lock, and only the third run would work.

I don't know which is more of a concern.

[artemcm]

Let's go with clearing the lock and re-trying here, because it seems more-likely to be the case than too much contention in this particular scenario, to me. If we see compiler hangs when this feature is used, we'll know where to look...

[artemcm]

@swift-ci please test

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - cf47257878b0f892da7f9c435cdffc2c2925c2bd

[artemcm]

@swift-ci please test

[artemcm]

@swift-ci please test Windows platform

[artemcm]

@swift-ci please test