Skip to content

[clang][modules] Introduce new ModuleCache interface #131193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 14, 2025

Conversation

jansvoboda11
Copy link
Contributor

This PR adds new ModuleCache interface to Clang's implicitly-built modules machinery. The main motivation for this change is to create a second implementation that uses a more efficient kind of llvm::AdvisoryLock during dependency scanning.

In addition to the lock abstraction, the ModuleCache interface also manages the existing InMemoryModuleCache instance. I found that compared to keeping these separate/independent, the code is a bit simpler now, since these are two tightly coupled concepts. I can envision a more efficient implementation of the InMemoryModuleCache for the single-process case too, which will be much easier to implement with the current setup.

This is not intended to be a functional change.

@llvmbot llvmbot added clang Clang issues not falling into any other category clang-tools-extra clangd clang:modules C++20 modules and Clang Header Modules labels Mar 13, 2025
@llvmbot
Copy link
Member

llvmbot commented Mar 13, 2025

@llvm/pr-subscribers-clang-modules

@llvm/pr-subscribers-clangd

Author: Jan Svoboda (jansvoboda11)

Changes

This PR adds new ModuleCache interface to Clang's implicitly-built modules machinery. The main motivation for this change is to create a second implementation that uses a more efficient kind of llvm::AdvisoryLock during dependency scanning.

In addition to the lock abstraction, the ModuleCache interface also manages the existing InMemoryModuleCache instance. I found that compared to keeping these separate/independent, the code is a bit simpler now, since these are two tightly coupled concepts. I can envision a more efficient implementation of the InMemoryModuleCache for the single-process case too, which will be much easier to implement with the current setup.

This is not intended to be a functional change.


Patch is 40.95 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/131193.diff

18 Files Affected:

  • (modified) clang-tools-extra/clangd/ModulesBuilder.cpp (+3-3)
  • (modified) clang/include/clang/Frontend/ASTUnit.h (+2-2)
  • (modified) clang/include/clang/Frontend/CompilerInstance.h (+6-7)
  • (modified) clang/include/clang/Serialization/ASTReader.h (+5-6)
  • (modified) clang/include/clang/Serialization/ASTWriter.h (+10-11)
  • (added) clang/include/clang/Serialization/ModuleCache.h (+45)
  • (modified) clang/include/clang/Serialization/ModuleManager.h (+6-6)
  • (modified) clang/lib/Frontend/ASTUnit.cpp (+10-11)
  • (modified) clang/lib/Frontend/CompilerInstance.cpp (+20-22)
  • (modified) clang/lib/Frontend/PrecompiledPreamble.cpp (+2-3)
  • (modified) clang/lib/Serialization/ASTReader.cpp (+25-21)
  • (modified) clang/lib/Serialization/ASTWriter.cpp (+6-6)
  • (modified) clang/lib/Serialization/CMakeLists.txt (+1)
  • (modified) clang/lib/Serialization/GeneratePCH.cpp (+4-4)
  • (added) clang/lib/Serialization/ModuleCache.cpp (+44)
  • (modified) clang/lib/Serialization/ModuleManager.cpp (+12-8)
  • (modified) clang/unittests/Frontend/FrontendActionTest.cpp (+5-2)
  • (modified) clang/unittests/Lex/HeaderSearchTest.cpp (-1)
diff --git a/clang-tools-extra/clangd/ModulesBuilder.cpp b/clang-tools-extra/clangd/ModulesBuilder.cpp
index 08a7b250a8119..44307b8a28b93 100644
--- a/clang-tools-extra/clangd/ModulesBuilder.cpp
+++ b/clang-tools-extra/clangd/ModulesBuilder.cpp
@@ -12,7 +12,7 @@
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Serialization/ASTReader.h"
-#include "clang/Serialization/InMemoryModuleCache.h"
+#include "clang/Serialization/ModuleCache.h"
 #include "llvm/ADT/ScopeExit.h"
 #include <queue>
 
@@ -206,9 +206,9 @@ bool IsModuleFileUpToDate(PathRef ModuleFilePath,
   Preprocessor PP(std::make_shared<PreprocessorOptions>(), *Diags, LangOpts,
                   SourceMgr, HeaderInfo, ModuleLoader);
 
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache = new InMemoryModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache = getCrossProcessModuleCache();
   PCHContainerOperations PCHOperations;
-  ASTReader Reader(PP, *ModuleCache, /*ASTContext=*/nullptr,
+  ASTReader Reader(PP, *ModCache, /*ASTContext=*/nullptr,
                    PCHOperations.getRawReader(), {});
 
   // We don't need any listener here. By default it will use a validator
diff --git a/clang/include/clang/Frontend/ASTUnit.h b/clang/include/clang/Frontend/ASTUnit.h
index 1f98c6ab328ba..248bbe1657f8b 100644
--- a/clang/include/clang/Frontend/ASTUnit.h
+++ b/clang/include/clang/Frontend/ASTUnit.h
@@ -70,7 +70,7 @@ class FileManager;
 class FrontendAction;
 class HeaderSearch;
 class InputKind;
-class InMemoryModuleCache;
+class ModuleCache;
 class PCHContainerOperations;
 class PCHContainerReader;
 class Preprocessor;
@@ -110,7 +110,7 @@ class ASTUnit {
   IntrusiveRefCntPtr<DiagnosticsEngine>   Diagnostics;
   IntrusiveRefCntPtr<FileManager>         FileMgr;
   IntrusiveRefCntPtr<SourceManager>       SourceMgr;
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache;
   std::unique_ptr<HeaderSearch>           HeaderInfo;
   IntrusiveRefCntPtr<TargetInfo>          Target;
   std::shared_ptr<Preprocessor>           PP;
diff --git a/clang/include/clang/Frontend/CompilerInstance.h b/clang/include/clang/Frontend/CompilerInstance.h
index 8b539dfc92960..4960d40ca7c37 100644
--- a/clang/include/clang/Frontend/CompilerInstance.h
+++ b/clang/include/clang/Frontend/CompilerInstance.h
@@ -51,8 +51,8 @@ class DiagnosticsEngine;
 class DiagnosticConsumer;
 class FileManager;
 class FrontendAction;
-class InMemoryModuleCache;
 class Module;
+class ModuleCache;
 class Preprocessor;
 class Sema;
 class SourceManager;
@@ -97,7 +97,7 @@ class CompilerInstance : public ModuleLoader {
   IntrusiveRefCntPtr<SourceManager> SourceMgr;
 
   /// The cache of PCM files.
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache;
 
   /// The preprocessor.
   std::shared_ptr<Preprocessor> PP;
@@ -209,7 +209,7 @@ class CompilerInstance : public ModuleLoader {
   explicit CompilerInstance(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps =
           std::make_shared<PCHContainerOperations>(),
-      InMemoryModuleCache *SharedModuleCache = nullptr);
+      ModuleCache *ModCache = nullptr);
   ~CompilerInstance() override;
 
   /// @name High-Level Operations
@@ -746,9 +746,8 @@ class CompilerInstance : public ModuleLoader {
   static IntrusiveRefCntPtr<ASTReader> createPCHExternalASTSource(
       StringRef Path, StringRef Sysroot,
       DisableValidationForModuleKind DisableValidation,
-      bool AllowPCHWithCompilerErrors, Preprocessor &PP,
-      InMemoryModuleCache &ModuleCache, ASTContext &Context,
-      const PCHContainerReader &PCHContainerRdr,
+      bool AllowPCHWithCompilerErrors, Preprocessor &PP, ModuleCache &ModCache,
+      ASTContext &Context, const PCHContainerReader &PCHContainerRdr,
       ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
       ArrayRef<std::shared_ptr<DependencyCollector>> DependencyCollectors,
       void *DeserializationListener, bool OwnDeserializationListener,
@@ -896,7 +895,7 @@ class CompilerInstance : public ModuleLoader {
 
   void setExternalSemaSource(IntrusiveRefCntPtr<ExternalSemaSource> ESS);
 
-  InMemoryModuleCache &getModuleCache() const { return *ModuleCache; }
+  ModuleCache &getModuleCache() const { return *ModCache; }
 };
 
 } // end namespace clang
diff --git a/clang/include/clang/Serialization/ASTReader.h b/clang/include/clang/Serialization/ASTReader.h
index 47301419c76c6..143b798a8348a 100644
--- a/clang/include/clang/Serialization/ASTReader.h
+++ b/clang/include/clang/Serialization/ASTReader.h
@@ -89,7 +89,7 @@ struct HeaderFileInfo;
 class HeaderSearchOptions;
 class LangOptions;
 class MacroInfo;
-class InMemoryModuleCache;
+class ModuleCache;
 class NamedDecl;
 class NamespaceDecl;
 class ObjCCategoryDecl;
@@ -1742,8 +1742,8 @@ class ASTReader
   ///
   /// \param ReadTimer If non-null, a timer used to track the time spent
   /// deserializing.
-  ASTReader(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
-            ASTContext *Context, const PCHContainerReader &PCHContainerRdr,
+  ASTReader(Preprocessor &PP, ModuleCache &ModCache, ASTContext *Context,
+            const PCHContainerReader &PCHContainerRdr,
             ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
             StringRef isysroot = "",
             DisableValidationForModuleKind DisableValidationKind =
@@ -1954,8 +1954,7 @@ class ASTReader
   ///
   /// \returns true if an error occurred, false otherwise.
   static bool readASTFileControlBlock(
-      StringRef Filename, FileManager &FileMgr,
-      const InMemoryModuleCache &ModuleCache,
+      StringRef Filename, FileManager &FileMgr, const ModuleCache &ModCache,
       const PCHContainerReader &PCHContainerRdr, bool FindModuleFileExtensions,
       ASTReaderListener &Listener, bool ValidateDiagnosticOptions,
       unsigned ClientLoadCapabilities = ARR_ConfigurationMismatch |
@@ -1964,7 +1963,7 @@ class ASTReader
   /// Determine whether the given AST file is acceptable to load into a
   /// translation unit with the given language and target options.
   static bool isAcceptableASTFile(StringRef Filename, FileManager &FileMgr,
-                                  const InMemoryModuleCache &ModuleCache,
+                                  const ModuleCache &ModCache,
                                   const PCHContainerReader &PCHContainerRdr,
                                   const LangOptions &LangOpts,
                                   const TargetOptions &TargetOpts,
diff --git a/clang/include/clang/Serialization/ASTWriter.h b/clang/include/clang/Serialization/ASTWriter.h
index 7371a679988e9..6b5cc181e2e13 100644
--- a/clang/include/clang/Serialization/ASTWriter.h
+++ b/clang/include/clang/Serialization/ASTWriter.h
@@ -60,7 +60,7 @@ class LangOptions;
 class MacroDefinitionRecord;
 class MacroInfo;
 class Module;
-class InMemoryModuleCache;
+class ModuleCache;
 class ModuleFileExtension;
 class ModuleFileExtensionWriter;
 class NamedDecl;
@@ -117,7 +117,7 @@ class ASTWriter : public ASTDeserializationListener,
   const SmallVectorImpl<char> &Buffer;
 
   /// The PCM manager which manages memory buffers for pcm files.
-  InMemoryModuleCache &ModuleCache;
+  ModuleCache &ModCache;
 
   /// The preprocessor we're writing.
   Preprocessor *PP = nullptr;
@@ -682,7 +682,7 @@ class ASTWriter : public ASTDeserializationListener,
   /// Create a new precompiled header writer that outputs to
   /// the given bitstream.
   ASTWriter(llvm::BitstreamWriter &Stream, SmallVectorImpl<char> &Buffer,
-            InMemoryModuleCache &ModuleCache,
+            ModuleCache &ModCache,
             ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
             bool IncludeTimestamps = true, bool BuildingImplicitModule = false,
             bool GeneratingReducedBMI = false);
@@ -986,9 +986,8 @@ class PCHGenerator : public SemaConsumer {
   virtual Module *getEmittingModule(ASTContext &Ctx);
 
 public:
-  PCHGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
-               StringRef OutputFile, StringRef isysroot,
-               std::shared_ptr<PCHBuffer> Buffer,
+  PCHGenerator(Preprocessor &PP, ModuleCache &ModCache, StringRef OutputFile,
+               StringRef isysroot, std::shared_ptr<PCHBuffer> Buffer,
                ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
                bool AllowASTWithErrors = false, bool IncludeTimestamps = true,
                bool BuildingImplicitModule = false,
@@ -1010,14 +1009,14 @@ class CXX20ModulesGenerator : public PCHGenerator {
 protected:
   virtual Module *getEmittingModule(ASTContext &Ctx) override;
 
-  CXX20ModulesGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
+  CXX20ModulesGenerator(Preprocessor &PP, ModuleCache &ModCache,
                         StringRef OutputFile, bool GeneratingReducedBMI,
                         bool AllowASTWithErrors);
 
 public:
-  CXX20ModulesGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
+  CXX20ModulesGenerator(Preprocessor &PP, ModuleCache &ModCache,
                         StringRef OutputFile, bool AllowASTWithErrors = false)
-      : CXX20ModulesGenerator(PP, ModuleCache, OutputFile,
+      : CXX20ModulesGenerator(PP, ModCache, OutputFile,
                               /*GeneratingReducedBMI=*/false,
                               AllowASTWithErrors) {}
 
@@ -1028,9 +1027,9 @@ class ReducedBMIGenerator : public CXX20ModulesGenerator {
   void anchor() override;
 
 public:
-  ReducedBMIGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
+  ReducedBMIGenerator(Preprocessor &PP, ModuleCache &ModCache,
                       StringRef OutputFile, bool AllowASTWithErrors = false)
-      : CXX20ModulesGenerator(PP, ModuleCache, OutputFile,
+      : CXX20ModulesGenerator(PP, ModCache, OutputFile,
                               /*GeneratingReducedBMI=*/true,
                               AllowASTWithErrors) {}
 };
diff --git a/clang/include/clang/Serialization/ModuleCache.h b/clang/include/clang/Serialization/ModuleCache.h
new file mode 100644
index 0000000000000..e5f55111259fd
--- /dev/null
+++ b/clang/include/clang/Serialization/ModuleCache.h
@@ -0,0 +1,45 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_SERIALIZATION_MODULECACHE_H
+#define LLVM_CLANG_SERIALIZATION_MODULECACHE_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+
+namespace llvm {
+class AdvisoryLock;
+} // namespace llvm
+
+namespace clang {
+class InMemoryModuleCache;
+
+/// The module cache used by implicitly-built modules.
+class ModuleCache : public RefCountedBase<ModuleCache> {
+public:
+  /// May perform any work that only needs to be performed once for multiple
+  /// calls \c getLock() with the same module filename.
+  virtual void prepareForGetLock(StringRef ModuleFilename) = 0;
+
+  /// Returns lock for the given module file. The lock is initially unlocked.
+  virtual std::unique_ptr<llvm::AdvisoryLock>
+  getLock(StringRef ModuleFilename) = 0;
+
+  /// Returns this process's view of the module cache.
+  virtual InMemoryModuleCache &getInMemoryModuleCache() = 0;
+  virtual const InMemoryModuleCache &getInMemoryModuleCache() const = 0;
+
+  // TODO: Virtualize writing/reading PCM files, timestamp files, etc.
+
+  virtual ~ModuleCache() = default;
+};
+
+IntrusiveRefCntPtr<ModuleCache> getCrossProcessModuleCache();
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Serialization/ModuleManager.h b/clang/include/clang/Serialization/ModuleManager.h
index f898dab39f06d..1eb74aee9787c 100644
--- a/clang/include/clang/Serialization/ModuleManager.h
+++ b/clang/include/clang/Serialization/ModuleManager.h
@@ -37,7 +37,7 @@ class FileEntry;
 class FileManager;
 class GlobalModuleIndex;
 class HeaderSearch;
-class InMemoryModuleCache;
+class ModuleCache;
 class PCHContainerReader;
 
 namespace serialization {
@@ -65,7 +65,7 @@ class ModuleManager {
   FileManager &FileMgr;
 
   /// Cache of PCM files.
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache;
 
   /// Knows how to unwrap module containers.
   const PCHContainerReader &PCHContainerRdr;
@@ -133,9 +133,9 @@ class ModuleManager {
       SmallVectorImpl<std::unique_ptr<ModuleFile>>::reverse_iterator>;
   using ModuleOffset = std::pair<uint32_t, StringRef>;
 
-  explicit ModuleManager(FileManager &FileMgr, InMemoryModuleCache &ModuleCache,
-                         const PCHContainerReader &PCHContainerRdr,
-                         const HeaderSearch &HeaderSearchInfo);
+  ModuleManager(FileManager &FileMgr, ModuleCache &ModCache,
+                const PCHContainerReader &PCHContainerRdr,
+                const HeaderSearch &HeaderSearchInfo);
 
   /// Forward iterator to traverse all loaded modules.
   ModuleIterator begin() { return Chain.begin(); }
@@ -306,7 +306,7 @@ class ModuleManager {
   /// View the graphviz representation of the module graph.
   void viewGraph();
 
-  InMemoryModuleCache &getModuleCache() const { return *ModuleCache; }
+  ModuleCache &getModuleCache() const { return *ModCache; }
 };
 
 } // namespace serialization
diff --git a/clang/lib/Frontend/ASTUnit.cpp b/clang/lib/Frontend/ASTUnit.cpp
index 8df5465ad990d..3eb9959f45857 100644
--- a/clang/lib/Frontend/ASTUnit.cpp
+++ b/clang/lib/Frontend/ASTUnit.cpp
@@ -61,7 +61,7 @@
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/ASTWriter.h"
 #include "clang/Serialization/ContinuousRangeMap.h"
-#include "clang/Serialization/InMemoryModuleCache.h"
+#include "clang/Serialization/ModuleCache.h"
 #include "clang/Serialization/ModuleFile.h"
 #include "clang/Serialization/PCHContainerOperations.h"
 #include "llvm/ADT/ArrayRef.h"
@@ -219,8 +219,8 @@ struct ASTUnit::ASTWriterData {
   llvm::BitstreamWriter Stream;
   ASTWriter Writer;
 
-  ASTWriterData(InMemoryModuleCache &ModuleCache)
-      : Stream(Buffer), Writer(Stream, Buffer, ModuleCache, {}) {}
+  ASTWriterData(ModuleCache &ModCache)
+      : Stream(Buffer), Writer(Stream, Buffer, ModCache, {}) {}
 };
 
 void ASTUnit::clearFileLevelDecls() {
@@ -829,7 +829,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(),
                                      AST->getFileManager(),
                                      UserFilesAreVolatile);
-  AST->ModuleCache = new InMemoryModuleCache;
+  AST->ModCache = getCrossProcessModuleCache();
   AST->HSOpts = HSOpts ? HSOpts : std::make_shared<HeaderSearchOptions>();
   AST->HSOpts->ModuleFormat = std::string(PCHContainerRdr.getFormats().front());
   AST->HeaderInfo.reset(new HeaderSearch(AST->HSOpts,
@@ -861,8 +861,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
   if (::getenv("LIBCLANG_DISABLE_PCH_VALIDATION"))
     disableValid = DisableValidationForModuleKind::All;
   AST->Reader = new ASTReader(
-      PP, *AST->ModuleCache, AST->Ctx.get(), PCHContainerRdr, {},
-      /*isysroot=*/"",
+      PP, *AST->ModCache, AST->Ctx.get(), PCHContainerRdr, {}, /*isysroot=*/"",
       /*DisableValidationKind=*/disableValid, AllowASTWithCompilerErrors);
 
   unsigned Counter = 0;
@@ -1546,7 +1545,7 @@ ASTUnit::create(std::shared_ptr<CompilerInvocation> CI,
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(), *AST->FileMgr,
                                      UserFilesAreVolatile);
-  AST->ModuleCache = new InMemoryModuleCache;
+  AST->ModCache = getCrossProcessModuleCache();
 
   return AST;
 }
@@ -1833,7 +1832,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromCommandLine(
   AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
   AST->StorePreamblesInMemory = StorePreamblesInMemory;
   AST->PreambleStoragePath = PreambleStoragePath;
-  AST->ModuleCache = new InMemoryModuleCache;
+  AST->ModCache = getCrossProcessModuleCache();
   AST->OnlyLocalDecls = OnlyLocalDecls;
   AST->CaptureDiagnostics = CaptureDiagnostics;
   AST->TUKind = TUKind;
@@ -1844,7 +1843,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromCommandLine(
   AST->Invocation = CI;
   AST->SkipFunctionBodies = SkipFunctionBodies;
   if (ForSerialization)
-    AST->WriterData.reset(new ASTWriterData(*AST->ModuleCache));
+    AST->WriterData.reset(new ASTWriterData(*AST->ModCache));
   // Zero out now to ease cleanup during crash recovery.
   CI = nullptr;
   Diags = nullptr;
@@ -2379,8 +2378,8 @@ bool ASTUnit::serialize(raw_ostream &OS) {
 
   SmallString<128> Buffer;
   llvm::BitstreamWriter Stream(Buffer);
-  InMemoryModuleCache ModuleCache;
-  ASTWriter Writer(Stream, Buffer, ModuleCache, {});
+  IntrusiveRefCntPtr<ModuleCache> ModCache = getCrossProcessModuleCache();
+  ASTWriter Writer(Stream, Buffer, *ModCache, {});
   return serializeUnit(Writer, Buffer, getSema(), OS);
 }
 
diff --git a/clang/lib/Frontend/CompilerInstance.cpp b/clang/lib/Frontend/CompilerInstance.cpp
index fc350f2ba42e0..b5c4de309c4bf 100644
--- a/clang/lib/Frontend/CompilerInstance.cpp
+++ b/clang/lib/Frontend/CompilerInstance.cpp
@@ -39,16 +39,17 @@
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
 #include "clang/Serialization/InMemoryModuleCache.h"
+#include "clang/Serialization/ModuleCache.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Config/llvm-config.h"
+#include "llvm/Support/AdvisoryLock.h"
 #include "llvm/Support/BuryPointer.h"
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/FileSystem.h"
-#include "llvm/Support/LockFileManager.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
@@ -66,11 +67,10 @@ using namespace clang;
 
 CompilerInstance::CompilerInstance(
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-    InMemoryModuleCache *SharedModuleCache)
-    : ModuleLoader(/* BuildingModule = */ SharedModuleCache),
+    ModuleCache *ModCache)
+    : ModuleLoader(/*BuildingModule=*/ModCache),
       Invocation(new CompilerInvocation()),
-      ModuleCache(SharedModuleCache ? SharedModuleCache
-                                    : new InMemoryModuleCache),
+      ModCache(ModCache ? ModCache : getCrossProcessModuleCache()),
       ThePCHContainerOperations(std::move(PCHContainerOps)) {}
 
 CompilerInstance::~CompilerInstance() {
@@ -205,7 +205,7 @@ IntrusiveRefCntPtr<ASTReader> CompilerInstance::getASTReader() const {
   return TheASTReader;
 }
 void CompilerInstance::setASTReader(IntrusiveRefCntPtr<ASTReader> Reader) {
-  assert(ModuleCache.get() == &Reader->getModuleManager().getModuleCache() &&
+  assert(ModCache.get() == &Reader->getModuleManager().getModuleCache() &&
          "Expected ASTReader to use the same PCM cache");
   TheASTReader = std::move(Reader);
 }
@@ -625,9 +625,8 @@ void CompilerInstance::createPCHExternalASTSource(
 IntrusiveRefCntPtr<ASTReader> CompilerInstance::createPCHExternalASTSource(
     StringRef Path, StringRef Sysroot,
     DisableValidationForModuleKind DisableValidation,
-    bool AllowPCHWithCompilerErrors, Preprocessor &PP,
-    InMemoryModuleCache &ModuleCache, ASTContext &Context,
-    const PCHContainerReader &PCHContainerRdr,
+    bool AllowPCHWithCompilerErrors, Preprocessor &PP, ModuleCache &ModCache,
+    ASTContext &Context, const PCHContainerReader &PCHContainerRdr,
     ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
     ArrayRef<std::shared_ptr<DependencyCollector>> DependencyCollectors,
     void *DeserializationListener, bool OwnDeserializationListener,
@@ -635,7 +634,7 @@ IntrusiveRefCntPtr<ASTReader> CompilerInstance::createPCHExternalASTSource(
   HeaderSearchOptions &HSOpts = PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
   IntrusiveRefCntPtr<ASTReader> Reader(new ASTReader(
-      PP, ModuleCache, &Context, PCHContainerRdr, Extensions,
+      PP, ModCache, &Context, PCHContainerRdr, Extensions,
       Sysro...
[truncated]

@llvmbot
Copy link
Member

llvmbot commented Mar 13, 2025

@llvm/pr-subscribers-clang

Author: Jan Svoboda (jansvoboda11)

Changes

This PR adds new ModuleCache interface to Clang's implicitly-built modules machinery. The main motivation for this change is to create a second implementation that uses a more efficient kind of llvm::AdvisoryLock during dependency scanning.

In addition to the lock abstraction, the ModuleCache interface also manages the existing InMemoryModuleCache instance. I found that compared to keeping these separate/independent, the code is a bit simpler now, since these are two tightly coupled concepts. I can envision a more efficient implementation of the InMemoryModuleCache for the single-process case too, which will be much easier to implement with the current setup.

This is not intended to be a functional change.


Patch is 40.95 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/131193.diff

18 Files Affected:

  • (modified) clang-tools-extra/clangd/ModulesBuilder.cpp (+3-3)
  • (modified) clang/include/clang/Frontend/ASTUnit.h (+2-2)
  • (modified) clang/include/clang/Frontend/CompilerInstance.h (+6-7)
  • (modified) clang/include/clang/Serialization/ASTReader.h (+5-6)
  • (modified) clang/include/clang/Serialization/ASTWriter.h (+10-11)
  • (added) clang/include/clang/Serialization/ModuleCache.h (+45)
  • (modified) clang/include/clang/Serialization/ModuleManager.h (+6-6)
  • (modified) clang/lib/Frontend/ASTUnit.cpp (+10-11)
  • (modified) clang/lib/Frontend/CompilerInstance.cpp (+20-22)
  • (modified) clang/lib/Frontend/PrecompiledPreamble.cpp (+2-3)
  • (modified) clang/lib/Serialization/ASTReader.cpp (+25-21)
  • (modified) clang/lib/Serialization/ASTWriter.cpp (+6-6)
  • (modified) clang/lib/Serialization/CMakeLists.txt (+1)
  • (modified) clang/lib/Serialization/GeneratePCH.cpp (+4-4)
  • (added) clang/lib/Serialization/ModuleCache.cpp (+44)
  • (modified) clang/lib/Serialization/ModuleManager.cpp (+12-8)
  • (modified) clang/unittests/Frontend/FrontendActionTest.cpp (+5-2)
  • (modified) clang/unittests/Lex/HeaderSearchTest.cpp (-1)
diff --git a/clang-tools-extra/clangd/ModulesBuilder.cpp b/clang-tools-extra/clangd/ModulesBuilder.cpp
index 08a7b250a8119..44307b8a28b93 100644
--- a/clang-tools-extra/clangd/ModulesBuilder.cpp
+++ b/clang-tools-extra/clangd/ModulesBuilder.cpp
@@ -12,7 +12,7 @@
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Serialization/ASTReader.h"
-#include "clang/Serialization/InMemoryModuleCache.h"
+#include "clang/Serialization/ModuleCache.h"
 #include "llvm/ADT/ScopeExit.h"
 #include <queue>
 
@@ -206,9 +206,9 @@ bool IsModuleFileUpToDate(PathRef ModuleFilePath,
   Preprocessor PP(std::make_shared<PreprocessorOptions>(), *Diags, LangOpts,
                   SourceMgr, HeaderInfo, ModuleLoader);
 
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache = new InMemoryModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache = getCrossProcessModuleCache();
   PCHContainerOperations PCHOperations;
-  ASTReader Reader(PP, *ModuleCache, /*ASTContext=*/nullptr,
+  ASTReader Reader(PP, *ModCache, /*ASTContext=*/nullptr,
                    PCHOperations.getRawReader(), {});
 
   // We don't need any listener here. By default it will use a validator
diff --git a/clang/include/clang/Frontend/ASTUnit.h b/clang/include/clang/Frontend/ASTUnit.h
index 1f98c6ab328ba..248bbe1657f8b 100644
--- a/clang/include/clang/Frontend/ASTUnit.h
+++ b/clang/include/clang/Frontend/ASTUnit.h
@@ -70,7 +70,7 @@ class FileManager;
 class FrontendAction;
 class HeaderSearch;
 class InputKind;
-class InMemoryModuleCache;
+class ModuleCache;
 class PCHContainerOperations;
 class PCHContainerReader;
 class Preprocessor;
@@ -110,7 +110,7 @@ class ASTUnit {
   IntrusiveRefCntPtr<DiagnosticsEngine>   Diagnostics;
   IntrusiveRefCntPtr<FileManager>         FileMgr;
   IntrusiveRefCntPtr<SourceManager>       SourceMgr;
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache;
   std::unique_ptr<HeaderSearch>           HeaderInfo;
   IntrusiveRefCntPtr<TargetInfo>          Target;
   std::shared_ptr<Preprocessor>           PP;
diff --git a/clang/include/clang/Frontend/CompilerInstance.h b/clang/include/clang/Frontend/CompilerInstance.h
index 8b539dfc92960..4960d40ca7c37 100644
--- a/clang/include/clang/Frontend/CompilerInstance.h
+++ b/clang/include/clang/Frontend/CompilerInstance.h
@@ -51,8 +51,8 @@ class DiagnosticsEngine;
 class DiagnosticConsumer;
 class FileManager;
 class FrontendAction;
-class InMemoryModuleCache;
 class Module;
+class ModuleCache;
 class Preprocessor;
 class Sema;
 class SourceManager;
@@ -97,7 +97,7 @@ class CompilerInstance : public ModuleLoader {
   IntrusiveRefCntPtr<SourceManager> SourceMgr;
 
   /// The cache of PCM files.
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache;
 
   /// The preprocessor.
   std::shared_ptr<Preprocessor> PP;
@@ -209,7 +209,7 @@ class CompilerInstance : public ModuleLoader {
   explicit CompilerInstance(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps =
           std::make_shared<PCHContainerOperations>(),
-      InMemoryModuleCache *SharedModuleCache = nullptr);
+      ModuleCache *ModCache = nullptr);
   ~CompilerInstance() override;
 
   /// @name High-Level Operations
@@ -746,9 +746,8 @@ class CompilerInstance : public ModuleLoader {
   static IntrusiveRefCntPtr<ASTReader> createPCHExternalASTSource(
       StringRef Path, StringRef Sysroot,
       DisableValidationForModuleKind DisableValidation,
-      bool AllowPCHWithCompilerErrors, Preprocessor &PP,
-      InMemoryModuleCache &ModuleCache, ASTContext &Context,
-      const PCHContainerReader &PCHContainerRdr,
+      bool AllowPCHWithCompilerErrors, Preprocessor &PP, ModuleCache &ModCache,
+      ASTContext &Context, const PCHContainerReader &PCHContainerRdr,
       ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
       ArrayRef<std::shared_ptr<DependencyCollector>> DependencyCollectors,
       void *DeserializationListener, bool OwnDeserializationListener,
@@ -896,7 +895,7 @@ class CompilerInstance : public ModuleLoader {
 
   void setExternalSemaSource(IntrusiveRefCntPtr<ExternalSemaSource> ESS);
 
-  InMemoryModuleCache &getModuleCache() const { return *ModuleCache; }
+  ModuleCache &getModuleCache() const { return *ModCache; }
 };
 
 } // end namespace clang
diff --git a/clang/include/clang/Serialization/ASTReader.h b/clang/include/clang/Serialization/ASTReader.h
index 47301419c76c6..143b798a8348a 100644
--- a/clang/include/clang/Serialization/ASTReader.h
+++ b/clang/include/clang/Serialization/ASTReader.h
@@ -89,7 +89,7 @@ struct HeaderFileInfo;
 class HeaderSearchOptions;
 class LangOptions;
 class MacroInfo;
-class InMemoryModuleCache;
+class ModuleCache;
 class NamedDecl;
 class NamespaceDecl;
 class ObjCCategoryDecl;
@@ -1742,8 +1742,8 @@ class ASTReader
   ///
   /// \param ReadTimer If non-null, a timer used to track the time spent
   /// deserializing.
-  ASTReader(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
-            ASTContext *Context, const PCHContainerReader &PCHContainerRdr,
+  ASTReader(Preprocessor &PP, ModuleCache &ModCache, ASTContext *Context,
+            const PCHContainerReader &PCHContainerRdr,
             ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
             StringRef isysroot = "",
             DisableValidationForModuleKind DisableValidationKind =
@@ -1954,8 +1954,7 @@ class ASTReader
   ///
   /// \returns true if an error occurred, false otherwise.
   static bool readASTFileControlBlock(
-      StringRef Filename, FileManager &FileMgr,
-      const InMemoryModuleCache &ModuleCache,
+      StringRef Filename, FileManager &FileMgr, const ModuleCache &ModCache,
       const PCHContainerReader &PCHContainerRdr, bool FindModuleFileExtensions,
       ASTReaderListener &Listener, bool ValidateDiagnosticOptions,
       unsigned ClientLoadCapabilities = ARR_ConfigurationMismatch |
@@ -1964,7 +1963,7 @@ class ASTReader
   /// Determine whether the given AST file is acceptable to load into a
   /// translation unit with the given language and target options.
   static bool isAcceptableASTFile(StringRef Filename, FileManager &FileMgr,
-                                  const InMemoryModuleCache &ModuleCache,
+                                  const ModuleCache &ModCache,
                                   const PCHContainerReader &PCHContainerRdr,
                                   const LangOptions &LangOpts,
                                   const TargetOptions &TargetOpts,
diff --git a/clang/include/clang/Serialization/ASTWriter.h b/clang/include/clang/Serialization/ASTWriter.h
index 7371a679988e9..6b5cc181e2e13 100644
--- a/clang/include/clang/Serialization/ASTWriter.h
+++ b/clang/include/clang/Serialization/ASTWriter.h
@@ -60,7 +60,7 @@ class LangOptions;
 class MacroDefinitionRecord;
 class MacroInfo;
 class Module;
-class InMemoryModuleCache;
+class ModuleCache;
 class ModuleFileExtension;
 class ModuleFileExtensionWriter;
 class NamedDecl;
@@ -117,7 +117,7 @@ class ASTWriter : public ASTDeserializationListener,
   const SmallVectorImpl<char> &Buffer;
 
   /// The PCM manager which manages memory buffers for pcm files.
-  InMemoryModuleCache &ModuleCache;
+  ModuleCache &ModCache;
 
   /// The preprocessor we're writing.
   Preprocessor *PP = nullptr;
@@ -682,7 +682,7 @@ class ASTWriter : public ASTDeserializationListener,
   /// Create a new precompiled header writer that outputs to
   /// the given bitstream.
   ASTWriter(llvm::BitstreamWriter &Stream, SmallVectorImpl<char> &Buffer,
-            InMemoryModuleCache &ModuleCache,
+            ModuleCache &ModCache,
             ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
             bool IncludeTimestamps = true, bool BuildingImplicitModule = false,
             bool GeneratingReducedBMI = false);
@@ -986,9 +986,8 @@ class PCHGenerator : public SemaConsumer {
   virtual Module *getEmittingModule(ASTContext &Ctx);
 
 public:
-  PCHGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
-               StringRef OutputFile, StringRef isysroot,
-               std::shared_ptr<PCHBuffer> Buffer,
+  PCHGenerator(Preprocessor &PP, ModuleCache &ModCache, StringRef OutputFile,
+               StringRef isysroot, std::shared_ptr<PCHBuffer> Buffer,
                ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
                bool AllowASTWithErrors = false, bool IncludeTimestamps = true,
                bool BuildingImplicitModule = false,
@@ -1010,14 +1009,14 @@ class CXX20ModulesGenerator : public PCHGenerator {
 protected:
   virtual Module *getEmittingModule(ASTContext &Ctx) override;
 
-  CXX20ModulesGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
+  CXX20ModulesGenerator(Preprocessor &PP, ModuleCache &ModCache,
                         StringRef OutputFile, bool GeneratingReducedBMI,
                         bool AllowASTWithErrors);
 
 public:
-  CXX20ModulesGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
+  CXX20ModulesGenerator(Preprocessor &PP, ModuleCache &ModCache,
                         StringRef OutputFile, bool AllowASTWithErrors = false)
-      : CXX20ModulesGenerator(PP, ModuleCache, OutputFile,
+      : CXX20ModulesGenerator(PP, ModCache, OutputFile,
                               /*GeneratingReducedBMI=*/false,
                               AllowASTWithErrors) {}
 
@@ -1028,9 +1027,9 @@ class ReducedBMIGenerator : public CXX20ModulesGenerator {
   void anchor() override;
 
 public:
-  ReducedBMIGenerator(Preprocessor &PP, InMemoryModuleCache &ModuleCache,
+  ReducedBMIGenerator(Preprocessor &PP, ModuleCache &ModCache,
                       StringRef OutputFile, bool AllowASTWithErrors = false)
-      : CXX20ModulesGenerator(PP, ModuleCache, OutputFile,
+      : CXX20ModulesGenerator(PP, ModCache, OutputFile,
                               /*GeneratingReducedBMI=*/true,
                               AllowASTWithErrors) {}
 };
diff --git a/clang/include/clang/Serialization/ModuleCache.h b/clang/include/clang/Serialization/ModuleCache.h
new file mode 100644
index 0000000000000..e5f55111259fd
--- /dev/null
+++ b/clang/include/clang/Serialization/ModuleCache.h
@@ -0,0 +1,45 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_SERIALIZATION_MODULECACHE_H
+#define LLVM_CLANG_SERIALIZATION_MODULECACHE_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+
+namespace llvm {
+class AdvisoryLock;
+} // namespace llvm
+
+namespace clang {
+class InMemoryModuleCache;
+
+/// The module cache used by implicitly-built modules.
+class ModuleCache : public RefCountedBase<ModuleCache> {
+public:
+  /// May perform any work that only needs to be performed once for multiple
+  /// calls \c getLock() with the same module filename.
+  virtual void prepareForGetLock(StringRef ModuleFilename) = 0;
+
+  /// Returns lock for the given module file. The lock is initially unlocked.
+  virtual std::unique_ptr<llvm::AdvisoryLock>
+  getLock(StringRef ModuleFilename) = 0;
+
+  /// Returns this process's view of the module cache.
+  virtual InMemoryModuleCache &getInMemoryModuleCache() = 0;
+  virtual const InMemoryModuleCache &getInMemoryModuleCache() const = 0;
+
+  // TODO: Virtualize writing/reading PCM files, timestamp files, etc.
+
+  virtual ~ModuleCache() = default;
+};
+
+IntrusiveRefCntPtr<ModuleCache> getCrossProcessModuleCache();
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Serialization/ModuleManager.h b/clang/include/clang/Serialization/ModuleManager.h
index f898dab39f06d..1eb74aee9787c 100644
--- a/clang/include/clang/Serialization/ModuleManager.h
+++ b/clang/include/clang/Serialization/ModuleManager.h
@@ -37,7 +37,7 @@ class FileEntry;
 class FileManager;
 class GlobalModuleIndex;
 class HeaderSearch;
-class InMemoryModuleCache;
+class ModuleCache;
 class PCHContainerReader;
 
 namespace serialization {
@@ -65,7 +65,7 @@ class ModuleManager {
   FileManager &FileMgr;
 
   /// Cache of PCM files.
-  IntrusiveRefCntPtr<InMemoryModuleCache> ModuleCache;
+  IntrusiveRefCntPtr<ModuleCache> ModCache;
 
   /// Knows how to unwrap module containers.
   const PCHContainerReader &PCHContainerRdr;
@@ -133,9 +133,9 @@ class ModuleManager {
       SmallVectorImpl<std::unique_ptr<ModuleFile>>::reverse_iterator>;
   using ModuleOffset = std::pair<uint32_t, StringRef>;
 
-  explicit ModuleManager(FileManager &FileMgr, InMemoryModuleCache &ModuleCache,
-                         const PCHContainerReader &PCHContainerRdr,
-                         const HeaderSearch &HeaderSearchInfo);
+  ModuleManager(FileManager &FileMgr, ModuleCache &ModCache,
+                const PCHContainerReader &PCHContainerRdr,
+                const HeaderSearch &HeaderSearchInfo);
 
   /// Forward iterator to traverse all loaded modules.
   ModuleIterator begin() { return Chain.begin(); }
@@ -306,7 +306,7 @@ class ModuleManager {
   /// View the graphviz representation of the module graph.
   void viewGraph();
 
-  InMemoryModuleCache &getModuleCache() const { return *ModuleCache; }
+  ModuleCache &getModuleCache() const { return *ModCache; }
 };
 
 } // namespace serialization
diff --git a/clang/lib/Frontend/ASTUnit.cpp b/clang/lib/Frontend/ASTUnit.cpp
index 8df5465ad990d..3eb9959f45857 100644
--- a/clang/lib/Frontend/ASTUnit.cpp
+++ b/clang/lib/Frontend/ASTUnit.cpp
@@ -61,7 +61,7 @@
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/ASTWriter.h"
 #include "clang/Serialization/ContinuousRangeMap.h"
-#include "clang/Serialization/InMemoryModuleCache.h"
+#include "clang/Serialization/ModuleCache.h"
 #include "clang/Serialization/ModuleFile.h"
 #include "clang/Serialization/PCHContainerOperations.h"
 #include "llvm/ADT/ArrayRef.h"
@@ -219,8 +219,8 @@ struct ASTUnit::ASTWriterData {
   llvm::BitstreamWriter Stream;
   ASTWriter Writer;
 
-  ASTWriterData(InMemoryModuleCache &ModuleCache)
-      : Stream(Buffer), Writer(Stream, Buffer, ModuleCache, {}) {}
+  ASTWriterData(ModuleCache &ModCache)
+      : Stream(Buffer), Writer(Stream, Buffer, ModCache, {}) {}
 };
 
 void ASTUnit::clearFileLevelDecls() {
@@ -829,7 +829,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(),
                                      AST->getFileManager(),
                                      UserFilesAreVolatile);
-  AST->ModuleCache = new InMemoryModuleCache;
+  AST->ModCache = getCrossProcessModuleCache();
   AST->HSOpts = HSOpts ? HSOpts : std::make_shared<HeaderSearchOptions>();
   AST->HSOpts->ModuleFormat = std::string(PCHContainerRdr.getFormats().front());
   AST->HeaderInfo.reset(new HeaderSearch(AST->HSOpts,
@@ -861,8 +861,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
   if (::getenv("LIBCLANG_DISABLE_PCH_VALIDATION"))
     disableValid = DisableValidationForModuleKind::All;
   AST->Reader = new ASTReader(
-      PP, *AST->ModuleCache, AST->Ctx.get(), PCHContainerRdr, {},
-      /*isysroot=*/"",
+      PP, *AST->ModCache, AST->Ctx.get(), PCHContainerRdr, {}, /*isysroot=*/"",
       /*DisableValidationKind=*/disableValid, AllowASTWithCompilerErrors);
 
   unsigned Counter = 0;
@@ -1546,7 +1545,7 @@ ASTUnit::create(std::shared_ptr<CompilerInvocation> CI,
   AST->UserFilesAreVolatile = UserFilesAreVolatile;
   AST->SourceMgr = new SourceManager(AST->getDiagnostics(), *AST->FileMgr,
                                      UserFilesAreVolatile);
-  AST->ModuleCache = new InMemoryModuleCache;
+  AST->ModCache = getCrossProcessModuleCache();
 
   return AST;
 }
@@ -1833,7 +1832,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromCommandLine(
   AST->FileMgr = new FileManager(AST->FileSystemOpts, VFS);
   AST->StorePreamblesInMemory = StorePreamblesInMemory;
   AST->PreambleStoragePath = PreambleStoragePath;
-  AST->ModuleCache = new InMemoryModuleCache;
+  AST->ModCache = getCrossProcessModuleCache();
   AST->OnlyLocalDecls = OnlyLocalDecls;
   AST->CaptureDiagnostics = CaptureDiagnostics;
   AST->TUKind = TUKind;
@@ -1844,7 +1843,7 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromCommandLine(
   AST->Invocation = CI;
   AST->SkipFunctionBodies = SkipFunctionBodies;
   if (ForSerialization)
-    AST->WriterData.reset(new ASTWriterData(*AST->ModuleCache));
+    AST->WriterData.reset(new ASTWriterData(*AST->ModCache));
   // Zero out now to ease cleanup during crash recovery.
   CI = nullptr;
   Diags = nullptr;
@@ -2379,8 +2378,8 @@ bool ASTUnit::serialize(raw_ostream &OS) {
 
   SmallString<128> Buffer;
   llvm::BitstreamWriter Stream(Buffer);
-  InMemoryModuleCache ModuleCache;
-  ASTWriter Writer(Stream, Buffer, ModuleCache, {});
+  IntrusiveRefCntPtr<ModuleCache> ModCache = getCrossProcessModuleCache();
+  ASTWriter Writer(Stream, Buffer, *ModCache, {});
   return serializeUnit(Writer, Buffer, getSema(), OS);
 }
 
diff --git a/clang/lib/Frontend/CompilerInstance.cpp b/clang/lib/Frontend/CompilerInstance.cpp
index fc350f2ba42e0..b5c4de309c4bf 100644
--- a/clang/lib/Frontend/CompilerInstance.cpp
+++ b/clang/lib/Frontend/CompilerInstance.cpp
@@ -39,16 +39,17 @@
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
 #include "clang/Serialization/InMemoryModuleCache.h"
+#include "clang/Serialization/ModuleCache.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Config/llvm-config.h"
+#include "llvm/Support/AdvisoryLock.h"
 #include "llvm/Support/BuryPointer.h"
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/FileSystem.h"
-#include "llvm/Support/LockFileManager.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
@@ -66,11 +67,10 @@ using namespace clang;
 
 CompilerInstance::CompilerInstance(
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
-    InMemoryModuleCache *SharedModuleCache)
-    : ModuleLoader(/* BuildingModule = */ SharedModuleCache),
+    ModuleCache *ModCache)
+    : ModuleLoader(/*BuildingModule=*/ModCache),
       Invocation(new CompilerInvocation()),
-      ModuleCache(SharedModuleCache ? SharedModuleCache
-                                    : new InMemoryModuleCache),
+      ModCache(ModCache ? ModCache : getCrossProcessModuleCache()),
       ThePCHContainerOperations(std::move(PCHContainerOps)) {}
 
 CompilerInstance::~CompilerInstance() {
@@ -205,7 +205,7 @@ IntrusiveRefCntPtr<ASTReader> CompilerInstance::getASTReader() const {
   return TheASTReader;
 }
 void CompilerInstance::setASTReader(IntrusiveRefCntPtr<ASTReader> Reader) {
-  assert(ModuleCache.get() == &Reader->getModuleManager().getModuleCache() &&
+  assert(ModCache.get() == &Reader->getModuleManager().getModuleCache() &&
          "Expected ASTReader to use the same PCM cache");
   TheASTReader = std::move(Reader);
 }
@@ -625,9 +625,8 @@ void CompilerInstance::createPCHExternalASTSource(
 IntrusiveRefCntPtr<ASTReader> CompilerInstance::createPCHExternalASTSource(
     StringRef Path, StringRef Sysroot,
     DisableValidationForModuleKind DisableValidation,
-    bool AllowPCHWithCompilerErrors, Preprocessor &PP,
-    InMemoryModuleCache &ModuleCache, ASTContext &Context,
-    const PCHContainerReader &PCHContainerRdr,
+    bool AllowPCHWithCompilerErrors, Preprocessor &PP, ModuleCache &ModCache,
+    ASTContext &Context, const PCHContainerReader &PCHContainerRdr,
     ArrayRef<std::shared_ptr<ModuleFileExtension>> Extensions,
     ArrayRef<std::shared_ptr<DependencyCollector>> DependencyCollectors,
     void *DeserializationListener, bool OwnDeserializationListener,
@@ -635,7 +634,7 @@ IntrusiveRefCntPtr<ASTReader> CompilerInstance::createPCHExternalASTSource(
   HeaderSearchOptions &HSOpts = PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
   IntrusiveRefCntPtr<ASTReader> Reader(new ASTReader(
-      PP, ModuleCache, &Context, PCHContainerRdr, Extensions,
+      PP, ModCache, &Context, PCHContainerRdr, Extensions,
       Sysro...
[truncated]

Copy link
Contributor

@Bigcheese Bigcheese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but it would be good to document a bit more about what the long term intent is for this class. It would be nice in the future to have this centralize all the handling of the on disk cache.

@jansvoboda11 jansvoboda11 merged commit c84d8e8 into llvm:main Mar 14, 2025
8 of 10 checks passed
@jansvoboda11 jansvoboda11 deleted the module-cache-interface branch March 14, 2025 18:32
@llvm-ci
Copy link
Collaborator

llvm-ci commented Mar 14, 2025

LLVM Buildbot has detected a new failure on builder lldb-arm-ubuntu running on linaro-lldb-arm-ubuntu while building clang-tools-extra,clang at step 6 "test".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/18/builds/12940

Here is the relevant piece of the build log for the reference
Step 6 (test) failure: build (failure)
...
PASS: lldb-api :: tools/lldb-dap/exception/cpp/TestDAP_exception_cpp.py (1155 of 2920)
UNSUPPORTED: lldb-api :: tools/lldb-dap/extendedStackTrace/TestDAP_extendedStackTrace.py (1156 of 2920)
PASS: lldb-api :: tools/lldb-dap/disconnect/TestDAP_disconnect.py (1157 of 2920)
PASS: lldb-api :: tools/lldb-dap/instruction-breakpoint/TestDAP_instruction_breakpoint.py (1158 of 2920)
PASS: lldb-api :: tools/lldb-dap/io/TestDAP_io.py (1159 of 2920)
PASS: lldb-api :: tools/lldb-dap/locations/TestDAP_locations.py (1160 of 2920)
PASS: lldb-api :: tools/lldb-dap/memory/TestDAP_memory.py (1161 of 2920)
PASS: lldb-api :: tools/lldb-dap/evaluate/TestDAP_evaluate.py (1162 of 2920)
PASS: lldb-api :: tools/lldb-dap/module/TestDAP_module.py (1163 of 2920)
PASS: lldb-api :: tools/lldb-dap/output/TestDAP_output.py (1164 of 2920)
FAIL: lldb-api :: tools/lldb-dap/launch/TestDAP_launch.py (1165 of 2920)
******************** TEST 'lldb-api :: tools/lldb-dap/launch/TestDAP_launch.py' FAILED ********************
Script:
--
/usr/bin/python3.10 /home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./lib --env LLVM_INCLUDE_DIR=/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/include --env LLVM_TOOLS_DIR=/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./bin --arch armv8l --build-dir /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex --lldb-module-cache-dir /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./bin/lldb --compiler /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./bin/clang --dsymutil /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./bin/dsymutil --make /usr/bin/gmake --llvm-tools-dir /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./bin --lldb-obj-root /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/tools/lldb --lldb-libs-dir /home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/./lib /home/tcwg-buildbot/worker/lldb-arm-ubuntu/llvm-project/lldb/test/API/tools/lldb-dap/launch -p TestDAP_launch.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 21.0.0git (https://github.com/llvm/llvm-project.git revision c84d8e8f1c406ab34d56efd4a9f8c5fbce70af2d)
  clang revision c84d8e8f1c406ab34d56efd4a9f8c5fbce70af2d
  llvm revision c84d8e8f1c406ab34d56efd4a9f8c5fbce70af2d
Skipping the following test categories: ['libc++', 'dsym', 'gmodules', 'debugserver', 'objc']
========= DEBUG ADAPTER PROTOCOL LOGS =========
1741979330.845331192 <-- (stdin/stdout) {"command":"initialize","type":"request","arguments":{"adapterID":"lldb-native","clientID":"vscode","columnsStartAt1":true,"linesStartAt1":true,"locale":"en-us","pathFormat":"path","supportsRunInTerminalRequest":true,"supportsVariablePaging":true,"supportsVariableType":true,"supportsStartDebuggingRequest":true,"sourceInitFile":false},"seq":1}
1741979330.847774267 --> (stdin/stdout) {"body":{"__lldb":{"version":"lldb version 21.0.0git (https://github.com/llvm/llvm-project.git revision c84d8e8f1c406ab34d56efd4a9f8c5fbce70af2d)\n  clang revision c84d8e8f1c406ab34d56efd4a9f8c5fbce70af2d\n  llvm revision c84d8e8f1c406ab34d56efd4a9f8c5fbce70af2d"},"completionTriggerCharacters":["."," ","\t"],"exceptionBreakpointFilters":[{"default":false,"filter":"cpp_catch","label":"C++ Catch"},{"default":false,"filter":"cpp_throw","label":"C++ Throw"},{"default":false,"filter":"objc_catch","label":"Objective-C Catch"},{"default":false,"filter":"objc_throw","label":"Objective-C Throw"}],"supportTerminateDebuggee":true,"supportsBreakpointLocationsRequest":true,"supportsCompletionsRequest":true,"supportsConditionalBreakpoints":true,"supportsConfigurationDoneRequest":true,"supportsDataBreakpoints":true,"supportsDelayedStackTraceLoading":true,"supportsDisassembleRequest":true,"supportsEvaluateForHovers":true,"supportsExceptionInfoRequest":true,"supportsExceptionOptions":true,"supportsFunctionBreakpoints":true,"supportsGotoTargetsRequest":false,"supportsHitConditionalBreakpoints":true,"supportsInstructionBreakpoints":true,"supportsLoadedSourcesRequest":false,"supportsLogPoints":true,"supportsModulesRequest":true,"supportsProgressReporting":true,"supportsReadMemoryRequest":true,"supportsRestartFrame":false,"supportsRestartRequest":true,"supportsRunInTerminalRequest":true,"supportsSetVariable":true,"supportsStepBack":false,"supportsStepInTargetsRequest":true,"supportsSteppingGranularity":true,"supportsValueFormattingOptions":true},"command":"initialize","request_seq":1,"seq":0,"success":true,"type":"response"}
1741979330.848397970 <-- (stdin/stdout) {"command":"launch","type":"request","arguments":{"program":"/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/tools/lldb-dap/launch/TestDAP_launch.test_args/a.out","args":["one","with space","'with single quotes'","\"with double quotes\""],"initCommands":["settings clear -all","settings set symbols.enable-external-lookup false","settings set target.inherit-tcc true","settings set target.disable-aslr false","settings set target.detach-on-error false","settings set target.auto-apply-fixits false","settings set plugin.process.gdb-remote.packet-timeout 60","settings set symbols.clang-modules-cache-path \"/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api\"","settings set use-color false"],"disableASLR":false,"enableAutoVariableSummaries":false,"enableSyntheticChildDebugging":false,"displayExtendedBacktrace":false,"commandEscapePrefix":null},"seq":2}
1741979330.848656893 --> (stdin/stdout) {"body":{"category":"console","output":"Running initCommands:\n"},"event":"output","seq":0,"type":"event"}
1741979330.848685503 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings clear -all\n"},"event":"output","seq":0,"type":"event"}
1741979330.848697901 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set symbols.enable-external-lookup false\n"},"event":"output","seq":0,"type":"event"}
1741979330.848708868 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set target.inherit-tcc true\n"},"event":"output","seq":0,"type":"event"}
1741979330.848719835 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set target.disable-aslr false\n"},"event":"output","seq":0,"type":"event"}
1741979330.848730326 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set target.detach-on-error false\n"},"event":"output","seq":0,"type":"event"}
1741979330.848740816 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set target.auto-apply-fixits false\n"},"event":"output","seq":0,"type":"event"}
1741979330.848771095 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set plugin.process.gdb-remote.packet-timeout 60\n"},"event":"output","seq":0,"type":"event"}
1741979330.848783016 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set symbols.clang-modules-cache-path \"/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/module-cache-lldb/lldb-api\"\n"},"event":"output","seq":0,"type":"event"}
1741979330.848793745 --> (stdin/stdout) {"body":{"category":"console","output":"(lldb) settings set use-color false\n"},"event":"output","seq":0,"type":"event"}
1741979330.986446381 --> (stdin/stdout) {"command":"launch","request_seq":2,"seq":0,"success":true,"type":"response"}
1741979330.986526012 --> (stdin/stdout) {"body":{"isLocalProcess":true,"name":"/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/tools/lldb-dap/launch/TestDAP_launch.test_args/a.out","startMethod":"launch","systemProcessId":1332613},"event":"process","seq":0,"type":"event"}
1741979330.986540556 --> (stdin/stdout) {"event":"initialized","seq":0,"type":"event"}
1741979330.986936808 <-- (stdin/stdout) {"command":"configurationDone","type":"request","arguments":{},"seq":3}
1741979330.986959457 --> (stdin/stdout) {"command":"configurationDone","request_seq":3,"seq":0,"success":true,"type":"response"}
1741979331.054115772 --> (stdin/stdout) {"body":{"category":"stdout","output":"arg[0] = \"/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/lldb-test-build.noindex/tools/lldb-dap/launch/TestDAP_launch.test_args/a.out\"\r\n"},"event":"output","seq":0,"type":"event"}
1741979331.054280519 --> (stdin/stdout) {"body":{"category":"stdout","output":"arg[1] = \"one\"\r\n"},"event":"output","seq":0,"type":"event"}
1741979331.054318428 --> (stdin/stdout) {"body":{"category":"stdout","output":"arg[2] = \"with space\"\r\n"},"event":"output","seq":0,"type":"event"}
1741979331.054350853 --> (stdin/stdout) {"body":{"category":"stdout","output":"arg[3] = \"'with single quotes'\"\r\n"},"event":"output","seq":0,"type":"event"}
1741979331.054382801 --> (stdin/stdout) {"body":{"category":"stdout","output":"arg[4] = \"\"with double quotes\"\"\r\n"},"event":"output","seq":0,"type":"event"}
1741979331.054415703 --> (stdin/stdout) {"body":{"category":"stdout","output":"env[0] = \"LLDB_LIB_DIR=/home/tcwg-buildbot/worker/lldb-arm-ubuntu/build/local/lib/python3.10/dist-packages/../..\"\r\n"},"event":"output","seq":0,"type":"event"}

jansvoboda11 added a commit to swiftlang/llvm-project that referenced this pull request Mar 18, 2025
This PR adds new `ModuleCache` interface to Clang's implicitly-built
modules machinery. The main motivation for this change is to create a
second implementation that uses a more efficient kind of
`llvm::AdvisoryLock` during dependency scanning.

In addition to the lock abstraction, the `ModuleCache` interface also
manages the existing `InMemoryModuleCache` instance. I found that
compared to keeping these separate/independent, the code is a bit
simpler now, since these are two tightly coupled concepts. I can
envision a more efficient implementation of the `InMemoryModuleCache`
for the single-process case too, which will be much easier to implement
with the current setup.

This is not intended to be a functional change.

(cherry picked from commit c84d8e8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
clang:modules C++20 modules and Clang Header Modules clang Clang issues not falling into any other category clang-tools-extra clangd
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants