Skip to content

[Dependency Scanning] Parallelize Clang module queries #76915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 5, 2024
Merged

[Dependency Scanning] Parallelize Clang module queries #76915

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
205 changes: 142 additions & 63 deletions include/swift/AST/ModuleDependencies.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ class ModuleDependencyInfoStorageBase {
ArrayRef<LinkLibrary> linkLibraries,
StringRef moduleCacheKey = "")
: dependencyKind(dependencyKind), linkLibraries(linkLibraries),
moduleCacheKey(moduleCacheKey.str()), resolved(false),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, we kept adding things to directDependencies and marked them resolved when done.
Now, instead, we keep track of dependencies of various kinds separately, meaning each dependency kind is only ever set once and therefore we no longer need to track which dependencies are "resolved".

moduleCacheKey(moduleCacheKey.str()),
finalized(false) {}

ModuleDependencyInfoStorageBase(
Expand All @@ -191,7 +191,7 @@ class ModuleDependencyInfoStorageBase {
: dependencyKind(dependencyKind), moduleImports(moduleImports),
optionalModuleImports(optionalModuleImports),
linkLibraries(linkLibraries), moduleCacheKey(moduleCacheKey.str()),
resolved(false), finalized(false) {}
finalized(false) {}

virtual ModuleDependencyInfoStorageBase *clone() const = 0;

Expand All @@ -209,11 +209,13 @@ class ModuleDependencyInfoStorageBase {
/// use this module.
std::vector<LinkLibrary> linkLibraries;

/// The set of modules on which this module depends, resolved
/// to Module IDs, qualified by module kind: Swift, Clang, etc.
std::vector<ModuleDependencyID> resolvedDirectModuleDependencies;

/// Dependencies comprised of Swift overlay modules of direct and
/// All directly-imported Swift module dependencies.
std::vector<ModuleDependencyID> importedSwiftModules;
/// All directly-imported Clang module dependencies.
std::vector<ModuleDependencyID> importedClangModules;
/// All cross-import overlay module dependencies.
std::vector<ModuleDependencyID> crossImportOverlayModules;
/// All dependencies comprised of Swift overlay modules of direct and
/// transitive Clang dependencies.
std::vector<ModuleDependencyID> swiftOverlayDependencies;

Expand All @@ -227,8 +229,6 @@ class ModuleDependencyInfoStorageBase {
/// The macro dependencies.
std::map<std::string, MacroPluginDependency> macroDependencies;

/// The direct dependency of the module is resolved by scanner.
bool resolved;
/// ModuleDependencyInfo is finalized (with all transitive dependencies
/// and inputs).
bool finalized;
Expand All @@ -239,6 +239,8 @@ struct CommonSwiftTextualModuleDependencyDetails {
ArrayRef<StringRef> extraPCMArgs, ArrayRef<StringRef> buildCommandLine,
StringRef CASFileSystemRootID)
: extraPCMArgs(extraPCMArgs.begin(), extraPCMArgs.end()),
bridgingHeaderFile(std::nullopt),
bridgingSourceFiles(), bridgingModuleDependencies(),
buildCommandLine(buildCommandLine.begin(), buildCommandLine.end()),
CASFileSystemRootID(CASFileSystemRootID) {}

Expand All @@ -254,7 +256,7 @@ struct CommonSwiftTextualModuleDependencyDetails {
std::vector<std::string> bridgingSourceFiles;

/// (Clang) modules on which the bridging header depends.
std::vector<std::string> bridgingModuleDependencies;
std::vector<ModuleDependencyID> bridgingModuleDependencies;

/// The Swift frontend invocation arguments to build the Swift module from the
/// interface.
Expand Down Expand Up @@ -426,7 +428,7 @@ class SwiftBinaryModuleDependencyStorage
std::vector<std::string> headerSourceFiles;

/// (Clang) modules on which the header inputs depend.
std::vector<std::string> headerModuleDependencies;
std::vector<ModuleDependencyID> headerModuleDependencies;

/// A flag that indicates this dependency is a framework
const bool isFramework;
Expand Down Expand Up @@ -660,42 +662,105 @@ class ModuleDependencyInfo {
return storage->optionalModuleImports;
}

/// Retreive the module-level dependencies.
const ArrayRef<ModuleDependencyID> getDirectModuleDependencies() const {
assert(storage->resolved);
return storage->resolvedDirectModuleDependencies;
}

std::string getModuleCacheKey() const {
assert(storage->resolved);
return storage->moduleCacheKey;
}

void updateModuleCacheKey(const std::string &key) {
storage->moduleCacheKey = key;
}

/// Resolve a dependency's set of `imports` with qualified Module IDs
void
resolveDirectDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
assert(!storage->resolved && "Resolving an already-resolved dependency");
storage->resolved = true;
storage->resolvedDirectModuleDependencies.assign(dependencyIDs.begin(),
dependencyIDs.end());
setImportedSwiftDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
assert(isSwiftModule());
storage->importedSwiftModules.assign(dependencyIDs.begin(),
dependencyIDs.end());
}
const ArrayRef<ModuleDependencyID> getImportedSwiftDependencies() const {
return storage->importedSwiftModules;
}

void
setImportedClangDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
storage->importedClangModules.assign(dependencyIDs.begin(),
dependencyIDs.end());
}
const ArrayRef<ModuleDependencyID> getImportedClangDependencies() const {
return storage->importedClangModules;
}

void
setHeaderClangDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
assert(isSwiftModule());
switch (getKind()) {
case swift::ModuleDependencyKind::SwiftInterface: {
auto swiftInterfaceStorage =
cast<SwiftInterfaceModuleDependenciesStorage>(storage.get());
swiftInterfaceStorage->textualModuleDetails.bridgingModuleDependencies.assign(dependencyIDs.begin(),
dependencyIDs.end());
break;
}
case swift::ModuleDependencyKind::SwiftSource: {
auto swiftSourceStorage =
cast<SwiftSourceModuleDependenciesStorage>(storage.get());
swiftSourceStorage->textualModuleDetails.bridgingModuleDependencies.assign(dependencyIDs.begin(),
dependencyIDs.end());
break;
}
case swift::ModuleDependencyKind::SwiftBinary: {
auto swiftBinaryStorage =
cast<SwiftBinaryModuleDependencyStorage>(storage.get());
swiftBinaryStorage->headerModuleDependencies.assign(dependencyIDs.begin(),
dependencyIDs.end());
break;
}
default: {
llvm_unreachable("Unexpected dependency kind");
}
}
}
const ArrayRef<ModuleDependencyID> getHeaderClangDependencies() const {
switch (getKind()) {
case swift::ModuleDependencyKind::SwiftInterface: {
auto swiftInterfaceStorage =
cast<SwiftInterfaceModuleDependenciesStorage>(storage.get());
return swiftInterfaceStorage->textualModuleDetails.bridgingModuleDependencies;
}
case swift::ModuleDependencyKind::SwiftSource: {
auto swiftSourceStorage =
cast<SwiftSourceModuleDependenciesStorage>(storage.get());
return swiftSourceStorage->textualModuleDetails.bridgingModuleDependencies;
}
case swift::ModuleDependencyKind::SwiftBinary: {
auto swiftBinaryStorage =
cast<SwiftBinaryModuleDependencyStorage>(storage.get());
return swiftBinaryStorage->headerModuleDependencies;
}
default:
return {};
}
}

/// Set this module's set of Swift Overlay dependencies
void
setOverlayDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
setSwiftOverlayDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
assert(isSwiftModule());
storage->swiftOverlayDependencies.assign(dependencyIDs.begin(),
dependencyIDs.end());
}

const ArrayRef<ModuleDependencyID> getSwiftOverlayDependencies() const {
return storage->swiftOverlayDependencies;
}

void
setCrossImportOverlayDependencies(const ArrayRef<ModuleDependencyID> dependencyIDs) {
assert(isSwiftModule());
storage->crossImportOverlayModules.assign(dependencyIDs.begin(),
dependencyIDs.end());
}
const ArrayRef<ModuleDependencyID> getCrossImportOverlayDependencies() const {
return storage->crossImportOverlayModules;
}

const ArrayRef<LinkLibrary> getLinkLibraries() const {
return storage->linkLibraries;
}
Expand Down Expand Up @@ -723,16 +788,6 @@ class ModuleDependencyInfo {
return {};
}

const ArrayRef<std::string> getHeaderDependencies() const {
if (auto *detail = getAsSwiftInterfaceModule())
return detail->textualModuleDetails.bridgingModuleDependencies;
if (auto *detail = getAsSwiftSourceModule())
return detail->textualModuleDetails.bridgingModuleDependencies;
if (auto *detail = getAsSwiftBinaryModule())
return detail->headerModuleDependencies;
return {};
}

std::vector<std::string> getCommandline() const {
if (auto *detail = getAsClangModule())
return detail->buildCommandLine;
Expand Down Expand Up @@ -798,9 +853,8 @@ class ModuleDependencyInfo {
llvm_unreachable("Unexpected type");
}

bool isResolved() const { return storage->resolved; }
void setIsResolved(bool isResolved) { storage->resolved = isResolved; }

/// Whether explicit input paths of all the module dependencies
/// have been specified on the command-line recipe for this module.
bool isFinalized() const { return storage->finalized; }
void setIsFinalized(bool isFinalized) { storage->finalized = isFinalized; }

Expand Down Expand Up @@ -876,10 +930,6 @@ class ModuleDependencyInfo {
const SourceManager *sourceManager = nullptr,
SourceLoc sourceLocation = SourceLoc());

/// Add a kind-qualified module dependency ID to the set of
/// module dependencies.
void addModuleDependency(ModuleDependencyID dependencyID);

/// Get the bridging header.
std::optional<std::string> getBridgingHeader() const;

Expand All @@ -904,10 +954,6 @@ class ModuleDependencyInfo {
/// Add source files that the header input depends on.
void addHeaderSourceFile(StringRef bridgingSourceFile);

/// Add (Clang) modules on which a non-bridging header input depends.
void addHeaderInputModuleDependency(StringRef module,
llvm::StringSet<> &alreadyAddedModules);

/// Add bridging header include tree.
void addBridgingHeaderIncludeTree(StringRef ID);

Expand Down Expand Up @@ -1184,6 +1230,8 @@ class ModuleDependenciesCache {
std::optional<ModuleDependencyKind> kind) const;
/// Whether we have cached dependency information for the given module Name.
bool hasDependency(StringRef moduleName) const;
/// Whether we have cached dependency information for the given Swift module.
bool hasSwiftDependency(StringRef moduleName) const;

SwiftDependencyScanningService &getScanService() {
return globalScanningService;
Expand All @@ -1200,17 +1248,29 @@ class ModuleDependenciesCache {
}
std::string getModuleOutputPath() const { return moduleOutputPath; }

/// Query all dependencies, direct and Swift overlay.
std::vector<ModuleDependencyID>
/// Query all dependencies
ModuleDependencyIDSetVector
getAllDependencies(const ModuleDependencyID &moduleID) const;

/// Query only direct import dependencies

/// Query all Clang module dependencies.
ModuleDependencyIDSetVector
getClangDependencies(const ModuleDependencyID &moduleID) const;

/// Query all directly-imported Swift dependencies
llvm::ArrayRef<ModuleDependencyID>
getOnlyDirectDependencies(const ModuleDependencyID &moduleID) const;

/// Query only Swift overlay dependencies
getImportedSwiftDependencies(const ModuleDependencyID &moduleID) const;
/// Query all directly-imported Clang dependencies
llvm::ArrayRef<ModuleDependencyID>
getImportedClangDependencies(const ModuleDependencyID &moduleID) const;
/// Query all Clang module dependencies of this module's imported (bridging) header
llvm::ArrayRef<ModuleDependencyID>
getOnlyOverlayDependencies(const ModuleDependencyID &moduleID) const;
getHeaderClangDependencies(const ModuleDependencyID &moduleID) const;
/// Query Swift overlay dependencies
llvm::ArrayRef<ModuleDependencyID>
getSwiftOverlayDependencies(const ModuleDependencyID &moduleID) const;
/// Query all cross-import overlay dependencies
llvm::ArrayRef<ModuleDependencyID>
getCrossImportOverlayDependencies(const ModuleDependencyID &moduleID) const;

/// Look for module dependencies for a module with the given ID
///
Expand All @@ -1231,6 +1291,12 @@ class ModuleDependenciesCache {
std::optional<const ModuleDependencyInfo *>
findDependency(StringRef moduleName) const;

/// Look for Swift module dependencies for a module with the given name
///
/// \returns the cached result, or \c None if there is no cached entry.
std::optional<const ModuleDependencyInfo *>
findSwiftDependency(StringRef moduleName) const;

/// Look for known existing dependencies.
///
/// \returns the cached result.
Expand All @@ -1246,19 +1312,32 @@ class ModuleDependenciesCache {

/// Update stored dependencies for the given module.
void updateDependency(ModuleDependencyID moduleID,
ModuleDependencyInfo dependencies);
ModuleDependencyInfo dependencyInfo);

/// Resolve a dependency module's set of imports
/// to a kind-qualified set of module IDs.
/// Resolve this module's set of directly-imported Swift module
/// dependencies
void
resolveDependencyImports(ModuleDependencyID moduleID,
const ArrayRef<ModuleDependencyID> dependencyIDs);

/// Resolve a dependency module's set of Swift module dependencies
setImportedSwiftDependencies(ModuleDependencyID moduleID,
const ArrayRef<ModuleDependencyID> dependencyIDs);
/// Resolve this module's set of directly-imported Clang module
/// dependencies
void
setImportedClangDependencies(ModuleDependencyID moduleID,
const ArrayRef<ModuleDependencyID> dependencyIDs);
/// Resolve this module's set of Swift module dependencies
/// that are Swift overlays of Clang module dependencies.
void
setSwiftOverlayDependencies(ModuleDependencyID moduleID,
const ArrayRef<ModuleDependencyID> dependencyIDs);
/// Resolve this Swift module's imported (bridging) header's
/// Clang module dependencies
void
setHeaderClangDependencies(ModuleDependencyID moduleID,
const ArrayRef<ModuleDependencyID> dependencyIDs);
/// Resolve this module's cross-import overlay dependencies
void
setCrossImportOverlayDependencies(ModuleDependencyID moduleID,
const ArrayRef<ModuleDependencyID> dependencyIDs);

StringRef getMainModuleName() const { return mainScanModuleName; }
};
Expand Down
17 changes: 14 additions & 3 deletions include/swift/ClangImporter/ClangImporter.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,9 @@ class SwiftLookupTable;
class TypeDecl;
class ValueDecl;
class VisibleDeclConsumer;
using ModuleDependencyIDSetVector =
llvm::SetVector<ModuleDependencyID, std::vector<ModuleDependencyID>,
std::set<ModuleDependencyID>>;
enum class SelectorSplitKind;

/// Kinds of optional types.
Expand Down Expand Up @@ -492,7 +495,11 @@ class ClangImporter final : public ClangModuleLoader {
ModuleDependencyInfo &MDI,
const clang::tooling::dependencies::TranslationUnitDeps &deps);

/// Add dependency information for header dependencies
void getBridgingHeaderOptions(
const clang::tooling::dependencies::TranslationUnitDeps &deps,
std::vector<std::string> &swiftArgs);

/// Query dependency information for header dependencies
/// of a binary Swift module.
///
/// \param moduleID the name of the Swift module whose dependency
Expand All @@ -505,10 +512,14 @@ class ClangImporter final : public ClangModuleLoader {
/// about new Clang modules discovered along the way.
///
/// \returns \c true if an error occurred, \c false otherwise
bool addHeaderDependencies(
bool getHeaderDependencies(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This got refactored to instead return a result that the client can use to update the dependency scanner cache, instead of doing it in-place.

ModuleDependencyID moduleID,
clang::tooling::dependencies::DependencyScanningTool &clangScanningTool,
ModuleDependenciesCache &cache);
ModuleDependenciesCache &cache,
ModuleDependencyIDSetVector &headerClangModuleDependencies,
std::vector<std::string> &headerFileInputs,
std::vector<std::string> &bridgingHeaderCommandLine,
std::optional<std::string> &includeTreeID);

clang::TargetInfo &getModuleAvailabilityTarget() const override;
clang::ASTContext &getClangASTContext() const override;
Expand Down
Loading