Skip to content

[clang][deps] Cherry-pick recent changes #5234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Sep 29, 2022

Conversation

jansvoboda11
Copy link

This patch cherry-picks some upstream changes to the dependency scanner.

@jansvoboda11
Copy link
Author

@swift-ci please test

Copy link

@benlangmuir benlangmuir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! I reviewed the list of commits comparing to all dependency scanner changes on "next". I didn't check the actual diff.

@jansvoboda11
Copy link
Author

@swift-ci please test
swiftlang/swift#60882

@benlangmuir
Copy link

swiftlang/swift#60882
@swift-ci please test

4 similar comments
@jansvoboda11
Copy link
Author

swiftlang/swift#60882
@swift-ci please test

@artemcm
Copy link

artemcm commented Sep 12, 2022

swiftlang/swift#60882
@swift-ci please test

@jansvoboda11
Copy link
Author

swiftlang/swift#60882
@swift-ci please test

@jansvoboda11
Copy link
Author

swiftlang/swift#60882
@swift-ci please test

@benlangmuir
Copy link

swiftlang/swift#60882
@swift-ci please test macOS

@benlangmuir
Copy link

New failures look like a real issue:

Test Case '-[SwiftDriverTests.ExplicitModuleBuildTests testExplicitModuleBuildEndToEnd]' started.
error: generate-pcm command failed with exit code 1 (use -v to see invocation)
/Users/ec2-user/jenkins/workspace/apple-llvm-project-pr-macos/branch-stable/20220421/swift-driver/Sources/SwiftDriverExecution/SwiftDriverExecutor.swift:88: error: -[SwiftDriverTests.ExplicitModuleBuildTests testExplicitModuleBuildEndToEnd] : failed: caught error: "fatalError"
Test Case '-[SwiftDriverTests.ExplicitModuleBuildTests testExplicitModuleBuildEndToEnd]' failed (0.800 seconds).

Test Case '-[SwiftDriverTests.ExplicitModuleBuildTests testModuleAliasingPrebuiltWithScanDeps]' started.
error: generate-pcm command failed with exit code 1 (use -v to see invocation)
/Users/ec2-user/jenkins/workspace/apple-llvm-project-pr-macos/branch-stable/20220421/swift-driver/Sources/SwiftDriverExecution/SwiftDriverExecutor.swift:88: error: -[SwiftDriverTests.ExplicitModuleBuildTests testModuleAliasingPrebuiltWithScanDeps] : failed: caught error: "fatalError"
Test Case '-[SwiftDriverTests.ExplicitModuleBuildTests testModuleAliasingPrebuiltWithScanDeps]' failed (0.245 seconds).

Test Case '-[SwiftDriverTests.ExplicitModuleBuildTests testModuleAliasingWithExplicitBuild]' started.
error: generate-pcm command failed with exit code 1 (use -v to see invocation)
/Users/ec2-user/jenkins/workspace/apple-llvm-project-pr-macos/branch-stable/20220421/swift-driver/Sources/SwiftDriverExecution/SwiftDriverExecutor.swift:88: error: -[SwiftDriverTests.ExplicitModuleBuildTests testModuleAliasingWithExplicitBuild] : failed: caught error: "fatalError"
Test Case '-[SwiftDriverTests.ExplicitModuleBuildTests testModuleAliasingWithExplicitBuild]' failed (0.245 seconds).

@artemcm
Copy link

artemcm commented Sep 16, 2022

We thought we addressed these with swiftlang/swift#61025.
Must be a slightly different issue.

@artemcm
Copy link

artemcm commented Sep 27, 2022

swiftlang/swift#60882
@swift-ci please test

1 similar comment
@artemcm
Copy link

artemcm commented Sep 28, 2022

swiftlang/swift#60882
@swift-ci please test

jansvoboda11 and others added 14 commits September 28, 2022 10:37
Since D129389 (and downstream PR #4965), the dependency scanner is responsible for generating full command-lines, including the modules paths. This patch removes the flag that was making this an opt-in behavior in clang-scan-deps.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D131420
Reduce unnecessary differences with upstream dep scanning code, and
group some of the changes so they are in larger regions that are easier
to follow.

(cherry picked from commit c42e2d8)
Instead of delaying the generation of command-lines to after all
dependencies are reported, compute them immediately. This is partly in
preparation for splitting the TU driver command into its constituent cc1
and other jobs, but it also just simplifies working with the compiler
invocation for modules if they are not "without paths".

Also change the computation of the default output path in
clang-scan-deps to scrape the implicit module cache from the
command-line rather than get it from the dependency, since that is now
unavailable at the time we make the callback.

Differential Revision: https://reviews.llvm.org/D131934
This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.

To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D132066
Move copying compiler arguments to a vector<string> and modifying
common module-related options into CompilerInvocation in preparation for
using some of them in more places and to avoid duplicating this code
accidentally in the future.

Differential Revision: https://reviews.llvm.org/D132419
jansvoboda11 and others added 7 commits September 28, 2022 10:38
When compiling a module, its semantics and Clang's behavior are affected by other modules. These modules are typically the **imported** ones. However, during implicit build, some modules end up being compiled and read without being actually imported. This patch starts tracking such modules and serializing them into `.pcm` files. This enables the dependency scanner to construct explicit compilations that mimic implicit build.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D132430
…ivate

When Clang encounters `@import M.Private` during implicit build, it precompiles module `M` and looks through its submodules. If the `Private` submodule is not found, Clang assumes `@import M_Private`. In the dependency scanner, we don't capture the dependency on `M`, since it's not imported. It's an affecting module, though: compilation of the import statement will fail when implicit modules are disabled and `M` is not precompiled and explicitly provided. This patch fixes that.

Depends on D132430.

Reviewed By: benlangmuir

Differential Revision: https://reviews.llvm.org/D132502
The invocation is only ever used to serialize cc1 arguments from, so
instead serialize the arguments inside the dep scanner to simplify the
interface.

Differential Revision: https://reviews.llvm.org/D132616
* Factor module map and module file path functions out
* Use a secondary mapping to lookup module deps by ID instead of the
  preprocessor module map.
* Sink DirectPrebuiltModularDeps into MDC.

Differential Revision: https://reviews.llvm.org/D132617
We can now use a DenseMap instead.

rdar://99151069
Fix missing include; the header was not actually standalone due to a an
incomplete type used in IntrusiveRefCntPtr.

(cherry picked from commit 4fd7b98)
@jansvoboda11 jansvoboda11 force-pushed the jan_svoboda/cherry-pick branch from 6045d7b to b1c74a0 Compare September 28, 2022 18:09
@jansvoboda11
Copy link
Author

swiftlang/swift#60882
@swift-ci please test

@artemcm artemcm merged commit 26319ba into stable/20220421 Sep 29, 2022
@jansvoboda11 jansvoboda11 deleted the jan_svoboda/cherry-pick branch September 29, 2022 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants