Turn On Cross-Module Incremental Builds! #34196
Conversation
llvm-bcanalyzer does *not* like it when there are multiple block info metadata blocks in the same bitstream file. This patch will skip the emission of that and just jump straight to the metadata block when we're not reading a "standalone" incremental dependency file. While I'm here, also add the right block abbreviation info so we can get a decent dump from llvm-bcanalyzer
…pendencies Register the module the external dependencies ride in with the pseudo-job in the Driver. This is a hack.
Cross-Module incremental dependencies are a new experimental mode of the Swift driver and frontend. Through a tight partnership between the two, we enable the driver to have far greater visibility into the dependency structure of a Swift module. Rather than invent a new model, we have chosen to extend the existing incremental compilation model that works for a single module to multiple modules. To do this, we need the frontend to emit Swift dependencies in a form the driver can consume. We could emit these metadata in the form of an extra supplementary output that summarizes the contents of a generated module. However, this approach comes with a number of downsides: - It requires additional integration with the build system - It assumes swiftmodule files will be consumed directly from the build directory; they are not - It incorrectly assumes a swiftmodule has but one interface. Taken in aggregate, a swiftmodule directory has one interface *per triple* Given this, the approach we take here is to encode these dependencies directly into the swiftmodule file itself. When frontends load these souped-up incremental swiftmodule files, they record in their own swiftdeps files that they depend on an incremental swiftmodule. Upon the next build, the driver is then able to read that module file, extract the swiftdeps information from it, and use it to influence the way it schedules jobs. The sum total is that we neatly extend the intra-module case of incremental builds to the inter-module case by treating swiftmodule inputs not as opaque entities, but as "big ol' flat Swift files" that just export an interface like any other Swift file within the module. As a further optimization, and because clients literally cannot observe this aspect of the incremental build, we only serialize the provides (the "defs" side of a "use-def" edge) when emitting swiftdeps metadata into swiftmodule files. rdar://69595010
|
@swift-ci test |
| assert(getSwiftDeps(iter->second) == swiftDeps.getValue() && | ||
| "jobsBySwiftDeps should be inverse of getSwiftDeps."); |
There was a problem hiding this comment.
I bet you have a good argument for dispensing with this assertion. What is it?
|
|
||
| if (dependencyTracker && file) { | ||
| auto DepPath = file->getFilename(); | ||
| // Don't record cached artifacts as dependencies. | ||
| if (!isCached(DepPath)) { | ||
| if (M->hasIncrementalInfo()) { | ||
| dependencyTracker->addIncrementalDependency(DepPath); | ||
| } else { | ||
| dependencyTracker->addDependency(DepPath, /*isSystem=*/false); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
my own style would be to factor these lines out into a function, "trackDependency" or some such.
Also, it would be helpful to add a "why" to the comment on 996. (Why not record cached artifacts?)
|
Build failed |
|
Is there going to be a file in the docs directory about Cross-Module Incremental Builds? |
|
@tschuett We're hoping to eventually just scrap this flag and turn this on for every incremental build after this gets battle tested. It should be completely seamless from an end-user's perspective. From an implementation perspective, I could write some words into |
It doesn't seem to like the output file maps I've written here.
|
@swift-ci test |
|
@swift-ci test Windows |
|
Anything is fine, but all the documentation is in the commit message, which nobody will ever find. I don't care about the flag. The feature/implementation seems to be interesting. |
|
Build failed |
|
Build failed |
|
Build failed |
Cross-Module incremental dependencies are a new experimental mode of the Swift driver and frontend. Through a tight partnership between the two, we enable the driver to have far greater visibility into the dependency structure of a Swift module.
Rather than invent a new model, we have chosen to extend the existing incremental compilation model that works for a single module to multiple modules. To do this, we need the frontend to emit Swift dependencies in a form the driver can consume. We could emit these metadata in the form of an extra supplementary output that summarizes the contents of a generated module. However, this approach comes with a number of downsides:
Given this, the approach we take here is to encode these dependencies directly into the swiftmodule file itself. When frontends load these souped-up incremental swiftmodule files, they record in their own swiftdeps files that they depend on an incremental swiftmodule. Upon the next build, the driver is then able to read that module file, extract the swiftdeps information from it, and use it to influence the way it schedules jobs.
The sum total is that we neatly extend the intra-module case of incremental builds to the inter-module case by treating swiftmodule inputs not as opaque entities, but as "big ol' flat Swift files" that just export an interface like any other Swift file within the module. As a further optimization, and because clients literally cannot observe this aspect of the incremental build, we only serialize the provides (the "defs" side of a "use-def" edge) when emitting swiftdeps metadata into swiftmodule files.
To build a module with cross-module incremental metadata, the unstable flag
-enable-experimental-cross-module-incremental-buildmust be provided to all driver invocations of the relevant targets. Any swiftmodule files without incremental metadata sections present are treated according to the status quo.rdar://69595010