[Driver] add -static-stdlib flag (Darwin platforms) #1817
Conversation
|
Bumpity bump. |
|
Any traction on this? It really sucks to have to keep deploying updated binaries with each drop in our tool chains. We want to keep up-to-date with Swift, but not be required to maintain multiple snapshots on our boxes and not be forced to use the latest for our tools. |
| @@ -0,0 +1,10 @@ | |||
| // Statically link a "hello world" program | |||
| // XFAIL: linux | |||
| // REQUIRE: static_stdlib | |||
|
@gparker42 Thanks! Fixed. |
|
Bumpity-bump-bump, don't let this patch slump. |
|
Looks OK to me. Any opinion, @jrose-apple? |
|
Rebased against swift-DEVELOPMENT-SNAPSHOT-2016-04-12-a |
|
ping @jrose-apple |
|
I'm sorry for letting this sit for so long. I'm actually still not sure this is the right way to expose this, particularly because it doesn't work correctly if you have a library target in addition to your executable. What's the plan of attack there? (It doesn't need to work yet. I just want to know that we have one. I'm concerned we'll end up ripping this flag back out next year because it's at the wrong level.) |
| Arguments.push_back("Foundation"); | ||
| Arguments.push_back("-force_load_swift_libs"); | ||
| } | ||
| else Arguments.push_back(context.Args.MakeArgString(RuntimeLibPath)); |
There was a problem hiding this comment.
Coding convention nitpick: please put the else on the same line as the close brace, and the body on the following line. I usually put braces around a single-line else-clause if the then-clause also had braces, too.
I would suggest the right thing to do is:
Whether either of these is useful is more of an open question, but I expect they are, for example if you need to create a Swift library for consumption by a C program (Go program, Rust program, etc.) that will not carry its own stdlib. But that problem is outside the scope of my present concern.
I'm unclear what you mean, level of abstraction? Level of the compiler? In what other level could we put this feature? |
|
I'm concerned about both the static and dynamic cases.
In practice @mxcl tells me that the package manager generally doesn't even bother building .a files now; it just links .o files directly (using swiftc). And we don't support building dylibs yet. But I wouldn't want to count either of those out in the future. Re: levels: The package manager may know if it's building an entire product, which means it could figure out how to only link in the stdlib once. But it might not. The main thing is that the compiler certainly doesn't (and shouldn't need to). If we decide that (1) the executable has to carry the runtime, and (2) all dependencies that use Swift will be statically linked, then we might be able to get away with just linking it in the executable. In that case, I wonder if a better patch would be to tell the Swift driver to disable automatic linking of the stdlib, and then have the package manager responsible for providing the standard library. (I haven't thought that all the way through, but I did think it was worth wondering aloud.) |
I agree, which is why I don't understand the above objections in the frontend context, since it cannot and should not solve the problem, but should instead simply accept the link directive it has been given. The chosen link directive may be silly; it may lead to all the problems you enumerated, and several more. But that would be user (or package manager, or third-party-build-tool) error. The flag has a legitimate use, it also has illegitimate uses, the frontend should not be playing detective. The use case here is that "somebody" (whether the package manager, a human, or a third-party tool) has already decided that we want a static stdlib and is prepared to resolve the multiple stdlib problem in some way, (if it can even arise in the situation, which it very well may not). We should have a supported path for an entity that "knows what it's doing".
True, one path would be to push this logic out to the caller. It is a little more work to replicate that (deeply platform-specific!) code across multiple callers, than it is to do it here. I do think it would be a good idea to have a The other issue is that by pushing out to callers, we lose the autolink behavior, unless we also replicate that across the callers. |
|
A third issue is that I don't know how it's even possible to do I may just be ignorant here, there is a lot I don't know about ELF and mach-o as formats. I've just never heard of the kind of executable that would be required to push this out to callers, and I don't know if linkers generally support them. |
|
Yeah. I'm bringing up these concerns because we do get complaints about the OS X command-line tool template not working with libraries, and there's no good way to opt out there. I didn't actually mean for You're correct that whatever's done here would be the same as what Xcode's doing, so that is some duplication. (Unfortunately there are other factors that keep Xcode from using swiftc as a linker right now, so we wouldn't get to consolidate there.) I think you've convinced me, though. @ddunbar, still on board after all this discussion? |
|
(Please do fix the coding style issue before the merge.) @swift-ci Please test |
This commit adds the flags -static-stdlib and -no-static-stdlib to create programs statically linked (or not) with the standard library. Not is the default, which is also the current behavior. These flags are currently placebos on non-Darwin platforms.
|
Style nit fixed. I don't recognize the test failure; I believe it's unrelated.
I may have misunderstood this, but still unclear to me what the output from |
swiftc a.o b.o -disable-link-stdlib -L /path/to/swift_staticI agree that the test failure looks harmless, but I'd still like to wait until master is back to passing before landing this and #1891. |
|
Oh, and for a dynamic library, I think you would have to leave symbols undefined (which can be done), but I'm likewise unsure if that would actually correctly work for the runtime. So it might make sense to ban this flag with dynamic libraries ( |
|
OIC, that makes a lot more sense than what I was thinking. 👍 wrt getting master passing |
|
Merged! Sorry again for the delay, and thanks for talking through my concerns. |
[Driver] implement -static-stdlib for Linux Implement the -static-stdlib driver flag for Linux, allowing the static linking of the standard library. This implementation largely follows the Darwin implementation in #1817, although some pecularities warrant extended discussion. The original "link with stdlib" implementation had some redundancies with getRuntimeLibraryPath; these redundancies are resolved, and the implementation alternates between getRuntimeLibraryPath and getStaticRuntimeLibraryPath cleanly as appropriate. A variety of libraries are required to link statically on Linux. The implementation currently dynamically links with them. We should probably support static linking of those as well, but I think that is beyond the scope of a -static-stdlib flag. The test coverage uses ldd here, as otool is not available on Linux. As a result, we currently have separate tests for Linux vs the other platforms; that isn't ideal, but it seems necessary. Perhaps the oddest part, and the one worth the most discussion, is the use of --dynamic-list. Inside stdlib/public/runtime/ProtocolConformances.cpp appears the following code: #elif defined(__ELF__) static int _addImageProtocolConformances(struct dl_phdr_info *info, size_t size, void *data) { // inspectArgs contains addImage*Block function and the section name InspectArgs *inspectArgs = reinterpret_cast<InspectArgs *>(data); void *handle; if (!info->dlpi_name || info->dlpi_name[0] == '\0') { handle = dlopen(nullptr, RTLD_LAZY); } else handle = dlopen(info->dlpi_name, RTLD_LAZY | RTLD_NOLOAD); auto conformances = reinterpret_cast<const uint8_t*>( dlsym(handle, inspectArgs->sectionName)); The effect of this is to search for protocol_conformances_start inside the images. However, dlsym only finds symbols that exist in the dynamic table. Failure to find the protocol conformances can be diagnosed by a "hello world" program printing String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none)) instead of "hello world". (And also by the test coverage in this commit.) Surprisingly, this behavior can still occur on ELF platforms even if `objdump -t` reports a valid `.protocol_conformances_start`. This is because `objdump -t` searches the global table, not the dynamic table, while dlsym only searches the dynamic table. To configure objdump to search only the dynamic table, use `-T`. Inquiring minds may wonder whether dynamically-linked programs (e.g. all Linux binaries up until now) also have a broken protocol conformance table on ELF. The answer is, surprisingly, no; I checked, and ordinary ELF programs are fine. The distinction is probably this, from the ld manpage: > the dynamic symbol table will normally contain only those symbols which are referenced by some dynamic object mentioned in the link. I think the linker sees `.protocol_conformances_start` inside libswiftCore.so and erroneously concludes the one in *the executable* is "referenced by some dynamic object" (e.g. the standard library). This behavior seems to save the dyanmically-linked executable from a broken protocol conformance table. I wonder if it would be wise to apply a similar fix to dynamically-linked programs to avoid relying on the linker "helping" us here, but that's out of scope of this commit. The linker manpage reflects that many people have been bitten by dlsym "surprise", and encourages the use of `--export-dynamic`: > If you use "dlopen" to load a dynamic object which needs to refer back > to the symbols defined by the program, rather than some other dynamic > object, then you will probably need to use [--export-dynamic] when > linking the program itself. However in this situation, the use of `--export-dynamic` causes the entire stdlib to be exported, which is not ideal. However, by combining with the `--exclude-libs ALL` argument, we avoid exporting the entire stdlib.
This commit adds the flags -static-stdlib and -no-static-stdlib to create programs statically linked (or not) with the standard library. Not is the default, which is also the current behavior. These flags are currently placebos on non-Darwin platforms.
[Driver] implement -static-stdlib for Linux Implement the -static-stdlib driver flag for Linux, allowing the static linking of the standard library. This implementation largely follows the Darwin implementation in swiftlang#1817, although some pecularities warrant extended discussion. The original "link with stdlib" implementation had some redundancies with getRuntimeLibraryPath; these redundancies are resolved, and the implementation alternates between getRuntimeLibraryPath and getStaticRuntimeLibraryPath cleanly as appropriate. A variety of libraries are required to link statically on Linux. The implementation currently dynamically links with them. We should probably support static linking of those as well, but I think that is beyond the scope of a -static-stdlib flag. The test coverage uses ldd here, as otool is not available on Linux. As a result, we currently have separate tests for Linux vs the other platforms; that isn't ideal, but it seems necessary. Perhaps the oddest part, and the one worth the most discussion, is the use of --dynamic-list. Inside stdlib/public/runtime/ProtocolConformances.cpp appears the following code: #elif defined(__ELF__) static int _addImageProtocolConformances(struct dl_phdr_info *info, size_t size, void *data) { // inspectArgs contains addImage*Block function and the section name InspectArgs *inspectArgs = reinterpret_cast<InspectArgs *>(data); void *handle; if (!info->dlpi_name || info->dlpi_name[0] == '\0') { handle = dlopen(nullptr, RTLD_LAZY); } else handle = dlopen(info->dlpi_name, RTLD_LAZY | RTLD_NOLOAD); auto conformances = reinterpret_cast<const uint8_t*>( dlsym(handle, inspectArgs->sectionName)); The effect of this is to search for protocol_conformances_start inside the images. However, dlsym only finds symbols that exist in the dynamic table. Failure to find the protocol conformances can be diagnosed by a "hello world" program printing String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none)) instead of "hello world". (And also by the test coverage in this commit.) Surprisingly, this behavior can still occur on ELF platforms even if `objdump -t` reports a valid `.protocol_conformances_start`. This is because `objdump -t` searches the global table, not the dynamic table, while dlsym only searches the dynamic table. To configure objdump to search only the dynamic table, use `-T`. Inquiring minds may wonder whether dynamically-linked programs (e.g. all Linux binaries up until now) also have a broken protocol conformance table on ELF. The answer is, surprisingly, no; I checked, and ordinary ELF programs are fine. The distinction is probably this, from the ld manpage: > the dynamic symbol table will normally contain only those symbols which are referenced by some dynamic object mentioned in the link. I think the linker sees `.protocol_conformances_start` inside libswiftCore.so and erroneously concludes the one in *the executable* is "referenced by some dynamic object" (e.g. the standard library). This behavior seems to save the dyanmically-linked executable from a broken protocol conformance table. I wonder if it would be wise to apply a similar fix to dynamically-linked programs to avoid relying on the linker "helping" us here, but that's out of scope of this commit. The linker manpage reflects that many people have been bitten by dlsym "surprise", and encourages the use of `--export-dynamic`: > If you use "dlopen" to load a dynamic object which needs to refer back > to the symbols defined by the program, rather than some other dynamic > object, then you will probably need to use [--export-dynamic] when > linking the program itself. However in this situation, the use of `--export-dynamic` causes the entire stdlib to be exported, which is not ideal. However, by combining with the `--exclude-libs ALL` argument, we avoid exporting the entire stdlib.
What's in this pull request?
Depends on #1806.
This PR adds a
-static-stdlibdriver flag to statically link the standard library into a Swift program (and-no-static-stdlibto not).-no-static-stdlibis default.The flags are only implemented for Darwin platforms at present. I do intend to PR a Linux implementation soon, but the implementation for Linux will be quite a bit more complex and IMO deserves its own discussion about the inner workings of linkers for ELF. As a consequence the test coverage in here is
XFAIL: linuxand can be reviewed (and merged?) independently of the linux conversation.Resolved bug number: (SR-)
This PR is related to, but does not completely resolve, SR-730.
Before merging this pull request to apple/swift repository:
Triggering Swift CI
The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:
Smoke Testing
Validation Testing
Note: Only members of the Apple organization can trigger swift-ci.