[cmake] Copy Dispatch to the SDK subdir on host.

[3405691582]

We have two directories for Swift libraries,

• SWIFT_SDK_<platform>_LIB_SUBDIR, a.k.a., the SDK subdir,
  a.k.a., swift-<platform>-<arch>/lib/swift/<platform>, and
• SWIFTLIB_SINGLE_SUBDIR,
  a.k.a., swift-<platform>-<arch>/lib/swift/<platform>/<arch>.

We unconditionally copy libdispatch.so and libBlocksRuntime.so to
SWIFTLIB_SINGLE_SUBDIR. But this is insufficient, because swiftc emits
binaries with an RPATH with the SDK subdir.

This does not cause problems when these libraries are installed on the
platform itself, but the platform loader will choke since it can't find
those libraries -- as they are not in the RPATH.

This itself isn't so much of an issue, but it is an inconsistency since
we normally do the needful to ensure we don't need to install binaries
and libraries to the system to run and execute the Swift compiler and
binaries, so this is indeed something of a regression.

Normally, we would require adding a -ldispatch flag to make this all
work but this is difficult to do correctly, and more
importantly because other platforms are using $ORIGIN to get the
dependency search to work properly, let's do so also here.

This seems like the best way to do this; please advise if there is some
better mechanism to go about achieving this goal.

[soloturn]

copy libdispatch to a subdir sounds counter-intuitive. should it not be a normal dependency on linux like any library?

[3405691582]

We are in a bit of a rock and a hard place here.

AIUI, the Unixlike toolchains specify rpath as .../lib/swift/<platform>. Swift builds libs that go into both dirs, and the Swift module files go only in .../lib/swift/<platform>.

Compiled binaries don't have direct dependencies on libdispatch, but do have implicit dependencies transitively through libswift_Concurrency.so. That gets resolved with $ORIGIN being set to ., but this means that libdispatch.so needs to be adjacent to libswift_Concurrency.so for that to work. On all platforms, libswift_Concurrency.so resides in both places, but we only copy libdispatch.so into .../lib/swift/<platform>/<arch>. This means that we have to have compiled binaries choose the libswift_Concurrency.so in .../lib/swift/<platform>/<arch>, otherwise the Dispatch dependency can't be resolved.

On Linux, the loader apparently doesn't just look at the rpath, but subdirectories of the rpath named x86_64 -- it even seems to prefer those subdirectories over the rpath itself. This means that the loader will pick the libswift_Concurrency.so in .../lib/swift/<platform>/<arch> where libdispatch.so resides. Everything is happy.

Elsewhere, e.g., on OpenBSD (at least), no rpath subdirectories are searched. The loader picks the version of libswift_Concurrency.so in .../lib/swift/<platform>, just as the rpath tells it to. But libdispatch.so is put in .../lib/swift/<platform>/<arch>, so then the libdispatch dependency breaks. The compiled binary doesn't load properly unless you set LD_LIBRARY_PATH. This means that with concurrency support enabled, all swiftc binaries don't load properly, even when not using concurrency features.

One alternative for us is to force the rpath to the .../lib/swift/<platform>/<arch>. That seems to work okay, but the problem here is that Swift populates the rpath from the runtime resource path. That path is overloaded -- it has the Swift module files, and it automagically appends the platform name to the path. So we have to inject a bit of a platform-specific hack in toolchains::GenericUnix::constructInvocation, which is ugly, but works, e.g.,

  if (addRuntimeRPath(getTriple(), context.Args)) {
    for (auto path : RuntimeLibPaths) {
      Arguments.push_back("-Xlinker");
      Arguments.push_back("-rpath");
      Arguments.push_back("-Xlinker");
      if (getTriple().isOSOpenBSD()) {
        path += "/" + getTriple().getArchName();
      }
      Arguments.push_back(context.Args.MakeArgString(path));
    }
  }

So ultimately, do we want to copy the dependency twice, or hack the rpath in the invocation? A rock and a hard place. Or is something else entirely that I haven't thought of?

[3405691582]

Actually, hacking the rpath in the invocation is wrong. When building the toolchain (not just swift in isolation), the .../lib/swift/<platform>/<arch> directory gets the .swiftmodule files and the actual .so files are in .../lib/swift/<platform>/. This suggests .../lib/swift/<platform>/<arch> is the wrong place for the .so files.

The question remains: why are we putting the Dispatch libraries in .../lib/swift/<platform>/<arch> altogether? Let me try changing this altogether and seeing what happens.

[finagolfin]

I think the goal is to move everything to lib/swift/<platform>/<arch>, so that we can support building multiple arches from one lib/swift/<platform> directory, see related issues in this forum thread.

If you're having issues with libdispatch not installed to the latter path, maybe it should be installed there too on OpenBSD, dunno?

[3405691582]

I think the goal is to move everything

Cool, sounds like my guess is correct.

If you're having issues with libdispatch not installed to the latter path, maybe it should be installed there too on OpenBSD, dunno?

Right. I'm just testing an update to this PR (not pushed yet) that copies libdispatch everywhere libswift_Concurrency.so resides (so unfortunately, we have to go from two copies to four copies always), because that is ultimately what is required.

[3405691582]

Okay, now this PR unconditionally makes four copies. This means that whichever library the platform loader picks, it can always resolve its libdispatch dependency.

[3405691582]

Ping.

[finagolfin]

@drexin, you added this libdispatch build file, could you review?

[drexin]

@swift-ci test