[5.9][CMake] Support Macros in Linux #68190

rintaro · 2023-08-29T17:59:31Z

Cherry-pick #66199 and #68082 into release/5.9

Explanation: Enable Macros in Linux. Swift parser integration is required for enabling macros. Since Linux doesn't have ABI stable swiftCore in /usr/lib or any other stable location, executables and shared libraries that contains Swift module need to have RUNPATH to the runtime in its own toolchain. This PR forces "hosttools" bootstrapping mode, but set the RUNPATH to the currently-being-built toolchain.
Scope: CMake build script
Risk: Mid. This PR is carefully crafted not to affect Darwin platforms. For Linux, this is necessary for enabling macros.
Testing: Passes current test suite.
Issues: rdar://104346187
Reviewers: TBA

When building Swift code into the compiler (e.g, the new Swift parser along with macros support), make sure we always add the appropriate paths to (1) link against the host Swift toolchain, and (2) find the host Swift libraries (such as SwiftSyntax) at runtime. The CMake code for doing this was only running for Darwin builds, so generalize it to also work on Linux.

This enables running macro tests on Linux.

For compiling codes required for macro support, we now need swiftc compiler in the build machine. Unlike Darwin OSes, where swiftCore runtime is guaranteed to be present in /usr/lib, Linux doesn't have ABI stability and the stdlib of the build machine is not at the specific location. So the built compiler cannot relies on the shared object in the toolchain. (cherry picked from commit 9c9010e)

Many shared libs and executables are only run after stdlib/runtime are built. They don't need to link with builders stdlib at all. (cherry picked from commit 2a2787b)

and Swift parser integration is enabled. If swift parser integration is enabled, SwiftSyntax libraries are always built with host tools. Other SwiftCompilerSources modules must use the same runtime with parser libraries. (cherry picked from commit 0f0c492)

(cherry picked from commit 9017ef5)

SourceKit components are never built while bootstrapping stages. So it doen't need to have stage 0 or 1 handling. (cherry picked from commit 914a6fd)

`$ORIGIN` is only supported in Linux, BSD, etc. (i.e. not in Windows) (cherry picked from commit 602604b)

In Linux. Instead of setting temporary "fallback" RUNPATH, Set LD_LIBRARY_PATH to builder's runtime when building standard library. So we don't need to strip the temporary RUNPATH when installing. (cherry picked from commit 757aaa3)

recursive-lipo requires shared libraries have +x permissions. Use PROGRAMS instead of FILES when installing swift-syntax libraries. (cherry picked from commit 319b8bd)

libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime library. (cherry picked from commit 4ccc866)

swift-compatibility-symbols, swift-def-to-strings-converter, and swift-serialize-diagnostics don't use any Swift modules. But when SWIFT_SWIFT_PARSER was enabled, they are linked with swiftCore. But these binaries can be executed before the runtime is being built. We need to stop them linking with swiftCore. (cherry picked from commit dc68773)

RPATH_SET is not available until cmake 3.21.0. Use RPATH_CHANGE instead. (cherry picked from commit f645069)

rintaro · 2023-08-29T18:27:14Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please test

rintaro · 2023-08-29T18:27:23Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please Build Toolchain

bnbarham · 2023-08-30T04:15:13Z

cmake/modules/SwiftUtils.cmake

@@ -107,6 +107,16 @@ function(get_bootstrapping_swift_lib_dir bs_lib_dir bootstrapping)
    elseif("${bootstrapping}" STREQUAL "")
      get_bootstrapping_path(bs_lib_dir ${lib_dir} "1")
    endif()
+  elseif(BOOTSTRAPPING_MODE STREQUAL "HOSTTOOLS")
+    if(SWIFT_HOST_VARIANT_SDK MATCHES "LINUX|ANDROID|OPENBSD|FREEBSD")


There's a bunch of places we're only checking for Linux and others where we check for DARWIN and then else. Given we haven't checked the other platforms, should we just check for LINUX for now?

bnbarham · 2023-08-30T04:17:44Z

lib/SILOptimizer/Utils/BasicBlockOptUtils.cpp

@@ -367,3 +367,12 @@ StaticInitCloner::clone(SingleValueInstruction *initVal) {
  }
  return cast<SingleValueInstruction>(getMappedValue(initVal));
 }
+
+// START: Workaround for C++ interop


Did you pick a random file or is this the one I randomly picked? Would probably be better to put this in its own file, but given it's only needed for 5.9 ... 🤔

Some non-stdlib thing e.g. swift-backtrace still might be built before the runtime is built. For building Swift code in Linux "hosttools", always set 'LD_LIBRARY_PATH' to the runtime in the builder. (cherry picked from commit 0aa0aac)

RPATH to 'lib/swift/host' was missing in Darwin platform. (cherry picked from commit 9d78971)

"Support Macros in Linux" patches was inconsistent with checking platforms. Some only checked 'LINUX' but some matches 'LINUX|ANDROID|OPENBSD|FREEBSD'. Although I don't have tested other platoforms than Linux at all, there's no reason to limit it to Linux. So use the consistent check to match 'LINUX|ANDROID|OPENBSD|FREEBSD' (cherry picked from commit 8049922)

When Swift parser integration is disabled (i.e. NOT SWIFT_SWIFT_PARSER), macro plugins targets are not created. So CMake should bail out. (cherry picked from commit ad2dacd)

Ubuntu 18.04 seems to be defaulting to 'ascii', causing an error when using line-directive combined with the utf-8 diagnostic output used when early swift syntax is enabled. (cherry picked from commit afd3f17)

rintaro · 2023-08-30T22:35:43Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please test

rintaro · 2023-08-30T22:35:48Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please Build Toolchain

rintaro · 2023-08-30T23:16:59Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please test

rintaro · 2023-08-30T23:17:05Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please Build Toolchain

rintaro · 2023-08-30T23:34:57Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please test

rintaro · 2023-08-30T23:35:06Z

swiftlang/llvm-project#7345
swiftlang/swift-syntax#2114
@swift-ci Please Build Toolchain

DougGregor and others added 14 commits August 29, 2023 04:07

[CMake] Fix host library builds and rpaths for testing macros

c573a7a

This enables running macro tests on Linux.

[SourceKit] Fix paths when including the new Swift compiler

defe9eb

Avoid adding redundant temporary RUNPATH to builder's stdlib

16bf7a3

Many shared libs and executables are only run after stdlib/runtime are built. They don't need to link with builders stdlib at all. (cherry picked from commit 2a2787b)

[CMake] Update for review comments

79f6bca

(cherry picked from commit 9017ef5)

[CMake] Fix weiredness in swift runtime linking for SourceKit libraries

42360db

SourceKit components are never built while bootstrapping stages. So it doen't need to have stage 0 or 1 handling. (cherry picked from commit 914a6fd)

[CMake] Further cleanup for RUNPATH setting

64c5430

`$ORIGIN` is only supported in Linux, BSD, etc. (i.e. not in Windows) (cherry picked from commit 602604b)

[CMake] Fix install for macOS toolchain

5bbab90

recursive-lipo requires shared libraries have +x permissions. Use PROGRAMS instead of FILES when installing swift-syntax libraries. (cherry picked from commit 319b8bd)

[CMake] Fix RUNPATH of lib_InternalSwiftScan in Linux

3badb3b

libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime library. (cherry picked from commit 4ccc866)

[CMake] Use RPATH_CHANGE instead RPATH_SET

c39f95b

RPATH_SET is not available until cmake 3.21.0. Use RPATH_CHANGE instead. (cherry picked from commit f645069)

rintaro requested a review from a team as a code owner August 29, 2023 17:59

This was referenced Aug 29, 2023

[5.9.0][CMake] Support Macros in Linux #68197

Merged

[5.9][CMake] Add padding to RUNPATH in linux swiftlang/swift-syntax#2114

Merged

[5.9.0][CMake] Add padding to RUNPATH in linux swiftlang/swift-syntax#2117

Merged

bnbarham approved these changes Aug 30, 2023

View reviewed changes

rintaro and others added 5 commits August 30, 2023 21:37

[CMake][SourceKit] Add RPATH to 'lib/swift/host'

c244dfc

RPATH to 'lib/swift/host' was missing in Darwin platform. (cherry picked from commit 9d78971)

[CMake] Handle cases where Swift parser integration is disabled in Linux

5125ac3

When Swift parser integration is disabled (i.e. NOT SWIFT_SWIFT_PARSER), macro plugins targets are not created. So CMake should bail out. (cherry picked from commit ad2dacd)

[stdlib] Set PYTHONIOENCODING when building the stdlib

f038280

Ubuntu 18.04 seems to be defaulting to 'ascii', causing an error when using line-directive combined with the utf-8 diagnostic output used when early swift syntax is enabled. (cherry picked from commit afd3f17)

rintaro force-pushed the 5.9-swift-swift-linux branch 2 times, most recently from 829c3b7 to 7c5bc7c Compare August 30, 2023 22:34

rintaro force-pushed the 5.9-swift-swift-linux branch from 7c5bc7c to 9e7c16f Compare August 30, 2023 23:16

[CMake] Instantiate missing C++ templates

71f5b47

rintaro force-pushed the 5.9-swift-swift-linux branch from 9e7c16f to 71f5b47 Compare August 30, 2023 23:34

shahmishal merged commit 38047ec into swiftlang:release/5.9 Aug 31, 2023

rintaro mentioned this pull request Oct 4, 2023

[5.9][SourceKit] Fix RPATH to lib/swift/host in SourceKit frameworks #68925

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[5.9][CMake] Support Macros in Linux #68190

[5.9][CMake] Support Macros in Linux #68190

Uh oh!

rintaro commented Aug 29, 2023 •

edited

Loading

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

bnbarham Aug 30, 2023

Uh oh!

bnbarham Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

Uh oh!

[5.9][CMake] Support Macros in Linux #68190

[5.9][CMake] Support Macros in Linux #68190

Uh oh!

Conversation

rintaro commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

bnbarham Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

bnbarham Aug 30, 2023

Choose a reason for hiding this comment

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

Uh oh!

rintaro commented Aug 29, 2023 •

edited

Loading