Skip to content

[CMake] Add broader support for cross-compiling the portions of the compiler that are written in Swift to non-Darwin Unix #71552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 15, 2024
Merged

[CMake] Add broader support for cross-compiling the portions of the compiler that are written in Swift to non-Darwin Unix #71552

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions SwiftCompilerSources/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,9 @@ function(add_swift_compiler_modules_library name)
list(APPEND sdk_option "-resource-dir" "${swift_exec_bin_dir}/../bootstrapping0/lib/swift")
endif()
elseif(BOOTSTRAPPING_MODE STREQUAL "CROSSCOMPILE")
get_filename_component(swift_exec_bin_dir ${ALS_SWIFT_EXEC} DIRECTORY)
# NOTE: prepending allows SWIFT_COMPILER_SOURCES_SDK_FLAGS to override the
# resource directory if needed.
list(PREPEND sdk_option "-resource-dir" "${swift_exec_bin_dir}/../lib/swift")
list(PREPEND sdk_option "-resource-dir" "${SWIFTLIB_DIR}")
endif()
get_versioned_target_triple(target ${SWIFT_HOST_VARIANT_SDK}
${SWIFT_HOST_VARIANT_ARCH} "${deployment_version}")
Expand Down Expand Up @@ -225,6 +224,9 @@ function(add_swift_compiler_modules_library name)
importedHeaderDependencies
COMMENT "Building swift module ${module}")

if(BOOTSTRAPPING_MODE STREQUAL "CROSSCOMPILE")
add_dependencies(${dep_target} swift-stdlib-${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}-${SWIFT_HOST_VARIANT_ARCH})
endif()
set("${module}_dep_target" ${dep_target})
set(all_module_targets ${all_module_targets} ${dep_target})
endforeach()
Expand Down
10 changes: 10 additions & 0 deletions cmake/modules/AddPureSwift.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,16 @@ function(_add_host_swift_compile_options name)
$<$<COMPILE_LANGUAGE:Swift>:none>)

target_compile_options(${name} PRIVATE $<$<COMPILE_LANGUAGE:Swift>:-target;${SWIFT_HOST_TRIPLE}>)
if(BOOTSTRAPPING_MODE STREQUAL "CROSSCOMPILE")
add_dependencies(${name} swift-stdlib-${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}-${SWIFT_HOST_VARIANT_ARCH})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the stdlib must be cross-compiled first by the host compiler,

The standard library must be built with the new compiler. Requiring the new stdlib to build the new compiler while also requiring the new compiler to build the stdlib will result in a situation where we can't build anything. I'm confused about how this is working for you at all given that changes landed literally yesterday that cause the stdlib to not build with the nightly toolchain compiler. You should be seeing something like this if you try:

swift/stdlib/public/core/KeyPath.swift:3696:5: error: type '(Any).Type' cannot conform to '_BitwiseCopyable'
3414 │     return (baseAddress, misalign)
3415 │   }
3416 │   mutating func pushDest<T : _BitwiseCopyable>(_ value: T) {
     │                 ╰─ note: required by instance method 'pushDest' where 'T' = '(Any).Type'
3417 │     let size = MemoryLayout<T>.size
3418 │     let (baseAddress, misalign) = adjustDestForAlignment(of: T.self)
     ┆
3694 │                      genericEnvironment: genericEnvironment,
3695 │                      arguments: patternArgs)
3696 │     pushDest(metadata)
     │     ├─ error: type '(Any).Type' cannot conform to '_BitwiseCopyable'
     │     ╰─ note: only concrete types such as structs, enums and classes can conform to protocols
3697 │     base = metadata
3698 │   }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standard library must be built with the new compiler. Requiring the new stdlib to build the new compiler while also requiring the new compiler to build the stdlib will result in a situation where we can't build anything. I'm confused about how this is working for you at all

Let me step back a bit and explain what I'm doing here. This CROSSCOMPILE bootstrap mode is only invoked when cross-compiling the Swift compiler itself for a different platform, say you want to build the Swift compiler for linux AArch64 on a linux x86_64 host. As such, these dependencies have to be reordered as I laid out in the commit message.

There are two ways to accomplish this, but the most likely is to first build the Swift compiler from source natively for the host, then turn around and cross-compile the exact same compiler source for another platform, using the freshly-built compiler to build the Swift portions of the cross-compiled compiler. That is what these reordered dependencies when cross-compiling the compiler are needed for.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@etcwilde, let me add more detail on why this is needed. Previously, the Swift compiler was written purely in C++ and the CMake dependency order was to always build the Swift compiler first, then the Swift stdlib, especially because for a native host build, that freshly-built Swift compiler is used to build the native Swift stdlib. However, cross-compiling the Swift compiler and stdlib is also supported and in that case, the freshly-built native clang and Swift compilers still cross-compile the Swift compiler first, then cross-compile the Swift stdlib. No dependency re-ordering was needed, because which was cross-compiled first didn't matter.

However, now that significant portions of the Swift compiler are written in Swift, the order does matter. You have to cross-compile the Swift stdlib first, use that cross-compiled stdlib to cross-compile SwiftCompilerSources/ and lib/ASTGen/ next, and only then can you link the cross-compiled Swift compiler. These CMake dependency changes make sure that re-ordering is done when cross-compiling the Swift stdlib and compiler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am aware of the Swift in the swift compiler. I am concerned about the abuse of CMake here (and I'm not blaming you, the whole build is bad). A given CMake invocation should only be building for one thing at a time. For cross-compiling, I would expect two CMake invocations. One to build your cross-compiling toolchain with the necessary SDKs enabled. Then, in a separate CMake invocation, use that just-built toolchain to cross compile the compiler and the SDKs that you plan on shipping.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am aware of the Swift in the swift compiler.

I know, I'm just making clear the implications of that change for cross-compiling.

A given CMake invocation should only be building for one thing at a time. For cross-compiling, I would expect two CMake invocations. One to build your cross-compiling toolchain with the necessary SDKs enabled. Then, in a separate CMake invocation, use that just-built toolchain to cross compile the compiler and the SDKs that you plan on shipping.

So, technically, that is mostly how the build-script currently works. It invokes CMake for each Swift repo separately, like the compiler or foundation, to natively build them, collects all the build products in one native toolchain directory, then uses that freshly-built native toolchain to cross-compile each of those repos with multiple CMake invocations with different cross-compilation flags.

The only difference from what you said is that rather than mixing cross-compiling the stdlib in with natively compiling initially, it does all the native compilation first, then cross-compiles for each cross-compilation non-Darwin platform separately.

I see no reason why your preference is better.

I am concerned about the abuse of CMake here (and I'm not blaming you, the whole build is bad).

What specifically are you worried about in the CMake usage here? Swift does not use CMake's built-in Swift support when building the Swift compiler, because obviously that CMake support didn't exist when this Swift project started.

target_compile_options(${name} PRIVATE
$<$<COMPILE_LANGUAGE:Swift>:-sdk;${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_ARCH_${SWIFT_HOST_VARIANT_ARCH}_PATH};>
$<$<COMPILE_LANGUAGE:Swift>:-resource-dir;${SWIFTLIB_DIR};>)
if(SWIFT_HOST_VARIANT_SDK STREQUAL "ANDROID" AND NOT "${SWIFT_ANDROID_NDK_PATH}" STREQUAL "")
swift_android_tools_path(${SWIFT_HOST_VARIANT_ARCH} tools_path)
target_compile_options(${name} PRIVATE $<$<COMPILE_LANGUAGE:Swift>:-tools-directory;${tools_path};>)
endif()
endif()
_add_host_variant_swift_sanitizer_flags(${name})

target_compile_options(${name} PRIVATE
Expand Down
18 changes: 11 additions & 7 deletions cmake/modules/AddSwift.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -529,15 +529,19 @@ function(_add_swift_runtime_link_flags target relpath_to_lib_dir bootstrapping)
elseif(SWIFT_HOST_VARIANT_SDK MATCHES "LINUX|ANDROID|OPENBSD|FREEBSD|WINDOWS")
set(swiftrt "swiftImageRegistrationObject${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_OBJECT_FORMAT}-${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}-${SWIFT_HOST_VARIANT_ARCH}")
if(ASRLF_BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS|CROSSCOMPILE")
# At build time and run time, link against the swift libraries in the
# installed host toolchain.
if(SWIFT_PATH_TO_SWIFT_SDK)
set(swift_dir "${SWIFT_PATH_TO_SWIFT_SDK}/usr")
if(ASRLF_BOOTSTRAPPING_MODE STREQUAL "HOSTTOOLS")
# At build time and run time, link against the swift libraries in the
# installed host toolchain.
if(SWIFT_PATH_TO_SWIFT_SDK)
set(swift_dir "${SWIFT_PATH_TO_SWIFT_SDK}/usr")
else()
get_filename_component(swift_bin_dir ${SWIFT_EXEC_FOR_SWIFT_MODULES} DIRECTORY)
get_filename_component(swift_dir ${swift_bin_dir} DIRECTORY)
endif()
set(host_lib_dir "${swift_dir}/lib/swift/${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}")
else()
get_filename_component(swift_bin_dir ${SWIFT_EXEC_FOR_SWIFT_MODULES} DIRECTORY)
get_filename_component(swift_dir ${swift_bin_dir} DIRECTORY)
set(host_lib_dir "${SWIFTLIB_DIR}/${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}")
endif()
set(host_lib_dir "${swift_dir}/lib/swift/${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}")
set(host_lib_arch_dir "${host_lib_dir}/${SWIFT_HOST_VARIANT_ARCH}")

set(swiftrt "${host_lib_arch_dir}/swiftrt${CMAKE_C_OUTPUT_EXTENSION}")
Expand Down
4 changes: 4 additions & 0 deletions lib/SwiftSyntax/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,10 @@ endif()
add_dependencies(swift-syntax-lib
${SWIFT_SYNTAX_MODULES})

if(BOOTSTRAPPING_MODE STREQUAL "CROSSCOMPILE")
add_dependencies(swift-syntax-lib swift-stdlib-${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}-${SWIFT_HOST_VARIANT_ARCH})
endif()

# Install Swift module interface files.
foreach(module ${SWIFT_SYNTAX_MODULES})
set(module_dir "${module}.swiftmodule")
Expand Down
5 changes: 3 additions & 2 deletions stdlib/cmake/modules/SwiftSource.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -836,8 +836,9 @@ function(_compile_swift_files
endif()

set(swift_compiler_tool_dep)
if(SWIFT_INCLUDE_TOOLS)
# Depend on the binary itself, in addition to the symlink.
if(SWIFT_INCLUDE_TOOLS AND NOT BOOTSTRAPPING_MODE STREQUAL "CROSSCOMPILE")
# Depend on the binary itself, in addition to the symlink, unless
# cross-compiling the compiler.
set(swift_compiler_tool_dep "swift-frontend${target_suffix}")
endif()

Expand Down
14 changes: 9 additions & 5 deletions tools/SourceKit/cmake/modules/AddSwiftSourceKit.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,15 @@ function(add_sourcekit_swift_runtime_link_flags target path HAS_SWIFT_MODULES)
elseif(SWIFT_HOST_VARIANT_SDK MATCHES "LINUX|ANDROID|OPENBSD" AND HAS_SWIFT_MODULES AND ASKD_BOOTSTRAPPING_MODE)
set(swiftrt "swiftImageRegistrationObject${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_OBJECT_FORMAT}-${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}-${SWIFT_HOST_VARIANT_ARCH}")
if(ASKD_BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS|CROSSCOMPILE")
# At build time and run time, link against the swift libraries in the
# installed host toolchain.
get_filename_component(swift_bin_dir ${SWIFT_EXEC_FOR_SWIFT_MODULES} DIRECTORY)
get_filename_component(swift_dir ${swift_bin_dir} DIRECTORY)
set(host_lib_dir "${swift_dir}/lib/swift/${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}")
if(ASKD_BOOTSTRAPPING_MODE MATCHES "HOSTTOOLS")
# At build time and run time, link against the swift libraries in the
# installed host toolchain.
get_filename_component(swift_bin_dir ${SWIFT_EXEC_FOR_SWIFT_MODULES} DIRECTORY)
get_filename_component(swift_dir ${swift_bin_dir} DIRECTORY)
set(host_lib_dir "${swift_dir}/lib/swift/${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}")
else()
set(host_lib_dir "${SWIFTLIB_DIR}/${SWIFT_SDK_${SWIFT_HOST_VARIANT_SDK}_LIB_SUBDIR}")
endif()

target_link_libraries(${target} PRIVATE ${swiftrt})
target_link_libraries(${target} PRIVATE "swiftCore")
Expand Down