Add support for cross-compiling Swift stdlib with an external sysroot #79090
Conversation
- This enables libdispatch to be built for the arch passed and not neccessarily the host arch.
- The name CROSS_COMPILE_SYSROOT is chosen since any time we use a sysroot not at "/" we are cross-compiling for some platform or arch.
- This forces it to use the external sysroot instead of the default
…fix-swift-build-support-typo Fix typo in get_linux_sysroot that makes cross-compilation fail
|
@MaxDesiatov here is my first PR that adds support for building the Swift stdlib using an external sysroot, using a new flag called |
|
To build this inside of a docker container (with Jammy as an example), assuming you have the Swift workspace checked out with my branch: docker run -it --name swift-armv7-builder -v $(pwd):/src swiftlang/swift:nightly-jammy /bin/bash
apt update && apt install cmake ninja-build libsqlite3-dev ncurses-dev git python3 libstdc++-11-dev-armhf-cross
cd /src
export SWIFT_NATIVE_PATH=/usr/bin
./swift/utils/build-script --cross-compile-hosts=linux-armv7 --cross-compile-sysroot=/ \
--build-llvm=0 --skip-local-build --build-swift-tools=0 --swift-enable-backtracing=0 --build-embedded-stdlib=0 \
--native-swift-tools-path=$SWIFT_NATIVE_PATH --native-clang-tools-path=$SWIFT_NATIVE_PATH \
--extra-cmake-options="-DSWIFT_ENABLE_RUNTIME_MODULE=FALSE"This will cross-compile for armv7 using the cross libraries installed with |
|
@swift-ci test |
|
@swift-ci build toolchain |
Yes, that is what I told you when you first tried it, that no other build preset uses that flag and while it was fine to use it as a temporary fix, you should look into fixing the need for that internally instead.
Makes sense, as that flag was only a temporary hack, to be used when cross-compiling the stdlib alone.
Yes and no, it is not the best to use initially when you're just cross-compiling the stdlib, but that flag is required later to cross-compile the corelibs, like Foundation.
There's nothing different, my Android CI uses that
I don't see the need for a new flag: the current build-script has been cross-compiling to linux and Android for a long time with the existing flags.
Seems fine, looks similar to my Android CI command linked above that I've been using for years now, except you have to disable those embedded/backtracing flags for linux also. Looks like you and Max are still getting this pull working, let me know when it's ready and I will review. One thing you'll want to watch out for is that your cross-compilation changes don't break the existing linux toolchain build. |
- This is where it will be installed, and we don't want to conflict with other gcc installations at /home or otherwise.
Thanks. That's a big deal, I want to ensure that all of these changes will not affect any other builds that we currently have. Thus, why it makes sense to introduce the We'll see what the solution can be to the other compilation issues. Worst case perhaps we can add a patch for those if cross-compiling them is an issue. Idk... |
|
@swift-ci test |
|
@swift-ci build toolchain |
Looks like it builds now- I updated my nightly-jammy container to the latest snapshot and Slab.swift now builds. Phew. That answers that. |
|
Most likely spurious, extremely unlikely that the few CMake changes have any effect on Windows and they don't use this |
|
@swift-ci test windows |
|
@finagolfin @MaxDesiatov looks like it built! 🎉 Let me know if these changes will be acceptable. My next work will be to ensure the core libs- dispatch, foundation, xctest, and swift testing build with a Linux cross compilation profile. We already know that Swift 6 and later builds for armv7, so now it comes down to supporting it from the build scripts 😁 |
|
@swift-ci test |
|
@swift-ci build toolchain |
|
@MaxDesiatov wording changes done. Let me know if there will be anything else! I have tested this PR with building against a variety of Ubuntu/Debian sysroots and arches and it seems to work really well, so now it comes down to any other reviewers who want to comment I guess... I'm now looking at building the corelibs and trying to figure out the best way to update the scripts to pass the correct flags, etc. @finagolfin can I get in touch with you again to ask you a few questions...? This is related to the targets.py stuff that we had talked about before related to |
|
Sure, the best way to reach me is my email, which you can get from my github patches. If you plan to use the CMake toolchain file, I don't know much about that format and the current Android build does not use it, but the flags both set should be pretty similar. |
|
@swift-ci build toolchain |
|
@swift-ci test |
|
@MaxDesiatov another fluke on the macOS run? I saw it last time where watchOS failed to build, but then it worked the next time you ran the CI. |
|
@swift-ci please test macos |
|
@edymtt @MaxDesiatov This has been sitting for a few days...I know this doesn't seem like a very exciting PR to review and that you guys are busy, but I'd like to see a little progress here. Maybe it'll help to show off what these small changes can do just inside of a container, and what is generated with the following commands: $ docker run -it --name swift-armv7-builder -v $(pwd):/src swiftlang/swift:nightly-jammy /bin/bash
$ apt update && apt install cmake ninja-build libsqlite3-dev ncurses-dev git python3 libstdc++-11-dev-armhf-cross uuid-dev
$ cd /src && export SWIFT_NATIVE_PATH=/usr/bin
$ ./swift/utils/build-script --cross-compile-hosts=linux-armv7 --cross-compile-sysroot=/ \
--build-llvm=0 --skip-local-build --build-swift-tools=0 --swift-enable-backtracing=0 --build-embedded-stdlib=0 \
--native-swift-tools-path=$SWIFT_NATIVE_PATH --native-clang-tools-path=$SWIFT_NATIVE_PATH \
--extra-cmake-options="-DSWIFT_ENABLE_RUNTIME_MODULE=FALSE" --release --install-swift --install-destdir=swift-armv7-installI ran the commands above on my MacBook Pro / M1 Pro. The following artifacts are generated: linux-armv7/usr/lib/swift/linux
├── Cxx.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── CxxStdlib.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── Distributed.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── Glibc.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── Observation.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── RegexBuilder.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── Swift.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── SwiftOnoneSupport.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── Synchronization.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── _Builtin_float.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── _Concurrency.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── _Differentiation.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── _RegexParser.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── _StringProcessing.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── _Volatile.swiftmodule
│ ├── armv7-unknown-linux-gnueabihf.abi.json
│ ├── armv7-unknown-linux-gnueabihf.private.swiftinterface
│ ├── armv7-unknown-linux-gnueabihf.swiftdoc
│ ├── armv7-unknown-linux-gnueabihf.swiftinterface
│ └── armv7-unknown-linux-gnueabihf.swiftmodule
├── armv7
│ ├── SwiftGlibc.h
│ ├── glibc.modulemap
│ └── swiftrt.o
├── libcxxshim.h
├── libcxxshim.modulemap
├── libcxxstdlibshim.h
├── libstdcxx.h
├── libstdcxx.modulemap
├── libswiftCore.so
├── libswiftCxx.a
├── libswiftCxxStdlib.a
├── libswiftDistributed.so
├── libswiftGlibc.so
├── libswiftObservation.so
├── libswiftRegexBuilder.so
├── libswiftRemoteMirror.so
├── libswiftSwiftOnoneSupport.so
├── libswiftSynchronization.so
├── libswift_Builtin_float.so
├── libswift_Concurrency.so
├── libswift_Differentiation.so
├── libswift_RegexParser.so
├── libswift_StringProcessing.so
└── libswift_Volatile.soEverything has the correct arch, and I can run it on the target (in my case a Raspberry Pi 2B) if I also manually scp the Can we get this completed so that this initial support is in, then I can work on the next step? |
|
@xtremekforever, the majority of this pull simply adding the new I see no reason you cannot continue on to building your next steps locally in the meantime: are you worried this will be rejected completely or what? The policy here is weekly pings, but I advise you that given how busy everybody is and how niche 32-bit armv7 is nowadays, this pull may take a little longer. One step you may want to take next that would give us more info and may help validate this pull is to run the host tests from the compiler validation suite with your new armv7 cross-compilation config. Take a look at the flags for how this is done for Android in the doc,, add the Basically, you'd build the Swift compiler and some other testing tools also by removing the |
|
@finagolfin thanks for the reply! My only worry is that it gets forgotten and by the time it's revisited we've already released Swift 6.2, etc. So I want to keep pinging on it, and if there's anything else to do to improve this or make it easier to approve I'll do it! I intend on seeing this through until the end. Let me try your suggestion to run some tests! I'm not sure if it'll work, but I'll let you know. I did send you an email the other day but got no response- is [email protected] the correct email? |
|
Yes, I got your email, just been a bit busy this week, will get back to you soon. |
|
@finagolfin okay no problem! I just wasn't sure if you had gotten it. You don't have to worry about that one since it was just me asking what you thought. I tried running this command: And as expected, it doesn't work: The scripts don't know how to compile for linux-armv7...as happened last time. Any ideas? Maybe we can continue this over email. |
|
Apologies for my late reply here -- I am currently not able to find the time to review this properly for two main reasons:
I will try to chip at this, but I can make no promises on when I will able to complete my review (assuming my priorities stay the same) |
Thanks for the reply. I understand that you're busy. I'll ping here again in a few weeks if nothing happens to see what's going on. In the meantime tho, while I wait for more review, I'll be working on other details like getting the |
|
Just a quick update that I am still busy with other efforts, so I did not get a chance to ponder on this PR. |
Thanks for the ping. I've been thinking about these changes myself and wondering what kind of work you and @etcwilde have been making to the build system- is there any info you guys can point me to like a forum post or a branch so I can see what's in progress? I'd like to understand where things are moving, and maybe that could influence these changes as well. If you have a moment... |
|
I believe you can see all the new work they're doing to change how the stdlib is built as it's merged here. |
|
So frankly, I'm not sure how to get the external sysroot to work without stepping on someone's toes (or some configuration that someone is using anyway). If we had separate scripts or at least some separation between building the compiler and the runtimes, it would be easier since all of the products in each phase should build against the same sysroot as they are ending up in the same place. As you have probably noticed, some configurations are using the just-built compiler to build the runtimes, but if the runtimes exist in the build directory, use the just-built runtimes instead. If you've cross-compiled the runtimes, that doesn't work. I'm working on a vision update to share with the community on where I'm hoping to see things end up. So in terms of direction of the new build. It's about breaking apart the projects based on where they need to land and what runtimes they need available to build. The first step is splitting the runtime/stdlib build out of the compiler build so that they are separate CMake projects entirely. I've added facilities to make it easier to build it with build-script, but that is for local development convenience more than for shipping. The intent is that you can point CMake at By using pure CMake, we're also able to take advantage of existing infrastructure and processes. This invocation should cross-compile the libswiftCore as a static archive for Android, for instance. cmake -B build-android -S swift/Runtimes/Core -G 'Ninja' \
-DCMAKE_Swift_COMPILER_WORKS=YES \
-DCMAKE_Swift_COMPILER=/Users/ewilde/Work/Swift-Project/build/Ninja-ReleaseAssert/swift-macosx-arm64/bin/swiftc \
-DCMAKE_Swift_COMPILER_TARGET=aarch64-linux-android31 \
--toolchain /Users/ewilde/AndroidNDK11579264.app/Contents/NDK/build/cmake/android.toolchain.cmake \
-DANDROID_ABI=arm64-v8a \
-DANDROID_PLATFORM=android-31
cmake --build build-androidThe toolchain and ABI flags come from the Android documentation from Google so there's no guessing about what they do.
From there, we can choose to enable or disable pieces. The readme in Hope that helps. |
|
I've been thinking about this, and I wonder if perhaps maybe it doesn't need to be so complicated to do all of it. I have researched using CMake cross compilation toolchain files, and at the very least it works well for the corelibs (Dispatch, Foundation, XCTest, Testing). For example, for cross-compiling to armv7: set(CMAKE_CXX_COMPILER_TARGET armv7-unknown-linux-gnueabihf)
set(CMAKE_C_COMPILER_TARGET armv7-unknown-linux-gnueabihf)
set(CMAKE_SYSROOT /src/sysroot-debian-bookworm)
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR armv7-a)
set(CMAKE_Swift_COMPILER_TARGET armv7-unknown-linux-gnueabihf)This can be passed to cmake invocations with If we modified the build scripts to be able to take this toolchain file and pass it to everything that's if a new I know @finagolfin has been very strongly recommending running tests and trying to get that to work for armv7, so on and so forth, but I've had no success doing any of that. It's quite the struggle! In the end I'm looking for any kind of solution that would give the ability to just cross compile all the projects using the build-scripts in a flexible and reusable way, and in a way that would be accepted by the Swift folks. @etcwilde & @edymtt , would you be open to the idea of perhaps adding a if [[ "${host}" == "openbsd-"* && -n "${OPENBSD_USE_TOOLCHAIN_FILE}" ]]; then
llvm_cmake_options=(
"${llvm_cmake_options[@]}"
-DCMAKE_TOOLCHAIN_FILE="${OPENBSD_USE_TOOLCHAIN_FILE}"
)
fiand if [[ $(is_cross_tools_host ${host}) && "${host}" == "openbsd-"* && -n "${OPENBSD_USE_TOOLCHAIN_FILE}" ]]; then
cmake_options=(
"${cmake_options[@]}"
-DCMAKE_TOOLCHAIN_FILE="${OPENBSD_USE_TOOLCHAIN_FILE}"
)
fiSo I wonder how acceptable it would be to do the same if you just provide that --cmake-toolchain-file instead. And it would be only when compiling for linux-* as well. Let me know if you have any thoughts on this idea. I suggest it as an alternative to what is provided in this PR- maybe a better solution? |
|
Both the linux and mac toolchain builds already use a CMake toolchain file, both for native and cross-compilation of a handful of build products, including the massive LLVM build and the new libTesting. I thought you were already using that linux toolchain file for this cross-compile, hence my referencing it earlier in this very thread, but searching for it now, I don't see it currently being used to build the stdlib or corelibs. If you want to use that file more, that is the direction others like Evan have expressed they'd like to move the build towards, so I suspect any pulls expanding and using that toolchain file more would be gladly accepted. I suggest you examine the Python code that currently generates the CMake toolchain files used more closely and compare it to the generated file, so you know exactly how the generation works, then modify the generation for your platform and use it more broadly, as you saw OpenBSD does. There is no substitute for reading the code and asking questions: I have not received any emails from you for the last month, including after a request to see your failing CMake config output. I was recently told about a |
Yeah I noticed that llvm does generate a cmake toolchain file for cross compilation, not sure if stdlib also does this, but for sure the corelibs do not- they are just built with cmake commands directly from the build-script-impl. So, to get them to use the swift build support products they'd all need to be brought in and built through that, which could be prone to a lot of errors... |
Would they? You've already noted that OpenBSD just passes the toolchain file as an additional flag to this compiler/stdlib repo in build-script-impl: why not try the same with linux armv7 for the stdlib and corelibs in build-script-impl? Considering OpenBSD is already doing it for this repo, I doubt you'll have much of a problem with all four stdlib/corelibs repos, plus you know libTesting already builds using it. |
This fixes #75344 and goes towards #78960.
So, here's what I have for building just the Swift stdlib for armv7. I will note that this also works for cross-compiling the same to aarch64 and x86_64, as long as you have the right sysroot.
./swift/utils/build-script --cross-compile-hosts=linux-armv7 --cross-compile-sysroots=sysroot-debian-bookworm \ --build-llvm=0 --skip-local-build --build-swift-tools=0 --swift-enable-backtracing=0 --build-embedded-stdlib=0 \ --native-swift-tools-path=$SWIFT_NATIVE_PATH --native-clang-tools-path=$SWIFT_NATIVE_PATHWith the new
--cross-compile-sysrootsparam, we now enable the following functionalities: