Add new options that control whether host toolchain is included #165

xtremekforever · 2024-12-20T23:53:13Z

I'm fully prepared for this to be torn apart, but this is hopefully a good "first PR" for this configuration. @euanh @MaxDesiatov

This adds a new --no-include-host-toolchain flag to the make-linux-sdk sub-command, making it possible to generate an SDK for x86_64 or aarch64 with only the target sysroot and Swift libraries, like this:

# x86_64
swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.3-RELEASE --target x86_64-unknown-linux-gnu --no-include-host-toolchain
# aarch64
swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.3-RELEASE --target aarch64-unknown-linux-gnu --no-include-host-toolchain

I tested the generated SDKs on both Ubuntu 22.04 and macOS 15 and they work just fine.

Ubuntu x86_64 -> aarch64:

> swift build --swift-sdk 6.0.3-RELEASE_ubuntu_jammy_aarch64
Building for debugging...
warning: Could not read SDKSettings.json for SDK at: ~/.swiftpm/swift-sdks/6.0.3-RELEASE_ubuntu_jammy_aarch64.artifactbundle/6.0.3-RELEASE_ubuntu_jammy_aarch64/aarch64-unknown-linux-gnu/ubuntu-jammy.sdk
[485/485] Linking ConsumePublishServices
Build complete! (25.37s)

macOS arm64 -> x86_64:

>  swift build --swift-sdk 6.0.3-RELEASE_ubuntu_jammy_x86_64
Building for debugging...
warning: Could not read SDKSettings.json for SDK at: /Users/xtremek/Library/org.swift.swiftpm/swift-sdks/6.0.3-RELEASE_ubuntu_jammy_x86_64.artifactbundle/6.0.3-RELEASE_ubuntu_jammy_x86_64/x86_64-unknown-linux-gnu/ubuntu-jammy.sdk
<unknown>:0: warning: libc not found for 'x86_64-unknown-linux-gnu'; C stdlib may be unavailable
[1171/1171] Linking ConsumePublishServices
Build complete! (39.43s)

Also, is there a need/desire to add some unit testing for this flag?

- So we end up with no swift.xctoolchain so that the toolchain installed on the host is used instead.

- Also don't download host LLVM if excluding host toolchain

- This way since 5.9 and 5.10 require the supportedTriples in info.json, they will still work, and all supported host versions of Swift can compile for the SDK.

xtremekforever · 2024-12-29T20:28:39Z

Update: since both 5.9 and 5.10 require the supportedTriples to be included in info.json, I've opted to add all the supported host triples when building the SDK with --no-include-host-toolchain:

          "supportedTriples" : [
            "x86_64-unknown-linux-gnu",
            "aarch64-unknown-linux-gnu",
            "x86_64-apple-macos",
            "arm64-apple-macos"
          ],

Then, either Linux x86_64/aarch64 or macOS x86_64/arm64 hosts can cross compile with the SDK.

Since the SDK doesn't contain the Swift toolchain, though, the open source Swift toolchain is required to be used on macOS for it to compile. I think this is also the case with the Linux Static SDK.

@euanh @MaxDesiatov please have a look here and see what you think!

Sources/GeneratorCLI/GeneratorCLI.swift

MaxDesiatov

Phrasing nit: we should not call these "SDK", but use "Swift SDK" wording instead, since these terms are separate from each other. In the Swift context, plain "SDK" is an Xcode-specific term, where its Darwin SDKs have a distinct file system layout that Xcode build system can consume. "Swift SDK" is a SwiftPM feature, unrelated to plain "SDK"s for Darwin. These are different features in different products and are not interchangeable with each other.

It's unfortunate though that swift.org website doesn't make this distinction, but that would have to be resolved with swift.org website maintainers.

- Pass --no-host-include-toolchain to build the Swift SDK without the host toolchain.

Sources/SwiftSDKGenerator/SwiftSDKRecipes/LinuxRecipe.swift

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator.swift

…nd 5.10 - Both of these versions require supportedTriples to be set and populated, but 6.0 and later can simply exclude the field for supporting all hosts.

MaxDesiatov · 2025-01-03T13:15:43Z

Also, is there a need/desire to add some unit testing for this flag?

Yes please, that would be much appreciated.

- Instead set rootPath to nil in Toolset if toolchain is preinstalled.

kateinoigakukun

Thanks!

…riples.

xtremekforever · 2025-01-05T19:16:46Z

Also, is there a need/desire to add some unit testing for this flag?

Yes please, that would be much appreciated.

@MaxDesiatov I added tests that check the following pieces of functionality in LinuxRecipe that are affected by the .preinstalled host Swift enum:

Toolset options
Downloadable artifacts
Host triples (supportedTriples in info.json)

I did not add tests for all of the functionality of LinuxRecipe, but I think that can be added later if desired.

kateinoigakukun · 2025-01-05T20:23:12Z

@swift-ci test

MaxDesiatov · 2025-01-06T13:19:26Z

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Metadata.swift

@@ -27,7 +27,6 @@ extension SwiftSDKGenerator {
    let toolsetJSONPath = pathsConfiguration.swiftSDKRootPath.appending("toolset.json")

    var relativeToolchainBinDir = pathsConfiguration.toolchainBinDirPath
-


nit: unnecessary newline that only adds to the diff as an unrelated change

MaxDesiatov

LGTM modulo a super minor formatting nit

…hain unpack

MaxDesiatov · 2025-01-06T13:52:03Z

@swift-ci test

xtremekforever · 2025-01-06T20:42:44Z

All good to merge since everything's in there?

MaxDesiatov · 2025-01-06T20:51:18Z

I wonder if @euanh would have any feedback, since he contributed to the Linux generator a lot recently and also enabled e2e tests for it?

euanh · 2025-01-07T10:36:27Z

I've run the EndToEnd tests. They didn't find any regressions (they take a while):

SWIFT_SDK_GENERATOR_RUN_SLOW_TESTS=1 swift test --disable-swift-testing --filter EndToEndTests
...
Test Suite 'Swift60_UbuntuEndToEndTests' passed at 2025-01-07 10:29:19.900.
         Executed 4 tests, with 0 failures (0 unexpected) in 447.902 (447.902) seconds
Test Suite 'swift-sdk-generatorPackageTests.xctest' passed at 2025-01-07 10:29:19.900.
         Executed 18 tests, with 0 failures (0 unexpected) in 2034.483 (2034.484) seconds
Test Suite 'Selected tests' passed at 2025-01-07 10:29:19.900.
         Executed 18 tests, with 0 failures (0 unexpected) in 2034.483 (2034.485) seconds

MaxDesiatov · 2025-01-07T14:00:56Z

Sources/GeneratorCLI/GeneratorCLI.swift

+      but requires exactly the same version of the swift.org toolchain to be installed for it to work.
+      """
+    )
+    var includeHostToolchain: Bool = true


Sorry about a request coming this late, just noticed that --no-include-host-toolchain reads quite weird to me? Although I'm not a native English speaker, so take this concern with a grain of salt. If you agree that --no-host-toolchain reads better, let's use that shorter wording and I'm ready to merge immediately after that.

Yes, --no-host-toolchain sounds better.

Done:

--host-toolchain/--no-host-toolchain Whether or not to include the host toolchain in the Swift SDK. If the host toolchain is not included, this makes the Swift SDK compatible with any host, but requires exactly the same version of the swift.org toolchain to be installed for it to work. (default: --host-toolchain)

MaxDesiatov · 2025-01-07T14:21:25Z

@swift-ci test

xtremekforever added 4 commits December 20, 2024 15:48

Initial changes to skip adding the host toolchain to the SDK

9cdb0b2

- So we end up with no swift.xctoolchain so that the toolchain installed on the host is used instead.

Don't set linker if host toolchain is not included

81392a2

Fix including of supportedTriples for Swift versions < 6.0

6d7e5bb

- Also don't download host LLVM if excluding host toolchain

Add all host triples to info.json with --no-host-toolchain

7a7853f

- This way since 5.9 and 5.10 require the supportedTriples in info.json, they will still work, and all supported host versions of Swift can compile for the SDK.

xtremekforever marked this pull request as ready for review December 29, 2024 20:28

xtremekforever requested a review from MaxDesiatov as a code owner December 29, 2024 20:28

MaxDesiatov requested review from euanh and kateinoigakukun January 2, 2025 11:01

MaxDesiatov added the enhancement New feature or request label Jan 2, 2025

MaxDesiatov changed the title ~~Add a new --no-host-toolchain flag that generates an SDK without the host Swift toolchain~~ Add new options that control whether host toolchain is included Jan 2, 2025

MaxDesiatov reviewed Jan 2, 2025

View reviewed changes

Sources/GeneratorCLI/GeneratorCLI.swift Outdated Show resolved Hide resolved

MaxDesiatov reviewed Jan 2, 2025

View reviewed changes

Update option to includeHostToolchain along with help description

76151b8

- Pass --no-host-include-toolchain to build the Swift SDK without the host toolchain.

xtremekforever requested a review from MaxDesiatov January 2, 2025 15:20

MaxDesiatov reviewed Jan 2, 2025

View reviewed changes

Sources/SwiftSDKGenerator/SwiftSDKRecipes/LinuxRecipe.swift Outdated Show resolved Hide resolved

Rename HostSwiftSource.skip -> .preinstalled

a8d5fd7

xtremekforever requested a review from MaxDesiatov January 2, 2025 17:36

kateinoigakukun reviewed Jan 3, 2025

View reviewed changes

Sources/SwiftSDKGenerator/SwiftSDKRecipes/LinuxRecipe.swift Outdated Show resolved Hide resolved

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator.swift Outdated Show resolved Hide resolved

Only set supportedTriples for preinstalled host toolchain under 5.9 a…

eb2db7b

…nd 5.10 - Both of these versions require supportedTriples to be set and populated, but 6.0 and later can simply exclude the field for supporting all hosts.

Remove includeHostToolchain from SwiftSDKGenerator

e8145d3

- Instead set rootPath to nil in Toolset if toolchain is preinstalled.

kateinoigakukun approved these changes Jan 4, 2025

View reviewed changes

Add LinuxRecipeTests with tests for toolset options, downloads, and t…

93a5396

…riples.

MaxDesiatov reviewed Jan 6, 2025

View reviewed changes

MaxDesiatov approved these changes Jan 6, 2025

View reviewed changes

Fix whitespace change, add creation of toolchainDirPath on host toolc…

2c5e853

…hain unpack

xtremekforever requested a review from MaxDesiatov January 6, 2025 13:50

xtremekforever requested a review from kateinoigakukun January 6, 2025 13:50

euanh approved these changes Jan 7, 2025

View reviewed changes

MaxDesiatov reviewed Jan 7, 2025

View reviewed changes

Use shorter wording for --host-toolchain/--no-host-toolchain option

12db5c0

MaxDesiatov enabled auto-merge (squash) January 7, 2025 14:22

MaxDesiatov merged commit f63f812 into swiftlang:main Jan 7, 2025
2 of 3 checks passed

xtremekforever deleted the #164-no-host-toolchain-option branch January 7, 2025 14:33

		@@ -27,7 +27,6 @@ extension SwiftSDKGenerator {
		let toolsetJSONPath = pathsConfiguration.swiftSDKRootPath.appending("toolset.json")

		var relativeToolchainBinDir = pathsConfiguration.toolchainBinDirPath

Add new options that control whether host toolchain is included #165

Add new options that control whether host toolchain is included #165

Conversation

xtremekforever commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xtremekforever commented Dec 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MaxDesiatov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaxDesiatov commented Jan 3, 2025

Uh oh!

kateinoigakukun left a comment

Choose a reason for hiding this comment

Uh oh!

xtremekforever commented Jan 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kateinoigakukun commented Jan 5, 2025

Uh oh!

MaxDesiatov Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

xtremekforever Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

MaxDesiatov left a comment

Choose a reason for hiding this comment

Uh oh!

MaxDesiatov commented Jan 6, 2025

Uh oh!

xtremekforever commented Jan 6, 2025

Uh oh!

MaxDesiatov commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

euanh commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaxDesiatov Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

euanh Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

xtremekforever Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

MaxDesiatov commented Jan 7, 2025

Uh oh!

Uh oh!

Uh oh!

xtremekforever commented Dec 20, 2024 •

edited

Loading

xtremekforever commented Dec 29, 2024 •

edited

Loading

MaxDesiatov left a comment •

edited

Loading

xtremekforever commented Jan 5, 2025 •

edited

Loading

MaxDesiatov commented Jan 6, 2025 •

edited

Loading

euanh commented Jan 7, 2025 •

edited

Loading