Build one test product for both Swift Testing and XCTest (with Swift Testing enabled by default.) #7766
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
grynspan
added
bug
build system
grynspan
force-pushed
the
jgrynspan/one-test-product-two-libraries
branch
from
de3ea52 to
6743d46
Compare
grynspan
commented
Jul 10, 2024
grynspan
commented
Jul 10, 2024
| @main | ||
| @available(*, deprecated, message: "Not actually deprecated. Marked as deprecated to allow inclusion of deprecated tests (which test deprecated functionality) without warnings") | ||
| struct Runner { | ||
| private static func testingLibrary() -> String { |
There was a problem hiding this comment.
There's probably a better way to disambiguate here. I am open to suggestions.
grynspan
commented
Jul 10, 2024
| @@ -175,23 +175,20 @@ public struct SwiftBuildCommand: AsyncSwiftCommand { | |||
| } | |||
|
|
|||
| if case .allIncludingTests = subset { | |||
| func updateTestingParameters(of buildParameters: inout BuildParameters, library: BuildParameters.Testing.Library) { | |||
| func updateTestingParameters(of buildParameters: inout BuildParameters) { | |||
There was a problem hiding this comment.
This helper function is probably redundant now.
grynspan
commented
Jul 10, 2024
grynspan
force-pushed
the
jgrynspan/one-test-product-two-libraries
branch
3 times, most recently
from
9ce3698 to
c89a951
Compare
|
@swift-ci please test |
grynspan
commented
Jul 10, 2024
2 tasks
grynspan
commented
Jul 10, 2024
| @@ -113,18 +109,14 @@ extension BuildParameters { | |||
| } | |||
| } | |||
|
|
|||
| /// Which testing library to use for this build. | |||
| public var library: Library | |||
There was a problem hiding this comment.
I should probably move the Library type out of BuildParameters (since it ain't one anymore) but I'm going to save that for a separate PR due to its potential impact to Xcode 16 betas.
|
@swift-ci please test |
1 similar comment
|
@swift-ci please test |
|
@swift-ci please test Windows |
|
Mac CI failures are due to the |
grynspan
force-pushed
the
jgrynspan/one-test-product-two-libraries
branch
from
244f1b4 to
7dfb5e8
Compare
grynspan
added a commit
that referenced
this pull request
Jul 14, 2024
This PR detects when a Swift Testing run returns `EXIT_NO_TESTS_FOUND` and treats it as a successful test run. This change handles the change from swiftlang/swift-testing#536. Separately, the refactor in #7766 stops Swift Package Manager from reporting `"No matching test cases were run"` when XCTest has no tests matching passed `--filter` arguments. A third PR after these two have been merged will restore that functionality by collating results from both XCTest and Swift Testing. This change partially resolves rdar://131704587.
|
Okay then. |
grynspan
added a commit
that referenced
this pull request
Jul 16, 2024
This PR detects when a Swift Testing run returns `EXIT_NO_TESTS_FOUND` and treats it as a successful test run. This change handles the change from swiftlang/swift-testing#536. Separately, the refactor in #7766 stops Swift Package Manager from reporting `"No matching test cases were run"` when XCTest has no tests matching passed `--filter` arguments. A third PR after these two have been merged will restore that functionality by collating results from both XCTest and Swift Testing. > [!NOTE] > Test coverage can be added after Swift Testing is in the Swift toolchain. This change partially resolves rdar://131704587.
grynspan
added a commit
that referenced
this pull request
Jul 16, 2024
…Testing enabled by default.) (#7766) This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time. On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project. Resolves rdar://120864035. --------- Co-authored-by: Pavel Yaskevich <[email protected]>
grynspan
added a commit
that referenced
this pull request
Jul 17, 2024
This PR turns Swift Testing back off by default when running `swift package init` because it's not available in the toolchain yet and the package dependency may come as a surprise (also, it breaks CI, but who's counting?) Follow-up to #7766.
bnbarham
pushed a commit
that referenced
this pull request
Jul 17, 2024
This PR turns Swift Testing back off by default when running `swift package init` because it's not available in the toolchain yet and the package dependency may come as a surprise (also, it breaks CI, but who's counting?) Follow-up to #7766.
grynspan
added a commit
that referenced
this pull request
Jul 17, 2024
This PR turns Swift Testing back off by default when running `swift package init` because it's not available in the toolchain yet and the package dependency may come as a surprise (also, it breaks CI, but who's counting?) Follow-up to #7766.
This was referenced Jul 17, 2024
grynspan
added a commit
that referenced
this pull request
Jul 17, 2024
…Testing enabled by default.) (#7766) This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time. On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project. This PR includes changes from the previous PRs numbered #7766, #7783, and #7787. Resolves rdar://120864035. --------- Co-authored-by: Pavel Yaskevich <[email protected]>
grynspan
added a commit
that referenced
this pull request
Jul 17, 2024
…Testing enabled by default.) This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time. On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project. This PR includes changes from the previous PRs numbered #7766, #7783, and #7787. Resolves rdar://120864035. --------- Co-authored-by: Pavel Yaskevich <[email protected]>
grynspan
added a commit
that referenced
this pull request
Jul 17, 2024
…Testing enabled by default.) This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time. On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project. This PR includes changes from the previous PRs numbered #7766, #7783, and #7787. Resolves rdar://120864035. --------- Co-authored-by: Pavel Yaskevich <[email protected]>
grynspan
added a commit
that referenced
this pull request
Jul 18, 2024
…Testing enabled by default.) (#7789) This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time. On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project. This PR includes changes from the previous PRs numbered #7766, #7783, #7784, and #7787. Resolves rdar://120864035. Co-authored-by: Pavel Yaskevich <[email protected]>
grynspan
added a commit
that referenced
this pull request
Jul 18, 2024
This PR detects when a Swift Testing run returns `EXIT_NO_TESTS_FOUND` and treats it as a successful test run. This change handles the change from swiftlang/swift-testing#536. Separately, the refactor in #7766 stops Swift Package Manager from reporting `"No matching test cases were run"` when XCTest has no tests matching passed `--filter` arguments. A third PR after these two have been merged will restore that functionality by collating results from both XCTest and Swift Testing. > [!NOTE] > Test coverage can be added after Swift Testing is in the Swift toolchain. This change partially resolves rdar://131704587.
grynspan
added a commit
that referenced
this pull request
Jul 18, 2024
…Testing enabled by default.) (#7789) This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time. On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project. This PR includes changes from the previous PRs numbered #7766, #7783, Resolves rdar://120864035. Co-authored-by: Pavel Yaskevich <[email protected]>
plemarquand
added a commit
to plemarquand/vscode-swift
that referenced
this pull request
Jul 25, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
plemarquand
added a commit
to plemarquand/vscode-swift
that referenced
this pull request
Jul 25, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
plemarquand
added a commit
to plemarquand/vscode-swift
that referenced
this pull request
Jul 25, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
plemarquand
added a commit
to plemarquand/vscode-swift
that referenced
this pull request
Jul 25, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
plemarquand
added a commit
to plemarquand/vscode-swift
that referenced
this pull request
Jul 25, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
plemarquand
added a commit
to plemarquand/vscode-swift
that referenced
this pull request
Jul 26, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
plemarquand
added a commit
to swiftlang/vscode-swift
that referenced
this pull request
Aug 7, 2024
With the merge of swiftlang/swift-package-manager#7766 a `swift build --build-tests` produces a single binary that contains both XCTests and swift-testing tests. Update the build configs when debugging tests to use this new unified test binary if it exists. To determine if we're running a unified binary on macOS we can check for the presence of the swiftpm-testing-helper, which is used to run the swift-testing tests in the unified binary. This utility isn't required on Windows and Linux as the test binary can be invoked directly. To determine if we're running a unified binary we check if the latest build produced a .swift-testing binary in the build folder. If it doesn't exist, or if it does exist but its `mtime` is older than the last build start time, we know its a unified binary. The `mtime` check is required in case the user updates their toolchain to a version that produces a unified binary but still has a .swift-testing binary in their .build folder left over from an old build. In the future we can remove these checks once we're confident users on 6.0 and main toolchains have moved on to versions that generate a unified test binary. Tentatively I propose this to be around when Swift 6.0 is finalized and released. This change only affects debugging tests. All other testing code paths use `swift test` which lets SwiftPM automatically handle the switch to the new unified binary.
This was referenced Oct 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors the previously-experimental Swift Testing support logic so that only a single build product is produced when using both XCTest and Swift Testing, and detection of Swift Testing usage is no longer needed at compile time.
On macOS, Xcode 16 is responsible for hosting Swift Testing content, so additional changes may be needed in Xcode to support this refactoring. Such changes are beyond the purview of the Swift open source project.
Resolves rdar://120864035.