Re-enable disabled tests #6422

neonichu · 2023-04-13T03:29:55Z

This re-enables a bunch of disabled tests, the hope is that they should now work that we have a newer superior Xcode installed on CI. If that happens to not be the case, we have SWIFTCI_DISABLE_SDK_DEPENDENT_TESTS to disable these tests specifically for the macOS CI if needed. This change introduces the variable where expected to be needed, but doesn't set it.

rdar://96920453

neonichu · 2023-04-13T03:30:05Z

@swift-ci please smoke test

Sources/PackageDescription/PackageDescriptionSerialization.swift

neonichu · 2023-04-13T03:31:23Z

Tests/FunctionalTests/DependencyResolutionTests.swift

@@ -78,6 +78,8 @@ class DependencyResolutionTests: XCTestCase {
    }

    func testMirrors() throws {
+        try XCTSkipIf(true, "test is broken and needs investigation rdar://107970938")


This test has broken while FunctionalTests were disabled and it wasn't clear to me what to do, so it remains disabled for now.

neonichu · 2023-04-13T03:31:44Z

Tests/FunctionalTests/PluginTests.swift

@@ -1022,7 +1024,7 @@ class PluginTests: XCTestCase {
            let (stdout, _) = try executeSwiftBuild(
                fixturePath.appending(component: "MySourceGenPlugin"),
                configuration: .Debug,
-                extraArgs: ["--product", "MyLocalTool", "-Xbuild-tools-swiftc", "-DUSE_CREATING"]
+                extraArgs: ["--product", "MyLocalTool", "-Xbuild-tools-swiftc", "-DUSE_CREATE"]


This test was not working, the source code uses USE_CREATE.

neonichu · 2023-04-13T03:32:05Z

Tests/WorkspaceTests/InitTests.swift

-            #else
-              XCTAssertBuilds(packageRoot)
-            #endif
+            XCTAssertBuilds(packageRoot)


Executable templates no longer contain tests.

neonichu · 2023-04-13T03:32:59Z

Tests/WorkspaceTests/WorkspaceTests.swift

@@ -4948,8 +4948,6 @@ final class WorkspaceTests: XCTestCase {

    // This verifies that the simplest possible loading APIs are available for package clients.
    func testSimpleAPI() throws {


This uses an executable template which doesn't contain tests anymore, so the original flaky failure should not be possible anymore regardless of whether the newer superior Xcode fixed it or not.

neonichu · 2023-04-13T04:51:59Z

Test Suite 'Selected tests' started at 2023-04-13 04:44:02.153Test Suite 'SwiftPMPackageTests.xctest' started at 2023-04-13 04:44:02.154Test Suite 'ResourcesTests' started at 2023-04-13 04:44:02.154Test Case '-[FunctionalTests.ResourcesTests testMovedBinaryResources]' started./Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/swift-tools-support-core/Sources/TSCBasic/Process.swift:1166: error: -[FunctionalTests.ResourcesTests testMovedBinaryResources] : failed: caught error: "signalled(6): /var/folders/7n/r31lzfjx6556jgc6vg55_cg80000gn/T/SwiftyResource.wmt1iS/SwiftyResource output:
    2023-04-13 04:44:11.816 SwiftyResource[59548:2909256] -[Swift.__SharedStringStorage hasSuffix:]: unrecognized selector sent to instance 0x6000014ec030
    2023-04-13 04:44:11.817 SwiftyResource[59548:2909256] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[Swift.__SharedStringStorage hasSuffix:]: unrecognized selector sent to instance 0x6000014ec030'

neonichu · 2023-04-13T04:53:25Z

This actually looks familiar, IIRC this was an issue when using inferior Swift libs but Foundation from the OS?

neonichu · 2023-04-13T04:59:05Z

@swift-ci please smoke test

neonichu · 2023-04-13T04:59:58Z

@swift-ci please smoke test windows

neonichu · 2023-04-13T15:37:04Z

@swift-ci please smoke test

neonichu · 2023-04-13T15:37:33Z

@swift-ci please smoke test windows

neonichu · 2023-04-13T15:37:46Z

@swift-ci please smoke test windows

neonichu · 2023-04-13T15:37:57Z

@swift-ci please smoke test windows

neonichu · 2023-04-13T15:38:11Z

¯\_(ツ)_/¯

neonichu · 2023-04-13T16:16:08Z

[1173/1235] Emitting module PackageCollectionsSigning
/home/build-user/swiftpm/Sources/PackageCollectionsSigning/Certificate/Certificate.swift:187:55: error: cannot find type 'EVP_PKEY' in scope
    private func keyType(of key: UnsafeMutablePointer<EVP_PKEY>) throws -> KeyType {
                                                      ^~~~~~~~

yim-lee · 2023-04-13T16:38:16Z

Changes in swift-crypto 2.4.1 cause the issue. Might need to force exact version 2.4.0 until we figure out how to fix it.

neonichu · 2023-04-13T16:50:14Z

We're still hitting the rpath/dyld issue :/

neonichu · 2023-04-13T16:51:30Z

Oh, I wonder whether we may need the macOS 12 deployment target update first?

neonichu · 2023-04-13T17:41:38Z

I should probably use SWIFTCI_DISABLE_SDK_DEPENDENT_TESTS in this PR. The main goal is to finally restore coverage for these tests with a workaround that we hand't thought of before. Making them work on macOS Swift CI is secondary and can happen on a follow-up PR.

This re-enables a bunch of disabled tests, the hope is that they should now work that we have a newer superior Xcode installed on CI. If that happens to not be the case, we have `SWIFTCI_DISABLE_SDK_DEPENDENT_TESTS` to disable these tests specifically for the macOS CI if needed. This change introduces the variable where expected to be needed, but doesn't set it. rdar://96920453

A macOS that contains the dyld issue worked around by swiftlang/swift#37978 has been deployed by Swift CI, so we can't do this anymore on Darwin.

neonichu · 2023-04-13T19:40:24Z

@swift-ci please smoke test

neonichu · 2023-04-13T19:40:39Z

@swift-ci please smoke test windows

neonichu · 2023-04-13T19:40:52Z

@swift-ci please smoke test windows

neonichu · 2023-04-13T21:12:02Z

Looks like deployment target didn't make a difference, so I'll update this to set SWIFTCI_DISABLE_SDK_DEPENDENT_TESTS on macOS when using bootstrap.

MaxDesiatov

Thanks for applying the cleanups as well!

neonichu · 2023-04-14T05:22:17Z

Actually found one more way how the inferior Swift runtime could be sneaking in on macOS, so trying to run tests once more.

neonichu · 2023-04-14T05:22:29Z

@swift-ci please smoke test

neonichu · 2023-04-14T05:22:45Z

@swift-ci please smoke test windows

neonichu · 2023-04-14T06:13:44Z

@swift-ci please smoke test

neonichu · 2023-04-14T06:14:04Z

@swift-ci please smoke test windows

neonichu · 2023-04-14T07:04:46Z

Looks like DYLD_LIBRARY_PATH was indeed the final piece to avoid the dynamic linker issue. There's still

[1543/1543] Testing WorkspaceTests.WorkspaceTests/testStateModified
Test Suite 'Selected tests' started at 2023-04-14 06:45:46.798
Test Suite 'SwiftPMPackageTests.xctest' started at 2023-04-14 06:45:46.799Test Suite 'WorkspaceTests' started at 2023-04-14 06:45:46.799Test Case '-[WorkspaceTests.WorkspaceTests testSimpleAPI]' started./Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/swiftpm/Tests/WorkspaceTests/WorkspaceTests.swift:4967: error: -[WorkspaceTests.WorkspaceTests testSimpleAPI] : failed: caught error: "fatalError"Test Case '-[WorkspaceTests.WorkspaceTests testSimpleAPI]' failed (2.746 seconds).Test Suite 'WorkspaceTests' failed at 2023-04-14 06:45:49.545.
	 Executed 1 test, with 1 failure (1 unexpected) in 2.746 (2.746) secondsTest Suite 'SwiftPMPackageTests.xctest' failed at 2023-04-14 06:45:49.545.
	 Executed 1 test, with 1 failure (1 unexpected) in 2.746 (2.746) seconds
Test Suite 'Selected tests' failed at 2023-04-14 06:45:49.545.
	 Executed 1 test, with 1 failure (1 unexpected) in 2.746 (2.747) seconds[debug]: evaluating manifest for 'mypkg' v. unknown
[error]: Invalid manifest (compiled with: ["/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/build/buildbot_incremental/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/7n/r31lzfjx6556jgc6vg55_cg80000gn/T/TemporaryDirectory.1QDKFO/vfs.yaml", "-L", "/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/build/buildbot_incremental/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/build/buildbot_incremental/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "x86_64-apple-macosx10.15", "-sdk", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk", "-F", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/build/buildbot_incremental/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk", "-package-description-version", "5.9.0", "/var/folders/7n/r31lzfjx6556jgc6vg55_cg80000gn/T/spm-tests-testSimpleAPI.t47rAD/MyPkg/Package.swift", "-o", "/var/folders/7n/r31lzfjx6556jgc6vg55_cg80000gn/T/TemporaryDirectory.J3kZ4K/mypkg-manifest"])
/var/folders/7n/r31lzfjx6556jgc6vg55_cg80000gn/T/spm-tests-testSimpleAPI.t47rAD/MyPkg/Package.swift:4:8: error: no such module 'PackageDescription'
import PackageDescription
       ^

and I wonder if this might explain why I had to make a change to PackageDescriptionSerialization to make this test work locally. It looks like we may not really be using the correct PD library when running this test.

neonichu · 2023-04-14T07:06:06Z

I also introduced an issue in bootstrap specific to non-Darwin:

TypeError: can only concatenate str (not "list") to str

neonichu · 2023-04-14T21:14:28Z

@swift-ci please smoke test

neonichu · 2023-04-14T21:14:49Z

@swift-ci please smoke test windows

neonichu · 2023-04-14T23:26:57Z

Tests are passing 🎉

tomerd · 2023-04-15T00:40:42Z

\o/

neonichu · 2023-04-15T00:51:34Z

Utilities/bootstrap

@@ -799,26 +793,6 @@ def get_swiftpm_flags(args):
            "-Xlinker", "@executable_path/../lib/swift/pm/llbuild",
        ])

-    # Add a relative rpath to find Swift libraries in toolchains.
-    # On non-Darwin platforms, a relative rpath is necessary because Swift
-    # libraries are not part of the OS.


I think this comment should actually stay since we're only changing this for Darwin.

Ah I had automerge enabled, going to restore this comment in a follow up.

Actually, we moved the non-Darwin part out of bootstrap and put it behind SWIFTCI_INSTALL_RPATH_OS, so this it is correct for the comment to be gone.

yim-lee · 2023-04-15T05:28:27Z

@swift-ci please test Windows platform

neonichu self-assigned this Apr 13, 2023

neonichu requested review from abertelrud and tomerd as code owners April 13, 2023 03:29

neonichu commented Apr 13, 2023

View reviewed changes

Sources/PackageDescription/PackageDescriptionSerialization.swift Outdated Show resolved Hide resolved

neonichu commented Apr 13, 2023

View reviewed changes

tomerd approved these changes Apr 13, 2023

View reviewed changes

neonichu added 3 commits April 13, 2023 12:39

Removed back deployment rpaths

70c7d85

Removed more Darwin rpaths into the toolchain

7ead791

A macOS that contains the dyld issue worked around by swiftlang/swift#37978 has been deployed by Swift CI, so we can't do this anymore on Darwin.

neonichu force-pushed the 96920453 branch from 18f2d29 to 7ead791 Compare April 13, 2023 19:40

MaxDesiatov approved these changes Apr 13, 2023

View reviewed changes

Do not include Swift runtime libs in DYLD_LIBRARY_PATH on Darwin

5f6ca3c

neonichu force-pushed the 96920453 branch from f3d7ca0 to 5f6ca3c Compare April 14, 2023 06:13

tomerd mentioned this pull request Apr 14, 2023

fix functional test #6437

Merged

neonichu added 2 commits April 14, 2023 14:12

Fix bootstrap

2ba498e

Undo incorrect change related to using superior PD libs

9a636f6

neonichu enabled auto-merge (squash) April 15, 2023 00:06

neonichu commented Apr 15, 2023

View reviewed changes

neonichu merged commit 16fb1d2 into main Apr 15, 2023

neonichu deleted the 96920453 branch April 17, 2023 16:04

		@@ -4948,8 +4948,6 @@ final class WorkspaceTests: XCTestCase {

		// This verifies that the simplest possible loading APIs are available for package clients.
		func testSimpleAPI() throws {

Re-enable disabled tests #6422

Re-enable disabled tests #6422

Uh oh!

Conversation

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

Uh oh!

neonichu Apr 13, 2023

Choose a reason for hiding this comment

Uh oh!

neonichu Apr 13, 2023

Choose a reason for hiding this comment

Uh oh!

neonichu Apr 13, 2023

Choose a reason for hiding this comment

Uh oh!

neonichu Apr 13, 2023

Choose a reason for hiding this comment

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

yim-lee commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

neonichu commented Apr 13, 2023

Uh oh!

MaxDesiatov left a comment

Choose a reason for hiding this comment

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 14, 2023

Uh oh!

neonichu commented Apr 13, 2023 •

edited

Loading

neonichu commented Apr 13, 2023 •

edited

Loading