Add "whole SDK" validation test #69928

beccadax · 2023-11-16T21:02:30Z

This test compares two API digests of large chunks of the installed SDK—one from the toolchain’s API digester, one from the just-built API digester—and flags any changes in behavior that would affect source compatibility.

I'm currently trying to evaluate its feasibility; I'm hoping we can run it on every pull request.

Thanks to @xymus for suggesting we generate a baseline at test time instead of including one in the test inputs, and @nkcsgexi for help with using the digester.

beccadax · 2023-11-16T21:02:50Z

@swift-ci please test

beccadax · 2023-11-17T00:31:17Z

<unknown>:0: remark: did not find a prebuilt standard library for target 'x86_64-apple-macos' compatible with this Swift compiler; building it may take a few minutes, but it should only happen once for this combination of compiler and target
Failed to load module: UIKit

real	1m29.328s
user	1m18.505s
sys	0m5.850s
Failed to load module: UIKit

real	1m30.140s
user	1m16.718s
sys	0m3.529s

/* Generic Signature Changes */

/* RawRepresentable Changes */

/* Removed Decls */
Constructor glob_t.__Unnamed_union___Anonymous_field5.init(gl_errblk:) has been removed
Var FLT_EVAL_METHOD has been removed

/* Moved Decls */

/* Renamed Decls */

/* Type Changes */
Var NSApp has declared type change from AppKit.NSApplication! to AppKit.NSApplication?
Var _CurrentRuneLocale has declared type change from Swift.UnsafeMutablePointer<Darwin._RuneLocale>! to Swift.UnsafeMutablePointer<Darwin._RuneLocale>?
Var __stderrp has declared type change from Swift.UnsafeMutablePointer<Darwin.FILE>! to Swift.UnsafeMutablePointer<Darwin.FILE>?
Var __stdinp has declared type change from Swift.UnsafeMutablePointer<Darwin.FILE>! to Swift.UnsafeMutablePointer<Darwin.FILE>?
Var __stdoutp has declared type change from Swift.UnsafeMutablePointer<Darwin.FILE>! to Swift.UnsafeMutablePointer<Darwin.FILE>?
Var _c_locale has declared type change from Darwin.locale_t! to Darwin.locale_t?
Var optarg has declared type change from Swift.UnsafeMutablePointer<Swift.CChar>! to Swift.UnsafeMutablePointer<Swift.CChar>?
Var port_obj_table has declared type change from Swift.UnsafeMutablePointer<Darwin.port_obj_tentry>! to Swift.UnsafeMutablePointer<Darwin.port_obj_tentry>?
Var suboptarg has declared type change from Swift.UnsafeMutablePointer<Swift.CChar>! to Swift.UnsafeMutablePointer<Swift.CChar>?
Var vprintf_stderr_func has declared type change from ((Swift.UnsafePointer<Swift.CChar>?, Swift.CVaListPointer?) -> Swift.Int32)! to ((Swift.UnsafePointer<Swift.CChar>?, Swift.CVaListPointer?) -> Swift.Int32)?

/* Decl Attribute changes */

/* Fixed-layout Type Changes */

/* Protocol Conformance Change */
Protocol URLSessionDelegate has removed inherited protocol Sendable
Struct NSAccessibility has removed conformance to Sendable

/* Protocol Requirement Change */

/* Class Inheritance Change */

/* Others */

real	0m11.956s
user	0m9.850s
sys	0m0.845s

Testing with Swift CI's installed Swift 5.7 found additional source-breaking changes compared to my local development environment, which uses Swift 5.9.

Slowest Tests:
--------------------------------------------------------------------------
280.82s: Swift(macosx-x86_64) :: Prototypes/CollectionTransformers.swift
238.14s: Swift-validation(macosx-x86_64) :: Python/swift_build_support.swift
191.46s: Swift-validation(macosx-x86_64) :: WholeSDK/api-digester.swift
187.06s: Swift-validation(macosx-x86_64) :: BuildSystem/generate_xcode.test
167.81s: Swift-validation(macosx-x86_64) :: Sema/type_checker_perf/slow/rdar19612086.swift

This is a fairly expensive test even with a limited number of modules, but might still be worthwhile.

beccadax · 2023-11-17T02:33:06Z

@swift-ci please smoke test

beccadax · 2023-11-17T02:36:14Z

Retesting with a lot more modules (i.e. most of the contents of /System/Library/Frameworks) to see how long it takes.

beccadax · 2023-11-17T09:27:58Z

Crashed. Wondering if it’s consistent.

@swift-ci please smoke test macOS platform

beccadax · 2023-11-17T22:41:23Z

@swift-ci please smoke test macOS platform

beccadax · 2023-11-17T23:40:08Z

The crash during attribute processing that I'm looking at occurs only in CI. I now know that it affects StoreKit.Transaction.environmentStringRepresentation:

  @available(iOS, introduced: 15.0, deprecated: 16.0, message: "Use the environment property instead")
  @available(macOS, introduced: 12.0, deprecated: 13.0, message: "Use the environment property instead")
  @available(tvOS, introduced: 15.0, deprecated: 16.0, message: "Use the environment property instead")
  @available(watchOS, introduced: 8.0, deprecated: 9.0, message: "Use the environment property instead")
  @available(macCatalyst, introduced: 15.0, deprecated: 16.0, message: "Use the environment property instead")
  @_backDeploy(before: iOS 16.0)
  @_backDeploy(before: macOS 13.0)
  @_backDeploy(before: tvOS 16.0)
  @_backDeploy(before: watchOS 9.0)
  @_backDeploy(before: macCatalyst 16.0)
  public var environmentStringRepresentation: Swift.String {
    get {
        if #available(iOS 16.0, macOS 13.0, tvOS 16.0, watchOS 9.0, *) {
            return environment.rawValue
        }
        else {
            return backing.value(
                atKeyPath: "environment",
                sentinel: "",
                transform: String.init
            )
        }
    }
  }

Certainly plenty of juicy attributes there for it to choke on.

I'll try disabling StoreKit for now.

beccadax · 2023-11-17T23:43:29Z

@swift-ci please smoke test macOS platform

beccadax · 2023-11-18T01:12:49Z

CI found a large number of additional "IUO became ordinary Optional" issues that I don't see locally, even when I build an Intel compiler and run it through Rosetta. I may need to try this test on an actual Intel Mac when I return from the break.

Slowest Tests:
--------------------------------------------------------------------------
532.00s: Swift-validation(macosx-x86_64) :: WholeSDK/api-digester.test
243.73s: Swift-validation(macosx-x86_64) :: Python/swift_build_support.swift
205.11s: Swift-validation(macosx-x86_64) :: BuildSystem/generate_xcode.test
176.29s: Swift-validation(macosx-x86_64) :: Sema/type_checker_perf/slow/rdar19612086.swift
155.90s: Swift(macosx-x86_64) :: Prototypes/CollectionTransformers.swift

Nearly nine minutes. The specific timings for the three jobs (baseline, just-built, comparison) are:

real	3m28.943s
user	2m57.470s
sys	0m18.954s

real	3m31.462s
user	3m10.350s
sys	0m12.988s

real	1m51.560s
user	1m50.118s
sys	0m1.432s

Some possible strategies here:

Combine the second and third steps, comparing the baseline digest directly against the SDK instead of against a digest of it. This might (or might not!) eliminate some overhead.
Generate separate digests for each module instead of one combined digest for all of them. This would help if the comparison operation is more than O(n) on the size of the digests it's comparing.
Eliminate the baseline step, at least in CI. The baseline should only ever change when the tools change; if we could regenerate it at tool update time, then that could shave more than a third of the time off this.

beccadax · 2023-11-28T03:46:33Z

@swift-ci smoke test

beccadax · 2023-11-29T00:39:29Z

The crash relates to a use of @backDeployed in the SDK. The old digester encoded this as BackDeploy; subsequent changes made it BackDeployed instead. This caused us to hit llvm_unreachable() which, on Intel machines, caused a crash.

I've fixed the unknown attribute crash in the general case, added an alias for the specific case of BackDeploy, fixed a bug that made swift-api-digester -diagnose-sdk emit diagnostics about bad content in the JSON files into the ether, and re-enabled an API digester test that has been disabled for quite a while. I've also re-enabled StoreKit, and I expect the WholeSDK test to avoid crashing on this run.

@swift-ci please smoke test

beccadax · 2023-11-29T18:12:09Z

lol, don't write too much in a swift-ci comment.

beccadax · 2023-11-29T18:12:12Z

@swift-ci please smoke test

This test compares two API digests of large chunks of the installed SDK—one from the toolchain’s API digester, one from the just-built API digester—and flags any changes in behavior that would affect source compatibility.

Some of these may be actual regressions!

Trying to gather information on a failure occurring in CI.

The diagnostic consumers for swift-api-digester -diagnose-sdk were being set up after JSON files were read in, so errors in those files were silently dropped. In at least one case this led to a crash with no hint about what was happening. Set up the diagnostic consumer before doing any serious work to avoid this problem. Fixing this required me to make `newCompilerInstance()` copy diagnostic consumers to the new instance’s diagnostic engine, since compiler instances were now being created after the consumers were added instead of before. I also had to make FilteringDiagnosticConsumer filter out clang warnings, remarks, and notes since those were now being emitted early enough to register in the digester.

The API digester represents unknown attributes as DAK_Count, but passing this value to the `is{Adding,Removing}Breaking{API,ABI}()` functions is not permitted. On some platforms this would cause a crash; on others it hit unreachable code which happened to end up working. Skip these unknown attributes during comparisons to avoid this problem.

Swift 5.7’s API digester supported the unofficial @_backDeploy attribute, writing it to digests under the name “BackDeploy”, but by the time it the proposal was accepted, the spelling in source code was @backDeployed and the spelling in digests became “BackDeployed”. Add a special case to the new digester to accept the old spelling. This should fix an issue seen in Swift CI where the WholeSDK API digester test couldn’t process the StoreKit framework.

Since the API digester is designed to tolerate them anyway.

The IUO-ness of imported declarations is not actually computed by IsImplicitlyUnwrappedOptionalRequest. Instead, ClangImporter manually sets the bit to `true` after the declaration’s type is imported and expects IsImplicitlyUnwrappedOptionalRequest to always set it to `false` for all other imported declarations. Normally, declaration types are imported greedily as soon as the declaration is created. However, a ClangImporter refactoring in #61026 deferred the import of a VarDecl’s type, and therefore the setting of its IUO bit, until the first time InterfaceTypeRequest is evaluated. It turns out that there is nothing to guarantee that InterfaceTypeRequest will be evaluated before IsImplicitlyUnwrappedOptionalRequest, so if isImplicitlyUnwrappedOptional() was fetched before getInterfaceType() was called, it would return an incorrect result. The only known client that accesses the information in this order is the API digester, but in theory any part of the compiler could fall into this trap. Force the evaluation of InterfaceTypeRequest during IsImplicitlyUnwrappedOptionalRequest when necessary to compute the IUO bit for an imported VarDecl, and add a test to prove that this fixes the observed bug in the API digester.

beccadax force-pushed the whole-sdk-validation branch from 1bbfd9a to c33d7df Compare November 17, 2023 00:34

beccadax changed the title ~~Add SDK validation test~~ Add "whole SDK" validation test Nov 17, 2023

beccadax mentioned this pull request Nov 30, 2023

[ClangImporter] Fix IUO ordering bug #70106

Merged

beccadax added 11 commits December 1, 2023 16:35

Prototype an SDK validation test

80c3bd2

This test compares two API digests of large chunks of the installed SDK—one from the toolchain’s API digester, one from the just-built API digester—and flags any changes in behavior that would affect source compatibility.

Add additional changes since Swift 5.7

da4d477

Some of these may be actual regressions!

Test all compatible frameworks in /S/L/F

144aec3

[NFC] Add some pretty stack traces to API digester

b083f8b

Trying to gather information on a failure occurring in CI.

Remove StoreKit from whole SDK validation

63cbaa3

Make API digester -deserialize-sdk early exit on failure

abb8795

api-digester: Make unknown decl attrs a warning

2379b44

Since the API digester is designed to tolerate them anyway.

beccadax force-pushed the whole-sdk-validation branch from c4f63e1 to 613bdc2 Compare December 2, 2023 00:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add "whole SDK" validation test #69928

Add "whole SDK" validation test #69928

Uh oh!

beccadax commented Nov 16, 2023 •

edited

Loading

Uh oh!

beccadax commented Nov 16, 2023

Uh oh!

beccadax commented Nov 17, 2023 •

edited

Loading

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023 •

edited

Loading

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 18, 2023

Uh oh!

beccadax commented Nov 28, 2023

Uh oh!

beccadax commented Nov 29, 2023

Uh oh!

beccadax commented Nov 29, 2023

Uh oh!

beccadax commented Nov 29, 2023

Uh oh!

Uh oh!

Add "whole SDK" validation test #69928

Are you sure you want to change the base?

Add "whole SDK" validation test #69928

Uh oh!

Conversation

beccadax commented Nov 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beccadax commented Nov 16, 2023

Uh oh!

beccadax commented Nov 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beccadax commented Nov 17, 2023

Uh oh!

beccadax commented Nov 18, 2023

Uh oh!

beccadax commented Nov 28, 2023

Uh oh!

beccadax commented Nov 29, 2023

Uh oh!

beccadax commented Nov 29, 2023

Uh oh!

beccadax commented Nov 29, 2023

Uh oh!

Uh oh!

beccadax commented Nov 16, 2023 •

edited

Loading

beccadax commented Nov 17, 2023 •

edited

Loading

beccadax commented Nov 17, 2023 •

edited

Loading