Skip to content

introduce archive index files #3302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 23, 2021
Merged

introduce archive index files #3302

merged 1 commit into from
Feb 23, 2021

Conversation

tomerd
Copy link
Contributor

@tomerd tomerd commented Feb 22, 2021

motivation: to support the package extensions feature, we need to support binary dependencies with tools that can be distributed across different platforms. in previous PR we intriduces the archive file, this PR adds the archive index files which helps control the size of the archive files by splitting them across key platforms

changes:

  • introduce new binary dependency file type named "Archive Index" with the extension "ari"
  • the structure of the file is a flat list of archive files, their checksums and the target-triples they support
  • add code to Workspace pre-downloading the binaryTarget "zip" files to first fetch "ari" files, parse them and add any relevant "zip" files to the host tripple to the list of candidate downloads
  • add and adjust tests

the new archive file name, extension and structure are all up to debate and would take final shape in an evolution proposal, but we can start discussing them now in preparation to such proposal.

the file structure proposed

{
    "schemaVersion": "1.0",
    "archives": [
        {
            "fileName": "<zipped archive file name>",
            "checksum": "<checksum of zipped archive>",
            "supportedTriples": ["<triple identifier>"]        
        }
    ]
}

concrete example:

web server directory structure

www
|- protocol-bufffer-compiler.ari
|  protoc-apple.zip
|  protoc-linux.zip

protocol-bufffer-compiler.ari content

{
    "schemaVersion": "1.0",
    "archives": [
        {
            "fileName": "protoc-apple.zip",
            "checksum": "<checksum of protoc-apple.zip>",
            "supportedTriples": ["x86_64-apple-macosx", "arm64-apple-macos", "x86_64-apple-macosx", "armv7-apple-ios", "arm64-apple-ios"]        
        },
        {
            "fileName": "protoc-linux.zip",
            "checksum": "<checksum of protoc-linux.zip>",
            "supportedTriples": ["x86_64-unknown-linux-gnu"]     
        }
    ]
}

@tomerd
Copy link
Contributor Author

tomerd commented Feb 22, 2021

@swift-ci please smoke test

@tomerd tomerd changed the title introduce archive index files [RFC] introduce archive index files Feb 22, 2021
@tomerd
Copy link
Contributor Author

tomerd commented Feb 22, 2021

@swift-ci please smoke test

@tomerd
Copy link
Contributor Author

tomerd commented Feb 22, 2021

SourceKitLSP build issue

@tomerd
Copy link
Contributor Author

tomerd commented Feb 22, 2021

@swift-ci please smoke test

@tomerd tomerd self-assigned this Feb 22, 2021
@tomerd
Copy link
Contributor Author

tomerd commented Feb 22, 2021 •
edited
Loading

more SourceKitLSP CI instability

@tomerd
Copy link
Contributor Author

tomerd commented Feb 22, 2021

@swift-ci please smoke test linux

@tomerd
Copy link
Contributor Author

tomerd commented Feb 23, 2021

@swift-ci please smoke test

@tomerd
Copy link
Contributor Author

tomerd commented Feb 23, 2021

more LSP issues on linux cc @benlangmuir

:37:47 2021-02-23 02:37:43.351 SourceKitLSPPackageTests.xctest[26841:21945700] manifest parse error(s):
18:37:47 
/home/buildnode/jenkins/workspace/swift-package-manager-Linux-smoke-test/branch-main/tmp/TemporaryDirectory.wL0O2a/pkg/Package.swift:3:5: error: type annotation missing in pattern
18:37:47 let pack
18:37:47     ^
18:37:47 
18:37:47 Test Suite 'Selected tests' started at 2021-02-23 02:37:43.185
18:37:47 Test Suite 'SourceKitDRegistryTests' started at 2021-02-23 02:37:43.187
18:37:47 Test Case 'SourceKitDRegistryTests.testRemoveResurrect' started at 2021-02-23 02:37:43.187
18:37:47 Test Case 'SourceKitDRegistryTests.testRemoveResurrect' passed (0.0 seconds)
18:37:47 Test Suite 'SourceKitDRegistryTests' passed at 2021-02-23 02:37:43.188
18:37:47 	 Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
18:37:47 Test Suite 'Selected tests' passed at 2021-02-23 02:37:43.188
18:37:47 	 Executed 1 test, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
18:37:47 
18:37:47 Test Suite 'Selected tests' started at 2021-02-23 02:37:43.167
18:37:47 Test Suite 'SwiftPMWorkspaceTests' started at 2021-02-23 02:37:43.169
18:37:47 Test Case 'SwiftPMWorkspaceTests.testUnknownFile' started at 2021-02-23 02:37:43.169
18:37:47 SKSwiftPMWorkspaceTests/SwiftPMWorkspaceTests.swift:309: Fatal error: 'try!' expression unexpectedly raised an error: toolchain is invalid: could not find swiftc
18:37:47 Current stack trace:
18:37:47 0    libswiftCore.so                    0x00007f808085e500 swift_reportError + 50
18:37:47 1    libswiftCore.so                    0x00007f80808e0400 _swift_stdlib_reportFatalErrorInFile + 109
18:37:47 2    libswiftCore.so                    0x00007f8080593712 <unavailable> + 1476370
18:37:47 3    libswiftCore.so                    0x00007f808059343b <unavailable> + 1475643
18:37:47 4    libswiftCore.so                    0x00007f8080593935 <unavailable> + 1476917
18:37:47 5    libswiftCore.so                    0x00007f80805921d0 _assertionFailure(_:_:file:line:flags:) + 367
18:37:47 6    libswiftCore.so                    0x00007f80805e5793 <unavailable> + 1812371
18:37:47 7    SourceKitLSPPackageTests.xctest    0x0000563aa897e181 <unavailable> + 9789825
18:37:47 8    SourceKitLSPPackageTests.xctest    0x0000563aa89789dc <unavailable> + 9767388
18:37:47 9    SourceKitLSPPackageTests.xctest    0x0000563aa89837d6 <unavailable> + 9811926
18:37:47 10   SourceKitLSPPackageTests.xctest    0x0000563aa8e02a72 <unavailable> + 14527090
18:37:47 11   SourceKitLSPPackageTests.xctest    0x0000563aa89830fa <unavailable> + 9810170
18:37:47 12   SourceKitLSPPackageTests.xctest    0x0000563aa898439c <unavailable> + 9814940
18:37:47 13   SourceKitLSPPackageTests.xctest    0x0000563aa8984546 <unavailable> + 9815366
18:37:47 14   libXCTest.so                       0x00007f807ed80bfc <unavailable> + 269308
18:37:47 15   libXCTest.so                       0x00007f807ed80c48 <unavailable> + 269384
18:37:47 16   libXCTest.so                       0x00007f807ed80bdf <unavailable> + 269279
18:37:47 17   libXCTest.so                       0x00007f807ed66578 <unavailable> + 161144
18:37:47 18   libXCTest.so                       0x00007f807ed7dcb0 XCTestCase.invokeTest() + 269
18:37:47 19   libXCTest.so                       0x00007f807ed7d9e0 XCTestCase.perform(_:) + 147
18:37:47 20   libXCTest.so                       0x00007f807ed84870 XCTest.run() + 144
18:37:47 21   libXCTest.so                       0x00007f807ed811f0 XCTestSuite.perform(_:) + 209
18:37:47 22   libXCTest.so                       0x00007f807ed84870 XCTest.run() + 144
18:37:47 23   libXCTest.so                       0x00007f807ed811f0 XCTestSuite.perform(_:) + 209
18:37:47 24   libXCTest.so                       0x00007f807ed84870 XCTest.run() + 144
18:37:47 25   libXCTest.so                       0x00007f807ed7a320 XCTMain(_:arguments:observers:) + 2356
18:37:47 26   libXCTest.so                       0x00007f807ed7a160 XCTMain(_:) + 133
18:37:47 27   SourceKitLSPPackageTests.xctest    0x0000563aa8a5cde7 <unavailable> + 10702311
18:37:47 28   libc.so.6                          0x00007f807dc7a750 __libc_start_main + 240
18:37:47 29   SourceKitLSPPackageTests.xctest    0x0000563aa817aec9 <unavailable> + 1388233
18:37:47 Exited with signal code 4

@tomerd
Copy link
Contributor Author

tomerd commented Feb 23, 2021

@swift-ci please smoke test linux

@tomerd
Copy link
Contributor Author

tomerd commented Feb 23, 2021

@swift-ci please smoke test

@tomerd tomerd added the ready Author believes the PR is ready to be merged & any feedback has been addressed label Feb 23, 2021
@tomerd tomerd changed the title [RFC] introduce archive index files introduce archive index files Feb 23, 2021
neonichu
neonichu approved these changes Feb 23, 2021
View reviewed changes
abertelrud
abertelrud approved these changes Feb 23, 2021
View reviewed changes
Copy link
Contributor

@abertelrud abertelrud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Just a question about whether the filename should be a subpath, but that can also be left for later.

@tomerd tomerd force-pushed the feature/binary-03 branch 3 times, most recently from db190e7 to b5cee3c Compare February 23, 2021 20:52
@tomerd
introduce archive index files
ee32f8a 
motivation: to support the package extensions feature, we need to support binary dependencies with tools that can be distributed across different platforms. in previous PR we intriduces the archive file, this PR adds the archive index files which helps control the size of the archive files by splitting them across key platforms

changes:
* introduce new binary dependency file type named "Archive Index" with the extension "ari"
* the structure of the file is a flat list of archive files, their checksums and the target-triples they support
* add code to Workspace pre-downloading the binaryTarget "zip" files to first fetch "ari" files, parse them and add any relevant "zip" files to the *host* tripple to the list of candidate downloads
* add and adjust tests
@tomerd
Copy link
Contributor Author

tomerd commented Feb 23, 2021

@swift-ci please smoke test

@tomerd tomerd merged commit 49bed82 into swiftlang:main Feb 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abertelrud abertelrud abertelrud approved these changes

@neonichu neonichu neonichu approved these changes

@friedbunny friedbunny Awaiting requested review from friedbunny

Assignees

@tomerd tomerd

Labels
ready Author believes the PR is ready to be merged & any feedback has been addressed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tomerd @neonichu @abertelrud