Skip to content

[Build] allow to generate symbols for a subset of binaries #37120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 13, 2021

Conversation

edymtt
Copy link
Contributor

@edymtt edymtt commented Apr 28, 2021

This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.

At the same time, introduce tests around the logic that handles symbols.

Addresses rdar://76865276

@edymtt edymtt force-pushed the darwin-symroot-path-filters branch from c3b5d73 to dc76660 Compare April 29, 2021 17:28
@edymtt edymtt changed the title Introduce a parameter to specify which paths to generate symbols for [Build] allow to generate symbols for a subset of binaries Apr 29, 2021
@edymtt edymtt force-pushed the darwin-symroot-path-filters branch from b040369 to f2b31c3 Compare April 30, 2021 17:50
@edymtt
Copy link
Contributor Author

edymtt commented Apr 30, 2021

@swift-ci please smoke test

@edymtt
Copy link
Contributor Author

edymtt commented Apr 30, 2021

@swift-ci Please Build Toolchain

@edymtt edymtt marked this pull request as ready for review April 30, 2021 18:47
@edymtt
Copy link
Contributor Author

edymtt commented Apr 30, 2021

@swift-ci please smoke test macOS

@swift-ci
Copy link
Contributor

Linux Toolchain (Ubuntu 16.04)
Download Toolchain
Git Sha - f2b31c3e7f321dea219331f16b35fe52827ee5c5

Install command
tar zxf swift-PR-37120-586-ubuntu16.04.tar.gz
More info

@edymtt
Copy link
Contributor Author

edymtt commented May 3, 2021

@swift-ci please smoke test macOS

@edymtt
Copy link
Contributor Author

edymtt commented May 3, 2021

@swift-ci Please Build Toolchain macOS

@swift-ci
Copy link
Contributor

swift-ci commented May 3, 2021

macOS Toolchain
Download Toolchain
Git Sha - f2b31c3e7f321dea219331f16b35fe52827ee5c5

Install command
tar -zxf swift-PR-37120-962-osx.tar.gz --directory ~/

@edymtt
Copy link
Contributor Author

edymtt commented May 3, 2021

@swift-ci please smoke test macOS

Copy link
Contributor

@gottesmm gottesmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far... can you make the changes/look into my questions. Then I want to do another quick go over/read the tests in a deeper way. Ping me to do another go around and assuming no further issues and the tests from first principles test everything we need, this will LGTM.

@edymtt edymtt force-pushed the darwin-symroot-path-filters branch from f2b31c3 to d3fb1c7 Compare May 5, 2021 17:00
@edymtt
Copy link
Contributor Author

edymtt commented May 5, 2021

@swift-ci please smoke test

@edymtt
Copy link
Contributor Author

edymtt commented May 5, 2021

@swift-ci Please Build Toolchain

@edymtt edymtt requested a review from gottesmm May 5, 2021 17:13
@swift-ci
Copy link
Contributor

swift-ci commented May 5, 2021

Linux Toolchain (Ubuntu 16.04)
Download Toolchain
Git Sha - d3fb1c78b24f2c913dd59c0774fa1dfe1835608a

Install command
tar zxf swift-PR-37120-587-ubuntu16.04.tar.gz
More info

@swift-ci
Copy link
Contributor

swift-ci commented May 5, 2021

macOS Toolchain
Download Toolchain
Git Sha - d3fb1c78b24f2c913dd59c0774fa1dfe1835608a

Install command
tar -zxf swift-PR-37120-969-osx.tar.gz --directory ~/

@edymtt
Copy link
Contributor Author

edymtt commented May 5, 2021

A quick check of the log shows that python files and compatibility libraries are copied to the symroot

> grep -A 100 cpio swift-PR-toolchain-osx-969.txt  | grep -e "\.a$" -e "\.py$"
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/bin/swift-api-checker.py
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/appletvsimulator/libswiftCompatibility51.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/appletvsimulator/libswiftCompatibilityDynamicReplacements.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/appletvsimulator/libswiftCompatibility50.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/watchos/libswiftCompatibility51.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/watchos/libswiftCompatibilityDynamicReplacements.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/watchos/libswiftCompatibility50.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/iphoneos/libswiftCompatibility51.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/iphoneos/libswiftCompatibilityDynamicReplacements.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/iphoneos/libswiftCompatibility50.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/appletvos/libswiftCompatibility51.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/appletvos/libswiftCompatibilityDynamicReplacements.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-x86_64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/appletvos/libswiftCompatibility50.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/bin/swift-api-checker.py
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/macosx/libswiftCompatibility51.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/macosx/libswiftCompatibilityDynamicReplacements.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/usr/lib/swift/macosx/libswiftCompatibility50.a
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/System/Library/PrivateFrameworks/LLDB.framework/Versions/A/Resources/Python/lldb/utils/symbolication.py
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/System/Library/PrivateFrameworks/LLDB.framework/Versions/A/Resources/Python/lldb/utils/in_call_stack.py
/Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift-nightly-symroot/macosx-arm64/.//Library/Developer/Toolchains/swift-PR-37120-969.xctoolchain/System/Library/PrivateFrameworks/LLDB.framework/Versions/A/Resources/Python/lldb/macosx/crashlog.py

Copy link
Contributor

@gottesmm gottesmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you address the comments about adding more context to the tests and then feel free to merge without coming back for further review!

Thank you for your work here!

edymtt added 2 commits May 12, 2021 09:52
...before adding the logic to filter paths

In particular:
* print the list of files that `cpio` copies, so we can test explicitly
  which files end up in the symroot (and also see those when asking for
  a toolchain in PR testing)
* use `find` instead of `grep` to filter files we want symbols generated
  for -- this is to avoid the script failing when there are no symbol to
  process, especially in lit tests
* remove an unnecessary check for `swift-api-digester` -- this is now a
  symlink to `swift-frontend` and we only process regular files.

Supports rdar://76865276
This would be needed to reduce overall build times in scenarios when
generating symbols for all binaries is too expensive and/or not needed.

Addresses rdar://76865276
@edymtt edymtt force-pushed the darwin-symroot-path-filters branch from d3fb1c7 to 1e5f790 Compare May 12, 2021 17:03
@edymtt
Copy link
Contributor Author

edymtt commented May 12, 2021

@swift-ci please smoke test

@edymtt
Copy link
Contributor Author

edymtt commented May 12, 2021

@swift-ci Please Build Toolchain

@swift-ci
Copy link
Contributor

Linux Toolchain (Ubuntu 16.04)
Download Toolchain
Git Sha - 1e5f790

Install command
tar zxf swift-PR-37120-592-ubuntu16.04.tar.gz
More info

@swift-ci
Copy link
Contributor

macOS Toolchain
Download Toolchain
Git Sha - 1e5f790

Install command
tar -zxf swift-PR-37120-981-osx.tar.gz --directory ~/

@edymtt edymtt merged commit c3f8dac into swiftlang:main May 13, 2021
@edymtt edymtt deleted the darwin-symroot-path-filters branch May 13, 2021 14:46
edymtt added a commit to edymtt/swift that referenced this pull request May 14, 2021
This was introduced to avoid indentation changes when making this logic
amenable to testing in swiftlang#37120 .
edymtt added a commit that referenced this pull request May 19, 2021
…37428)

This was introduced to avoid indentation changes when making this logic
amenable to testing in #37120 .
LucianoPAlmeida pushed a commit to LucianoPAlmeida/swift that referenced this pull request May 27, 2021
…wiftlang#37428)

This was introduced to avoid indentation changes when making this logic
amenable to testing in swiftlang#37120 .
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants