[5.0] Build and install 'clang' and 'clangd' in the macOS & linux toolchains #22273

hyp · 2019-01-31T22:47:33Z

Build script now builds clang_tools_extra as part of LLVM's build.
Build script now has a new libc++ build step to allow libc++ headers to be installed in the
resulting toolchain.
'clang', 'clangd', 'clang-headers' & 'compiler-rt' targets are now installed for
the package build configurations for macOS and linux.
'clang-resource-dir-symlink' is used in the package build configuration for macOS and linux
to avoid duplication of Clang's headers and compiler-rt archives.

rdar://24912710
(cherry picked from commit 1941aaa)

hyp · 2019-01-31T22:48:26Z

@swift-ci build toolchain

shahmishal

LGTM! We will need to also update the toolchain bots before merging this PR, I will merge it once bots have been updated.

shahmishal · 2019-01-31T22:59:37Z

@swift-ci test

swift-ci · 2019-02-01T00:18:45Z

Build failed
Swift Test Linux Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

shahmishal · 2019-02-01T00:29:42Z

#22282

shahmishal · 2019-02-04T07:35:12Z

@swift-ci build toolchain

shahmishal · 2019-02-04T07:35:25Z

@swift-ci test

swift-ci · 2019-02-04T08:11:33Z

Linux Toolchain (Ubuntu 16.04)
Download Toolchain
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

Install command
tar zxf swift-PR-22273-150-ubuntu16.04.tar.gz
More info

swift-ci · 2019-02-04T08:58:42Z

macOS Toolchain
Download Toolchain
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

Install command
tar -zxf swift-PR-22273-197-osx.tar.gz --directory ~/

swift-ci · 2019-02-04T10:43:54Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

benlangmuir · 2019-02-04T17:38:28Z

There's an issue with the libc++ header installation: https://bugs.swift.org/browse/SR-9861

benlangmuir · 2019-02-04T18:05:59Z

Nevermind, that bug should not affect the swift-5 branch.

hyp · 2019-02-06T01:34:03Z

@swift-ci test

swift-ci · 2019-02-06T04:46:26Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

hyp · 2019-02-11T19:00:20Z

@swift-ci test

benlangmuir · 2019-02-13T18:28:53Z

@hyp while the Product issue doesn't specifically affect swift-5.0-branch, I think for consistency we should include the changes from #22404 since they affect the way that build-script-impl is invoked. That PR also depends on #22346.

swift-ci · 2019-02-13T21:44:56Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

hyp · 2019-02-15T22:56:18Z

@swift-ci test

swift-ci · 2019-02-16T02:08:22Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

tkremenek · 2019-02-17T18:18:47Z

@swift-ci clean test

swift-ci · 2019-02-17T23:24:34Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

ianpartridge · 2019-02-20T12:55:04Z

Any news on when this might land? It will make a big improvement to the Swift 5 Docker images: https://forums.swift.org/t/which-clang-package-should-we-install/20542/12

tkremenek · 2019-02-20T20:09:31Z

@swift-ci clean test macos

swift-ci · 2019-02-20T23:26:51Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

hyp · 2019-02-22T23:06:29Z

@swift-ci clean test macos

swift-ci · 2019-02-23T03:28:03Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

jrose-apple · 2019-02-26T01:36:51Z

Heads-up that we'll need #22911 too.

swift-ci · 2019-03-05T18:30:42Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

akyrtzi · 2019-03-05T21:34:38Z

Discussed with @shahmishal about the OSX bot failure, we think the additional PRs @benlangmuir suggested may help to resolve the failure, @hyp will cherry-pick them to this PR.

akyrtzi · 2019-03-05T22:54:40Z

Update: discussed with @benlangmuir and he says that clangd is problematic and is hitting an assertion hit. Given that clangd is not usable there's no reason to put all of clang+clangd in the swift-5.0 macOS toolchain, so @hyp is going to look into updating the PR to only put clang in the linux toolchain.

ianpartridge · 2019-03-05T23:00:11Z

That sounds great - thanks @akyrtzi @benlangmuir @hyp!

- Build script now builds clang_tools_extra as part of LLVM's build. - Build script now has a new libc++ build step to allow libc++ headers to be installed in the resulting toolchain. - 'clang', 'clang-headers' & 'compiler-rt' targets are now installed for the package build configurations for linux. - 'clang-resource-dir-symlink' is used in the package build configuration for linux to avoid duplication of Clang's headers and compiler-rt archives. rdar://24912710 (cherry picked from commit 1941aaa)

hyp · 2019-03-06T01:00:06Z

Updated the PR to only install clang on Linux

hyp · 2019-03-06T01:00:12Z

@swift-ci please test

swift-ci · 2019-03-06T01:02:01Z

Build failed
Swift Test OS X Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

swift-ci · 2019-03-06T01:02:15Z

Build failed
Swift Test Linux Platform
Git Sha - dcf5e208ddf5cd8cf6124617e0d8061a8448945b

hyp · 2019-03-06T01:03:33Z

@shahmishal Why do the builds fail immediately?

hyp · 2019-03-06T01:05:22Z

@swift-ci build toolchain

shahmishal · 2019-03-06T01:12:25Z

@hyp, it's due to force push

Automatically restarting the build, due to invalid sha

hyp · 2019-03-06T18:48:12Z

@shahmishal is macOS failure the same one we had the last time? I took out the Mac builds, but maybe the problem is in the build script?

shahmishal · 2019-03-06T19:12:00Z

We need to cherry-pick these two PRs into #22273. Otherwise, we are going to try to install libcxx headers during PR testing.

#22404
#22346

These two PRs enable support to only install libcxx headers in package preset, the current solution in PR#22273 will try to install the headers if libcxx directory exists.

https://github.com/apple/swift/pull/22273/files#diff-65b44eb6cb88af2161d8e1176231aad0R1217

# Build libcxx, unless it doesn't exist.
	LIBCXX_SOURCE_DIR="${WORKSPACE}/libcxx"
	if [[ ! -e "${LIBCXX_SOURCE_DIR}" ]] ; then
	  SKIP_BUILD_LIBCXX=1
	fi

This is the reason we are seeing PR failing due to "--install-destdir is required to install products."

hyp · 2019-03-06T20:59:35Z

@swift-ci please test

hyp · 2019-03-06T20:59:48Z

Cherry-picked the changes

swift-ci · 2019-03-06T21:01:46Z

Build failed
Swift Test Linux Platform
Git Sha - 76e66e9

swift-ci · 2019-03-06T21:03:29Z

Build failed
Swift Test OS X Platform
Git Sha - 76e66e9

Now we build libcxx if and only if --libcxx is passed, like with other optional products. This loses the intended "build if sources are checked out" behaviour, but behaves more predictably. (cherry picked from commit e180006) Revert "Revert "[build-script] Add Product for libc++"" add libcxx-cmake-options

hyp · 2019-03-06T21:20:30Z

error: unknown setting: libcxx-cmake-options, fixed.

hyp · 2019-03-06T21:20:36Z

@swift-ci please test

swift-ci · 2019-03-06T21:22:42Z

Build failed
Swift Test Linux Platform
Git Sha - 71d88c57c4066d216696c4cdb587a388fa17e9c7

swift-ci · 2019-03-06T21:24:40Z

Build failed
Swift Test OS X Platform
Git Sha - 71d88c57c4066d216696c4cdb587a388fa17e9c7

hyp · 2019-03-06T22:38:42Z

@swift-ci build toolchain

hyp · 2019-03-07T01:00:11Z

@shahmishal tests passed, please merge

ianpartridge · 2019-03-07T09:33:25Z

Awesome, thanks @hyp and @shahmishal!

hyp requested review from shahmishal and benlangmuir January 31, 2019 22:47

hyp requested a review from a team as a code owner January 31, 2019 22:47

shahmishal approved these changes Jan 31, 2019

View reviewed changes

hyp force-pushed the buildInstallClangAndClangd-swift5.0 branch from dcf5e20 to 76e66e9 Compare March 6, 2019 00:59

hyp force-pushed the buildInstallClangAndClangd-swift5.0 branch from 71d88c5 to c221dd6 Compare March 6, 2019 21:20

akyrtzi merged commit 624112d into swiftlang:swift-5.0-branch Mar 7, 2019

ianpartridge mentioned this pull request Mar 27, 2019

Add Swift 5.0 Dockerfiles for Ubuntu 16.04 and Ubuntu 18.04 swiftlang/swift-docker#134

Merged

[5.0] Build and install 'clang' and 'clangd' in the macOS & linux toolchains #22273

[5.0] Build and install 'clang' and 'clangd' in the macOS & linux toolchains #22273

Uh oh!

Conversation

hyp commented Jan 31, 2019

Uh oh!

hyp commented Jan 31, 2019

Uh oh!

shahmishal left a comment

Choose a reason for hiding this comment

Uh oh!

shahmishal commented Jan 31, 2019

Uh oh!

swift-ci commented Feb 1, 2019

Uh oh!

shahmishal commented Feb 1, 2019

Uh oh!

shahmishal commented Feb 4, 2019

Uh oh!

shahmishal commented Feb 4, 2019

Uh oh!

swift-ci commented Feb 4, 2019

Uh oh!

swift-ci commented Feb 4, 2019

Uh oh!

swift-ci commented Feb 4, 2019

Uh oh!

benlangmuir commented Feb 4, 2019

Uh oh!

benlangmuir commented Feb 4, 2019

Uh oh!

hyp commented Feb 6, 2019

Uh oh!

swift-ci commented Feb 6, 2019

Uh oh!

hyp commented Feb 11, 2019

Uh oh!

benlangmuir commented Feb 13, 2019

Uh oh!

swift-ci commented Feb 13, 2019

Uh oh!

hyp commented Feb 15, 2019

Uh oh!

swift-ci commented Feb 16, 2019

Uh oh!

tkremenek commented Feb 17, 2019

Uh oh!

swift-ci commented Feb 17, 2019

Uh oh!

ianpartridge commented Feb 20, 2019

Uh oh!

tkremenek commented Feb 20, 2019

Uh oh!

swift-ci commented Feb 20, 2019

Uh oh!

hyp commented Feb 22, 2019

Uh oh!

swift-ci commented Feb 23, 2019

Uh oh!

jrose-apple commented Feb 26, 2019

Uh oh!

swift-ci commented Mar 5, 2019

Uh oh!

akyrtzi commented Mar 5, 2019

Uh oh!

akyrtzi commented Mar 5, 2019

Uh oh!

ianpartridge commented Mar 5, 2019

Uh oh!

hyp commented Mar 6, 2019

Uh oh!

hyp commented Mar 6, 2019

Uh oh!

swift-ci commented Mar 6, 2019

Uh oh!

swift-ci commented Mar 6, 2019

Uh oh!

hyp commented Mar 6, 2019

Uh oh!

hyp commented Mar 6, 2019