build-script: build static version of Foundation #20824

compnerd · 2018-11-28T05:56:24Z

Build and install Foundation static. We now build Foundation using
CMake, which does not easily generate static and shared versions of
libraries. Create two builds to populate the toolchain
distribution.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

compnerd · 2018-11-28T05:56:35Z

CC: @kevints

compnerd · 2018-11-28T05:57:32Z

Please test with following PRs:
swiftlang/swift-corelibs-foundation#1787

@swift-ci please smoke test

compnerd · 2018-11-28T05:57:52Z

Please test with following PRs:
swiftlang/swift-corelibs-foundation#1787

@swift-ci please build toolchain

swift-ci · 2018-11-28T07:52:48Z

macOS Toolchain
Download Toolchain
Git Sha - af2254610abdc37f6363d3ae1eac67b01b1a7054

Install command
tar -zxf swift-PR-20824-121-osx.tar.gz --directory ~/

compnerd · 2018-11-28T15:57:56Z

Please test with following PRs:
swiftlang/swift-corelibs-foundation#1787

@swift-ci please build toolchain

compnerd · 2018-11-28T15:58:06Z

Please test with following PRs:
swiftlang/swift-corelibs-foundation#1787

@swift-ci please smoke test

compnerd · 2018-11-28T16:50:16Z

CC: @shahmishal @gottesmm @kevints @spevans

It seems that the ICU version on the system is not the version that we are building against?

10:31:52 -- Found ICU: /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/libicu-linux-x86_64/tmp_install/include (found version "61.1") 
...
10:33:03 cd /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/foundation_static-linux-x86_64 && /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/swift-linux-x86_64/bin/swiftc -L/home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/foundation_static-linux-x86_64 -L /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/libdispatch-linux-x86_64 -L /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/libdispatch-linux-x86_64/src -ldispatch -Xlinker -rpath -Xlinker /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/libdispatch-linux-x86_64/src -lFoundation -L/home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/foundation_static-linux-x86_64/CoreFoundation-prefix/usr/lib -lCoreFoundation /usr/lib/x86_64-linux-gnu/libcurl.so /usr/lib/x86_64-linux-gnu/libicuuc.so /usr/lib/x86_64-linux-gnu/libicui18n.so /usr/lib/x86_64-linux-gnu/libxml2.so /usr/lib/x86_64-linux-gnu/libuuid.so -o /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/foundation_static-linux-x86_64/plutil.dir/plutil /home/buildnode/jenkins/workspace/swift-PR-toolchain-Linux/branch-master/buildbot_linux/foundation_static-linux-x86_64/plutil.dir/main.swift.o

Note that we are trying to link against the system ICU rather than the version in tmp_install

compnerd · 2018-11-28T16:55:15Z

Is there a way to get the CMakeCache.txt and build.ninja from the build?

spevans · 2018-11-28T16:57:32Z

There should be an built ICU in build/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/ which is the one to link against, rather than the system one which is older.

compnerd · 2018-11-28T17:04:25Z

@spevans - should be. I definitely see that being passed to CMake, but, I don't see that in the install logs? Could it be that we aren't populating the library at the time that we think that we are?

compnerd · 2018-11-28T17:06:36Z

No, interesting, I did find it, I think I wasn't searching for the right name. It seems that CMake doesn't believe that to be available.

swift-ci · 2018-11-28T17:07:09Z

macOS Toolchain
Download Toolchain
Git Sha - af2254610abdc37f6363d3ae1eac67b01b1a7054

Install command
tar -zxf swift-PR-20824-122-osx.tar.gz --directory ~/

spevans · 2018-11-28T17:08:10Z

@compnerd Im just doing a local build with your 2 PRs on ubuntu18 so I will see If I can replicate the issue

compnerd · 2018-11-28T17:54:12Z

Oh, I think that this might be due to the fact that we are renaming the libraries :/

Build and install Foundation static. We now build Foundation using CMake, which does not easily generate static and shared versions of libraries. Create two builds to populate the toolchain distribution.

compnerd · 2018-11-28T18:13:10Z

@swift-ci please smoke test

compnerd · 2018-11-28T18:13:16Z

@swift-ci please build toolchain

kevints · 2018-11-28T18:14:58Z

utils/build-script-impl

@@ -2604,7 +2608,7 @@ for host in "${ALL_HOSTS[@]}"; do
                esac

                ;;
-            foundation)
+            foundation|foundation_static)


Should we build a static ICU as well?

Yeah, that would be a good idea to do given the rest of the static linking, and the behaviour of glibc's libdl. But, I think that is beyond the scope of this change.

A static libICU is already built on linux as part of the --libicu option:

if [ $(true_false "${BUILD_SWIFT_STATIC_STDLIB}") == "TRUE" ]; then libicu_enable_static="--enable-static" else libicu_enable_static="" fi

Oh, cool. Um, why is that controlled by static stdlib though? Lets get this working and then adjust the static build to use the static ICU.

Since ICU is now bundled (presumably so that "🏴󠁧󠁢󠁥󠁮󠁧󠁿".count == 1 instead of possibly 7) it's on us to build the static version of it as well. -static-stdlib changes the library search path to lib/swift_static instead of lib/swift so if there's no icu libs there it won't work.

What should it be controlled by? Are we adding an extra --static- flag for each lib? maybe it would be better with a --static-libs flag instead to build them all together, -static-stdlib requires all of them anyway

IOW it seems ICU is now considered part of the stdlib, so flags that control the linkage of the stdlib should cover ICU now too

@kevints, this is about the build-script, not the the driver flag. The driver flag remains -static-stdlib. The icu libs are already present in the swift_static directory.

Ok @compnerd I was confused then. This looks good to me

spevans · 2018-11-28T18:41:12Z

@compnerd I got this to work by undoing the ICU_UC lib change in the other PR, it looks good except that the libFoundation.a ends up in the wrong directory:

$ find ~/swift-install/usr/lib -name libFoundation.a
/home/spse/swift-install/usr/lib/swift/linux/libFoundation.a

swift-ci · 2018-11-28T18:46:26Z

Linux Toolchain (Ubuntu 16.04)
Download Toolchain
Git Sha - 69e2018

Install command
tar zxf swift-PR-20824-105-ubuntu16.04.tar.gz
More info

compnerd · 2018-11-28T18:48:54Z

@spevans - the location thing is fixed with swiftlang/swift-corelibs-foundation#1788 ... I think that if this passes the smoke test/toolchain builds, this should be good to go. We should fix up the ICU location in a follow up. Note that the only thing impacted will be plutil.

swift-ci · 2018-11-28T19:16:43Z

macOS Toolchain
Download Toolchain
Git Sha - 69e2018

Install command
tar -zxf swift-PR-20824-124-osx.tar.gz --directory ~/

compnerd · 2018-11-28T19:53:40Z

@kevints, @spevans - thoughts on this? I think that fixing the static build of plutil is nice, but not a requirement for this.

spevans · 2018-11-28T20:12:04Z

@compnerd I think this is ok now, I just did a test of the latest version along with swift-corelibs-foundation/master and it looks to have worked ok including installing the libraries in the right places:

$ find ~/swift-install/ -name 'libFoundation*'
/home/spse/swift-install/usr/lib/swift_static/linux/libFoundation.a
/home/spse/swift-install/usr/lib/swift/linux/libFoundation.so

As to plutil its currently completely broken anyway, it has the wrong file permissions and needs an extra RUNPATH entry so that can be dealt with in a followup PR:

$ ls -l swift-DEVELOPMENT-SNAPSHOT-2018-11-26-a-ubuntu18.04/usr/bin/plutil
-rw-r--r-- 1 spse spse 77320 Nov 26 14:18 swift-DEVELOPMENT-SNAPSHOT-2018-11-26-a-ubuntu18.04/usr/bin/plutil

gottesmm · 2018-11-28T23:57:49Z

@spevans @compnerd is there an integration test that we can add to make sure that this doesn't break again?

spevans · 2018-11-29T00:19:04Z

We could compile something with -static-stdlib. If it did import Foundation it should validate libFoundation.a libdispatch.a and the libicu*.a libraries

gottesmm · 2018-11-29T01:38:18Z

@spevans SGTM. Can you put that together?

build-script: build static version of Foundation

69e2018

Build and install Foundation static. We now build Foundation using CMake, which does not easily generate static and shared versions of libraries. Create two builds to populate the toolchain distribution.

compnerd force-pushed the static-foundation branch from af22546 to 69e2018 Compare November 28, 2018 18:12

kevints reviewed Nov 28, 2018

View reviewed changes

compnerd merged commit 8935b89 into swiftlang:master Nov 28, 2018

compnerd deleted the static-foundation branch November 28, 2018 20:16

build-script: build static version of Foundation #20824

build-script: build static version of Foundation #20824

Uh oh!

Conversation

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

swift-ci commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

spevans commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

swift-ci commented Nov 28, 2018

Uh oh!

spevans commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spevans commented Nov 28, 2018

Uh oh!

swift-ci commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swift-ci commented Nov 28, 2018

Uh oh!

compnerd commented Nov 28, 2018

Uh oh!

spevans commented Nov 28, 2018

Uh oh!

gottesmm commented Nov 28, 2018

Uh oh!

spevans commented Nov 29, 2018

Uh oh!

gottesmm commented Nov 29, 2018

Uh oh!

Uh oh!

compnerd commented Nov 28, 2018 •

edited

Loading