SR-8876: Always build libicu on Linux #19860

spevans · 2018-10-12T20:24:10Z

Uses version 61.1 from ICU Github unicode-org/icu repository.
Updates mixin_linux_installation to add liblic option.
Use -j when building libicu.
Resolves SR-8876.
build-script-impl updates to make all tests work.

Also resolves:

SR-5618: libicu compilation should happen in parallel respecting -j.

Note, ICU 62.1 currently causes a regression with swift-corelibs-foundation on Linux which will be looked at separately once this PR is merged.

spevans · 2018-10-12T20:25:56Z

@swift-ci test

swift-ci · 2018-10-12T20:28:10Z

Build failed
Swift Test OS X Platform
Git Sha - 739ce893a64eaa62939ad5fb380f411c21911667

swift-ci · 2018-10-12T20:29:26Z

Build failed
Swift Test Linux Platform
Git Sha - 739ce893a64eaa62939ad5fb380f411c21911667

parkera · 2018-10-12T21:57:14Z

This is great!

spevans · 2018-10-12T22:17:38Z

@swift-ci test

swift-ci · 2018-10-12T22:19:23Z

Build failed
Swift Test Linux Platform
Git Sha - 739ce893a64eaa62939ad5fb380f411c21911667

swift-ci · 2018-10-12T22:19:25Z

Build failed
Swift Test OS X Platform
Git Sha - 739ce893a64eaa62939ad5fb380f411c21911667

milseman · 2018-10-12T22:58:19Z

Will this change to update-checkout cause it to be cloned on Darwin as well, and if so can we make it only do so on Linux?

spevans · 2018-10-12T23:21:40Z

@milseman Currently it will clone on all platforms, I will fix update-checkout to allow specifying platforms in utils/update_checkout/update-checkout-config.json

spevans · 2018-10-13T09:27:53Z

@swift-ci test

swift-ci · 2018-10-13T09:28:20Z

Build failed
Swift Test OS X Platform
Git Sha - ebffe7b672c0896f63511a00a80ad25b62169db0

swift-ci · 2018-10-13T09:30:02Z

Build failed
Swift Test Linux Platform
Git Sha - ebffe7b672c0896f63511a00a80ad25b62169db0

spevans · 2018-10-13T11:05:05Z

@swift-ci test

swift-ci · 2018-10-13T11:05:29Z

Build failed
Swift Test OS X Platform
Git Sha - 9d2f108bcc6cc65ac073b503f1b590bd78c69411

swift-ci · 2018-10-13T11:07:03Z

Build failed
Swift Test Linux Platform
Git Sha - 9d2f108bcc6cc65ac073b503f1b590bd78c69411

spevans · 2018-10-14T21:17:53Z

@swift-ci test

swift-ci · 2018-10-14T21:19:37Z

Build failed
Swift Test Linux Platform
Git Sha - fe08b5c34d04b21f7a455760d5f258db5ba60c19

swift-ci · 2018-10-14T21:19:43Z

Build failed
Swift Test OS X Platform
Git Sha - fe08b5c34d04b21f7a455760d5f258db5ba60c19

spevans · 2018-10-15T05:46:26Z

cc @millenomi I added the change to namespace the symbols with a _swift suffix and the library files have swift appended to them using the ICU configure option --with-library-suffix=swift.

millenomi · 2018-10-15T19:52:28Z

@shahmishal who could review this?

tkremenek · 2018-10-16T00:58:26Z

As this is adding a new library dependency, this falls under:

https://swift.org/contributing/#adding-external-library-dependencies

I'll need to go review this for any legal concerns, but this is on me.

spevans · 2018-10-26T20:35:05Z

@tkremenek Have you had a chance to look at this? thanks

spevans · 2018-10-28T14:23:28Z

@swift-ci test

swift-ci · 2018-10-28T14:25:31Z

Build failed
Swift Test Linux Platform
Git Sha - 26b1b59478d0a1c49c28e0777acff76a743723be

swift-ci · 2018-10-28T14:27:24Z

Build failed
Swift Test OS X Platform
Git Sha - 26b1b59478d0a1c49c28e0777acff76a743723be

tkremenek · 2018-10-30T18:56:57Z

Sorry for the delay here; I should have an answer shortly.

tkremenek · 2018-11-02T03:59:55Z

OK, I got legal confirmation that this is OK. There is no problem with statically linking in ICU into the Standard Library in this way; ICU's license clearly allows this.

However, we should also include the ICU license in the distribution of Swift when we do this, as the license apples to the ICU data files that will now need to be bundled with the Standard Library. Does this PR do that?

- Uses version 61.1 from ICU Github unicode-org/icu repository. - Updates mixin_linux_installation to add libicu option. - Use -j when building libicu. - When buiding ICU, use --with-library-suffix=swift This suffixes the ICU symbols with _swift. The libaries are now named libicuucswift, libicui18nswift and libicudataswift. - Add the contents of uconfig.h.prepend into uconfig.h. This avoids passing the renaming CFLAGS to swift and swift-corelibs-foundation. Also resolves: SR-5618: libicu compilation should happen in parallel respecting -j.

- Update --reset-to-remote to support tags. - Add optional platform entry to a repository. This allows specifying which platforms a repo should be checked out on.

- Copy ICU licence into final install.

spevans · 2018-11-02T17:38:54Z

Ive updated the PR to copy the ICU license into the distribution. It gets copied into usr/share/icuswift/61.1/LICENSE

spevans · 2018-11-02T17:41:11Z

Please test with the following PRs:
swiftlang/swift-corelibs-foundation#1744

@swift-ci please test

swift-ci · 2018-11-02T17:41:42Z

Build failed
Swift Test OS X Platform
Git Sha - 839afef7ac82ed752417616fbebc2aaaa71bb4e9

swift-ci · 2018-11-02T17:42:56Z

Build failed
Swift Test Linux Platform
Git Sha - 839afef7ac82ed752417616fbebc2aaaa71bb4e9

milseman · 2018-11-04T00:25:59Z

🍾 Congrats @spevans

shahmishal · 2018-11-15T00:08:33Z

@spevans we are seeing this issue with recent development toolchain https://bugs.swift.org/browse/SR-9251. It might be related to this PR changes.

millenomi · 2018-11-16T01:20:47Z

@shahmishal Do you know if we’re distributing these binaries? If not, can we add them to the toolchain?

millenomi · 2018-11-16T01:22:56Z

Also, is there any documentation of the installation setup?

shahmishal · 2018-11-16T01:23:38Z

@millenomi Yes we are distributing them on https://swift.org/download/#snapshots

shahmishal · 2018-11-16T01:28:02Z

Let me know if you need help installing the toolchain, you can also find the info on https://swift.org/download/#using-downloads

millenomi · 2018-11-16T01:49:03Z

@shahmishal I meant whether we were distributing specifically the libicu*swift binaries, which apparently we are:

millenomi $ find /Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04 -name libicu\*
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift_static/linux/libicuucswift.a
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift_static/linux/libicui18nswift.a
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift_static/linux/libicudataswift.a
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicui18nswift.so
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicudataswift.so.61
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicuucswift.so.61.1
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicuucswift.so.61
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicudataswift.so
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicui18nswift.so.61.1
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicudataswift.so.61.1
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicuucswift.so
/Users/millenomi/Downloads/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicui18nswift.so.61

@spevans Can you investigate? Perhaps the final RPATHs are incorrect?

spevans · 2018-11-16T07:08:20Z

@millenomi This should have been fixed by swiftlang/swift-corelibs-foundation#1769 but it is not working correctly. Im taking a look

spevans · 2018-11-16T07:34:20Z

Looks like there are 2 libFoundation.so files in the tar:

$ find swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib -name libFoundation.so
swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/x86_64/libFoundation.so
swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libFoundation.so

And the one in the x86_64 directory is being loaded:

$ ldd hello|grep libFoundation
	libFoundation.so => /home/spse/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/x86_64/libFoundation.so (0x00007f07ae29d000)

so this is setting the $ORIGIN to the x86_64 subdirectory and then failing to find libicu*

$ ./hello
./hello: error while loading shared libraries: libicuucswift.so.61: cannot open shared object file: No such file or directory

Deleting this extra libFoundation.so makes it fall back to the one in usr/lib/swift/linux and everything works:

$ rm /home/spse/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/x86_64/libFoundation.so
ubuntu1804:~ spse$ ldd hello|grep libFoundation
	libFoundation.so => /home/spse/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libFoundation.so (0x00007f5986fcc000)
ubuntu1804:~ spse$ ldd hello|grep libicu.*61
	libicuucswift.so.61 => /home/spse/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicuucswift.so.61 (0x00007f393c962000)
	libicui18nswift.so.61 => /home/spse/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicui18nswift.so.61 (0x00007f393c46c000)
	libicudataswift.so.61 => /home/spse/swift-DEVELOPMENT-SNAPSHOT-2018-11-15-a-ubuntu18.04/usr/lib/swift/linux/libicudataswift.so.61 (0x00007f393a0f7000)
ubuntu1804:~ spse$ ./hello
Hello

I'll see if I can set see what is adding the libFoundation.so into the x86_64 directory and remove it as no other .so libraries are stored in there. However I still cant work out why the x86_64 directory is preferred over its parent looking at the RUNPATHs of all the other libraries.

spevans · 2018-11-16T10:43:50Z

Hopefully this second fix to solve the issue: swiftlang/swift-corelibs-foundation#1776

libicu is always built on Linux, because libicu of system is sometimes very old. swiftlang/swift#19860

spevans requested a review from shahmishal October 15, 2018 07:34

tkremenek self-assigned this Oct 16, 2018

tkremenek self-requested a review October 16, 2018 00:58

nevil mentioned this pull request Oct 20, 2018

Implement NSTextCheckingResult.range(withName:) swiftlang/swift-corelibs-foundation#1522

Merged

drodriguez mentioned this pull request Oct 26, 2018

Allow cross-compile Android while compiling libicu in Linux. #20083

Closed

MaxDesiatov mentioned this pull request Oct 28, 2018

Unable to compile a simple Swift file (maybe not possible?) emscripten-core/emscripten#2427

Closed

spevans mentioned this pull request Oct 29, 2018

NumberFormatter: Fix the pattern default .nostyle pattern for ICU62+ swiftlang/swift-corelibs-foundation#1736

Merged

spevans mentioned this pull request Nov 2, 2018

Update CMake build to pass -I ICU_INCLUDE_DIR swiftlang/swift-corelibs-foundation#1744

Merged

spevans force-pushed the pr_sr_8876 branch from 839afef to e6aa564 Compare November 2, 2018 17:32

spevans added 3 commits November 2, 2018 17:37

update_checkout.py: Add support for tags and optional repositories.

cb32792

- Update --reset-to-remote to support tags. - Add optional platform entry to a repository. This allows specifying which platforms a repo should be checked out on.

SR-8876: Update for CMake build for Foundation

6188980

- Copy ICU licence into final install.

spevans force-pushed the pr_sr_8876 branch from e6aa564 to 6188980 Compare November 2, 2018 17:37

tkremenek merged commit 7af28e8 into swiftlang:master Nov 2, 2018

shogo82148 added a commit to shogo82148/wandbox-builder that referenced this pull request Jun 2, 2019

workaround for swiftlang/swift#19860

2bc9372

libicu is always built on Linux, because libicu of system is sometimes very old. swiftlang/swift#19860

MrLotU mentioned this pull request Oct 5, 2020

Make Unicode.Scalar.Properties added after ICU 55 available on Linux #34182

Closed

MaxDesiatov mentioned this pull request Nov 4, 2018

[SR-3819] ICU often needs -ldl, even if linked statically #46404

Closed

This was referenced Nov 3, 2018

[SR-5618] libicu compilation should happen in parallel respecting -j #48188

Closed

[SR-8876] Build recent ICU on Linux swiftlang/swift-corelibs-foundation#3625

Closed

SR-8876: Always build libicu on Linux #19860

SR-8876: Always build libicu on Linux #19860

Uh oh!

Conversation

spevans commented Oct 12, 2018

Uh oh!

spevans commented Oct 12, 2018

Uh oh!

swift-ci commented Oct 12, 2018

Uh oh!

swift-ci commented Oct 12, 2018

Uh oh!

parkera commented Oct 12, 2018

Uh oh!

spevans commented Oct 12, 2018

Uh oh!

swift-ci commented Oct 12, 2018

Uh oh!

swift-ci commented Oct 12, 2018

Uh oh!

milseman commented Oct 12, 2018

Uh oh!

spevans commented Oct 12, 2018

Uh oh!

spevans commented Oct 13, 2018

Uh oh!

swift-ci commented Oct 13, 2018

Uh oh!

swift-ci commented Oct 13, 2018

Uh oh!

spevans commented Oct 13, 2018

Uh oh!

swift-ci commented Oct 13, 2018

Uh oh!

swift-ci commented Oct 13, 2018

Uh oh!

spevans commented Oct 14, 2018

Uh oh!

swift-ci commented Oct 14, 2018

Uh oh!

swift-ci commented Oct 14, 2018

Uh oh!

spevans commented Oct 15, 2018

Uh oh!

millenomi commented Oct 15, 2018

Uh oh!

tkremenek commented Oct 16, 2018

Uh oh!

spevans commented Oct 26, 2018

Uh oh!

spevans commented Oct 28, 2018

Uh oh!

swift-ci commented Oct 28, 2018

Uh oh!

swift-ci commented Oct 28, 2018

Uh oh!

tkremenek commented Oct 30, 2018

Uh oh!

tkremenek commented Nov 2, 2018

Uh oh!

spevans commented Nov 2, 2018

Uh oh!

spevans commented Nov 2, 2018

Uh oh!

swift-ci commented Nov 2, 2018

Uh oh!

swift-ci commented Nov 2, 2018

Uh oh!

milseman commented Nov 4, 2018

Uh oh!

shahmishal commented Nov 15, 2018

Uh oh!

millenomi commented Nov 16, 2018

Uh oh!

millenomi commented Nov 16, 2018

Uh oh!

shahmishal commented Nov 16, 2018

Uh oh!

shahmishal commented Nov 16, 2018

Uh oh!