build static foundation

[weissi]

the first commit introduces a StaticAndDynamicLibrary product (sorry about the multiple inheritance 😨, more than happy to change if someone has a better idea) and which builds libFoundation.a on top of libFoundation.so.

The other commit is just a simple fix, the ar command shouldn't be given the compiler flags.

[alblue]

@swift-ci please test

[spevans]

@weissi Good to see this being worked, I just have a couple of points:

1. Should there be an option to enable building a static lib rather than doing it by default? There may be some platforms that dont support them.

2. I just did a test and install on my local linux host and whilst it built both libFoundation.so and libFoundation.a correctly, only libFoundation.so was installed in the final install directory.

3. Given that there are some specific linker flags eg defsym=__CFConstantStringClassReference=_TMC10Foundation19_NSCFConstantString, should these be inserted into the final libFoundation.a so they can be extracted using swift-autolink-extract? I noticed the following flags are embedded,

-lswiftCore
-ldispatch
-lswiftGlibc
-lpthread
-lutil
-ldl

but it may be useful to insert the others so that they dont need to be added to each build script that uses libFoundation.

[weissi]

hey @spevans thanks very much for your comments, replies individually

1. well, all systems I know of that support dynamic libs, support static ones too. So I thought that if we're building the dynamic one, the static one should be fine. But it's a change so if there's strong feelings towards making this configurable, I don't have a problem with that at all.

2. thanks! should be fixed

3. that's a very good point, definitely want to get that working. Unfortunately I don't have time to research that today (and will be on holiday until the end of next week) but will work on that after I'm back. Any pointers on how to achieve that would also be appreciated.

Another change I was thinking about is to remove the ugly multiple inheritance and create a StaticLibrary("Foundation") object from the DynamicLibrary("Foundation") object. I could for example just assign the .phases list but that also feels like a hack.

Any ideas how to improve this patch would be greatly appreciated.

[spevans]

@weissi I was only suggesting it as the other repos (libdispatch and swift) take an option to build static libs. However I agree with you that its pretty easy to fix if it needs to be, and building both together does save on having to add another option into swift/utils/build-script-impl for now.

2. Looks good on the install, I now see

./usr/lib/swift_static/linux/libFoundation.a
./usr/lib/swift/linux/libFoundation.so

3. I dont know if there is an official way but you could use something like the following (or add it into the python code), although I havent actually tested it with linking

autolink.sh:

#!/bin/bash

output=$1
shift

read -r -d '' CODE <<EOF
// These commands will get extracted by 'swift-autolink-extract'
#define AUTOLINK_ENTRY(x) asm(".asciz \"" x "\"")

asm(".section .swift1_autolink_entries");
EOF

for arg in "$@"
do
    CODE="$CODE"$'\n'"AUTOLINK_ENTRY("\"${arg}\"");"
done

clang++ -Wall -Werror -c -x c++ -o $output - <<<"$CODE"

$ ./autolink.sh foo.o -some -linker -arguments
$ swift-autolink-extract foo.o
-some
-linker
-arguments
$ ar r libFoundation.a foo.o
$ swift-autolink-extract libFoundation.a|tail                                   
-ldispatch
-lswiftGlibc
-lpthread
-lutil
-ldl
-lswiftCore
-ldispatch
-some
-linker
-arguments

[spevans]

@weissi Just to let you know that I was able to use this PR and some other changes I did to build a static version of TestFoundation on linux which runs ok. Also, I think you can ignore my pt 3) for now, some kind of autolink data does need to be added but Im still trying to see what the best way to do the linking is.

[alblue]

@swift-ci please test

[alblue]

@phausler can you review this change please?

[weissi]

@spevans thanks for letting me know and sorry for not responding earlier 🤕. Would you want to add your changes to this PR and make it a shared one or rather open another one after this goes it (after potentially some changes).

[spevans]

@weissi I'll do my changes in a separate PR since they are not finished yet, and I still need to check if some of the changes I made are acceptable 😀

[phausler]

overall the changes here look reasonable

[spevans]

@weissi 1 quick question, is it possible to use different compile flags when compiling the libFoundation.a specifically to add some -D flags. I realise that this will involve a 2nd compile of CoreFoundation but when linking statically I compile foundation with something like

#if BUILD_STATIC_LIBRARY
#undef BINARY_SUPPORT_DYLD
#undef BINARY_SUPPORT_DLFCN
#undef BINARY_SUPPORT_DLL
#endif

(there are some other changes as well)

and pass -DBUILD_STATIC_LIBRARY to disable the use of libdl functions used by the Bundle code which obviously cant be disabled in the shared library.

[phausler]

BINARY_SUPPORT_DYLD, BINARY_SUPPORT_DLFCN, BINARY_SUPPORT_DLL all have to do with what CFBundle can do when loading a bundle not about when the library itself is compiled as static.

[spevans]

@phausler Agreed, its just that I have been testing creating a fully statically linked TestFoundation and this meant not linking in libdl or using any dl functions. Of course it may end up that ultimately I may need to build a libFoundationStatic.a and libFoundationShared.a (as was done with the swift stdlib with libswiftImageInspectionShared.a and libswiftImageInspectionStatic.a)

[phausler]

the issue I think that might be at hand here is the dropping of functions like the initialization routines ala __CFInitialize (which is a pre-requiste for making CF and consequently Foundation work)

[spevans]

I actually managed to build TestFoundation as a fully static binary including libcurl and dependancies and it ran all of the tests successfully. Of course not everything is fully covered by the tests and I understand that plugin capability is lost but for Linux it seems to be ok. Obviously this wouldn't apply to Darwin

[phausler]

If the tests run it is a good sign that __CFInitialize is being called; if it wasn't it would be crashing all over the place since we assume layouts are setup by that time.

[spevans]

Yes __CFInitialize is not wrapped inside any BINARY_SUPPORT conditionals. The only code that seems to get excluded is dlopen() etc used for bundle & plugin loading.

[phausler]

static libraries can still call dlopen; I don't see why we should disallow that when building as a .a versus a .so

[spevans]

Sorry, I was referring to using static libraries inside a statically linked binary but Ive just thought of a solution that doesnt require undef'ing the BINARY_SUPPORT conditionals anyway, so I'll stop wittering on now 😀

[weissi]

@spevans With the current implementation it'd be hard to pass additional compilation flags to the static version. The reason being that I use the same .o files for the .so and the .a version. And given that the .a file is really just an archive of .os I can't currently add any -D or other flags. If that's really needed then we'd need to compile them twice.

[spevans]

@weissi That's fine, I have thought of a different solution for static linking instead of using additional compilation flags so I dont have any issues with this PR.

[weissi]

@phausler any reservations to merge this one?

[aciidgh]

@phausler @parkera Can someone please take a look here?

[parkera]

Let's give this a try.

[parkera]

@swift-ci test and merge

[pushkarnk]

I guess this is generating warnings of the kind: builds involving this target will not be correct; continuing anyway for every target.

[pushkarnk]

The build.ninja has two rules for each target. Example:
$grep "CompileSwift Foundation/NSString.swift" build.ninja | wc -l returns 2