On Linux build LLVM and subprojects with -gsplit-dwarf which is more …

[adrian-prantl]

…space/time

efficient than -g on that platform. This improves time spent to link products
built with debug info quite a bit.

[adrian-prantl]

@swift-ci test

[JDevlieghere]

The change looks good, but the indention seems off.

[adrian-prantl]

Turns out that file mixes tabs & spaces quite a bit.

[adrian-prantl]

@swift-ci test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - e25a751ef68796ad2cf66367d360fbfdc9ac1263

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - e25a751ef68796ad2cf66367d360fbfdc9ac1263

[adrian-prantl]

@swift-ci test

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 9b6ff03

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 9b6ff03

[CodaFi]

Y'all picked up source kit LSP failures

@swift-ci please test

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 9b6ff03

[adrian-prantl]

@swift-ci test

[drodriguez]

@adrian-prantl: Hi. I babysit the Android community builds (https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-16.04-android-arm64/ and https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-16.04-android/). They have been failing from this afternoon. From the commits of the first failing build (https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-16.04-android/4076/), this one is probably the one that introduced the problem, since it seems that the disk space is involved.

From what I can find online, and my understanding, -gsplit-dwarf should be disk space neutral (some sources says it should be beneficial), but the machines were building LLVM, Swift, and stdlib for Linux and Android in less than 30 GB of disk. I tried to increase the disk space to 40 GB, and it managed to get pass LLVM, but hits the disk limit while building Swift (if I understand the changes correctly, Swift should not be affected, but I suppose the extra space of LLVM is taking the space that Swift used to take). When I checked the machine LLVM build directory was a whooping 27 GB. Sadly I cannot increase the disk space further for some hours (sigh).

Does this result makes sense? I don't know if the linking time is better, but at least in those machines the disk space seems to be much worse (by the way, the machines are using ld.bfd for linking, but that seems to be the case for the official Linux CI too).

If you are planning more changes like this (enabling the option for more products, or something like that), can you give me a heads up, so I can have a closer look to the builds and react faster? Thanks!

PD: please, do not revert this, I should be able to find the right amount of disk space. My questions are more about curiosity than anything.

[adrian-prantl]

That is unexpected, I'm sorry! As you said, this is expected to be roughly disk space neutral, with a much lower RAM and CPU usage during link time.

I'd be happy to revert this change, too, if it causes problems. After all it was meant to be to the benefit Linux users, not the opposite.

Why are those bots building with debug info? Is that so you can get better crash backtraces? Would it make sense to turn debug info off on the bots?

Can you post some numbers on disk usage before & after? I'd be curious where the difference comes from. (Whether it's the .o files or the final build products, etc.) I can also try to collect them, but I'm assuming you're already set up for this.

[drodriguez]

Please, as I wrote before, I don't think this should be reverted. I was just asking to know if it makes sense what I was seeing or not. Increasing the disk is easy, and knowing how much is LLVM now makes easy to do the math.

Those bots, as far as I know do not build with debug info, even if the preset is called buildbot_linux_crosscompile_android,tools=RA,stdlib=RD,build, everything is build with release and with assertions, but as far as I know, no debug (it is basically buildbot_linux with the Android stdlib, and disabling many other products).

I have another machine with a very similar configuration to those two where I can do those tests you ask.

[drodriguez]

@adrian-prantl: I am probably wrong, but I was looking at https://github.com/apple/swift-llvm/blob/stable/cmake/modules/HandleLLVMOptions.cmake, and it seems that enabling LLVM_USE_SPLIT_DWARF will add the flag independently of the build type, and if I read https://reviews.llvm.org/D52296 correctly, it seems that passing -gsplit-dwarf implies -g. Shouldn't LLVM_USE_SPLIT_DWARF only be used in build types like Debug and RelWithDebInfo? We might be generating debug info in Linux all the time.

[adrian-prantl]

Good find! I'll update build script then.