[presets] Introduce slimmer preset for lldb sanitizer testing #34504
Conversation
Add a new "buildbot_incremental,lldb_asan_ubsan" preset. This differs from the existing preset used to do sanitized lldb testing on ci.swift.org, "buildbot_incremental_asan_ubsan,tools=RDA,stdlib=RDA", in the following ways: - It builds the swift benchmarks, with an unoptimized benchmark driver. This is to enable stepper testing on CI. For more context, see: swiftlang/llvm-project#2059. - It does not build a sanitized clang or swift. This can save a large amount of time, and make reports from oss-lldb-asan bots more useful. This is still WIP pending more testing. Note: A dry run for the existing buildbot_incremental_asan_ubsan preset does not apply '--skip-build-benchmarks': ``` ./swift/utils/build-script -n --preset "buildbot_incremental_asan_ubsan,tools=RDA,stdlib=RDA" ``` So I'm not yet sure where that comes from. Perhaps the oss-lldb-asan-osx bot is using a different preset? rdar://69589638
vedantk
requested review from
kastiglione,
JDevlieghere,
adrian-prantl and
shahmishal
|
I just realized: if the oss-lldb-asan-osx bot is indeed using the "buildbot_incremental_asan_ubsan,tools=RDA,stdlib=RDA" preset, it might be simpler to just edit that preset instead of introducing a new preset and changing the bot config. |
|
I'm still not 100% sure, but it's starting to look like the oss-lldb-asan-osx job doesn't use a preset from build-presets.ini. E.g., I don't see it passing a '--preset' option, and I do see it passing '--lldb-build-with-cmake' which no preset does anymore (afaik). I'll follow up with @shahmishal to figure out whether we want to check in a new preset or simply tweak the bot config. |
|
@vedantk It's not using preset, if you have a preset I can update the job. |
vedantk
changed the title
|
Thanks @shahmishal! Can we change the oss-lldb-*asan-osx configs to: |
|
@shahmishal would it be more maintainable/sustainable to check in the preset, and to then change the oss-lldb-asan-osx bots to do: That way, we can modify the preset when the testing needs change without bugging you, and we retain the sccache integration on the bot. |
|
Sounds good, let's get this PR merged and I will update the bot. |
|
preset=buildbot_incremental,lldb_asan_ubsan |
|
Looks like java.io.EOFException, doesn't look like any of the compiler invocations failed. swift is (correctly) not being built with sanitization enabled. |
|
preset=buildbot_incremental,lldb_asan_ubsan |
|
Progress! This time we time out after hitting some container-overflow FPs in the unit tests. That will be addressed by swiftlang/llvm-project#2087. |
|
preset=buildbot_incremental,lldb_asan_ubsan |
|
@shahmishal it looks like the preset is behaving exactly as expected/desired (https://ci.swift.org/job/swift-PR-osx-preset/130/consoleText); however, the test job fails because the bot is not configured to allow debugging ("(this is a non-interactive debug session, cannot get permission to debug processes.)"). Can we merge this and switch the oss-lldb-*-asan-osx bots over? |
|
Sounds good, let's merge it. |
|
@swift-ci please smoke test and merge |
Add a new "buildbot_incremental,lldb_asan_ubsan" preset.
This differs from the existing "buildbot_incremental_asan_ubsan,tools=RDA,stdlib=RDA"
preset in the following ways:
It builds the swift benchmarks, with an unoptimized benchmark driver.
This is to enable stepper testing on CI. For more context, see:
[test/Shell] Add stepper test for Benchmark_Onone llvm-project#2059.
It does not build a sanitized clang or swift. This can save a lot of
time, and make reports from oss-lldb-asan bots more actionable.
It also cuts down on redundant work, since clang & swift already have
sanitizer bots. This does trade away some sanitized test coverage.
rdar://69589638