Manually merge apple/stable/20190619-> swift/master up to bitstream remarks #298
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
so we only call ModulesDidLoad at the end of the method after the new module has been added to the target and the sections have all been adjusted to their actual load addresses. Solves a problem where an operating system plugin in the kernel could be loaded multiple times; the first before the binary had even been added to the target. <rdar://problem/50523558> (cherry picked from commit 6602e1f)
Currently nothing prevents you from continuing your debug session after generating the reproducer. This can cause the reproducer to end up in an inconsistent state. Most of the time this doesn't matter, but I want to prevent this from causing bugs in the future. (cherry picked from commit 0cf86da)
Remove stale copy of moved test. (NFC)
…-read-script-fix Reordering KextImageInfo::LoadImageUsingMemoryModule
…b1a4278dc31dfc8f1538c9186892 [Reproducer] Quit the debugger after generating a reproducer
…latforms. Attempt llvm#3. The main problem here is that `-*-version_min=` was not being passed to the compiler when building test cases. This can cause problems when testing on devices running older OSs because Clang would previously assume the minimum deployment target is the the latest OS in the SDK which could be much newer than what the device is running. Previously the generated value looked like this: `-arch arm64 -isysroot <path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk` With this change it now looks like: `-arch arm64 -stdlib=libc++ -miphoneos-version-min=8.0 -isysroot <path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk` This mirrors the setting of config.target_cflags on macOS. This change is made for ASan, LibFuzzer, TSan, and UBSan. To implement this a new `get_test_cflags_for_apple_platform()` function has been added that when given an Apple platform name and architecture returns a string containing the C compiler flags to use when building tests. This also calls a new helper function `is_valid_apple_platform()` that validates Apple platform names. This is the third attempt at landing the patch. The first attempt (r359305) had to be reverted (r359327) due to a buildbot failure. The problem was that calling `get_test_cflags_for_apple_platform()` can trigger a CMake error if the provided architecture is not supported by the current CMake configuration. Previously, this could be triggered by passing `-DCOMPILER_RT_ENABLE_IOS=OFF` to CMake. The root cause is that we were generating test configurations for a list of architectures without checking if the relevant Sanitizer actually supported that architecture. We now intersect the list of architectures for an Apple platform with `<SANITIZER>_SUPPORTED_ARCH` (where `<SANITIZER>` is a Sanitizer name) to iterate through the correct list of architectures. The second attempt (r363633) had to be reverted (r363779) due to a build failure. The failed build was using a modified Apple toolchain where the iOS simulator SDK was missing. This exposed a bug in the existing UBSan test generation code where it was assumed that `COMPILER_RT_ENABLE_IOS` implied that the toolchain supported both iOS and the iOS simulator. This is not true. This has been fixed by using the list `SANITIZER_COMMON_SUPPORTED_OS` for the list of supported Apple platforms for UBSan. For consistency with the other Sanitizers we also now intersect the list of architectures with UBSAN_SUPPORTED_ARCH. rdar://problem/50124489 Differential Revision: https://reviews.llvm.org/D61242 llvm-svn: 373405 (cherry picked from commit bbde056)
The bitstream reader handles errors poorly. This has two effects: * Bugs in file handling (especially modules) manifest as an "unexpected end of file" crash * Users of clang as a library end up aborting because the code unconditionally calls `report_fatal_error` The bitstream reader should be more resilient and return Expected / Error as soon as an error is encountered, not way late like it does now. This patch starts doing so and adopting the error handling where I think it makes sense. There's plenty more to do: this patch propagates errors to be minimally useful, and follow-ups will propagate them further and improve diagnostics. https://bugs.llvm.org/show_bug.cgi?id=42311 <rdar://problem/33159405> Differential Revision: https://reviews.llvm.org/D63518 llvm-svn: 364464 (cherry picked from commit 0e82895) Conflicts: clang-tools-extra/clangd/ClangdUnit.cpp clang/lib/Serialization/GlobalModuleIndex.cpp
On the armv8 bot the failure is slightly different in the number it prints. Don't check the numbers. This was caused by r364464. llvm-svn: 364488 (cherry picked from commit cf9ee06df81e18f0863a71a32bd4f9a8f16ce0d6)
It was failing with:
In file included from /b/s/w/ir/cache/builder/src/third_party/llvm/llvm/lib/Bitcode/Reader/BitstreamReader.cpp:9:0:
/b/s/w/ir/cache/builder/src/third_party/llvm/llvm/include/llvm/Bitcode/BitstreamReader.h:
In member function 'llvm::Expected<long unsigned int> llvm::SimpleBitstreamCursor::ReadVBR64(unsigned int)':
/b/s/w/ir/cache/builder/src/third_party/llvm/llvm/include/llvm/Bitcode/BitstreamReader.h:262:14:
error: could not convert 'MaybeRead' from 'llvm::Expected<unsigned int>' to 'llvm::Expected<long unsigned int>'
return MaybeRead;
^
/b/s/w/ir/cache/builder/src/third_party/llvm/llvm/include/llvm/Bitcode/BitstreamReader.h:279:16:
error: could not convert 'MaybeRead' from 'llvm::Expected<unsigned int>' to 'llvm::Expected<long unsigned int>'
return MaybeRead;
^
llvm-svn: 364504
(cherry picked from commit 527c9713aa1db3028a8047ddbf60a9bbd506aefd)
/work/llvm.monorepo/llvm/lib/Bitcode/Reader/BitcodeReader.cpp: In function
‘llvm::Expected<std::basic_string<char> > readIdentificationBlock(llvm::BitstreamCursor&)’:
/work/llvm.monorepo/llvm/lib/Bitcode/Reader/BitcodeReader.cpp:205:22:
warning: unused variable ‘BitCode’ [-Wunused-variable]
switch (unsigned BitCode = MaybeBitCode.get()) {
^
/work/llvm.monorepo/llvm/lib/Bitcode/Reader/BitcodeReader.cpp: In member function
‘llvm::Error {anonymous}::ModuleSummaryIndexBitcodeReader::parseModule()’:
/work/llvm.monorepo/llvm/lib/Bitcode/Reader/BitcodeReader.cpp:5367:26:
warning: unused variable ‘BitCode’ [-Wunused-variable]
switch (unsigned BitCode = MaybeBitCode.get()) {
^
llvm-svn: 364505
(cherry picked from commit ff4e0ac0a4e9596fd532fc93b895f03f2e8d4b7a)
Summary:
After rL364464 the following tests started to fail when
running the clang-doc tests with an ubsan instrumented
build of clang-doc:
Clang Tools :: clang-doc/single-file-public.cpp
Extra Tools Unit Tests :: clang-doc/./ClangDocTests/BitcodeTest.emitEnumInfoBitcode
Extra Tools Unit Tests :: clang-doc/./ClangDocTests/BitcodeTest.emitMethodInfoBitcode
Extra Tools Unit Tests :: clang-doc/./ClangDocTests/BitcodeTest.emitRecordInfoBitcode
Extra Tools Unit Tests :: clang-doc/./ClangDocTests/SerializeTest.emitInfoWithCommentBitcode
We need to check that the read value is in range for being
casted to the llvm::bitc::FixedAbbrevIDs enum, before the
cast in ClangDocBitcodeReader::skipUntilRecordOrBlock.
SerializedDiagnosticReader::skipUntilRecordOrBlock was updated
in the same way.
Reviewers: jfb
Reviewed By: jfb
Subscribers: Bigcheese, vsapsai, bruno, ilya-biryukov, dexonsmith, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64262
llvm-svn: 365239
(cherry picked from commit 401fcb5422fd1a463bf77c2322e52eed976ac509)
Upstream change in rL364464 broke downstream APINotes. apple-llvm-split-commit: c0ab2b05f0bbbb9c667e3f2421bfdb140ae0c119 apple-llvm-split-dir: clang/ (cherry picked from commit af93446)
Upstream change in rL364464 broke downstream index while building. apple-llvm-split-commit: 994094aa63b43bc0bc2d900ce27c0f376ac10f5c apple-llvm-split-dir: clang/ (cherry picked from commit d779dda)
This moves Bitcode/Bitstream*, Bitcode/BitCodes.h to Bitstream/. This is needed to avoid a circular dependency when using the bitstream code for parsing optimization remarks. Since Bitcode uses Core for the IR part: libLLVMRemarks -> Bitcode -> Core and Core uses libLLVMRemarks to generate remarks (see IR/RemarkStreamer.cpp): Core -> libLLVMRemarks we need to separate the Bitstream and Bitcode part. For clang-doc, it seems that it doesn't need the whole bitcode layer, so I updated the CMake to only use the bitstream part. Differential Revision: https://reviews.llvm.org/D63899 llvm-svn: 365091 (cherry picked from commit e030827) rdar://57040003
This allows us to use the analyzer from unit tests. * Refactor the interface to use proper error handling for most functions after JF's work. * Move everything into a BitstreamAnalyzer class. * Move that to Bitcode/BitcodeAnalyzer.h. Differential Revision: https://reviews.llvm.org/D64116 llvm-svn: 365286 (cherry picked from commit 4cdb68e) Conflicts: llvm/tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp
llvm-svn: 365340 (cherry picked from commit d6fd354)
This fixes a modules issue: error: declaration of 'bitc' must be imported from module 'Clang_Serialization.ASTBitCodes' before it is required Stream.EmitRecord(llvm::bitc::BLOCKINFO_CODE_SETBID, Record); llvm-svn: 365405 (cherry picked from commit 315ce83)
This fixes a modules issue. llvm-svn: 365580 (cherry picked from commit 1a697aa)
This header is not in upstream llvm and needs to be updated to use the new paths for bitstream. apple-llvm-split-commit: 2b6648be8ebe2964c93e454764dfdec49c3abbd2 apple-llvm-split-dir: llvm/ (cherry picked from commit 2826129)
apple-llvm-split-commit: 152b357e96ce96dfe46ac419c51e1ced09dd7d5f apple-llvm-split-dir: clang/ (cherry picked from commit 9e2de38)
It was showing up in a lot of unrelated tools. llvm-svn: 365647 (cherry picked from commit 3700736)
…uctor Make the parser take an already-parsed string table. llvm-svn: 365101 (cherry picked from commit e6ba313)
Make the parser require an explicit format. This allows new formats to be easily added by following YAML as an example. llvm-svn: 365102 (cherry picked from commit 312f1d7)
It's useless to have both. llvm-svn: 366216 (cherry picked from commit cc90981)
Before, everything was based on some kind of type erased parser implementation which container a lot of boilerplate code when multiple formats were to be supported. This simplifies it by: * the remark now owns its arguments * *always* returning an error from the implementation side * working around the way the YAML parser reports errors: catch them through callbacks and re-insert them in a proper llvm::Error * add a CParser wrapper that is used when implementing the C API to avoid cluttering the C++ API with useless state * LLVMRemarkParserGetNext now returns an object that needs to be released to avoid leaking resources * add a new API to dispose of a remark entry: LLVMRemarkEntryDispose llvm-svn: 366217 (cherry picked from commit 94bad22)
Add tests for both the string table and non string table case. llvm-svn: 366832 (cherry picked from commit 78c92d2)
llvm-svn: 366842 (cherry picked from commit cbbdc41)
This exposes better support to use a string table with a format through an actual new remark::Format, called yaml-strtab. This can now be used with -fsave-optimization-record=yaml-strtab. llvm-svn: 366849 (cherry picked from commit c5b5cc4)
Emit a remarks section by default for the following formats: * bitstream * yaml-strtab while still providing -remarks-section=<bool> to override the defaults. (cherry picked from commit 209d5a1)
There is no ELF support at the moment. Remove all the references to the `.remarks` section. (cherry picked from commit c7557dd)
(cherry picked from commit 025166c)
(cherry picked from commit f5094e1)
Remarks are usually emitted per-TU, and for generating a standalone remark file that can be shipped with the linked binary we need some kind of tool to merge everything together. The remarks::RemarkLinker class takes care of this and: * Deduplicates remarks * Filters remarks with no debug location * Merges string tables from all the entries As an output, it provides an iterator range that can be used to serialize the remarks to a file. Differential Revision: https://reviews.llvm.org/D69141 (cherry picked from commit e72e59e)
Always compare using lowercase to support multiple platforms. (cherry picked from commit 6b82988)
This adds support to dsymutil for linking remark files and placing them in the final .dSYM bundle. The result will be placed in: * a.out.dSYM/Contents/Resources/Remarks/a.out or * a.out.dSYM/Contents/Resources/Remarks/a.out-<arch> for universal binaries When multi-threaded, this runs a third thread which loops over all the object files and parses remarks as it finds __remarks sections. Testing this involves running dsymutil on pre-built binaries and object files, then running llvm-bcanalyzer on the final result to check for remarks. Differential Revision: https://reviews.llvm.org/D69142 (cherry picked from commit 1ee18f2)
dsymutil uses lipo(1) to build the fat binary, which it invokes as a process. For that, we need to only run this test on darwin systems. Should fix: http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-ubuntu/builds/181/steps/test-check-all/logs/stdio (cherry picked from commit 86cdf74)
(cherry picked from commit cc9e963)
The other paremeters appear to be sufficient to determine which modules have just been loaded and need to be removed, so stop collecting and sending in that set explicitly.
…light As part of an audit of whether all errors are being reported from the ASTReader, delay err_module_file_conflict if a diagnostic is already in flight when it is hit. This required plumbing an extra argument through the delayed diagnostic mechanics in DiagnosticsEngine.
Split a loop in ReadAST that visits the just-loaded module chain, between an initial loop that reads further from the ASTs (and can fail) and a second loop that does some preloading (and cannot fail). This makes it less likely for a reading failure to affect the AST. This is not fixing a known bug and the behaviour change may not be observable, it's just part of an audit to look at all of the error handling in the ASTReader. https://reviews.llvm.org/D70056
These were found via an audit. In the case of `ParseLineTable` this is actually dead code, since parsing the line table always succeeds, but it's prudent to be defensive since it's possible an assertion there could be converted to a `true` return in the future.
ReadASTBlock and ReadASTExtensions can both return failures. Be consistent and remove all the just-loaded modules, just like when ReadASTCore returns failures. https://reviews.llvm.org/D70055
If ReadASTBlock does not find its top-level submodule, there's something wrong the with the PCM. Error in that case, to avoid hitting problems further from the source. Note that the Swift compiler sometimes hits a case in CompilerInstance::loadModule where the top-level submodule mysteriously does not have Module::IsFromModuleFile set. That will emit a confusing warn_missing_submodule, which was never intended for the main module. The recent audit of error-handling in ReadAST may have rooted out the real problem. If not, this commit will help to clarify the real problem, and replace a confusing warning with an error pointing at the malformed PCM file. We're specifically sniffing out whether the top-level submodule was found/processed, in case there is a malformed module file that is missing it. If there is an error encountered during ReadSubmoduleBlock the return status should already propagate through. It would be nice to detect other missing submodules around here to catch other instances of warn_missing_submodule closer to the source, but that's left as a future exercise. https://reviews.llvm.org/D70063
…able/20190619 Cherry-pick remark infrastructure work
…ry-picks-for-missing-submodule
…hat require scanning symbols Performance issues lead to the libc++ std::function formatter to be disabled. This change is the first of two changes that should address the performance issues and allow us to enable the formatter again. In some cases we end up scanning the symbol table for the callable wrapped by std::function for those cases we will now cache the results and used the cache in subsequent look-ups. This still leaves a large cost for the initial lookup which will be addressed in the next change. Differential Revision: https://reviews.llvm.org/D67111 (cherry picked from commit e18f4db)
…on_formatter [LLDB] Adding caching to libc++ std::function formatter for lookups that require scanning symbols
…erry-picks-for-missing-submodule apple/stable/20190619: Another round of modules fixes for 'missing submodule X' error on Linux
…slots. `Selector::getIdentifierInfoForSlot` returns NULL if a slot has no corresponding identifier. Add a boolean to the hash and a NULL check. rdar://problem/51615164 Reviewers: rtrieu Reviewed By: rtrieu Subscribers: dexonsmith, cfe-commits, jkorous Differential Revision: https://reviews.llvm.org/D63789 llvm-svn: 364664 (cherry picked from commit 5f8b909)
…prove non-cached lookup performance Performance issues lead to the libc++ std::function formatter to be disabled. We addressed some of those performance issues by adding caching see D67111 This PR fixes the first lookup performance by not using FindSymbolsMatchingRegExAndType(...) and instead finding the compilation unit the std::function wrapped callable should be in and then searching for the callable directly in the CU. Differential Revision: https://reviews.llvm.org/D69913 (cherry picked from commit 91e94a7) Conflicts: lldb/source/Plugins/LanguageRuntime/CPlusPlus/CPPLanguageRuntime.cpp
|
@swift-ci please test |
…rmatter_fix_non_cache_performance [LLDB][Formatters] Re-enable std::function formatter with fixes to improve non-cached lookup performance
With all the previous refactorings this slipped through and now we always dump the contents of the bitcode files, even if -dump is not passed. (cherry picked from commit 1ca85b3)
(cherry picked from commit 3dfe4cf)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to be able to test the related swift changes we need to manually create a merge PR and test the two together.
Unfortunately, this also comes with some other commits that are tested together, but at this point we can't do anything about it.