[SR-755] Tool for re-symbolicating fatal stacktraces on Linux. #4479

gmilos · 2016-08-24T17:51:59Z

What's in this pull request?

On Linux stack traces are full of symbols. This provides a tool to re-symbolicate such stack trace.

The cause is difference in behaviour of dladdr between Linux & Darwin. See SR.

Resolved bug number: (SR-755)

Before merging this pull request to apple/swift repository:

Test pull request on Swift continuous integration.

Triggering Swift CI

The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:

Smoke Testing

Platform	Comment
All supported platforms	@swift-ci Please smoke test
All supported platforms	@swift-ci Please smoke test and merge
OS X platform	@swift-ci Please smoke test OS X platform
Linux platform	@swift-ci Please smoke test Linux platform

A smoke test on macOS does the following:

Builds the compiler incrementally.
Builds the standard library only for macOS. Simulator standard libraries and
device standard libraries are not built.
lldb is not built.
The test and validation-test targets are run only for macOS. The optimized
version of these tests are not run.

A smoke test on Linux does the following:

Builds the compiler incrementally.
Builds the standard library incrementally.
lldb is built incrementally.
The swift test and validation-test targets are run. The optimized version of these
tests are not run.
lldb is tested.

Validation Testing

Platform	Comment
All supported platforms	@swift-ci Please test
All supported platforms	@swift-ci Please test and merge
OS X platform	@swift-ci Please test OS X platform
OS X platform	@swift-ci Please benchmark
Linux platform	@swift-ci Please test Linux platform

Lint Testing

Language	Comment
Python	@swift-ci Please Python lint

Note: Only members of the Apple organization can trigger swift-ci.

DougGregor · 2016-08-24T17:57:55Z

@swift-ci please smoke test

gmilos · 2016-08-26T11:06:48Z

The failure on Linux was a timeout, doesn't seem to be related to the change I made. Trying to kick off another round of ci.

gmilos · 2016-08-26T11:06:53Z

@swift-ci please smoke test

gmilos · 2016-08-26T11:08:52Z

@swift-ci Please test Linux platform

DougGregor · 2016-08-26T16:17:44Z

@swift-ci please smoke test Linux

modocache · 2016-08-26T19:47:18Z

stdlib/public/runtime/Errors.cpp

@@ -72,7 +72,6 @@ static bool getSymbolNameAddr(llvm::StringRef libraryName, Dl_info dlinfo,
  // If the address is unavailable, just use <unavailable> as the symbol


nit-pick: Since you remove the "" string below, I think this comment here might no longer accurate. Could you update this comment as well?

modocache · 2016-08-27T03:34:11Z

You'll want to run the Python linter on this new code, which contains a few violations. There's a handy utility to run the linter, just run utils/python-lint.

modocache · 2016-08-27T03:37:31Z

utils/symbolicate-linux-fatal.py

+
+dynlibs = {}
+def process_ldd(lddoutput):
+    global dynlibs


This line, global dynlibs, is not needed. Since you define dynlibs in a global scope above, you are able to reference it throughout this file. This is why you're able to reference it in the process_stack function below.

Also, I believe the Python linter will raise this issue, but I think that typically Python globals tend to be in UPPER_SNAKE_CASE.

gmilos · 2016-08-31T18:38:40Z

@modocache re utils/python-lint. I run it + fixed up all the issues it found. Thanks for the suggestion.

gmilos · 2016-09-01T10:54:26Z

@DougGregor could you kick off another round of tests? I've addressed all the comments, so hopefully this is ready to be merged, pending successful tests.

modocache · 2016-09-01T13:51:13Z

utils/symbolicate-linux-fatal

+
+
+def main():
+    global BINARY


Instead of storing args.binary below into this global BINARY variable, I would suggest passing binary as an argument into the create_lldb_target() and process_stack() functions. I think doing so limits the scope where BINARY is used and makes the control flow of the program easier to understand.

gmilos · 2016-09-01T15:32:59Z

@modocache All comments addressed. @DougGregor ready for CI again.

gribozavr · 2016-09-01T16:39:03Z

stdlib/public/runtime/Errors.cpp

+  if (foundSymbol) {
+    static const char *backtraceEntryFormat = "%-4u %-34s 0x%0.16lx %s + %td\n";
+    fprintf(stderr, backtraceEntryFormat, index, libraryName.data(), symbolAddr,
+          symbolName.c_str(), ptrdiff_t(uintptr_t(framePC) - symbolAddr));


In C++ code, the line after the linebreak should be aligned with the opening paren. (Here, symbolName should start in the same column as stderr.)

gribozavr · 2016-09-01T16:39:33Z

The code changes generally LGTM, but it would be good to add some tests.

gribozavr · 2016-09-01T16:39:37Z

@swift-ci Please smoke test

gribozavr · 2016-09-01T16:39:41Z

@swift-ci Please Python lint

gribozavr · 2016-09-01T20:31:22Z

@swift-ci Please smoke test

gribozavr · 2016-09-01T20:31:53Z

@swift-ci Please Python lint

rintaro · 2016-09-14T02:28:27Z

@swift-ci Please smoke test

modocache · 2016-09-14T03:11:27Z

The tests appear to have passed, but in reality the new test was never run. From the logs:

12:04:04 UNSUPPORTED: Swift(linux-x86_64) :: Runtime/linux-fatal-backtrace.swift (266 of 8611)

gmilos · 2016-09-14T15:58:34Z

@modocache @rintaro There was a bug working out if lldb PRODUCT is enabled. This is now fixed. Could you please kick off Smoke Tests (where I expect the test to be run) and regular tests (where I don't). I have now tested this locally (I finally understand enough about how the presets work to reproduce, hopefully equivalent, test run locally).

…l-stacktrace-symbolication

gmilos · 2016-09-15T13:20:12Z

@modocache @rintaro Could either one of you kick off Smoke Tests + regular Tests? (I don't have the rights to do that :( ). I've just resolved a superficial merge conflict.

gmilos · 2016-09-16T14:22:18Z

@DougGregor @gribozavr Any chance either one of you could kick off Smoke + regular tests please?

gribozavr · 2016-09-16T16:15:27Z

@swift-ci Please smoke test

gribozavr · 2016-09-16T16:16:22Z

@swift-ci Please test

gribozavr · 2016-09-16T16:18:58Z

@swift-ci Please smoke test

gribozavr · 2016-09-16T16:19:17Z

@swift-ci Please test

DougGregor · 2016-09-20T03:24:21Z

Tests pass, but now we have a conflict. Argh! @gmilos , can you clear up the conflict so we can merge?

…l-stacktrace-symbolication

gmilos · 2016-09-20T17:26:14Z

@DougGregor yep, simple conflict in .pep8 now fixed (hit that before too).

Another round of tests would be very useful. Could you kick them off for me please? Alternatively, if you added me to the Apple group, I could make sure the tests are all green before asking for review/merge.

DougGregor · 2016-09-20T22:41:31Z

@swift-ci please smoke test and merge

DougGregor · 2016-09-20T22:52:10Z

@swift-ci please test

swift-ci · 2016-09-20T22:55:55Z

Build failed
Jenkins build - Swift Test Linux Platform
Git Commit - 74ff4da
Test requested by - @DougGregor

gmilos · 2016-09-20T22:57:40Z

Build failure is unrelated to the patch:

15:56:37 FAILED: /usr/bin/clang++   -DCLANG_ENABLE_ARCMT -DCLANG_ENABLE_OBJC_REWRITER -DCLANG_ENABLE_STATIC_ANALYZER -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/lib/CodeGen -I/home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/llvm/tools/clang/lib/CodeGen -I/home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/llvm/tools/clang/include -Itools/clang/include -Iinclude -I/home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/llvm/include -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Werror=date-time -std=c++11 -fcolor-diagnostics -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3    -UNDEBUG  -fno-exceptions -fno-rtti -MMD -MT tools/clang/lib/CodeGen/CMakeFiles/clangCodeGen.dir/CGDebugInfo.cpp.o -MF tools/clang/lib/CodeGen/CMakeFiles/clangCodeGen.dir/CGDebugInfo.cpp.o.d -o tools/clang/lib/CodeGen/CMakeFiles/clangCodeGen.dir/CGDebugInfo.cpp.o -c /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/llvm/tools/clang/lib/CodeGen/CGDebugInfo.cpp
15:56:37 
/home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/llvm/tools/clang/lib/CodeGen/CGDebugInfo.cpp:3572:64: error: cannot initialize a parameter of type 'llvm::DIExpression *' with an lvalue of type 'llvm::GlobalVariable *'
15:56:37                                        Var->hasLocalLinkage(), Var, nullptr);

gmilos · 2016-09-20T23:07:27Z

Same failure is present here:
https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-15_10/7724/consoleFull#-1191272582ee1a197b-acac-4b17-83cf-a53b95139a76
(that shows the offending set of changes too).

@DougGregor would you mind kicking of another set of tests on this PR (both smoke and regular) once the above llvm issue is fixed?

swift-ci · 2016-09-20T23:09:15Z

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - 74ff4da
Test requested by - @DougGregor

DougGregor · 2016-09-21T16:05:52Z

@swift-ci please smoke test

DougGregor · 2016-09-21T17:13:16Z

Linux PR test passed here: https://ci.swift.org/view/Pull%20Request/job/swift-PR-Linux-smoke-test/1340/

macOS PR test passed here: https://ci.swift.org/view/Pull%20Request/job/swift-PR-osx-smoke-test/1845/

Document how symbolicate-linux-fatal tool from swiftlang/swift#4479 can be used to add symbols to backtraces on Linux

[pull] swiftwasm from main

modocache reviewed Aug 26, 2016
View reviewed changes

modocache reviewed Aug 27, 2016
View reviewed changes

gmilos force-pushed the SR-755-linux-fatal-stacktrace-symbolication branch 2 times, most recently from d252514 to 84cc469 Compare August 31, 2016 18:37

modocache reviewed Sep 1, 2016
View reviewed changes

gribozavr reviewed Sep 1, 2016
View reviewed changes

Fix bash PRODUCT enumeration, this makes test run OK.

e819ae9

Merge remote-tracking branch 'upstream/master' into SR-755-linux-fata…

74ff4da

…l-stacktrace-symbolication

Merge remote-tracking branch 'upstream/master' into SR-755-linux-fata…

f96b655

…l-stacktrace-symbolication

DougGregor merged commit f1a4dd7 into swiftlang:master Sep 21, 2016

rintaro mentioned this pull request Feb 23, 2017

Use get_python_lib to look for lldb python lib dir. #7685

Merged

jmikola added a commit to mongodb/mongo-swift-driver that referenced this pull request Jul 23, 2018

Instructions for adding symbols to Linux backtraces

ccd4e98

Document how symbolicate-linux-fatal tool from swiftlang/swift#4479 can be used to add symbols to backtraces on Linux

jmikola mentioned this pull request Jul 23, 2018

Instructions for adding symbols to Linux backtraces mongodb/mongo-swift-driver#98

Merged

jmikola added a commit to mongodb/mongo-swift-driver that referenced this pull request Jul 24, 2018

Instructions for adding symbols to Linux backtraces (#98)

f9553cd

Document how symbolicate-linux-fatal tool from swiftlang/swift#4479 can be used to add symbols to backtraces on Linux

swift-ci mentioned this pull request Aug 24, 2016

[SR-755] [Linux] Stack traces do not include debug symbols #43370

Open

marcrasi mannequin mentioned this pull request Jul 17, 2018

[SR-8293] report better error location for Python import errors in swift-juptyer #50821

Closed

kateinoigakukun added a commit that referenced this pull request Aug 31, 2022

Merge pull request #4479 from swiftwasm/main

7af6775

[pull] swiftwasm from main

		@@ -72,7 +72,6 @@ static bool getSymbolNameAddr(llvm::StringRef libraryName, Dl_info dlinfo,
		// If the address is unavailable, just use <unavailable> as the symbol

[SR-755] Tool for re-symbolicating fatal stacktraces on Linux. #4479

[SR-755] Tool for re-symbolicating fatal stacktraces on Linux. #4479

Uh oh!

Conversation

gmilos commented Aug 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's in this pull request?

Resolved bug number: (SR-755)

Uh oh!

DougGregor commented Aug 24, 2016

Uh oh!

gmilos commented Aug 26, 2016

Uh oh!

gmilos commented Aug 26, 2016

Uh oh!

gmilos commented Aug 26, 2016

Uh oh!

DougGregor commented Aug 26, 2016

Uh oh!

modocache Aug 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

modocache commented Aug 27, 2016

Uh oh!

modocache Aug 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmilos commented Aug 31, 2016

Uh oh!

gmilos commented Sep 1, 2016

Uh oh!

modocache Sep 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmilos commented Sep 1, 2016

Uh oh!

gribozavr Sep 1, 2016

Choose a reason for hiding this comment

Uh oh!

gribozavr commented Sep 1, 2016

Uh oh!

gribozavr commented Sep 1, 2016

Uh oh!

gribozavr commented Sep 1, 2016

Uh oh!

gribozavr commented Sep 1, 2016

Uh oh!

gribozavr commented Sep 1, 2016

Uh oh!

rintaro commented Sep 14, 2016

Uh oh!

modocache commented Sep 14, 2016

Uh oh!

gmilos commented Sep 14, 2016

Uh oh!

gmilos commented Sep 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmilos commented Sep 16, 2016

Uh oh!

gribozavr commented Sep 16, 2016

Uh oh!

gribozavr commented Sep 16, 2016

Uh oh!

gribozavr commented Sep 16, 2016

Uh oh!

gribozavr commented Sep 16, 2016

Uh oh!

DougGregor commented Sep 20, 2016

Uh oh!

gmilos commented Sep 20, 2016

Uh oh!

DougGregor commented Sep 20, 2016

Uh oh!

DougGregor commented Sep 20, 2016

Uh oh!

swift-ci commented Sep 20, 2016

gmilos commented Aug 24, 2016 •

edited

Loading

modocache Aug 26, 2016 •

edited

Loading

modocache Aug 27, 2016 •

edited

Loading

modocache Sep 1, 2016 •

edited

Loading

gmilos commented Sep 15, 2016 •

edited

Loading