[libunwind][AIX] Call dlclose only when dlsym() fails

[xingxue-ibm]

The personality routine __xlcxx_personality_v0 in libc++abi is hard-coded in the unwinder as the handler for EH in applications generated by the legacy IBM C++ compiler. The symbol is resolved dynamically using dlopen to avoid a hard dependency of libunwind on libc++abi for cases such as non-C++ applications. However, dlclose was incorrectly called after dlsym succeeded, potentially invalidating the function pointer obtained from dlsym when the memory allocated for the dlopen is reclaimed. This PR changes to call dlclose only when dlsym fails.

[llvmbot]

@llvm/pr-subscribers-libunwind

Author: Xing Xue (xingxue-ibm)

Changes

The personality routine __xlcxx_personality_v0 in libc++abi is hard-coded in the unwinder as the handler for EH in applications generated by the legacy IBM C++ compiler. The symbol is resolved dynamically using dlopen to avoid a hard dependency of libunwind on libc++abi for cases such as non-C++ applications. However, dlclose was incorrectly called after dlsym succeeded, potentially invalidating the function pointer obtained from dlsym when the memory allocated for the dlopen is reclaimed. This PR changes to call dlclose only when dlsym fails.

---

Full diff: https://github.com/llvm/llvm-project/pull/112768.diff

1 Files Affected:

• (modified) libunwind/src/UnwindCursor.hpp (+1-1)

diff --git a/libunwind/src/UnwindCursor.hpp b/libunwind/src/UnwindCursor.hpp
index 2a3aba28fb6ca5..9f3ffd19e1746a 100644
--- a/libunwind/src/UnwindCursor.hpp
+++ b/libunwind/src/UnwindCursor.hpp
@@ -2151,8 +2151,8 @@ bool UnwindCursor<A, R>::getInfoFromTBTable(pint_t pc, R &registers) {
         if (xlcPersonalityV0 == NULL) {
           _LIBUNWIND_TRACE_UNWINDING("dlsym() failed with errno=%d\n", errno);
           assert(0 && "dlsym() failed");
+          dlclose(libHandle);
         }
-        dlclose(libHandle);
         errno = saveErrno;
       }
       xlcPersonalityV0InitLock.unlock();

[compnerd]

Should the call be prior to the assert?

[xingxue-ibm]

Should the call be prior to the assert?

Good catch!

[hubert-reinterpretcast]

LGTM; thanks!