Bug#35728261 Autotest testNodeRestart -n WatchdogSlowShutdown T1

fails occasionally

Context:
Data nodes has a mechanism (implemented in
ErrorReporter.cpp :: prepare_to_crash) to prevent multiple threads to
start processing the crash handling in parallel. That works for ndbmtd,
where we can have multiple threads crashing at the same time, but for
ndbd case the prepare_to_crash function is empty since there is only
one exec thread. But, in addition to exec threads, the watchdog thread
can also start the crash handling, if this thread 'crash' at same time
as the exec thread we can have both threads trying to start the crash
handling in parallel.

Problem:
TestNodeRestart -n WatchdogSlowShutdown fail in .2ndbd due to
a race between the Watchdog thread and the signal exec thread trying
to start a crash handling in parallel.

Solution:
Singlethreaded version of prepare_to_crash changed in order to ensure
that, if both Watchdog and exec thread crash 'at same time', only one
thread (the first one) proceed with the crash handling, the second one
is stopped immediately.

This patch also fixes an issue in the FastScheduler implementation
of traceDumpGetJam() used by the watchdog thread when it is the first
thread to start the crash handling. Function uses a local
EmulatedJamBuffer that is not set in the watchdog thread.

Change-Id: I5c9ffb6d06bf74cd38b65268171ed9c9718265e4

Author: vinc13e

storage/ndb/src/kernel/error/ErrorReporter.cpp

@@ -293,6 +293,12 @@ int WriteMessage(int thrdMessageID, const char *thrdProblemData,
    * crash handler then we will never return from this first call.
    * Otherwise we will return, write the error log and never return
    * from the second call to prepare_to_crash below.
+   *
+   * In singlethreaded case first call of prepare_to_crash does nothing.
+   * In the second call we ensure that, if there are two threads (watchdog
+   * and signal execution) processing the crash handling in parallel,
+   * only one thread (the first one) will proceed with the crash
+   * handling, the second one will stop immediately.
    */
   ErrorReporter::prepare_to_crash(true, (nst == NST_ErrorInsert));
 

storage/ndb/src/kernel/vm/Emulator.cpp

@@ -722,6 +722,7 @@ EmulatorData::EmulatorData() {
   theShutdownMutex = 0;
   m_socket_server = 0;
   m_mem_manager = 0;
+  m_st_jam_buffer = 0;
 }
 
 void EmulatorData::create() {
@@ -735,6 +736,7 @@ void EmulatorData::create() {
   EmulatedJamBuffer *jamBuffer = nullptr;
 #endif
   NDB_THREAD_TLS_JAM = jamBuffer;
+  m_st_jam_buffer = jamBuffer;
 
   theConfiguration = new Configuration();
   theWatchDog = new WatchDog();
@@ -769,4 +771,5 @@ void EmulatorData::destroy() {
   NdbMutex_Destroy(theShutdownMutex);
   if (m_mem_manager) delete m_mem_manager;
   m_mem_manager = 0;
+  m_st_jam_buffer = 0;
 }

storage/ndb/src/kernel/vm/Emulator.hpp

@@ -134,6 +134,9 @@ struct EmulatorData {
   class SocketServer *m_socket_server;
   class Ndbd_mem_manager *m_mem_manager;
 
+  // Single threaded only
+  EmulatedJamBuffer *m_st_jam_buffer;
+
   /**
    * Constructor
    *

storage/ndb/src/kernel/vm/FastScheduler.cpp

@@ -455,9 +455,9 @@ bool FastScheduler::traceDumpGetJam(Uint32 thr_no,
   thrdTheEmulatedJam = NULL;
   thrdTheEmulatedJamIndex = 0;
 #else
-  const EmulatedJamBuffer *jamBuffer = NDB_THREAD_TLS_JAM;
-  thrdTheEmulatedJam = jamBuffer->theEmulatedJam;
-  thrdTheEmulatedJamIndex = jamBuffer->theEmulatedJamIndex;
+  thrdTheEmulatedJam = globalEmulatorData.m_st_jam_buffer->theEmulatedJam;
+  thrdTheEmulatedJamIndex =
+      globalEmulatorData.m_st_jam_buffer->theEmulatedJamIndex;
 #endif
   return true;
 }

storage/ndb/src/kernel/vm/SimulatedBlock.cpp

@@ -4950,14 +4950,22 @@ Uint32 SimulatedBlock::get_recv_thread_idx(TrpId trp_id) {
 
 #ifndef NDBD_MULTITHREADED
 /**
- * Add a stub for this function since we have some code in ErrorReporter.cpp
- * that needs this function, it's only really needed for ndbmtd, so need an
- * empty function in ndbd.
+ * Function for ndbd only. ndbmtd version of this function
+ * is implemented in ErrorReporter.cpp
  */
 void ErrorReporter::prepare_to_crash(bool first_phase,
                                      bool error_insert_crash) {
   (void)first_phase;
   (void)error_insert_crash;
+
+  static bool crash_handling_started = false;
+  if (!first_phase) {
+    if (crash_handling_started) {
+      /* Someone else handling the crash, exit now */
+      my_thread_exit(nullptr);
+    }
+    crash_handling_started = true;
+  }
 }
 #endif