BUG#33934013: MYSQL CRASHES IF YOU STOP REPLICATION WHILE IT

              IS REPLICATING A CTAS.

DESCRIPTION:
============
If a CREATE TABLE ... AS SELECT (CTAS) is being applied on a
Replica and is interrupted by stopping replication, it could
sometimes cause a server exit.

ANALYSIS:
=========
CTAS is implemented as an atomic operation for Storage
Engines that support Atomic DDLs. Applying binlog for CTAS
at replica is also atomic. CTAS (CREATE + INSERT) is now
executed as a single binlog transaction on Replica. Thus, if
the server crashes while executing it on the Replica, the
transaction is rolled back.

The class Transactional_ddl_context is used to keep the
context of transactional DDL statements. Currently, only
CREATE TABLE with START TRANSACTION uses this context.

When a CTAS query is executed, it is written to binary log
in the following format.
BEGIN
CREATE TABLE t1 (c1 INT) START TRANSACTION
Write_rows events on t1
COMMIT

Consider a case of CTAS where the CREATE has succeeded and
the Replica is interrupted by manually stopping replication
(STOP REPLICA) just after the applier thread has executed
atleast one WRITE_ROWS_EVENT. As expected, the SQL thread is
killed and the transaction is forced to rollback.

The rollback process involves removing all the instances of
TABLE and the TABLE_SHARE instance from the Table cache and
Table definition cache respectively. To remove the TABLE
object from the Table cache, the expectation is that the
TABLE object must be unused. In this scenario, before
applying the WRITE_ROWS_EVENT the table was opened and the
TABLE object was added to the used list in the Table cache
and also appropriate locks were taken. Since the SQL thread
was stopped abruptly, the TABLE object was not marked as
free. Hence, as part of the cleanup, when
Relay_log_info::cleanup_context() attempts to close the
thread tables thereby free the TABLE object from the Table
cache, an assert is triggered in
Table_cache_manager::free_table().

Freeing the TABLE object from Table cache also involves
closing the table handler. It expects the handler to have
released its lock (i.e handler::m_lock_type == F_UNLCK). In
this scenario, handler::m_lock_type was set to F_WRLCK while
opening and locking the table earlier and thus triggering an
assert while closing the handler.

FIX:
====
The fix is to unlock and move the TABLE object into the free
list in the Table cache just before freeing the TABLE and
TABLE_SHARE objects from their respective caches.

Change-Id: I40ed95efc28db4fb6231a38724787d3e3b8d202d

Author: karthik-kamath

sql/rpl_replica.cc

@@ -5020,6 +5020,16 @@ static int exec_relay_log_event(THD *thd, Relay_log_info *rli,
         return 1;
     }
 
+    DBUG_EXECUTE_IF("wait_on_exec_relay_log_event", {
+      if (ev->get_type_code() == binary_log::WRITE_ROWS_EVENT) {
+        const char act[] =
+            "now SIGNAL signal.waiting_on_event_execution "
+            "WAIT_FOR signal.can_continue_execution";
+        assert(opt_debug_sync_timeout > 0);
+        assert(!debug_sync_set_action(current_thd, STRING_WITH_LEN(act)));
+      }
+    };);
+
     /* ptr_ev can change to NULL indicating MTS coorinator passed to a Worker */
     exec_res = apply_event_and_update_pos(ptr_ev, thd, rli);
     /*

sql/sql_class.cc

@@ -3230,6 +3230,11 @@ void Transactional_ddl_context::init(dd::String_type db,
                                      dd::String_type tablename,
                                      const handlerton *hton) {
   assert(m_hton == nullptr);
+  /*
+    Currently, Transactional_ddl_context is used only for CREATE TABLE ... START
+    TRANSACTION statement.
+  */
+  assert(m_thd->lex->sql_command == SQLCOM_CREATE_TABLE);
   m_db = db;
   m_tablename = tablename;
   m_hton = hton;
@@ -3241,7 +3246,15 @@ void Transactional_ddl_context::init(dd::String_type db,
 */
 void Transactional_ddl_context::rollback() {
   if (!inited()) return;
+  /*
+    Since the transaction is being rolledback, We need to unlock and close the
+    table belonging to this transaction.
+  */
+  if (m_thd->lock) mysql_unlock_tables(m_thd, m_thd->lock);
+  m_thd->lock = nullptr;
+  if (m_thd->open_tables) close_thread_table(m_thd, &m_thd->open_tables);
   table_cache_manager.lock_all_and_tdc();
+
   TABLE_SHARE *share =
       get_cached_table_share(m_db.c_str(), m_tablename.c_str());
   if (share) {