WL#15280: HEATWAVE SUPPORT FOR MDS HA

  # This is the 1st commit message:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA

  Problem Statement
  -----------------
  Currently customers cannot enable heatwave analytics service to their
  HA DBSystem or enable HA if they are using Heatwave enabled DBSystem.
  In this change, we attempt to remove this limitation and provide
  failover support of heatwave in an HA enabled DBSystem.

  High Level Overview
  -------------------
  To support heatwave with HA, we extended the existing feature of auto-
  reloading of tables to heatwave on MySQL server restart (WL-14396). To
  provide seamless failover functionality to tables loaded to heatwave,
  each node in the HA cluster (group replication) must have the latest
  view of tables which are currently loaded to heatwave cluster attached
  to the primary, i.e., the secondary_load flag should be in-sync always.

  To achieve this, we made following changes -
    1. replicate secondary load/unload DDL statements to all the active
       secondary nodes by writing the DDL into the binlog, and
    2. Control how secondary load/unload is executed when heatwave cluster
       is not attached to node executing the command

  Implementation Details
  ----------------------
  Current implementation depends on two key assumptions -
   1. All MDS DBSystems will have RAPID plugin installed.
   2. No non-MDS system will have the RAPID plugin installed.

  Based on these assumptions, we made certain changes w.r.t. how server
  handles execution of secondary load/unload statements.
   1. If secondary load/unload command is executed from a mysql client
      session on a system without RAPID plugin installed (i.e., non-MDS),
      instead of an error, a warning message will be shown to the user,
      and the DDL is allowed to commit.
   2. If secondary load/unload command is executed from a replication
      connection on an MDS system without heatwave cluster attached,
      instead of throwing an error, the DDL is allowed to commit.
   3. If no error is thrown from secondary engine, then the DDL will
      update the secondary_load metadata and write a binlog entry.

  Writing to binlog implies that all the consumer of binlog now need to
  handle this DDL gracefully. This has an adverse effect on Point-in-time
  Recovery. If the PITR backup is taken from a DBSystem with heatwave, it
  may contain traces of secondary load/unload statements in its binlog.
  If such a backup is used to restore a new DBSystem, it will cause failure
  while trying to execute statements from its binlog because
   a) DBSystem will not heatwave cluster attached at this time, and
   b) Statements from binlog are executed from standard mysql client
      connection, thus making them indistinguishable from user executed
      command.
  Customers will be prevented (by control plane) from using PITR functionality
  on a heatwave enabled DBSystem until there is a solution for this.

  Testing
  -------
  This commit changes the behavior of secondary load/unload statements, so it
   - adjusts existing tests' expectations, and
   - adds a new test validating new DDL behavior under different scenarios

  Change-Id: Ief7e9b3d4878748b832c366da02892917dc47d83

  # This is the commit message #2:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA (PITR SUPPORT)

  Problem
  -------
  A PITR backup taken from a heatwave enabled system could have traces
  of secondary load or unload statements in binlog. When such a backup
  is used to restore another system, it can cause failure because of
  following two reasons:

  1. Currently, even if the target system is heatwave enabled, heatwave
  cluster is attached only after PITR restore phase completes.
  2. When entries from binlogs are applied, a standard mysql client
  connection is used. This makes it indistinguishable from other user
  session.

  Since secondary load (or unload) statements are meant to throw error
  when they are executed by user in the absence of a healthy heatwave
  cluster, PITR restore workflow will fail if binlogs from the backup
  have any secondary load (or unload) statements in them.

  Solution
  --------
  To avoid PITR failure, we are introducing a new system variable
  rapid_enable_delayed_secondary_ops. It controls how load or unload
  commands are to be processed by rapid plugin.

    - When turned ON, the plugin silently skips the secondary engine
      operation (load/unload) and returns success to the caller. This
      allows secondary load (or unload) statements to be executed by the
      server in the absence of any heatwave cluster.
    - When turned OFF, it follows the existing behavior.
    - The default value is OFF.
    - The value can only be changed when rapid_bootstrap is IDLE or OFF.
    - This variable cannot be persisted.

  In PITR workflow, Control Plane would set the variable at the start of
  PITR restore and then reset it at the end of workflow. This allows the
  workflow to complete without failure even when heatwave cluster is not
  attached. Since metadata is always updated when secondary load/unload
  DDLs are executed, when heatwave cluster is attached at a later point
  in time, the respective tables get reloaded to heatwave automatically.

  Change-Id: I42e984910da23a0e416edb09d3949989159ef707

  # This is the commit message mysql#3:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA (TEST CHANGES)

  This commit adds new functional tests for the MDS HA + HW integration.

  Change-Id: Ic818331a4ca04b16998155efd77ac95da08deaa1

  # This is the commit message mysql#4:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA
  BUG#34776485: RESTRICT DEFAULT VALUE FOR rapid_enable_delayed_secondary_ops

  This commit does two things:
  1. Add a basic test for newly introduced system variable
  rapid_enable_delayed_secondary_ops, which controls the behavior of
  alter table secondary load/unload ddl statements when rapid cluster
  is not available.

  2. It also restricts the DEFAULT value setting for the system variable
  So, following is not allowed:
  SET GLOBAL rapid_enable_delayed_secondary_ops = default
  This variable is to be used in restricted scenarios and control plane
  only sets it to ON/OFF before and after PITR apply. Allowing set to
  default has no practical use.

  Change-Id: I85c84dfaa0f868dbfc7b1a88792a89ffd2e81da2

  # This is the commit message mysql#5:

  Bug#34726490: ADD DIAGNOSTICS FOR SECONDARY LOAD / UNLOAD DDL

  Problem:
  --------
  If secondary load or unload DDL gets rolled back due to some error after
  it had loaded / unloaded the table in heatwave cluster, there is no undo
  of the secondary engine action. Only secondary_load flag update is
  reverted and binlog is not written. From User's perspective, the table
  is loaded and can be seen on performance_schema. There are also no
  error messages printed to notify that the ddl didn't commit. This
  creates a problem to debug any issue in this area.

  Solution:
  ---------
  The partial undo of secondary load/unload ddl will be handled in
  bug#34592922. In this commit, we add diagnostics to reveal if the ddl
  failed to commit, and from what stage.

  Change-Id: I46c04dd5dbc07fc17beb8aa2a8d0b15ddfa171af

  # This is the commit message mysql#6:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA (TEST FIX)

  Since ALTER TABLE SECONDARY LOAD / UNLOAD DDL statements now write
  to binlog, from Heatwave's perspective, SCN is bumped up.

  In this commit, we are adjusting expected SCN values in certain
  tests which does secondary load/unload and expects SCN to match.

  Change-Id: I9635b3cd588d01148d763d703c72cf50a0c0bb98

  # This is the commit message mysql#7:

  Adding MTR tests for ML in rapid group_replication suite

  Added MTR tests with Heatwave ML queries with in
  an HA setup.

  Change-Id: I386a3530b5bbe6aea551610b6e739ab1cf366439

  # This is the commit message mysql#8:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA (MTR TEST ADJUSTMENT)

  In this commit we have adjusted the existing test to work with the
  new MTR test infrastructure which extends the functionalities to
  HA landscape. With this change, a lot of mannual settings have now
  become redundant and thus removed in this commit.

  Change-Id: Ie1f4fcfdf047bfe8638feaa9f54313d509cbad7e

  # This is the commit message mysql#9:

  WL#15280: HEATWAVE SUPPORT FOR MDS HA (CLANG-TIDY FIX)

  Fix clang-tidy warnings found in previous change#16530, patch#20

  Change-Id: I15d25df135694c2f6a3a9146feebe2b981637662

Change-Id: I3f3223a85bb52343a4619b0c2387856b09438265

Author: Souvik Pal

mysql-test/suite/rpl/include/rpl_secondary_engine_load.inc

@@ -68,14 +68,6 @@ ALTER TABLE t1 SECONDARY_LOAD;
 --source include/sync_slave_sql_with_master.inc
 --echo [ Connection Slave ]
 
-# Check that SECONDARY_LOAD was not propagated and that the table has not been
-# loaded into its secondary engine.
-FLUSH STATUS;
-SELECT * FROM t1;
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-
-# Load table also on slave.
-ALTER TABLE t1 SECONDARY_LOAD;
 
 # Check that the query is offloaded.
 FLUSH STATUS;
@@ -150,12 +142,6 @@ ALTER TABLE t2 SECONDARY_LOAD;
 SHOW CREATE TABLE t2;
 SHOW CREATE TABLE t3;
 
-FLUSH STATUS;
-SELECT * FROM t2;
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-
-# Load the table on slave.
-ALTER TABLE t2 SECONDARY_LOAD;
 
 FLUSH STATUS;
 SELECT * FROM t2;
@@ -176,14 +162,6 @@ DROP TABLE t3;
 --source include/sync_slave_sql_with_master.inc
 --echo [ Connection Slave ]
 
-# Check that SECONDARY_UNLOAD was not propagated and that the table is
-# still loaded into its secondary engine.
-FLUSH STATUS;
-SELECT * FROM t1;
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-
-# Unload table also on slave.
-ALTER TABLE t1 SECONDARY_UNLOAD;
 
 # Check that the table is unloaded.
 FLUSH STATUS;

mysql-test/suite/rpl/r/rpl_mix_or_row_secondary_engine_load.result

@@ -47,18 +47,6 @@ include/sync_slave_sql_with_master.inc
 FLUSH STATUS;
 SELECT * FROM t1;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-Variable_name	Value
-Secondary_engine_execution_count	0
-ALTER TABLE t1 SECONDARY_LOAD;
-FLUSH STATUS;
-SELECT * FROM t1;
-pk	vc	j	b	vb
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
 Secondary_engine_execution_count	1
@@ -172,18 +160,6 @@ t3	CREATE TABLE `t3` (
 FLUSH STATUS;
 SELECT * FROM t2;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-Variable_name	Value
-Secondary_engine_execution_count	0
-ALTER TABLE t2 SECONDARY_LOAD;
-FLUSH STATUS;
-SELECT * FROM t2;
-pk	vc	j	b	vb
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
 Secondary_engine_execution_count	1
@@ -201,13 +177,6 @@ include/sync_slave_sql_with_master.inc
 FLUSH STATUS;
 SELECT * FROM t1;
 pk	vc	j	b	vb
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-Variable_name	Value
-Secondary_engine_execution_count	1
-ALTER TABLE t1 SECONDARY_UNLOAD;
-FLUSH STATUS;
-SELECT * FROM t1;
-pk	vc	j	b	vb
 1	Poodle	{"key": "value"}	Pug	4
 2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
 3	Beagle	[1, 2, 3]	Pitbull	��4
@@ -243,14 +212,9 @@ include/wait_for_slave_param.inc [Slave_SQL_Running_State]
 FLUSH STATUS;
 SELECT * FROM t2;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
-Secondary_engine_execution_count	0
+Secondary_engine_execution_count	1
 [ Connection Master ]
 ALTER TABLE t1 SECONDARY_ENGINE=NULL;
 include/sync_slave_sql_with_master.inc
@@ -351,14 +315,9 @@ include/wait_for_slave_param.inc [Slave_SQL_Running_State]
 FLUSH STATUS;
 SELECT * FROM t1;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
-Secondary_engine_execution_count	0
+Secondary_engine_execution_count	1
 [ Connection Master ]
 DROP TABLE t1;
 DROP TABLE t2;

mysql-test/suite/rpl/r/rpl_stmt_secondary_engine_load.result

@@ -47,18 +47,6 @@ include/sync_slave_sql_with_master.inc
 FLUSH STATUS;
 SELECT * FROM t1;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-Variable_name	Value
-Secondary_engine_execution_count	0
-ALTER TABLE t1 SECONDARY_LOAD;
-FLUSH STATUS;
-SELECT * FROM t1;
-pk	vc	j	b	vb
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
 Secondary_engine_execution_count	1
@@ -172,18 +160,6 @@ t3	CREATE TABLE `t3` (
 FLUSH STATUS;
 SELECT * FROM t2;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-Variable_name	Value
-Secondary_engine_execution_count	0
-ALTER TABLE t2 SECONDARY_LOAD;
-FLUSH STATUS;
-SELECT * FROM t2;
-pk	vc	j	b	vb
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
 Secondary_engine_execution_count	1
@@ -201,13 +177,6 @@ include/sync_slave_sql_with_master.inc
 FLUSH STATUS;
 SELECT * FROM t1;
 pk	vc	j	b	vb
-SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
-Variable_name	Value
-Secondary_engine_execution_count	1
-ALTER TABLE t1 SECONDARY_UNLOAD;
-FLUSH STATUS;
-SELECT * FROM t1;
-pk	vc	j	b	vb
 1	Poodle	{"key": "value"}	Pug	4
 2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
 3	Beagle	[1, 2, 3]	Pitbull	��4
@@ -243,14 +212,9 @@ include/wait_for_slave_param.inc [Slave_SQL_Running_State]
 FLUSH STATUS;
 SELECT * FROM t2;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
-Secondary_engine_execution_count	0
+Secondary_engine_execution_count	1
 [ Connection Master ]
 ALTER TABLE t1 SECONDARY_ENGINE=NULL;
 include/sync_slave_sql_with_master.inc
@@ -351,14 +315,9 @@ include/wait_for_slave_param.inc [Slave_SQL_Running_State]
 FLUSH STATUS;
 SELECT * FROM t1;
 pk	vc	j	b	vb
-1	Poodle	{"key": "value"}	Pug	4
-2	Labrador	[{"key": "value"}, {"key": "value2"}]	Boxer	��
-3	Beagle	[1, 2, 3]	Pitbull	��4
-4	Collie	["ab", "bc", "cd"]	Huskey	���
-5	Shiba	{"1": "one"}	Mastiff	��
 SHOW SESSION STATUS LIKE 'Secondary_engine_execution_count';
 Variable_name	Value
-Secondary_engine_execution_count	0
+Secondary_engine_execution_count	1
 [ Connection Master ]
 DROP TABLE t1;
 DROP TABLE t2;

mysql-test/suite/secondary_engine/r/define.result

@@ -10,7 +10,8 @@ DROP TABLE t1;
 #
 CREATE TABLE t1 (a INT) SECONDARY_ENGINE NONEXISTENT;
 ALTER TABLE t1 SECONDARY_LOAD;
-ERROR 42000: Unknown storage engine 'NONEXISTENT'
+Warnings:
+Warning	1286	Unknown storage engine 'NONEXISTENT'
 DROP TABLE t1;
 #
 # Attempt to load table into an unsupported secondary storage engine.

mysql-test/suite/secondary_engine/t/define.test

@@ -12,7 +12,7 @@ DROP TABLE t1;
 --echo # Attempt to load table into an unknown storage engine.
 --echo #
 CREATE TABLE t1 (a INT) SECONDARY_ENGINE NONEXISTENT;
---error ER_UNKNOWN_STORAGE_ENGINE
+# expect warning
 ALTER TABLE t1 SECONDARY_LOAD;
 DROP TABLE t1;
 

sql/sql_class.h

@@ -1897,6 +1897,11 @@ class THD : public MDL_context_owner,
   /* MTS: method inserts a new unique name into binlog_updated_dbs */
   void add_to_binlog_accessed_dbs(const char *db);
 
+  bool is_applier_thread() const {
+    return system_thread == SYSTEM_THREAD_SLAVE_SQL ||
+           system_thread == SYSTEM_THREAD_SLAVE_WORKER;
+  }
+
  private:
   std::unique_ptr<Transaction_ctx> m_transaction;
 

sql/sql_table.cc

@@ -49,8 +49,6 @@
 #include "mysql/psi/psi_table.h"  // IWYU pragma: keep
 
 /* PSI_TABLE_CALL() with WITH_LOCK_ORDER */
-#include "mysql/psi/mysql_table.h"  // IWYU pragma: keep
-
 #include "decimal.h"
 #include "field_types.h"  // enum_field_types
 #include "lex_string.h"
@@ -81,6 +79,7 @@
 #include "mysql/plugin.h"
 #include "mysql/psi/mysql_mutex.h"
 #include "mysql/psi/mysql_stage.h"
+#include "mysql/psi/mysql_table.h"  // IWYU pragma: keep
 #include "mysql_com.h"
 #include "mysql_time.h"
 #include "mysqld_error.h"  // ER_*
@@ -2667,8 +2666,10 @@ static bool secondary_engine_load_table(THD *thd, const TABLE &table) {
       ha_resolve_by_name(thd, &table.s->secondary_engine, false);
   if ((plugin == nullptr) || !plugin_is_ready(table.s->secondary_engine,
                                               MYSQL_STORAGE_ENGINE_PLUGIN)) {
-    my_error(ER_UNKNOWN_STORAGE_ENGINE, MYF(0), table.s->secondary_engine.str);
-    return true;
+    push_warning_printf(
+        thd, Sql_condition::SL_WARNING, ER_UNKNOWN_STORAGE_ENGINE,
+        ER_THD(thd, ER_UNKNOWN_STORAGE_ENGINE), table.s->secondary_engine.str);
+    return false;
   }
 
   // The engine must support being used as a secondary engine.
@@ -2730,9 +2731,12 @@ static bool secondary_engine_unload_table(THD *thd, const char *db_name,
   plugin_ref plugin = ha_resolve_by_name(thd, &secondary_engine, false);
   if ((plugin == nullptr) ||
       !plugin_is_ready(secondary_engine, MYSQL_STORAGE_ENGINE_PLUGIN)) {
-    if (error_if_not_loaded)
-      my_error(ER_UNKNOWN_STORAGE_ENGINE, MYF(0), secondary_engine);
-    return error_if_not_loaded;
+    if (error_if_not_loaded) {
+      push_warning_printf(
+          thd, Sql_condition::SL_WARNING, ER_UNKNOWN_STORAGE_ENGINE,
+          ER_THD(thd, ER_UNKNOWN_STORAGE_ENGINE), secondary_engine);
+    }
+    return false;
   }
   handlerton *hton = plugin_data<handlerton *>(plugin);
   if (hton == nullptr) {
@@ -11484,6 +11488,15 @@ bool Sql_cmd_secondary_load_unload::mysql_secondary_load_or_unload(
          hton->flags & HTON_SUPPORTS_SECONDARY_ENGINE &&
          hton->post_ddl != nullptr);
 
+  // cache table name locally for future use
+  const size_t name_len = table_list->db_length +
+                          table_list->table_name_length +
+                          5;  // for backticks, dot `db`.`tab`
+  // allocated on thread, freed-up on thread exit
+  char *full_tab_name = (char *)sql_calloc(name_len + 1);  // for \0 at the end
+  sprintf(full_tab_name, "`%s`.%s`", table_list->db, table_list->table_name);
+  full_tab_name[name_len] = '\0';  // may not needed, since inited with 0
+
   dd::cache::Dictionary_client::Auto_releaser releaser(thd->dd_client());
   dd::Table *table_def = nullptr;
   if (thd->dd_client()->acquire_for_modification(
@@ -11518,6 +11531,14 @@ bool Sql_cmd_secondary_load_unload::mysql_secondary_load_or_unload(
     cleanup();
   });
 
+  /* Currently, if the DDL rolls back after completing the secondary engine
+   * operation, then during undo only the secondary_flag update is undone, but
+   * the secondary engine operation (load/unload) is not undone. This creates
+   * a temporary inconsistency in the system where the secondary_load flag is
+   * not in sync with the actual status of the table in the secondary engine. */
+
+  // TODO: undo handling of secondary engine operation
+
   // Load if SECONDARY_LOAD, unload if SECONDARY_UNLOAD
   const bool is_load = m_alter_info->flags & Alter_info::ALTER_SECONDARY_LOAD;
 
@@ -11540,19 +11561,71 @@ bool Sql_cmd_secondary_load_unload::mysql_secondary_load_or_unload(
                                       table_list->table_name, *table_def, true))
       return true;
   }
+  DBUG_PRINT("sec_load_unload", ("secondary engine %s succeeded for table %s",
+                                 (is_load ? "load" : "unload"), full_tab_name));
+
+  DBUG_EXECUTE_IF("sim_fail_before_metadata_update", {
+    DBUG_PRINT("sec_load_unload", ("Force exit before metadata update"));
+    my_error(
+        ER_SECONDARY_ENGINE, MYF(0),
+        "Simulated failure of sec_{un}load before secondary_load flag update");
+    return true;
+  });
 
   // Update the secondary_load flag based on the current operation.
   if (table_def->options().set("secondary_load", is_load) ||
-      thd->dd_client()->update(table_def))
+      thd->dd_client()->update(table_def)) {
+    LogErr(ERROR_LEVEL, ER_SECONDARY_ENGINE,
+           "SECONDARY_LOAD metadata update failed for table %s", full_tab_name);
     return true;
+  }
+
+  DBUG_PRINT("sec_load_unload", ("secondary_load flag %s for table %s",
+                                 (is_load ? "set" : "reset"), full_tab_name));
+
+  DBUG_EXECUTE_IF("sim_fail_before_write_bin_log", {
+    DBUG_PRINT("sec_load_unload", ("Force exit before binlog write"));
+    my_error(ER_SECONDARY_ENGINE, MYF(0),
+             "Simulated failure of sec_{un}load before write_bin_log()");
+    return true;
+  });
+
+  /* Write binlog to maintain replication consistency. Read-Replica's may not
+   * have binlog enabled. write_bin_log API takes care of such cases. */
+  if (write_bin_log(thd, true, thd->query().str, thd->query().length, true) !=
+      0) {
+    LogErr(ERROR_LEVEL, ER_SECONDARY_ENGINE,
+           "Unable to write to binlog for alter table %s secondary_%s",
+           full_tab_name, (is_load ? "load" : "unload"));
+    return true;
+  }
+
+  DBUG_PRINT("sec_load_unload",
+             ("binlog entry added for alter table %s secondary_%s",
+              full_tab_name, (is_load ? "load" : "unload")));
+
+  DBUG_EXECUTE_IF("sim_fail_after_write_bin_log", {
+    DBUG_PRINT("sec_load_unload", ("Force exit after binlog write"));
+    my_error(ER_SECONDARY_ENGINE, MYF(0),
+             "Simulated failure of sec_{un}load after write_bin_log()");
+    return true;
+  });
 
   // Close primary table.
   close_all_tables_for_name(thd, table_list->table->s, false, nullptr);
   table_list->table = nullptr;
 
   // Commit transaction if no errors.
-  if (trans_commit_stmt(thd) || trans_commit_implicit(thd)) return true;
+  if (trans_commit_stmt(thd) || trans_commit_implicit(thd)) {
+    LogErr(ERROR_LEVEL, ER_SECONDARY_ENGINE,
+           "Commit failed for alter table %s secondary_%s", full_tab_name,
+           (is_load ? "load" : "unload"));
+    return true;
+  }
 
+  DBUG_PRINT("sec_load_unload",
+             ("commit succeeded for alter table %s secondary_%s", full_tab_name,
+              (is_load ? "load" : "unload")));
   // Transaction committed successfully, no rollback will be necessary.
   rollback_guard.commit();