Skip to content

Commit d2d4b66

Browse files
author
Vitor Oliveira
committed
WL#9556 - Writeset-based MTS dependency tracking on master
This worklog introduces the ability to use transaction write sets to find out which transactions can be safely executed concurrently on the slave. The transactions are tracked on the master and the binary log commit_parent field, which is used by the MTS applier on the slave, is changed to consider writeset dependencies between transactions. Relevant changes to the code: - introduces two new options: binlog-transaction-dependency-tracking, which selects the tracking mode between COMMIT_ORDER, WRITESET and WRITESET_SESSION, and binlog-transaction-dependency-history-size. - adds the Transaction_dependency_tracker and associated classes Commit_order_trx_dependency_tracker, Writeset_trx_dependency_tracker and Writeset_session_trx_dependency_tracker, specific to each of the tracking modes (in the new rpl_trx_tracking.h/.cc files). - to support the WRITESET mode, the hashes of each row touched by a transaction are stored in the m_writeset_history map, up to the size determined by the binlog-transaction-dependency-history-size option. - to support the WRITESET_SESSION mode, the Dependency_tracker_ctx is added to the THD so that the last sequence_number used by a client is available for future transactions of the same session. - extends the function add_pke to track transactions that touch tables with primary keys and tables without them, so that partial writesets are detected.
1 parent a8d0b2d commit d2d4b66

File tree

38 files changed

+4958
-185
lines changed

38 files changed

+4958
-185
lines changed

‎mysql-test/include/assert_grep.inc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,10 @@ if ($assert_match != '')
9494
my $assert_only_after= $ENV{'_AG_ASSERT_ONLY_AFTER'};
9595
my $out= $ENV{'_AG_OUT'};
9696

97+
if (!defined($assert_count)) {
98+
$assert_count = '';
99+
}
100+
97101
my $result= '';
98102
my $count= 0;
99103
open(FILE, "$file") or die("Error $? opening $file: $!\n");

‎mysql-test/include/assert_logical_timestamps.inc

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
# --let $binlog_file= FILENAME
99
# [--let $binlog_position= OFFSET]
1010
# --let $logical_timestamps= LC1 SN1[;LC2 SN2[;LC3 SN3...]]
11+
# [--let $save_binlog_position= 1]
12+
# [--let $dont_print_logical_timestamps= 1]
1113
# --source include/assert_logical_timestamps.inc
1214
#
1315
# Parameters:
@@ -26,9 +28,23 @@
2628
# integer. Note that this is whitespace-sensitive so there should
2729
# not be any other space characters than those separating
2830
# last_committed from sequence_number.
29-
31+
#
32+
# $save_binlog_position
33+
# If this flag is set, the script will source
34+
# include/save_binlog_position.inc after it has finished the
35+
# assert, making the next invocation automatically continue where
36+
# this invocation ended. This is convenient if the script is to
37+
# be called multiple times in a sequence.
38+
#
39+
# $dont_print_logical_timestamps
40+
# By default, $logical_timestamps is printed to the result log. If
41+
# this flag is set, that is suppressed.
3042

3143
--let $include_filename= include/assert_logical_timestamps.inc [$logical_timestamps]
44+
if ($dont_print_logical_timestamps)
45+
{
46+
--let $include_filename= include/assert_logical_timestamps.inc
47+
}
3248
--source include/begin_include_file.inc
3349

3450

@@ -53,5 +69,11 @@ if ($binlog_position)
5369
--remove_file $assert_file
5470

5571

72+
if ($save_binlog_position)
73+
{
74+
--source include/save_binlog_position.inc
75+
}
76+
77+
5678
--let $include_filename= include/assert_logical_timestamps.inc [$logical_timestamps]
5779
--source include/end_include_file.inc

‎mysql-test/r/mysqld--help-notwin.result

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,13 @@ The following options may be given as the first argument:
141141
non-transactional engines for the binary log. If you
142142
often use statements updating a great number of rows, you
143143
can increase this to get more performance
144+
--binlog-transaction-dependency-history-size=#
145+
Maximum number of rows to keep in the writeset history.
146+
--binlog-transaction-dependency-tracking=name
147+
Selects the source of dependency information from which
148+
to assess which transactions can be executed in parallel
149+
by the slave's multi-threaded applier. Possible values
150+
are COMMIT_ORDER, WRITESET and WRITESET_SESSION.
144151
--block-encryption-mode=name
145152
mode for AES_ENCRYPT/AES_DECRYPT
146153
--bulk-insert-buffer-size=#
@@ -1263,6 +1270,8 @@ binlog-row-event-max-size 8192
12631270
binlog-row-image FULL
12641271
binlog-rows-query-log-events FALSE
12651272
binlog-stmt-cache-size 32768
1273+
binlog-transaction-dependency-history-size 25000
1274+
binlog-transaction-dependency-tracking COMMIT_ORDER
12661275
block-encryption-mode aes-128-ecb
12671276
bulk-insert-buffer-size 8388608
12681277
character-set-client-handshake TRUE

‎mysql-test/r/mysqld--help-win.result

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,13 @@ The following options may be given as the first argument:
141141
non-transactional engines for the binary log. If you
142142
often use statements updating a great number of rows, you
143143
can increase this to get more performance
144+
--binlog-transaction-dependency-history-size=#
145+
Maximum number of rows to keep in the writeset history.
146+
--binlog-transaction-dependency-tracking=name
147+
Selects the source of dependency information from which
148+
to assess which transactions can be executed in parallel
149+
by the slave's multi-threaded applier. Possible values
150+
are COMMIT_ORDER, WRITESET and WRITESET_SESSION.
144151
--block-encryption-mode=name
145152
mode for AES_ENCRYPT/AES_DECRYPT
146153
--bulk-insert-buffer-size=#
@@ -1262,6 +1269,8 @@ binlog-row-event-max-size 8192
12621269
binlog-row-image FULL
12631270
binlog-rows-query-log-events FALSE
12641271
binlog-stmt-cache-size 32768
1272+
binlog-transaction-dependency-history-size 25000
1273+
binlog-transaction-dependency-tracking COMMIT_ORDER
12651274
block-encryption-mode aes-128-ecb
12661275
bulk-insert-buffer-size 8388608
12671276
character-set-client-handshake TRUE

‎mysql-test/suite/binlog/r/binlog_transaction_dependency_tracking_on_start.result

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#
2+
# Test 1.a: Try starting server with TWSE=OFF and BTDT=COMMIT_ORDER. Expect SUCCESS.
3+
#
4+
# restart: --transaction_write_set_extraction=OFF --binlog_transaction_dependency_tracking=COMMIT_ORDER
5+
include/assert.inc [transaction_write_set_extraction should be OFF]
6+
include/assert.inc [binlog_transaction_dependency_tracking should be COMMIT_ORDER]
7+
#
8+
# Test 1.b: Try starting server with TWSE=OFF and BTDT=WRITESET. Expect FAILURE.
9+
#
10+
Occurrences of 'The transaction_write_set_extraction must be set to XXHASH64 or MURMUR32 when binlog_transaction_dependency_tracking is WRITESET or WRITESET_SESSION.' in the input file: 1
11+
Occurrences of '\[ERROR\] Aborting' in the input file: 1
12+
#
13+
# Test 1.c: Try starting server with TWSE=OFF and BTDT=WRITESET_SESSION. Expect FAILURE.
14+
#
15+
Occurrences of 'The transaction_write_set_extraction must be set to XXHASH64 or MURMUR32 when binlog_transaction_dependency_tracking is WRITESET or WRITESET_SESSION.' in the input file: 1
16+
Occurrences of '\[ERROR\] Aborting' in the input file: 1
17+
#
18+
# Test 1.d: Try starting server with TWSE=MURMUR32 and BTDT=COMMIT_ORDER. Expect SUCCESS.
19+
#
20+
# restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=COMMIT_ORDER
21+
include/assert.inc [transaction_write_set_extraction should be MURMUR32]
22+
include/assert.inc [binlog_transaction_dependency_tracking should be COMMIT_ORDER]
23+
#
24+
# Test 1.e: Try starting server with TWSE=MURMUR32 and BTDT=WRITESET. Expect SUCCESS.
25+
#
26+
# restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=WRITESET
27+
include/assert.inc [transaction_write_set_extraction should be MURMUR32]
28+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET]
29+
#
30+
# Test 1.f: Try starting server with TWSE=MURMUR32 and BTDT=WRITESET_SESSION. Expect SUCCESS.
31+
#
32+
# restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=WRITESET_SESSION
33+
include/assert.inc [transaction_write_set_extraction should be MURMUR32]
34+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET_SESSION]
35+
#
36+
# Test 1.g: Try starting server with TWSE=XXHASH64 and BTDT=COMMIT_ORDER. Expect SUCCESS.
37+
#
38+
# restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=COMMIT_ORDER
39+
include/assert.inc [transaction_write_set_extraction should be XXHASH64]
40+
include/assert.inc [binlog_transaction_dependency_tracking should be COMMIT_ORDER]
41+
#
42+
# Test 1.h: Try starting server with TWSE=XXHASH64 and BTDT=WRITESET. Expect SUCCESS
43+
#
44+
# restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=WRITESET
45+
include/assert.inc [transaction_write_set_extraction should be XXHASH64]
46+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET]
47+
#
48+
# Test 1.i: Try starting server with TWSE=XXHASH64 and BTDT=WRITESET_SESSION. Expect SUCCESS.
49+
#
50+
# restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=WRITESET_SESSION
51+
include/assert.inc [transaction_write_set_extraction should be XXHASH64]
52+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET_SESSION]
53+
SET GLOBAL binlog_transaction_dependency_tracking= COMMIT_ORDER;
54+
SET GLOBAL transaction_write_set_extraction= OFF;

‎mysql-test/suite/binlog/t/binlog_transaction_dependency_tracking_on_start-master.opt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
--no-console --log-error=$MYSQLTEST_VARDIR/tmp/binlog_transaction_dependency_tracking.err

‎mysql-test/suite/binlog/t/binlog_transaction_dependency_tracking_on_start.test

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
################################################################################
2+
# The testcase aims to test possible correct and incorrect settings of the
3+
# following variables together during the start of the server:
4+
# transaction_write_set_extraction (TWSE)
5+
# binlog_transaction_dependency_tracking (BTDT)
6+
#
7+
# Test:
8+
# 0. This test requires one server.
9+
# 1. Try starting server with:
10+
# a. TWSE=OFF and BTDT=COMMIT_ORDER. Expect SUCCESS.
11+
# b. TWSE=OFF and BTDT=WRITESET. Expect FAILURE.
12+
# c. TWSE=OFF and BTDT=WRITESET_SESSION. Expect FAILURE.
13+
# d. TWSE=MURMUR32 and BTDT=COMMIT_ORDER. Expect SUCCESS.
14+
# e. TWSE=MURMUR32 and BTDT=WRITESET. Expect SUCCESS.
15+
# f. TWSE=MURMUR32 and BTDT=WRITESET_SESSION. Expect SUCCESS.
16+
# g. TWSE=XXHASH64 and BTDT=COMMIT_ORDER. Expect SUCCESS.
17+
# h. TWSE=XXHASH64 and BTDT=WRITESET. Expect SUCCESS
18+
# i. TWSE=XXHASH64 and BTDT=WRITESET_SESSION. Expect SUCCESS.
19+
#
20+
# ==== References ====
21+
# WL#9556: Writeset-based MTS dependency tracking on master
22+
################################################################################
23+
--source include/not_embedded.inc
24+
--source include/big_test.inc
25+
--source include/have_binlog_format_row.inc
26+
27+
--let $saved_binlog_transaction_dependency_tracking= `SELECT @@GLOBAL.binlog_transaction_dependency_tracking`
28+
--let $saved_transaction_write_set_extraction= `SELECT @@GLOBAL.transaction_write_set_extraction`
29+
30+
--echo #
31+
--echo # Test 1.a: Try starting server with TWSE=OFF and BTDT=COMMIT_ORDER. Expect SUCCESS.
32+
--echo #
33+
--let $restart_parameters="restart: --transaction_write_set_extraction=OFF --binlog_transaction_dependency_tracking=COMMIT_ORDER"
34+
--source include/restart_mysqld.inc
35+
--source include/wait_until_connected_again.inc
36+
--let $assert_text= transaction_write_set_extraction should be OFF
37+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "OFF"
38+
--source include/assert.inc
39+
--let $assert_text= binlog_transaction_dependency_tracking should be COMMIT_ORDER
40+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "COMMIT_ORDER"
41+
--source include/assert.inc
42+
43+
--echo #
44+
--echo # Test 1.b: Try starting server with TWSE=OFF and BTDT=WRITESET. Expect FAILURE.
45+
--echo #
46+
# Let server restart attempts write to the file $error_log.
47+
--let $error_log= $MYSQLTEST_VARDIR/tmp/binlog_transaction_dependency_tracking.err
48+
--source include/shutdown_mysqld.inc
49+
50+
let $mysqld=$MYSQLD_CMD --core-file --console > $error_log 2>&1 --transaction_write_set_extraction=OFF --binlog_transaction_dependency_tracking=WRITESET;
51+
52+
--error 1
53+
--exec $mysqld
54+
--let $grep_file=$error_log
55+
--let $grep_pattern=The transaction_write_set_extraction must be set to XXHASH64 or MURMUR32 when binlog_transaction_dependency_tracking is WRITESET or WRITESET_SESSION.
56+
--let $grep_output= print_count
57+
--source include/grep_pattern.inc
58+
59+
# The server restart aborts
60+
--let $grep_pattern= \[ERROR\] Aborting
61+
--let $grep_output= print_count
62+
--source include/grep_pattern.inc
63+
64+
--echo #
65+
--echo # Test 1.c: Try starting server with TWSE=OFF and BTDT=WRITESET_SESSION. Expect FAILURE.
66+
--echo #
67+
let $mysqld=$MYSQLD_CMD --core-file --console > $error_log 2>&1 --transaction_write_set_extraction=OFF --binlog_transaction_dependency_tracking=WRITESET_SESSION;
68+
69+
--error 1
70+
--exec $mysqld
71+
--let $grep_file=$error_log
72+
--let $grep_pattern=The transaction_write_set_extraction must be set to XXHASH64 or MURMUR32 when binlog_transaction_dependency_tracking is WRITESET or WRITESET_SESSION.
73+
--let $grep_output= print_count
74+
--source include/grep_pattern.inc
75+
76+
# The server restart aborts
77+
--let $grep_pattern= \[ERROR\] Aborting
78+
--let $grep_output= print_count
79+
--source include/grep_pattern.inc
80+
81+
--echo #
82+
--echo # Test 1.d: Try starting server with TWSE=MURMUR32 and BTDT=COMMIT_ORDER. Expect SUCCESS.
83+
--echo #
84+
--let $restart_parameters="restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=COMMIT_ORDER"
85+
--source include/start_mysqld.inc
86+
--source include/wait_until_connected_again.inc
87+
--let $assert_text= transaction_write_set_extraction should be MURMUR32
88+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "MURMUR32"
89+
--source include/assert.inc
90+
--let $assert_text= binlog_transaction_dependency_tracking should be COMMIT_ORDER
91+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "COMMIT_ORDER"
92+
--source include/assert.inc
93+
94+
--echo #
95+
--echo # Test 1.e: Try starting server with TWSE=MURMUR32 and BTDT=WRITESET. Expect SUCCESS.
96+
--echo #
97+
--let $restart_parameters="restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=WRITESET"
98+
--source include/restart_mysqld.inc
99+
--source include/wait_until_connected_again.inc
100+
--let $assert_text= transaction_write_set_extraction should be MURMUR32
101+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "MURMUR32"
102+
--source include/assert.inc
103+
--let $assert_text= binlog_transaction_dependency_tracking should be WRITESET
104+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "WRITESET"
105+
--source include/assert.inc
106+
107+
--echo #
108+
--echo # Test 1.f: Try starting server with TWSE=MURMUR32 and BTDT=WRITESET_SESSION. Expect SUCCESS.
109+
--echo #
110+
--let $restart_parameters="restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=WRITESET_SESSION"
111+
--source include/restart_mysqld.inc
112+
--source include/wait_until_connected_again.inc
113+
--let $assert_text= transaction_write_set_extraction should be MURMUR32
114+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "MURMUR32"
115+
--source include/assert.inc
116+
--let $assert_text= binlog_transaction_dependency_tracking should be WRITESET_SESSION
117+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "WRITESET_SESSION"
118+
--source include/assert.inc
119+
120+
--echo #
121+
--echo # Test 1.g: Try starting server with TWSE=XXHASH64 and BTDT=COMMIT_ORDER. Expect SUCCESS.
122+
--echo #
123+
--let $restart_parameters="restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=COMMIT_ORDER"
124+
--source include/restart_mysqld.inc
125+
--source include/wait_until_connected_again.inc
126+
--let $assert_text= transaction_write_set_extraction should be XXHASH64
127+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "XXHASH64"
128+
--source include/assert.inc
129+
--let $assert_text= binlog_transaction_dependency_tracking should be COMMIT_ORDER
130+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "COMMIT_ORDER"
131+
--source include/assert.inc
132+
133+
--echo #
134+
--echo # Test 1.h: Try starting server with TWSE=XXHASH64 and BTDT=WRITESET. Expect SUCCESS
135+
--echo #
136+
--let $restart_parameters="restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=WRITESET"
137+
--source include/restart_mysqld.inc
138+
--source include/wait_until_connected_again.inc
139+
--let $assert_text= transaction_write_set_extraction should be XXHASH64
140+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "XXHASH64"
141+
--source include/assert.inc
142+
--let $assert_text= binlog_transaction_dependency_tracking should be WRITESET
143+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "WRITESET"
144+
--source include/assert.inc
145+
146+
--echo #
147+
--echo # Test 1.i: Try starting server with TWSE=XXHASH64 and BTDT=WRITESET_SESSION. Expect SUCCESS.
148+
--echo #
149+
--let $restart_parameters="restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=WRITESET_SESSION"
150+
--source include/restart_mysqld.inc
151+
--source include/wait_until_connected_again.inc
152+
--let $assert_text= transaction_write_set_extraction should be XXHASH64
153+
--let $assert_cond= "[SELECT @@GLOBAL.transaction_write_set_extraction]" = "XXHASH64"
154+
--source include/assert.inc
155+
--let $assert_text= binlog_transaction_dependency_tracking should be WRITESET_SESSION
156+
--let $assert_cond= "[SELECT @@GLOBAL.binlog_transaction_dependency_tracking]" = "WRITESET_SESSION"
157+
--source include/assert.inc
158+
159+
# Clean up
160+
--let $restart_parameters=
161+
--let $error_log=
162+
--remove_file $MYSQLTEST_VARDIR/tmp/binlog_transaction_dependency_tracking.err
163+
--eval SET GLOBAL binlog_transaction_dependency_tracking= $saved_binlog_transaction_dependency_tracking
164+
--eval SET GLOBAL transaction_write_set_extraction= $saved_transaction_write_set_extraction

0 commit comments

Comments
 (0)