Skip to content

Commit 117dde1

Browse files
kennethdyerrohshar
kennethdyer
and
rohshar
authored
DOCSP-34813 Data Verification (#241)
* DOCSP-34813 Data Verification * Updates directives * Fixes build issue * Fixes build issue * Fixes build issue * Vale checks * Vale checks * Fixes build issue * Fixes per Jeff * Fixes per Jeff * Fixes per Jeff * Fixes per Jeff * Fixes per Jeff * Fixes per Jeff * Build issue * Minor edits * Minor edits * Minor edits * Fixes build error * Adds sidebars * Fixes toc depth * Fixes count admonition * Fixes count admonition * Fixes hash indent * Fixes hash indent * Fixes per Jeff * Fixes per Jeff * Fixes per Rohan * Fixes per Rohan * Fixes per Rohan * Fixes per Tim * Fixes per Tim * Adds facets to verification subpages * Fixes per Rohan * Fixes build issue * Fixes build issue * Fixes build issue * Fixes line emphasis * Fixes per Rohan * Fixes build issue * Fixes per Rohan Co-authored-by: Rohan Sharan <[email protected]> * Fixes per Tim --------- Co-authored-by: Rohan Sharan <[email protected]>
1 parent da8437a commit 117dde1

File tree

10 files changed

+519
-3
lines changed

10 files changed

+519
-3
lines changed

‎snooty.toml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ intersphinx = [ "https://www.mongodb.com/docs/atlas/objects.inv",
55
"https://www.mongodb.com/docs/php-library/upcoming/objects.inv",
66
"https://www.mongodb.com/docs/compass/current/objects.inv",
77
"https://www.mongodb.com/docs/mongodb-shell/objects.inv",
8-
"https://www.mongodb.com/docs/master/objects.inv",
8+
"https://www.mongodb.com/docs/master/objects.inv",
99
"https://www.mongodb.com/docs/drivers/objects.inv",
1010
"https://www.mongodb.com/docs/database-tools/objects.inv",
1111
"https://www.mongodb.com/docs/mongocli/stable/objects.inv",
@@ -20,7 +20,8 @@ toc_landing_pages = ["/quickstart",
2020
"/multiple-mongosyncs",
2121
"/release-notes/release-notes",
2222
"/faq",
23-
"/reference/collection-level-filtering"
23+
"/reference/collection-level-filtering",
24+
"/reference/verification"
2425
]
2526

2627
[constants]

‎source/includes/fact-verification.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
2+
Before transferring your application load from the source cluster to the
3+
destination, check your data to ensure that the sync was successful.
4+
5+
For more information, see :ref:`c2c-verification`.
6+

‎source/multiple-mongosyncs.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,11 @@ These commands only check progress and commit synchronization for the
272272
to ``progress`` and ``commit`` on any other ``mongosync`` instances
273273
that may be running.
274274

275+
Data Verification
276+
-----------------
277+
278+
.. include:: /includes/fact-verification
279+
275280
.. _c2c-sharded-reverse:
276281

277282
Reverse the Synchronization Direction

‎source/quickstart.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ Quickstart
66

77
.. default-domain:: mongodb
88

9+
.. facet::
10+
:name: genre
11+
:values: tutorial
12+
913
.. contents:: On this page
1014
:local:
1115
:backlinks: none
@@ -272,6 +276,11 @@ If the ``commit`` request is successful, ``mongosync`` returns
272276
is complete, ``mongosync`` enters the ``COMMITTED`` state and the
273277
clusters are no longer in continuous sync.
274278

279+
Data Verification
280+
-----------------
281+
282+
.. include:: /includes/fact-verification
283+
275284
Synchronization Notes
276285
---------------------
277286

‎source/reference.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Reference
55
=========
66

77
.. toctree::
8-
:titlesonly:
8+
:titlesonly:
99

1010
/reference/mongosync
1111
/reference/configuration
@@ -18,6 +18,7 @@ Reference
1818
/reference/logging
1919
/reference/permissions
2020
/reference/telemetry
21+
/reference/verification
2122
/reference/versioning
2223
/reference/supported-server-version
2324

‎source/reference/api/commit.txt

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,9 +144,17 @@ Response
144144
Behavior
145145
--------
146146

147+
State
148+
~~~~~
149+
147150
If the ``commit`` request is successful, ``mongosync`` enters the
148151
``COMMITTING`` state, then automatically transitions to the
149152
``COMMITTED`` state. Once ``mongosync`` enters the ``COMMITTED`` state,
150153
continuous sync between the clusters stops.
151154

155+
Data Verification
156+
~~~~~~~~~~~~~~~~~
157+
158+
.. include:: /includes/fact-verification
159+
152160

‎source/reference/verification.txt

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
.. _c2c-verification:
2+
3+
====================
4+
Verify Data Transfer
5+
====================
6+
7+
.. default-domain:: mongodb
8+
9+
.. facet::
10+
:name: genre
11+
:values: reference
12+
13+
.. contents:: On this page
14+
:local:
15+
:backlinks: none
16+
:depth: 2
17+
:class: singlecol
18+
19+
When :program:`mongosync` has fully committed, verify the successful transfer
20+
of your data before you switch your application to using the destination
21+
cluster. You can verify your data transfer using document counts, hash
22+
comparison, document comparison, or the Migration Verifier.
23+
24+
Use Cases
25+
---------
26+
27+
You should verify your data after every sync. This is important in
28+
cases where you plan to move your application load from the source to the
29+
destination cluster.
30+
31+
Verification methods:
32+
33+
- :ref:`c2c-verify-with-doc-counts`
34+
35+
- :ref:`c2c-verify-with-hash-comp`
36+
37+
- :ref:`c2c-verify-with-doc-comp`
38+
39+
- :ref:`c2c-verify-with-verifier`
40+
41+
The specific method you use to verify your data depends on your application
42+
workload and the complexity of the data.
43+
44+
.. _c2c-verify-method:
45+
46+
Tasks
47+
-------
48+
49+
.. _c2c-verify-with-doc-counts:
50+
51+
Document Counts
52+
~~~~~~~~~~~~~~~
53+
54+
The most basic method of verification is to compare the number of documents
55+
in each synced collection on the source cluster to the number on the
56+
destination cluster.
57+
58+
This method only verifies a successful sync when run against clusters with
59+
insert-only workloads.
60+
61+
For more information, see :ref:`c2c-verify-doc-counts`.
62+
63+
.. _c2c-verify-with-hash-comp:
64+
65+
Hash Comparison
66+
~~~~~~~~~~~~~~~
67+
68+
You can verify sync by comparing MD5 hashes of collections synced from the
69+
source cluster to the destination cluster.
70+
71+
While hash comparison ensures that the destination cluster has received all
72+
changes from the source, the :dbcommand:`dbHash` command locks the cluster,
73+
preventing additional writes until it completes.
74+
75+
.. note::
76+
77+
Hash comparison is not possible with sharded clusters. It also does not
78+
work for standalone servers and replica sets that use MongoDB 4.4 or earlier
79+
releases, since the document field order can vary.
80+
81+
For more information, see :ref:`c2c-verify-hash-comp`.
82+
83+
.. _c2c-verify-with-doc-comp:
84+
85+
Document Comparison
86+
~~~~~~~~~~~~~~~~~~~
87+
88+
You can verify sync by comparing documents on the source and destination
89+
clusters. Write a script that queries collections on the source cluster and
90+
then checks that the document exists with the same values on the destination
91+
cluster.
92+
93+
.. _c2c-verify-with-verifier:
94+
95+
Migration Verifier
96+
~~~~~~~~~~~~~~~~~~
97+
98+
Migration Verifier connects to the source and destination clusters and performs
99+
a series of verification checks, comparing documents, views, and indexes to
100+
confirm the sync was successful.
101+
102+
.. important::
103+
104+
Migration Verifier is an experimental and unsupported tool.
105+
106+
For installation instructions, see
107+
`GitHub <https://github.com/mongodb-labs/migration-verifier>`__.
108+
109+
Unlike other verification methods, Migration Verifier can run concurrent
110+
with ``mongosync``, checking documents on the destination cluster as they
111+
sync.
112+
113+
For more information, see :ref:`c2c-migration-verifier`.
114+
115+
.. toctree::
116+
:hidden:
117+
118+
/reference/verification/count
119+
/reference/verification/hash
120+
/reference/verification/verifier
121+
122+
Learn More
123+
----------
124+
125+
For more information, see:
126+
127+
- :method:`db.collection.countDocuments`
128+
129+
- :dbcommand:`dbHash`
130+
131+
132+

‎source/reference/verification/count.txt

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
.. _c2c-verify-doc-counts:
2+
3+
===========================
4+
Verify with Document Counts
5+
===========================
6+
7+
.. default-domain:: mongodb
8+
9+
.. facet::
10+
:name: genre
11+
:values: tutorial
12+
13+
.. contents:: On this page
14+
:local:
15+
:backlinks: none
16+
:depth: 1
17+
:class: singlecol
18+
19+
The most basic method of verification is to compare the number of documents
20+
in each synced collection on the source cluster to the number on the
21+
destination cluster.
22+
23+
This method only verifies a successful sync when run against clusters with
24+
insert-only workloads. If you do not have an insert-only workload for the
25+
cluster, use a different :ref:`verification method <c2c-verify-method>`.
26+
27+
Steps
28+
-----
29+
30+
.. procedure::
31+
32+
.. step:: Count Source Documents
33+
34+
Run the :method:`~db.collection.countDocuments` method on each synced
35+
collection on the source cluster:
36+
37+
.. io-code-block::
38+
39+
.. input::
40+
:language: javascript
41+
42+
db.us_accounts.countDocuments()
43+
44+
.. output::
45+
:language: javascript
46+
47+
345
48+
49+
Note the number of documents.
50+
51+
.. step:: Count Destination Documents
52+
53+
Run the :method:`~db.collection.countDocuments` method on each synced
54+
collection on the destination cluster:
55+
56+
.. io-code-block::
57+
58+
.. input::
59+
:language: javascript
60+
61+
db.us_accounts.countDocuments()
62+
63+
.. output::
64+
:language: javascript
65+
66+
345
67+
68+
Note the number of documents.
69+
70+
.. step:: Compare Counts
71+
72+
Compare the results of each document count:
73+
74+
- If the document count is different, not all changes synced to the
75+
destination cluster.
76+
77+
- If the document count is the same *and* your cluster has an insert-only
78+
workload, the sync was successful.
79+
80+
- If your source cluster does *not* have an insert-only workload, use a
81+
different method to verify sync.
82+
83+
Learn More
84+
----------
85+
86+
- :ref:`c2c-verify-hash-comp`
87+
- :ref:`c2c-migration-verifier`

0 commit comments

Comments
 (0)