Skip to content

Commit 3cf4a1c

Browse files
author
Dave Cuthbert
authored
DOCSP-23262 BACKPORT (#71)
* DOCSP-23262 check oplog window size v0.9.0 (#67) * DOCSP-23262 oplog size * DOCSP-23262 oplog window sizing * Staging fixes * Staging fixes * Staging fixes * Review feedback * Staging fixes
1 parent b433e1b commit 3cf4a1c

File tree

7 files changed

+118
-15
lines changed

7 files changed

+118
-15
lines changed

‎source/faq.txt

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,10 @@ clusters.
2828

2929
.. _c2c-faq-increase-oplog:
3030

31-
Should I increase the size of the oplog in the source cluster?
32-
--------------------------------------------------------------
31+
Should I increase the size of the ``oplog`` in the source cluster?
32+
------------------------------------------------------------------
3333

34-
If you anticipate synchronizing a large data set, or if you plan to
35-
pause synchronization for an extended period of time, increase the size
36-
of the replica set :term:`oplog` in the source cluster.
37-
38-
The ``oplog`` in the source cluster must be large enough to track
39-
events that happen during the time it takes to complete the initial
40-
sync to the destination cluster.
34+
.. include:: /includes/fact-oplog-background
4135

4236
The proper ``oplog`` size depends on system hardware, network speed,
4337
and other factors including system workload. However, assuming network
@@ -48,6 +42,9 @@ required ``oplog`` size is:
4842

4943
minimumRetentionHours = dataSizeInGB / 30
5044

45+
To estimate the size of ``oplog`` needed for initial synchronization,
46+
see: :ref:`c2c-oplog-sizing`.
47+
5148
To learn more about how to increase the size of the ``oplog``, see:
5249
:ref:`tutorial-change-oplog-size`.
5350

‎source/includes/fact-oplog-background.rst

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
The :term:`oplog` in the source cluster must be large enough to track
2+
events that happen during the time it takes to complete the initial
3+
sync to the destination cluster.
4+
5+
If you anticipate synchronizing a large data set, or if you plan to
6+
pause synchronization for an extended period of time, increase the size
7+
of the replica set ``oplog`` in the source cluster.

‎source/quickstart.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,3 +250,6 @@ Synchronization Notes
250250
but macOS in particular has low defaults. See :ref:`UNIX ulimit
251251
settings <system-resource-utilization>`.
252252

253+
- To estimate the size of ``oplog`` needed for initial synchronization,
254+
see: :ref:`c2c-oplog-sizing`.
255+

‎source/reference.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,6 @@ Reference
1212
/reference/api
1313
/reference/mongosync-states
1414
/reference/configuration
15+
/reference/oplog-sizing
1516
/reference/limitations
1617

‎source/reference/limitations.txt

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,14 @@ Limitations
55
===========
66

77

8+
.. default-domain:: mongodb
9+
10+
.. contents:: On this page
11+
:local:
12+
:backlinks: none
13+
:depth: 1
14+
:class: twocols
15+
816
{+c2c-full-product-name+} is under active development. This preview
917
release has several limitations. Review the list for changes after the
1018
{+c2c-product-name+} General Availability release.
@@ -138,8 +146,3 @@ In these cases, the replication of ``dropDatabase`` removes all user-created
138146
collections from the database, but leaves its system collections on the
139147
destination cluster.
140148

141-
.. contents:: On this page
142-
:local:
143-
:backlinks: none
144-
:depth: 1
145-
:class: singlecol

‎source/reference/mongosync-states.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@
1212
:depth: 2
1313
:class: singlecol
1414

15-
1615
``mongosync`` enters different states depending on the requests it
1716
receives. ``mongosync`` can only be in a single state at a given time.
1817
The current ``mongosync`` states determines which API operations you can

‎source/reference/oplog-sizing.txt

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
.. _c2c-oplog-sizing:
2+
3+
================
4+
``oplog`` Sizing
5+
================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
The :ref:`mongosync <c2c-mongosync>` program uses :ref:`change streams
16+
<changeStreams>` to synchronize data between source and destination
17+
clusters. ``mongosync`` does not access the :term:`oplog` directly,
18+
but when a change stream returns events from the past, the events must
19+
be within the :term:`oplog` time range.
20+
21+
.. include:: /includes/fact-oplog-background.txt
22+
23+
Estimate ``oplog`` Size Needed for Initial Sync
24+
-----------------------------------------------
25+
26+
.. procedure::
27+
:style: normal
28+
29+
.. step:: Determine the ``oplog`` Time Span
30+
31+
To get the difference in seconds between the first and last entry
32+
in the ``oplog`` run :method:`db.getReplicationInfo()`. If you
33+
are replicating a sharded cluster, run the command on each shard.
34+
35+
.. code-block:: javascript
36+
37+
db.getReplicationInfo().timeDiff
38+
39+
The value returned is the minimum ``oplog`` window of the
40+
cluster. If there are multiple shards, the smallest number is the
41+
minimum ``oplog`` window.
42+
43+
.. step:: Estimate Copy Rate During Synching
44+
45+
.. _c2c-step-est-size:
46+
47+
To gather performance data while synching, start the sync process
48+
and monitor how fast data is transferred between clusters.
49+
50+
To start syncing, run the :ref:`/start <c2c-api-start>` command.
51+
52+
To get the ``copy_rate``:
53+
54+
- run the :ref:`/progress <c2c-api-progress>` command to get
55+
``estimatedCopiedBytes_time01``
56+
- wait a second or two
57+
- run the :ref:`/progress <c2c-api-progress>` command to get
58+
``estimatedCopiedBytes_time02``
59+
60+
The ``copy_rate`` is:
61+
62+
.. code-block:: shell
63+
64+
copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
65+
66+
.. step:: Estimate Copy Time
67+
68+
Estimate the time needed to copy the entire collection. The
69+
estimated copy time is:
70+
71+
.. code-block:: shell
72+
73+
estimatedCopyTime = estimatedTotalBytes / copy_rate
74+
75+
.. step:: Validate ``oplog`` Size
76+
77+
If the estimated time is larger than the minimum oplog window you
78+
must cancel synchronization. Before restarting, make one of the
79+
following changes:
80+
81+
- Increase the oplog window. Use :dbcommand:`replSetResizeOplog`
82+
to set ``minRetentionHours`` greater than the estimated copy
83+
time.
84+
- Scale up the ``mongosync`` instance. Add cpu or memory to scale
85+
up the ``mongosync`` node so that it has a higher copy rate.
86+
87+
.. note::
88+
89+
The copy rate may vary during synchronization. To monitor progress,
90+
repeat the :ref:`steps to estimate the copy rate
91+
<c2c-step-est-size>` and verify that the copy rate stays about the
92+
same.
93+

0 commit comments

Comments
 (0)