Skip to content

Commit cd6b96e

Browse files
jason-price-mongodbjason-price-mongodb
andauthored
DOCSP-26914 sharding defragmentation refactor (#2594)
* DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor * DOCSP-26914-sharding-defragmentation-refactor --------- Co-authored-by: jason-price-mongodb <[email protected]>
1 parent ce08ff9 commit cd6b96e

16 files changed

+624
-0
lines changed

snooty.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ toc_landing_pages = [
8787
"/core/sharded-cluster-components",
8888
"/core/sharding-change-a-shard-key",
8989
"/core/sharding-balancer-administration",
90+
"/core/defragment-sharded-collections",
9091
"/core/sharding-data-partitioning",
9192
"/core/sharding-shard-key",
9293
"/core/storage-engines",
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
.. _defragment-sharded-collections:
2+
3+
==============================
4+
Defragment Sharded Collections
5+
==============================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 1
13+
:class: singlecol
14+
15+
.. include:: /includes/defragment-sharded-collections-conditions.rst
16+
17+
To defragment a sharded collection, use the
18+
:dbcommand:`configureCollectionBalancing` command's
19+
``defragmentCollection`` option. The option is available starting in
20+
MongoDB 6.0.
21+
22+
Before you Begin
23+
----------------
24+
25+
Consider these issues before you defragment collections:
26+
27+
- Defragmentation might cause many metadata updates on the shards. If
28+
your CRUD operations are already taking longer than usual during
29+
migrations, you should only run defragmentation during a :ref:`shard
30+
balancing window <sharding-schedule-balancing-window>` to reduce the
31+
system workload.
32+
- If defragmentation is impacting workload and CRUD latency on the
33+
cluster, you can reduce the impact using the
34+
:parameter:`chunkDefragmentationThrottlingMS` parameter.
35+
- Merged chunks lose their placement history.
36+
37+
- This means that while defragmentation is running, snapshot reads and
38+
indirectly, transactions, could fail with stale chunk history
39+
errors.
40+
- Placement history records the shards that a chunk was stored on.
41+
Defragmentation erases the placement history and some operations
42+
could fail, but will typically resolve after around five minutes.
43+
44+
- Defragmentation affects the locality of the documents in a collection
45+
by moving data between shards. If a collection has ranges of data that
46+
are frequently accessed, after defragmenting the collection it is
47+
possible that the frequently accessed data will be on one shard. This
48+
might decrease the performance of CRUD operations by placing the
49+
workload on one shard instead of multiple shards.
50+
51+
Tasks
52+
-----
53+
54+
- :ref:`Manually start defragmenting a sharded collection
55+
<start-defragmenting-sharded-collection>`
56+
- :ref:`Monitor defragmentation of a sharded collection
57+
<monitor-defragmentation-sharded-collection>`
58+
- :ref:`Manually stop defragmenting a sharded collection
59+
<stop-defragmenting-sharded-collection>`
60+
61+
.. note::
62+
63+
Typically, you should use a :ref:`shard balancing window
64+
<sharding-schedule-balancing-window>` to specify when the balancer
65+
runs instead of manually starting and stopping defragmentation.
66+
67+
Details
68+
-------
69+
70+
This section describes additional details related to defragmenting
71+
sharded collections.
72+
73+
Configure Collection Balancing Status
74+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
75+
76+
The ``defragmentCollection`` field returned by the
77+
:dbcommand:`configureCollectionBalancing` command is only ``true`` when
78+
defragmentation is running.
79+
80+
After defragmentation automatically ends or you manually stop
81+
defragmentation, the ``defragmentCollection`` field is removed from the
82+
returned document.
83+
84+
Operations
85+
~~~~~~~~~~
86+
87+
Secondary node reads are permitted during defragmentation, but might
88+
take longer to complete until metadata updates on the primary node are
89+
replicated to the secondary nodes.
90+
91+
Chunk Size, Balancing, and Defragmentation
92+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
93+
94+
For details about the MongoDB balancer, see :ref:`sharding-balancing`.
95+
96+
.. include:: /includes/chunk-size-and-balancing.rst
97+
98+
Learn More
99+
----------
100+
101+
- Introduction to sharding, see :ref:`sharding-introduction`
102+
- Partition data with chunks, see :ref:`sharding-data-partitioning`
103+
- Configure collection balancing, see
104+
:dbcommand:`configureCollectionBalancing`
105+
- Examine balancer collection status, see
106+
:dbcommand:`balancerCollectionStatus`
107+
- Configure shard balancing windows, see
108+
:ref:`sharding-schedule-balancing-window`
109+
- Monitor shards using MongoDB Atlas, see `Review Sharded Clusters
110+
<https://www.mongodb.com/docs/atlas/review-sharded-cluster-metrics/#review-sharded-clusters/>`__
111+
112+
.. toctree::
113+
:titlesonly:
114+
115+
/core/defragment-sharded-collections/start-defragmenting-sharded-collection
116+
/core/defragment-sharded-collections/monitor-defragmentation-sharded-collection
117+
/core/defragment-sharded-collections/stop-defragmenting-sharded-collection
Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
.. _monitor-defragmentation-sharded-collection:
2+
3+
===============================================
4+
Monitor Defragmentation of a Sharded Collection
5+
===============================================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 1
13+
:class: singlecol
14+
15+
To monitor defragmentation of a sharded collection, use the
16+
:dbcommand:`balancerCollectionStatus` command.
17+
18+
You can see the current defragmentation state and the number of
19+
remaining chunks to process. This shows you the defragmentation
20+
progress.
21+
22+
About this Task
23+
---------------
24+
25+
.. include:: /includes/defragment-sharded-collections-status.rst
26+
27+
.. include:: /includes/defragment-sharded-collections-example.rst
28+
29+
In the procedure for this task, you monitor the phases and see the
30+
defragmentation progress.
31+
32+
Before you Begin
33+
----------------
34+
35+
- Start defragmenting a sharded collection. For details, see
36+
:ref:`start-defragmenting-sharded-collection`.
37+
- Connect to :binary:`~bin.mongos`.
38+
39+
Procedure
40+
---------
41+
42+
.. procedure::
43+
:style: normal
44+
45+
.. step:: Monitor defragmentation
46+
47+
Run:
48+
49+
.. code-block:: javascript
50+
51+
db.adminCommand(
52+
{
53+
balancerCollectionStatus: "test.ordersShardedCollection"
54+
}
55+
)
56+
57+
.. step:: Examine output document
58+
59+
The previous command returns a document with information about
60+
defragmentation status, current phase, and the defragmentation
61+
work remaining. For example:
62+
63+
.. code-block:: javascript
64+
:copyable: false
65+
66+
{
67+
"balancerCompliant": false,
68+
"firstComplianceViolation": "defragmentingChunks",
69+
"details": {
70+
"currentPhase": "moveAndMergeChunks",
71+
"progress": { "remainingChunksToProcess": 1 }
72+
}
73+
}
74+
75+
The following table describes the document fields.
76+
77+
.. list-table::
78+
:header-rows: 1
79+
:widths: 20 20 60
80+
81+
* - Field
82+
- Type
83+
- Description
84+
85+
* - ``balancerCompliant``
86+
- Boolean
87+
- ``false`` if collection chunks must be moved. Otherwise,
88+
``true``.
89+
90+
* - ``firstComplianceViolation``
91+
- String
92+
- Indicates the reason that chunks for the namespace must be
93+
moved or merged. Only returned if ``balancerCompliant`` is
94+
``false``.
95+
96+
* - ``details``
97+
- Object
98+
- Addtional information about the current defragmentation
99+
state. Only returned if ``firstComplianceViolation`` is
100+
``defragmentingChunks``.
101+
102+
* - ``currentPhase``
103+
- String
104+
- Current defragmentation phase:
105+
106+
- For phase one, ``currentPhase`` is
107+
``mergeAndMeasureChunks``.
108+
109+
Phase one merges contiguous chunks located on the same
110+
shard and calculates the data size for those chunks.
111+
112+
- For phase two, ``currentPhase`` is
113+
``moveAndMergeChunks``.
114+
115+
After phase one is complete, there might be some small
116+
chunks remaining. Phase two migrates those small chunks
117+
to other shards and merges the chunks on those shards.
118+
119+
* - ``remainingChunksToProcess``
120+
- Integer
121+
- Number of remaining chunks to process in the current phase.
122+
123+
For additional information about the returned document fields, see
124+
the :ref:`balancer collection status output document
125+
<cmd-balancer-CollectionStatus-output>`.
126+
127+
.. step:: Confirm that defragmentation is complete
128+
129+
After defragmentation completes, the command returns either:
130+
131+
- ``balancerCompliant: true`` if your collection is balanced.
132+
- ``balancerCompliant: false`` with ``firstComplianceViolation``
133+
set to a string other than ``defragmentingChunks`` if
134+
your collection is not balanced.
135+
136+
Example output for a balanced collection after defragmentation
137+
completes:
138+
139+
.. code-block:: javascript
140+
:copyable: false
141+
:emphasize-lines: 3
142+
143+
{
144+
chunkSize: 0.2,
145+
balancerCompliant: true,
146+
ok: 1,
147+
'$clusterTime': {
148+
clusterTime: Timestamp({ t: 1677543079, i: 1 }),
149+
signature: {
150+
hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
151+
keyId: Long("0")
152+
}
153+
},
154+
operationTime: Timestamp({ t: 1677543079, i: 1 })
155+
}
156+
157+
Next Steps
158+
----------
159+
160+
If defragmentation has not yet completed, you can stop it. For details,
161+
see :ref:`stop-defragmenting-sharded-collection`.
162+
163+
Learn More
164+
----------
165+
166+
- :ref:`Start defragmenting a sharded collection
167+
<start-defragmenting-sharded-collection>`
168+
- :ref:`Stop defragmenting a sharded collection
169+
<stop-defragmenting-sharded-collection>`
170+
- To view the balancer collection status output document, see
171+
:ref:`Balancer collection status output document
172+
<cmd-balancer-CollectionStatus-output>`
173+
174+
.. include:: /includes/defragment-sharded-collections-learn-more.rst
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
.. _start-defragmenting-sharded-collection:
2+
3+
========================================
4+
Start Defragmenting a Sharded Collection
5+
========================================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 1
13+
:class: singlecol
14+
15+
To start defragmenting a sharded collection, use the
16+
:dbcommand:`configureCollectionBalancing` command with the
17+
``defragmentCollection`` option set to ``true``.
18+
19+
About this Task
20+
---------------
21+
22+
.. include:: /includes/defragment-sharded-collections-conditions.rst
23+
24+
.. include:: /includes/defragment-sharded-collections-example.rst
25+
26+
Before you Begin
27+
----------------
28+
29+
Connect to :binary:`~bin.mongos`.
30+
31+
Procedure
32+
---------
33+
34+
.. procedure::
35+
:style: normal
36+
37+
.. step:: Start defragmenting the collection
38+
39+
Run:
40+
41+
.. code-block:: javascript
42+
43+
db.adminCommand(
44+
{
45+
configureCollectionBalancing: "test.ordersShardedCollection",
46+
defragmentCollection: true
47+
}
48+
)
49+
50+
.. step:: Ensure defragmentation started
51+
52+
Ensure ``ok`` is ``1`` in the command output, which indicates the
53+
command execution was successful:
54+
55+
.. code-block:: javascript
56+
:copyable: false
57+
:emphasize-lines: 2
58+
59+
{
60+
ok: 1,
61+
'$clusterTime': {
62+
clusterTime: Timestamp({ t: 1677616966, i: 8 }),
63+
signature: {
64+
hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
65+
keyId: Long("0")
66+
}
67+
},
68+
operationTime: Timestamp({ t: 1677616966, i: 8 })
69+
}
70+
71+
Next Steps
72+
----------
73+
74+
You can monitor the collection's defragmentation progress. For details,
75+
see :ref:`monitor-defragmentation-sharded-collection`.
76+
77+
Learn More
78+
----------
79+
80+
.. include:: /includes/defragment-sharded-collections-learn-more.rst

0 commit comments

Comments
 (0)