Skip to content

Commit 18c9721

Browse files
schmallisoschmalliso
committed
DOCS-4739: adds production checklist
DOCS-4739: adds redirect
1 parent 66fea66 commit 18c9721

File tree

4 files changed

+333
-0
lines changed

4 files changed

+333
-0
lines changed

‎config/redirects.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,4 +132,11 @@ type: 'redirect'
132132
code: 301
133133
outputs:
134134
- 'before-v2.6'
135+
---
136+
from: '/administration/production-checklist'
137+
to: '/administration/production-notes'
138+
type: 'redirect'
139+
code: 301
140+
outputs:
141+
- 'before-v3.0'
135142
...

‎source/administration/production-checklist.txt

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
====================
2+
Production Checklist
3+
====================
4+
5+
.. default-domain:: mongodb
6+
7+
The following checklists provide recommendations that will help you
8+
avoid issues in your production MongoDB deployment.
9+
10+
Operations Checklist
11+
--------------------
12+
13+
Filesystem
14+
~~~~~~~~~~
15+
16+
.. cssclass:: checklist
17+
18+
- Align your disk partitions with your RAID configuration.
19+
20+
- Avoid using NFS drives for your :setting:`~storage.dbPath`.
21+
Using NFS drives can result in degraded and unstable performance.
22+
See: :ref:`production-nfs` for more information.
23+
24+
- VMWare users should use VMWare virtual drives over NFS.
25+
26+
- Linux/Unix: format your drives into XFS or EXT4. If using RAID,
27+
you may need to configure XFS with your RAID geometry.
28+
29+
- Windows: use the NTFS file system.
30+
**Do not** use any FAT file system (i.e. FAT 16/32/exFAT).
31+
32+
Replication
33+
~~~~~~~~~~~
34+
35+
.. cssclass:: checklist
36+
37+
- Verify that all non-hidden replica set members are identically
38+
provisioned in terms of their RAM, CPU, disk, network setup, etc.
39+
40+
- :doc:`Configure the oplog size </tutorial/change-oplog-size>` to
41+
suit your use case:
42+
43+
- The replication oplog window should cover normal maintenance and
44+
downtime windows to avoid the need for a full resync.
45+
46+
- The replication oplog window should cover the time needed to
47+
restore a replica set member, either by an initial sync, or
48+
restoring from the last backup.
49+
50+
- Ensure that your replica set includes at least three data-bearing nodes
51+
with ``w:majority`` :doc:`write concern
52+
</reference/write-concern>`. Three data-bearing nodes are
53+
required for replica set wide data durability.
54+
55+
- Use hostnames when configuring replica set members, rather than IP
56+
addresses.
57+
58+
- Ensure full bidirectional network connectivity between all
59+
:program:`mongod` instances.
60+
61+
- Ensure that each host can resolve itself.
62+
63+
- Ensure that your replica set contains an odd number of voting members.
64+
65+
.. TODO: add link to fault tolerance page when WRITING-1222 closes
66+
67+
- Ensure that :program:`mongod` instances have ``0`` or ``1`` votes.
68+
69+
- For high availability, deploy your replica set into a *minimum* of
70+
three data centers.
71+
72+
.. TODO: add link to fault tolerance page when WRITING-1222 closes
73+
74+
Sharding
75+
~~~~~~~~
76+
77+
.. cssclass:: checklist
78+
79+
- Pplace your :doc:`config servers
80+
</core/sharded-cluster-config-servers>` on dedicated hardware for
81+
optimal performance in large clusters. Ensure that the hardware has
82+
enough RAM to hold the data files entirely in memory and that it
83+
has dedicated storage.
84+
85+
- Use NTP to synchronize the clocks on all components of your sharded
86+
cluster.
87+
88+
- Ensure full bidirectional network connectivity between
89+
:program:`mongod`, :program:`mongos` and config servers.
90+
91+
- Use CNAMEs to identify your config servers to the cluster so that
92+
you can rename and renumber your config servers without downtime.
93+
94+
Journaling: MMAPv1 Storage Engine
95+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
96+
97+
.. TODO: change heading to use an H4 for MMAPv1 and WT once WT
98+
journaling notes added
99+
100+
.. cssclass:: checklist
101+
102+
- Ensure that all instances use :doc:`journaling </core/journaling>`.
103+
104+
- Place the journal on its own low-latency disk for write-intensive
105+
workloads. Note that this will affect snapshot-style backups as the
106+
files constituting the state of the database will reside on
107+
separate volumes.
108+
109+
Hardware
110+
~~~~~~~~
111+
112+
.. cssclass:: checklist
113+
114+
- Use RAID10 and SSD drives for optimal performance.
115+
116+
- SAN and Virtualization:
117+
118+
- Ensure that each :program:`mongod` has provisioned IOPS for its
119+
:setting:`~storage.dbPath`, or has its own physical drive or LUN.
120+
121+
- Avoid dynamic memory features, such as memory ballooning, when
122+
running in virtual environments.
123+
124+
- Avoid placing all replica set members on the same SAN, as the SAN
125+
can be a single point of failure.
126+
127+
Deployments to Cloud Hardware
128+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
129+
130+
.. cssclass:: checklist
131+
132+
- Windows Azure: Adjust the TCP keepalive (``tcp_keepalive_time``) to
133+
100-120. The default TTL for TCP connections on Windows Azure load
134+
balancers is too show for MongoDB's connection pooling behavior.
135+
136+
- Use MongoDB version 2.6.4 or later on systems with high-latency
137+
storage, such as Windows Azure, as these versions include
138+
performance improvements for those systems. See: :ecosystem:`Azure
139+
Deployment Recommendations </platforms/windows-azure>` for more
140+
information.
141+
142+
Operating System Configuration
143+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
144+
145+
Linux
146+
`````
147+
148+
.. cssclass:: checklist
149+
150+
- Turn off transparent hugepages and defrag. See: :ref:`Recommended
151+
Configuration for MongoDB on Linux <linux-recommended-configuration>`
152+
for more information.
153+
154+
- :ref:`Adjust the readahead settings <readahead>` on the devices
155+
storing your database files to suit your use case. If your working
156+
set is bigger that the available RAM, and the document access
157+
pattern is random, consider lowering the readahead to 32 or 16.
158+
Evaluate different settings to find an optimal value that maximizes
159+
the resident memory and lowers the number of page faults.
160+
161+
- Use the ``noop`` or ``deadline`` disk schedulers for SSD drives.
162+
163+
- Use the ``noop`` disk scheduler for virtualized drives in guest VMs.
164+
165+
- Disable NUMA or set vm.zone_reclaim_mode to 0 and run :program:`mongod`
166+
instances with node interleaving. See: :ref:`production-numa`
167+
for more information.
168+
169+
- Adjust the ``ulimit`` values on your hardware to suit your use case. If
170+
multiple :program:`mongod` or :program:`mongos` instances are
171+
running under the same user, scale the ``ulimit`` values
172+
accordingly. See: :doc:`/reference/ulimit` for more information.
173+
174+
- Use ``noatime`` for the :setting:`~storage.dbPath` mount point.
175+
176+
- Configure sufficient file handles (``fs.file-max``), kernel pid limit
177+
(``kernel.pid_max``), and maximum threads per process
178+
(``kernel.threads-max``) for your deployment. For large systems,
179+
values of 98000, 32768, and 64000 are a good starting point.
180+
181+
- Ensure that your system has swap space configured. Refer to your
182+
operating system's documentation for details on appropriate sizing.
183+
184+
- Ensure that the system default TCP keepalive is set correctly. A
185+
value of 300 often provides better performance for replica sets and
186+
sharded clusters. See: :ref:`faq-keepalive` in the Frequently Asked
187+
Questions for more information.
188+
189+
Windows
190+
```````
191+
192+
.. cssclass:: checklist
193+
194+
- Consider disabling NTFS "last access time" updates. This is
195+
analogous to diabling ``atime`` on Unix-like systems.
196+
197+
Backups
198+
~~~~~~~
199+
200+
.. cssclass:: checklist
201+
202+
- Schedule periodic tests of your back up and restore process to have
203+
time estimates on hand, and to verify its functionality.
204+
205+
Monitoring
206+
~~~~~~~~~~
207+
208+
.. cssclass:: checklist
209+
210+
- Use :mms-home:`MMS </>` or another monitoring system to monitor
211+
key database metrics and set up alerts for them. Include alerts
212+
for the following metrics:
213+
214+
- lock percent (for the :ref:`MMAPv1 storage engine <storage-mmapv1>`)
215+
- replication lag
216+
- replication oplog window
217+
- assertions
218+
- queues
219+
- page faults
220+
221+
- Monitor hardware statistics for your servers. In particular,
222+
pay attention to the disk use, CPU, and available disk space.
223+
224+
In the absence of disk space monitoring, or as a precaution:
225+
226+
- Create a dummy 4GB file on the :setting:`storage.dbPath` drive to
227+
ensure available space if the disk becomes full.
228+
229+
- A combination of ``cron+df`` can alert when disk space hits a
230+
high-water mark, if no other monitoring tool is available.
231+
232+
Load Balancing
233+
~~~~~~~~~~~~~~
234+
235+
.. cssclass:: checklist
236+
237+
- Configure load balancers to enable "sticky sessions" or "client
238+
affinity", with a sufficient timeout for existing connections.
239+
240+
- Avoid placing load balancers between MongoDB cluster or replica set
241+
components.
242+
243+
Development
244+
-----------
245+
246+
Data Durability
247+
~~~~~~~~~~~~~~~
248+
249+
.. cssclass:: checklist
250+
251+
- Ensure that your replica set includes at least three data-bearing nodes
252+
with ``w:majority`` :doc:`write concern
253+
</reference/write-concern>`. Three data-bearing nodes are
254+
required for replica-set wide data durability.
255+
256+
- Ensure that all instances use :doc:`journaling </core/journaling>`.
257+
258+
Schema Design
259+
~~~~~~~~~~~~~
260+
261+
.. cssclass:: checklist
262+
263+
- Ensure that your schema design does not rely on indexed arrays that
264+
grow in length without bound. Typically, best performance can
265+
be achieved when such indexed arrays have fewer than 1000 elements.
266+
267+
Replication
268+
~~~~~~~~~~~
269+
270+
.. cssclass:: checklist
271+
272+
- Do not use secondary reads to scale overall read throughput. See:
273+
`Can I use more replica nodes to scale`_ for an overview of read
274+
scaling. For information about secondary reads, see:
275+
:doc:`/core/read-preference`.
276+
277+
.. _Can I use more replica nodes to scale: http://askasya.com/post/canreplicashelpscaling
278+
279+
Sharding
280+
~~~~~~~~
281+
282+
.. cssclass:: checklist
283+
284+
- Ensure that your shard key distributes the load evenly on your shards.
285+
See: :doc:`/tutorial/choose-a-shard-key` for more information.
286+
287+
- Use :doc:`targeted queries </core/sharded-cluster-query-router>`
288+
for workloads that need to scale with the number of shards.
289+
290+
- Always read from primary nodes for non-targeted queries that may
291+
be sensitive to `stale or orphaned data <http://blog.mongodb.org/post/74730554385/background-indexing-on-secondaries-and-orphaned>`_.
292+
293+
- :doc:`Pre-split and manually balance chunks
294+
</tutorial/create-chunks-in-sharded-cluster>` when inserting large
295+
data sets into a new non-hashed sharded collection. Pre-splitting
296+
and manually balancing enables the insert load to be distributed
297+
among the shards, increasing performance for the initial load.
298+
299+
Drivers
300+
~~~~~~~
301+
302+
.. cssclass:: checklist
303+
304+
- Make use of connection pooling. Most MongoDB drivers support
305+
connection pooling. Adjust the connection pool size to suit your
306+
use case, beginning at 110-115% of the typical number of concurrent
307+
database requests.
308+
309+
- Ensure that your applications handle transient write and read errors
310+
during replica set elections.
311+
312+
- Ensure that your applications handle failed requests and retry them if
313+
applicable. Drivers **do not** automatically retry failed requests.
314+
315+
- Use exponential backoff logic for database request retries.
316+
317+
- Use :method:`cusor.maxTimeMS()` for reads and :ref:`wc-wtimeout` for
318+
writes if you need to cap execution time for database operations.
319+

‎source/administration/production-notes.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -442,6 +442,8 @@ should use the Ext4 and XFS file systems:
442442
*on directories*. For example, HGFS and Virtual Box's shared
443443
folders do *not* support this operation.
444444

445+
.. _linux-recommended-configuration:
446+
445447
Recommended Configuration
446448
`````````````````````````
447449

‎source/includes/toc-administration-landing.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,9 @@ file: /reference/administration
1212
description: |
1313
Reference and documentation of internal mechanics of administrative
1414
features, systems and functions and operations.
15+
---
16+
file: /administration/production-checklist
17+
description: |
18+
Checklist of recommendations for deploying
19+
MongoDB in production.
1520
...

0 commit comments

Comments
 (0)