DOCSP-23354 disaster recovery v1.0 (#75)

* DOCSP-23354 Disaster recovery

* DOCSP-23354 Disaster recovery

* Staging fixes

* review feedback

* Review feedback

* Staging fixes

* Reveiw feedback

Author: Dave Cuthbert

source/reference.txt

@@ -12,5 +12,7 @@ Reference
    /reference/api
    /reference/mongosync-states
    /reference/oplog-sizing
+   /reference/configuration
+   /reference/failure-recovery
    /reference/limitations
 

source/reference/failure-recovery.txt

@@ -0,0 +1,122 @@
+.. _c2c-failure-recovery:
+
+==========================
+Recovery From Sync Failure
+==========================
+
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 2
+   :class: singlecol
+
+:program:`mongosync` temporarily alters some collection characteristics
+during synchronization. The original values are restored during the
+:ref:`commit process <c2c-api-commit>`. However, a permanent system
+failure could disrupt synchronization before the sync is committed. If
+the sync ends prematurely, you must manually reset the collection
+characteristics on the destination cluster.
+
+Manually revert the following temporary changes:
+
+- :ref:`Unique indexes set as non-unique <c2c-dr-unique>`
+- :ref:`TTL indexes with expireAfterSeconds set to MAX_INT
+  <c2c-dr-ttl>`
+- :ref:`Hidden indexes set as non-hidden <c2c-dr-hidden>`
+- :ref:`Capped collections with size/max set to the maximum possible
+  value <c2c-dr-capped>`
+- :ref:`Write blocking <c2c-dr-write-blocking>`
+
+.. _c2c-dr-unique:
+
+Unique Indexes
+--------------
+
+:program:`mongosync` replicates writes in parallel, which can cause
+transient uniqueness violations on the destination cluster. To avoid
+these errors, unique indexes are replicated as non-unique. If
+synchronization ends early, manually resolve any uniqueness violations
+and convert the non-unique indexes back to unique indexes.
+
+Prepare the Index for Conversion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each non-unique index on the destination cluster that should be
+unique, run :dbcommand:`collMod` with ``prepareUnique`` set to
+``true``.
+
+After running ``collMod``, the index rejects any new writes that
+would introduce duplicate keys.
+
+Resolve Uniqueness Violations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There may still be uniqueness conflicts in the index after you run
+``prepareUnique``. To find conflicts and convert the index to a unique
+index, run ``collMod`` with ``unique`` set to ``true`` on each of the
+non-unique indexes that you want to set as unique.
+
+If the call to ``collMod`` succeeds, there are no uniqueness violations
+on that index and the index is converted to an unique index.
+
+If the call to ``collMod`` fails with a ``CannotConvertIndexToUnique``
+error, correct the uniqueness violations and rerun ``collMod``.
+
+See: `Convert an Existing Index
+<https://www.mongodb.com/docs/v6.0/reference/command/collMod/#convert-an-existing-index-to-a-unique-index>`__
+
+.. _c2c-dr-ttl:
+
+TTL Indexes
+-----------
+
+During synchronization, ``expireAfterSeconds`` is set
+to ``MAX_INT`` for :ref:`TTL indexes
+<index-feature-ttl>`. To reset ``expireAfterSeconds``, use the
+``collMod`` command. See: :ref:`Change Expiration Value for Indexes
+<ex-change-exp-value>`.
+
+
+.. _c2c-dr-hidden:
+
+Hidden Indexes
+--------------
+During synchronization, :ref:`hidden indexes <index-type-hidden>` are
+not hidden on the destination cluster. Use the ``collMod`` command to
+hide indexes that should be hidden.
+
+.. _c2c-dr-capped:
+
+Capped Collections
+------------------
+
+During synchronization, :ref:`capped collections
+<manual-capped-collection>` are set to the maximum allowable size. Use
+the ``collMod`` command to set the correct maximum size.
+
+.. _c2c-dr-write-blocking:
+
+Write blocking
+--------------
+
+When write blocking is enabled as part of the :ref:`/start
+<c2c-api-start>` command, ``mongosync`` blocks user writes on the
+destination cluster during replication. If replication ends
+prematurely, you must manually unblock writing on the destination.
+
+After writing is enabled, you can redirect writes from your client
+application to the destination cluster.
+
+To enable writes, update :dbcommand:`setUserWriteBlockMode`:
+
+.. code-block:: javascript
+
+   db.adminCommand(
+       {
+           setUserWriteBlockMode: 1,
+           global: false
+       }
+   )
+