DOCSP-35686 Embedded Verifier (#339)

* DOCSP-35686 Embedded Verifier

---------

Co-authored-by: Jessica <[email protected]>
Co-authored-by: John DuBois <[email protected]>

Author: 3 people

source/includes/api/requests/start-no-verifier.sh

@@ -0,0 +1,9 @@
+curl localhost:27182/api/v1/start -XPOST \
+--data '
+   {
+      "source": "cluster0",
+      "destination": "cluster1",
+      "verification": {
+         "enabled": false
+      }
+   } '

source/includes/api/responses/progress.json

@@ -15,6 +15,27 @@
             {
                "Source":"cluster0: localhost:27017",
                "Destination":"cluster1: localhost:27018"
+            },
+         "verification":
+            {
+               "source":
+                  {
+                     "estimatedDocumentCount": 42,
+                     "hashedDocumentCount": 42,
+                     "lagTimeSeconds": 2,
+                     "totalCollectionCount": 42,
+                     "scannedCollectionCount": 10,
+                     "phase": "stream hashing"
+
+                  },
+               "destination": {
+                     "estimatedDocumentCount": 42,
+                     "hashedDocumentCount": 42,
+                     "lagTimeSeconds": 2,
+                     "totalCollectionCount": 42,
+                     "scannedCollectionCount": 10,
+                     "phase": "stream hashing"
+               }
             }
       },
    "success": true

source/includes/api/tables/progress-response.rst

@@ -93,7 +93,6 @@
        and divide the result by the value of the ``estimatedTotalBytes`` field
        . Then, multiply the result by 100.
 
-
        The value of ``estimatedCopiedBytes`` may be larger than the
        value of the ``estimatedTotalBytes`` due to retried operations.
        A comparison of ``estimatedTotalBytes`` and
@@ -138,3 +137,104 @@
 
        .. versionadded:: 1.3
 
+   * - ``verification``
+     - document
+     - Provides information on the phase and progress of
+       verification checks performed by the embedded verifier.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source``
+     - document
+     - Provides information on the phase and progress of
+       verification checks running on the source cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source.`` ``estimatedDocumentCount``
+     - integer
+     - Estimated number of documents on the source cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source.`` ``hashedDocumentCount``
+     - integer
+     - Number of documents hashed by the verifier on the
+       source cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source.`` ``lagTimeSeconds``
+     - integer
+     - Time in seconds after the last verification check was
+       performed on the source cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source.`` ``phase``
+     - string
+     - Current phase of the verification process on the source
+       cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source.`` ``scannedCollectionCount``
+     - integer
+     - Number of collections scanned by the embedded verifier on
+       the source cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.source.`` ``totalCollectionCount``
+     - integer
+     - Number of collections on the source cluster to include in
+       verification checks.
+
+   * - ``verification.destination``
+     - document
+     - Provides information on the phase and progress of
+       verification checks running on the destination cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.destination.`` ``estimatedDocumentCount``
+     - integer
+     - Estimated number of documents on the destination cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.destination.`` ``hashedDocumentCount``
+     - integer
+     - Number of documents hashed by the verifier on the
+       destination cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.destination.`` ``lagTimeSeconds``
+     - integer
+     - Time in seconds since the last verification check
+       performed on the destination cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.destination.`` ``phase``
+     - string
+     - Current phase of the verification process on the
+       destination cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.destination.`` ``scannedCollectionCount``
+     - integer
+     - Number of collections scanned by the embedded verifier on
+       the destination cluster.
+
+       .. versionadded:: 1.9
+
+   * - ``verification.destination.`` ``totalCollectionCount``
+     - integer
+     - Number of collections on the destination cluster to
+       include in verification checks.
+
+       .. versionadded:: 1.9
+

source/includes/fact-verifier-beta-limitations.rst

@@ -0,0 +1,11 @@
+
+The verifier isn't compatible with these beta features:
+
+- :ref:`c2c-beta-many-to-one`
+- :ref:`c2c-beta-abc-migration`
+- :ref:`c2c-beta-document-filtering`
+- :ref:`c2c-beta-namespace-remapping`
+- :ref:`c2c-beta-destination-data-handling`
+- Clusters that run a release of MongoDB that
+  ``mongosync`` doesn't support.
+

source/includes/fact-verifier-beta-unsupport.rst

@@ -0,0 +1,7 @@
+
+|beta-feature| is not compatible with the :ref:`Embedded Verifier <c2c-embedded-verifier>`.
+The :ref:`/start <c2c-api-start>` command returns an error if you enable both the verifier
+and |beta-feature|.
+
+For more information, see :ref:`c2c-verification`.
+

source/includes/fact-verifier-buildIndexes.rst

@@ -0,0 +1,13 @@
+
+If you start sync with verification enabled and ``buildIndexes``
+set to ``never``, the migration will fail if ``mongosync`` finds
+a TTL collection on the source cluster. This can happen after
+you call the ``/start`` endpoint or much later, such as where a
+user creates a TTL index on the source cluster while a migration
+is in progress.
+
+To sync TTL collections without building indexes on the
+destination cluster, you must start sync with the :ref:`verifier
+disabled <c2c-disabled-verifier>`.
+
+

source/includes/fact-verifier-limitations.rst

@@ -0,0 +1,43 @@
+
+Compatibility
+~~~~~~~~~~~~~
+
+The embedded verifier is not available in mongosync 1.8 and
+earlier.
+
+For alternative verification methods, see :ref:`c2c-verification`.
+
+Limitations
+~~~~~~~~~~~
+
+The embedded verifier has the following limitations:
+
+- The verifier doesn't support sharded clusters. If the
+  migration includes a sharded cluster, ``mongosync`` disables
+  the verifier.
+
+- ``mongosync`` stores the verifier state in memory, which can
+  result in a significant memory overhead. To run the verifier,
+  ``mongosync`` requires approximately 10 GB of memory, plus an
+  additional 500 MB for every 1 million documents.
+
+- The verifier cannot be resumed. If a user stops or pauses sync
+  and then starts ``mongosync`` again for any reason, the
+  verification process restarts from the beginning. This can
+  cause verification to fall substantially behind the migration.
+
+- The :ref:`/reverse <c2c-api-reverse>` endpoint disables the
+  verifier. It remains disabled after additional calls to the
+  ``/reverse`` endpoint.
+
+- .. include:: /includes/fact-verifier-buildIndexes
+
+Unsupported Verification Checks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. include:: /includes/fact-verifier-unsupported
+
+Unsupported Beta Features
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. include:: /includes/fact-verifier-beta-limitations

source/includes/fact-verifier-memory.rst

@@ -0,0 +1,14 @@
+
+Verification requires 10 GB of memory plus an additional 500 MB
+for every 1 million documents in the migration. 
+
+If the available memory is insufficient, the ``/start`` endpoint
+returns an error. If this occurs, to use ``mongosync`` with the
+verifier you must first increase the memory of the server and
+resume the migration. 
+
+If increasing server memory isn't an option, restart
+``mongosync`` with the :ref:`verifier disabled
+<c2c-disabled-verifier>`.
+
+

source/includes/fact-verifier-unsupported.rst

@@ -0,0 +1,16 @@
+The verifier doesn't check the following namespaces: 
+
+- Capped collections
+- Collections with TTL indexes
+- Collections that don't use the default collation
+- Views
+
+The verifier doesn't check the following collection features:
+
+- Collection metadata
+- Indexes
+
+To verify the above data and metadata, script additional checks
+for these collections and collection features. For more
+information, see :ref:`c2c-verification`.
+

source/includes/opts/acceptDisclaimer.rst

@@ -0,0 +1,11 @@
+
+Accepts disclaimers for the :ref:`embedded verifier
+<c2c-verifier-disclaimer>` and, when using
+{+c2c-beta-program-short+}, :ref:`beta features <c2c-beta-program>`.
+
+When the ``mongosync`` process starts without this |opt-term|, the
+user is prompted to accept each disclaimer.
+
+This |opt-term| allows users who have already read and accepted
+disclaimers to accept the disclaimer and start the ``mongosync``
+process.

source/reference/api/reverse.txt

@@ -152,6 +152,15 @@ To view the mapping direction for the synchronization of the source and
 destination clusters, use the :ref:`progress <c2c-api-progress>`
 endpoint and check the ``directionMapping`` object.
 
+Embedded Verifier
+~~~~~~~~~~~~~~~~~
+
+The embedded verifier is enabled by default for replica set
+migrations and performs verification checks for the forward
+direction of reversible sync. When you call the ``/reverse``
+endpoint, ``mongosync`` disables the verifier. The verifier
+remains disabled, even after additional calls to the
+``/reverse`` endpoint.
 
 Endpoint Protection
 ~~~~~~~~~~~~~~~~~~~