DOCSP-26021 sharding metadata consistency (#3114)

* Fixes per Ali

* DOCSP-26021 Adds checkMetadataConsistency command

* Adds example

* Fixes build issue

* Fixes links

* Documents batching checkMetadataConsistency

* Documents checkIndexes option

* Fixes example

* Fixes table

* Fixes per Ian

* Fixes per Ian

* Fixes example issue

* Fixes per Ian

* Fix build issue

* Fixes build issue

* Fixes per Tocci

* Fixes per Tocci

* Fixes per Tocci

* Fixes per Tocci

* Fixes per Tocci

* Fixes build issues

* Fixes per Sarah

Co-authored-by: Sarah Olson <[email protected]>

* Fixes per Sarah

---------

Co-authored-by: Sarah Olson <[email protected]>

Author: kennethdyer

source/core/sharded-cluster-requirements.txt

@@ -56,6 +56,25 @@ Unique Indexes in Sharded Collections
 
 .. include:: /includes/limits-sharding-unique-indexes.rst
 
+Consistent Indexes
+~~~~~~~~~~~~~~~~~~
+
+MongoDB does not guarantee consistent indexes across shards.  Index creation
+during :dbcommand:`addShard` operations or chunk migrations may not propagate
+to new shards.
+
+To check a sharded cluster for consistent indexes, use the
+:dbcommand:`checkMetadataConsistency` command:
+
+
+.. code-block:: javascript
+
+   db.runCommand( {
+      checkMetadataConsistency: 1,
+      checkIndexes: true
+   } )
+
+
 Sharding Existing Collection Data Size
 --------------------------------------
 

source/reference/command.txt

@@ -447,6 +447,12 @@ Sharding Commands
 
      - Stops the balancer thread.
 
+   * - :dbcommand:`checkMetadataConsistency`
+
+     - Performs a series of consistency checks on sharding metadata.
+
+       .. versionadded:: 7.0
+
    * - :dbcommand:`checkShardingIndex`
 
      - Internal command that validates index on shard key.

source/reference/command/checkMetadataConsistency.txt

@@ -0,0 +1,204 @@
+========================
+checkMetadataConsistency
+========================
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 2
+   :class: singlecol
+
+
+Definition
+----------
+
+.. dbcommand:: checkMetadataConsistency
+
+   Performs a series of consistency checks on sharding metadata for a cluster,
+   database, or collection.  The command returns a cursor with either all or a
+   batch of the inconsistency results found.
+
+   Run this command after major maintenance operations, such as upgrades and
+   downgrades, to check the state of the catalog.
+
+   By default, the command does not check indexes for consistency across 
+   the shards.  To check indexes, set the ``checkIndexes`` option.
+
+   .. versionadded:: 7.0
+
+Syntax
+------
+
+-  To check the entire cluster for sharding metadata inconsistencies, run
+   the command from the ``admin`` database.
+
+   .. code-block:: javascript
+
+      db.getSiblingDB("admin").runCommand( {
+         checkMetadataConsistency: 1
+      } )
+
+-  To check the database for sharding metadata inconsistencies, run the command
+   from the database context:
+
+   .. code-block:: javascript
+
+      use cars
+      db.runCommand( {
+         checkMetadataConsistency: 1
+      } )
+
+-  To check a collection for sharding metadata inconsistencies, run the command
+   with the ``coll`` option:
+
+   .. code-block:: javascript
+
+      use library
+      db.runCommand( {
+         checkMetadataConsistency: 1,
+         coll: "authors",
+      } )
+
+
+Command Fields
+~~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+
+   * - Field
+     - Type
+     - Description
+
+   * - ``checkIndexes``
+     - boolean 
+     - Sets whether the command also checks indexes in sharding metadata.
+
+       For more information, see :ref:`checkMetadataConsistency-indexes`.
+
+   * - ``coll``
+     - string
+     - Sets the collection to check for sharding metadata inconsistencies.
+
+   * - ``cursor``
+     - document
+     - Configures the return cursor.
+
+   * - ``cursor.batchSize``
+     - integer
+     - Maximum number of inconsistency results to include in each batch.
+
+Output
+~~~~~~
+
+The ``checkMetadataConsistency`` command returns a cursor with a document for
+each inconsistency found in sharding metadata.
+
+The return document has the following fields:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 30 45
+
+   * - Field
+     - Type
+     - Description
+
+   * - ``cursor``
+     - document
+     - Cursor with the results of the inconsistency checks.
+
+   * - ``cursor.id``
+     - integer
+     - A 64-bit integer indicated the cursor ID.  Use the ``cursor.id`` value
+       with the :dbcommand:`getMore` command to retrieve the next batch 
+       of inconsistencies.
+
+       If the cursor returns an ID of ``0``, it indicates that there are no
+       more batches of information.
+
+
+   * - ``cursor.ns``
+     - string
+     - The database and collection checked for inconsistencies.
+
+   * - ``cursor.firstBatch``
+     - array
+     - Results of metadata consistency checks.
+
+   * - ``ok``
+     - boolean
+     - Indicates whether the command was successful.
+
+Behavior
+--------
+
+Batch Results
+~~~~~~~~~~~~~
+
+The ``checkMetadataConsistency`` command returns results in batches. To
+customize the batch size, the ``batchSize`` option:
+
+.. code-block:: javascript
+
+   var cur = db.runCommand( {
+      checkMetadataConsistency: 1,
+      cursor: {
+         batchSize: 10
+      }
+   } )
+
+If the ``cursor.id`` field is greater than 0, you can use with the
+:dbcommand:`getMore` command to retrieve the next batch of results.
+
+
+.. _checkMetadataConsistency-indexes:
+
+Check Indexes
+~~~~~~~~~~~~~
+
+The ``checkMetadataConsistency`` command does not check indexes by default.  
+To check metadata consistency and indexes, use the ``checkIndexes`` option:
+
+.. code-block:: javascript
+
+   db.runCommand( {
+      checkMetadataConsistency: 1,
+      checkIndexes: true
+   } )
+
+
+Example
+-------
+
+
+Use :method:`~db.runCommand` to run the ``checkMetadataConsistency`` command:
+
+.. code-block:: javascript
+
+   db.runCommand( { checkMetadataConsistency: 1 } )
+
+Example Output:
+
+.. code-block:: json
+   :copyable: false
+
+   {
+      cursor: {
+         id: Long("0"),
+         ns: "test.$cmd.aggregate",
+         firstBatch: [
+            {
+               type: "MisplacedCollection",
+               description: "Unsharded collection found on shard different from database primary shard",
+               details: {
+                  namespace: "test.authors",
+                  shard: "shard02",
+                  localUUID: new UUID("1ad56770-61e2-48e9-83c6-8ecefe73cfc4")
+               }
+            }
+         ],
+      },
+      ok: 1
+   }
+

source/reference/command/nav-sharding.txt

@@ -55,6 +55,12 @@ Sharding Commands
 
      - Stops the balancer thread.
 
+   * - :dbcommand:`checkMetadataConsistency`
+
+     - Performs a series of consistency checks on sharding metadata.
+
+       .. versionadded:: 7.0
+
    * - :dbcommand:`checkShardingIndex`
 
      - Internal command that validates index on shard key.
@@ -215,6 +221,7 @@ Sharding Commands
    /reference/command/balancerStart
    /reference/command/balancerStatus
    /reference/command/balancerStop
+   /reference/command/checkMetadataConsistency
    /reference/command/checkShardingIndex
    /reference/command/clearJumboFlag
    /reference/command/cleanupOrphaned

source/reference/explain-results.txt

@@ -225,7 +225,7 @@ representative. Your output may differ significantly.
                         }
                         ]
                      }
-                  },
+                  }
                }
 
 .. data:: explain.queryPlanner