DOCS-11962,DOCS-12387,DOCS-11959,DOCS-11965: queryHash and planCacheKey

Author: Kay Kim

source/administration/analyzing-mongodb-performance.txt

@@ -136,6 +136,8 @@ The following profiling levels are available:
 
 .. note::
    .. include:: /includes/fact-log-slow-queries.rst
+   
+.. include:: /includes/extracts/4.2-changes-log-query-shapes-plan-cache-key.rst
 
 .. _ftdc-stub:
 

source/core/index-partial.txt

@@ -46,6 +46,8 @@ You can specify a ``partialFilterExpression`` option for all MongoDB
 Behavior
 --------
 
+.. _partial-index-query-coverage:
+
 Query Coverage
 ~~~~~~~~~~~~~~
 

source/core/query-plans.txt

@@ -42,17 +42,9 @@ The following diagram illustrates the query planner logic:
 See :ref:`query-plans-plan-cache-flushes` for additional scenarios that trigger
 changes to the plan cache.
 
-You can use the :method:`db.collection.explain()` or the
-:method:`cursor.explain()` method to view statistics about the query
-plan for a given query. This information can help as you develop
-:doc:`indexing strategies </applications/indexes>`.
-
-.. include:: /includes/fact-explain-collection-method.rst
-
-.. versionchanged:: 2.6
-
-   :method:`~db.collection.explain()` operations no longer read from or write
-   to the query planner cache.
+See :method:`db.collection.explain()` or the :method:`cursor.explain()`
+method for information about viewing query plan information for a given
+query.
 
 .. _query-plans-query-revision:
 
@@ -61,16 +53,85 @@ plan for a given query. This information can help as you develop
 Plan Cache Flushes
 ------------------
 
-Catalog operations like index or collection drops flush the plan cache.
+The query plan cache does not persist if a :binary:`~bin.mongod`
+restarts or shuts down. In addition, catalog operations like index or
+collection drops clear the plan cache.
 
-The plan cache does not persist if a :binary:`~bin.mongod` restarts or shuts down.
+Users can also:
 
-.. versionadded:: 2.6
+- Manually clear the entire plan cache using the
+  :method:`PlanCache.clear()` method.
+
+- Manually clear specific plan cache entries using the
+  :method:`PlanCache.clearPlansByQuery()` method.
+
+.. seealso:: :ref:`query-hash-plan-cache-key`
+
+.. _query-hash-plan-cache-key:
+
+``queryHash`` and ``planCacheKey``
+----------------------------------
+
+.. _query-hash:
+
+
+``queryHash``
+~~~~~~~~~~~~~
+
+.. include:: /includes/extracts/4.2-changes-query-shapes.rst
+
+.. _plan-cache-key:
+
+``planCacheKey``
+~~~~~~~~~~~~~~~~
+
+.. include:: /includes/extracts/4.2-changes-plan-cache-key.rst
 
-   MongoDB provides :doc:`/reference/method/js-plan-cache` to view and modify
-   the cached query plans. The :method:`PlanCache.clear()` method flushes the
-   entire plan cache. Users can also clear particular plan cache entries using
-   :method:`PlanCache.clearPlansByQuery()`.
+For example, consider a collection ``foo`` with the following indexes:
+ 
+.. code-block:: javascript
+
+   db.foo.createIndex( { x: 1 } )
+   db.foo.createIndex( { x: 1, y: 1 } )
+   db.foo.createIndex( { x: 1, z: 1 }, { partialFilterExpression: { x: { $gt: 10 } } } )
+
+The following queries on the collection have the same shape:
+
+.. code-block:: javascript
+
+   db.foo.explain().find( { x: { $gt: 5 } } )  // Query Operation 1
+   db.foo.explain().find( { x: { $gt: 20 } } ) // Query Operation 2
+
+Given these queries, the index with the :ref:`partial filter expression
+<partial-index-query-coverage>` can support query operation 2 but *not*
+support query operation 1. Since the indexes available to support query operation 1
+differs from query operation 2, the two queries have different
+``planCacheKey``.
+
+If one of the indexes were dropped, or if a new index ``{ x: 1, a: 1
+}`` were added, the ``planCacheKey`` for both query operations will
+change.
+
+Availability
+~~~~~~~~~~~~
+
+The ``queryHash`` and ``planCacheKey`` are available in:
+
+- :doc:`explain() output </reference/explain-results>` fields:
+  :data:`queryPlanner.queryHash <explain.queryPlanner.queryHash>` and
+  :data:`queryPlanner.planCacheKey <explain.queryPlanner.planCacheKey>`
+
+- :doc:`profiler log messages </tutorial/manage-the-database-profiler>`
+  and :doc:`diagnostic log messages (i.e. mongod/mongos log
+  messages)</reference/log-messages>` when logging slow queries.
+
+- :pipeline:`$planCacheStatus` aggregation stage (*New in MongoDB 4.2*)
+
+- :method:`PlanCache.listQueryShapes()`
+  method/:dbcommand:`planCacheListQueryShapes` command
+
+- :method:`PlanCache.getPlansByQuery()`
+  method/:dbcommand:`planCacheListPlans` command
 
 .. _index-filters:
 
@@ -105,3 +166,7 @@ sparingly.
 
 See :dbcommand:`planCacheListFilters`,
 :dbcommand:`planCacheClearFilters`, and :dbcommand:`planCacheSetFilter`.
+
+.. seealso::
+
+   :doc:`/applications/indexes`

source/includes/extracts-4.2-changes.yaml

@@ -105,6 +105,53 @@ content: |
    entries </tutorial/manage-the-database-profiler>` and
    :doc:`diagnostic log messages </reference/log-messages>` include
    :data:`~system.profile.storage` information.
+---
+ref: 4.2-changes-log-query-shapes-plan-cache-key
+content: |
+
+   Starting in MongoDB 4.2, the :doc:`profiler entries
+   </tutorial/manage-the-database-profiler>` and the :ref:`diagnostic
+   log messages (i.e. mongod/mongos log
+   messages)<log-message-slow-ops>` for read/write operations include:
+
+   - ``queryHash`` to help identify slow queries with the same
+     :term:`query shape`. 
+
+   - ``planCacheKey`` to provide more insight into the :doc:`query plan
+     cache </core/query-plans>` for slow queries.
+
+---
+ref: 4.2-changes-query-shapes
+content: |
+
+   To help identify slow queries with the same :term:`query shape`,
+   starting in MongoDB 4.2, each :term:`query shape` is associated with
+   a :ref:`queryHash <4.2-query-hash>`. The ``queryHash`` is a
+   hexadecimal string that represents a hash of the query shape and
+   is dependent only on the query shape.
+
+   .. note::
+
+      As with any hash function, two different query shapes may result
+      in the same hash value. However, the occurrence of hash
+      collisions between different query shapes is unlikely.
+
+---
+ref: 4.2-changes-plan-cache-key
+content: |
+
+   To provide more insight into the :doc:`query plan cache
+   </core/query-plans>`, MongoDB 4.2 introduces the :ref:`planCacheKey
+   <4.2-plan-cache-key>`. ``planCacheKey`` is a hexadecimal string that
+   represents a query shape and the possible plans for that shape.
+   
+   .. note::
+
+      Unlike the ``queryHash``, the ``planCacheKey`` is dependent on
+      both the query shape and the currently available indexes for the
+      shape. That is, if indexes that can support the query shape are
+      added/dropped, the ``planCacheKey`` value changes whereas the
+      ``queryHash`` value would not change.
 
 ...
 

source/reference/command/planCacheClear.txt

@@ -60,7 +60,8 @@ If a collection ``orders`` has the following query shape:
      {
        "query" : { "qty" : { "$gt" : 10 } },
        "sort" : { "ord_date" : 1 },
-       "projection" : { }
+       "projection" : { },
+       "queryHash" : "9AAD95BE"  // Available starting in MongoDB 4.2
      }
 
 The following operation clears the query plan cached for the shape:

source/reference/command/planCacheListPlans.txt

@@ -15,11 +15,11 @@ Definition
 
 .. dbcommand:: planCacheListPlans
 
-   .. versionadded:: 2.6
-
    Displays the cached query plans for the specified :term:`query
    shape`.
 
+   .. include:: /includes/extracts/4.2-changes-query-shapes.rst
+
    .. include:: /includes/fact-query-optimizer-cache-behavior.rst
 
    The :binary:`~bin.mongo` shell provides the wrapper
@@ -65,7 +65,8 @@ If a collection ``orders`` has the following query shape:
      {
        "query" : { "qty" : { "$gt" : 10 } },
        "sort" : { "ord_date" : 1 },
-       "projection" : { }
+       "projection" : { },
+       "queryHash" : "9AAD95BE"   // Available starting in MongoDB 4.2
      }
 
 The following operation displays the query plan cached for the shape:

source/reference/command/planCacheListQueryShapes.txt

@@ -15,11 +15,11 @@ Definition
 
 .. dbcommand:: planCacheListQueryShapes
 
-   .. versionadded:: 2.6
-
    Displays the :term:`query shapes <query shape>` for which cached
    query plans exist for a collection.
 
+   .. include:: /includes/extracts/4.2-changes-query-shapes.rst
+
    .. include:: /includes/fact-query-optimizer-cache-behavior.rst
 
    The :binary:`~bin.mongo` shell provides the wrapper
@@ -74,7 +74,8 @@ query plans associated with the following shapes:
         {
           "query" : { "qty" : { "$gt" : 10 } },
            "sort" : { "ord_date" : 1 },
-           "projection" : { }
+           "projection" : { },
+           "queryHash" : "9AAD95BE"  // Available starting in MongoDB 4.2
         },
         {
            "query" : { "$or" : [ { "qty" : { "$gt" : 15 } }, { "status" : "A" } ] },
@@ -89,7 +90,8 @@ query plans associated with the following shapes:
              ]
            },
            "sort" : { },
-           "projection" : { }
+           "projection" : { },
+           "queryHash" : "0A087AD0"  // Available starting in MongoDB 4.2
          }
       ],
       "ok" : 1

source/reference/database-profiler.txt

@@ -441,6 +441,42 @@ operation.
    see :ref:`the FAQ on when operations yield
    <faq-concurrency-yielding>`.
 
+.. data:: system.profile.queryHash
+
+   A hexadecimal string that represents a hash of the :term:`query
+   shape` and is dependent only on the query shape. ``queryHash`` can
+   help identify slow queries (including the query filter of write
+   operations) with the same query shape.
+
+   .. note::
+
+      As with any hash function, two different query shapes may result
+      in the same hash value. However, the occurrence of hash
+      collisions between different query shapes is unlikely.
+
+   For more information on ``queryHash`` and ``planCacheKey``, see
+   :ref:`query-hash-plan-cache-key`.
+
+   .. versionadded:: 4.2
+
+.. data:: system.profile.planCacheKey
+
+   A hexadecimal string that represents a query shape and the possible
+   plans for that shape. ``planCacheKey`` can provide insight into the
+   plan cache.
+
+   Unlike the :data:`~system.profile.queryHash`, the
+   :data:`~system.profile.planCacheKey` is dependent on both the query
+   shape and the currently available indexes for the shape. That is, if
+   indexes that can support the query shape are added/dropped, the
+   ``planCacheKey`` value changes whereas the ``queryHash`` value would
+   not change.
+
+   For more information on ``queryHash`` and ``planCacheKey``, see
+   :ref:`query-hash-plan-cache-key`.
+
+   .. versionadded:: 4.2
+
 .. data:: system.profile.locks
 
    .. versionadded:: 3.0.0

source/reference/explain-results.txt

@@ -97,6 +97,8 @@ the ``explain`` operation.
                      "parsedQuery" : {
                         ...
                      },
+                     "queryHash" : <hexadecimal string>,
+                     "planCacheKey" : <hexadecimal string>,
                      "winningPlan" : {
                         "stage" : <STAGE1>,
                         ...
@@ -139,7 +141,9 @@ the ``explain`` operation.
                               },
                               "plannerVersion" : <int>,
                               "namespace" : <string>,
-                              ...
+                              "parsedQuery" : <document>,
+                              "queryHash" : <hexadecimal string>,
+                              "planCacheKey" : <hexadecimal string>,
                               "winningPlan" : {
                                  "stage" : <STAGE2>,
                                  "inputStage" : {
@@ -173,6 +177,42 @@ the ``explain`` operation.
          A boolean that specifies whether MongoDB applied an :ref:`index
          filter <index-filters>` for the :term:`query shape`.
 
+      .. data:: explain.queryPlanner.queryHash
+
+         A hexadecimal string that represents the hash of the
+         :term:`query shape` and is dependent only on the query shapes.
+         ``queryHash`` can help identify slow queries (including the
+         query filter of write operations) with the same query shape.
+
+         .. note::
+
+            As with any hash function, two different query shapes may result
+            in the same hash value. However, the occurrence of hash
+            collisions between different query shapes is unlikely.
+
+         For more information on ``queryHash`` and ``planCacheKey``,
+         see :ref:`query-hash-plan-cache-key`.
+
+         .. versionadded:: 4.2
+
+      .. data:: explain.queryPlanner.planCacheKey
+
+         A hexadecimal string that represents the mapping between a
+         query shape and the query plan. ``planCacheKey`` can provide
+         insight into the plan cache.
+
+         Unlike the :data:`~explain.queryPlanner.queryHash`, the
+         :data:`~explain.queryPlanner.planCacheKey` is dependent on
+         both the query shape and the currently available indexes for
+         the shape. That is, if indexes that can support the query
+         shape are added/dropped, the ``planCacheKey`` value changes
+         whereas the ``queryHash`` value would not change.
+
+         For more information on ``queryHash`` and ``planCacheKey``,
+         see :ref:`query-hash-plan-cache-key`.
+
+         .. versionadded:: 4.2
+
       .. data:: explain.queryPlanner.winningPlan
 
          A document that details the plan selected by the :doc:`query

source/reference/glossary.txt

@@ -725,15 +725,16 @@ Glossary
       query plan. See :ref:`read-operations-query-optimization`.
 
    query shape
-      A combination of query predicate, sort, and projection
-      specifications.
+      A combination of query predicate, sort, and projection.
 
       For the query predicate, only the structure of the predicate,
       including the field names, are significant; the values in the
       query predicate are insignificant. As such, a query predicate ``{
       type: 'food' }`` is equivalent to the query predicate ``{ type:
       'utensil' }`` for a query shape.
 
+      .. include:: /includes/extracts/4.2-changes-query-shapes.rst
+
    read concern
       Specifies a level of isolation for read operations. For example,
       you can use read concern to only read data that has propagated to