DOCSP-28729 default meta time index (#3087) (#4255)

* Added info on default meta, time index and detailed use case for creating additional secondary indexes

* Removed mention of default clustered index for time series collections, per SERVER-54965

* Removed mention of default clustered index for time series collections, per SERVER-54965

* Removed redundant 'internal indexes' heading since info is already presented earlier and the ref to adding secondary indexes is also present immediately prior

* Updated limitations topic and expanded secondary index options in 6.0 and higher

* Updated limitations topic and expanded secondary index options in 6.0 and higher--amend

* Updated limitations topic and expanded secondary index options in 6.0 and higher--amend

* Updated limitations topic and expanded secondary index options in 6.0 and higher--amend

* Spelling and grammar check

* Self review

* internal PR feedback

* Internal PR feedback

* External PR feedback

Author: nvillahermosa-mdb

source/core/timeseries-collections.txt

@@ -73,11 +73,12 @@ Benefits
 Compared to normal collections, storing time series data in time series
 collections improves query efficiency and reduces the disk usage for
 time series data and :term:`secondary indexes <secondary index>`.
+MongoDB 6.3 and later automatically creates a :ref:`compound index
+<index-type-compound>` on the time and metadata
+fields for new time series collections. 
 
 Time series collections use an underlying columnar storage format and
-store data in time-order with an automatically created :ref:`clustered
-index <clustered-collections>`. The columnar storage format provides the
-following benefits:
+store data in time-order. This format provides the following benefits:
 
 - Reduced complexity for working with time series data
 - Improved query efficiency
@@ -116,26 +117,6 @@ optimized internal storage format and return results faster.
 
 .. _manual-timeseries-internal-index:
 
-Internal Index
-``````````````
-
-When you create a time series collection, MongoDB automatically creates
-an internal :ref:`clustered index <clustered-collections>` on the time
-field. This index improves query efficiency and reduces disk 
-usage. To learn more about the performance benefits of clustered 
-indexes, see :ref:`Clustered Collections <clustered-collections>`.
-
-The internal index is not listed when you run :dbcommand:`listIndexes`.
-
-.. note::
-   If you insert a document into a collection with a ``timeField`` 
-   value before ``1970-01-01T00:00:00.000Z`` or after 
-   ``2038-01-19T03:14:07.000Z``, 
-   MongoDB logs a warning and prevents some query optimizations from 
-   using the internal index. :ref:`Create a secondary index <timeseries-add-secondary-index>` 
-   on the ``timeField`` to regain query performance and resolve the log 
-   warning.
-
 Get Started
 -----------
 

source/core/timeseries/timeseries-limitations.txt

@@ -100,33 +100,32 @@ To remove all documents from a collection, use the
 Time Series Secondary Indexes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-There is improved support for secondary indexes in MongoDB 6.0.
+MongoDB 6.0 and later features improved support for secondary indexes on
+time series collections. For a summary of these options, see :ref:`timeseries-add-secondary-index-mongodb-6.0`.
 
-Time Series Secondary Indexes with MongoDB 6.0 and Later
-````````````````````````````````````````````````````````
+Time Series Secondary Indexes
+`````````````````````````````
+
+Starting in MongoDB 6.3, time series collections support the ``expireAfterSeconds`` index property for :ref:`partial
+indexes <index-type-partial>` on the ``metaField``. For earlier versions
+of MongoDB, use the collection level :ref:`expireAfterSeconds
+<db.createCollection.expireAfterSeconds>` parameter.
 
 Starting in MongoDB 6.0, you can add a :term:`secondary index <secondary
 index>` to any field.
 
+These index types are partially supported:
+ 
+- You can only create :ref:`multikey indexes <index-type-multikey>` on
+  the ``metaField``.
+- You can only create :ref:`sparse indexes <index-type-sparse>` on the ``metaField``.
+
 These index types aren't supported:
 
 - :ref:`Text indexes <index-type-text>`
 - :ref:`2d indexes <2d-index>`
 - :ref:`Unique indexes <index-type-unique>`
 
-You can only use the :ref:`multikey index <index-type-multikey>` type on
-the ``metaField``.
-
-These index properties are partially supported. You can create:
-
-- :ref:`Partial indexes <index-type-partial>` on every field except
-  ``metaField`` and ``timeField``.
-
-- :ref:`Sparse indexes <index-type-sparse>` on the ``metaField``.
-
-For improvements to time series secondary indexes available starting in
-MongoDB 6.0, see :ref:`timeseries-add-secondary-index-mongodb-6.0`.
-
 .. include:: /includes/time-series-secondary-indexes-downgrade-FCV.rst
 
 Time Series Secondary Indexes with MongoDB 5.0 and Earlier

source/core/timeseries/timeseries-secondary-index.txt

@@ -18,26 +18,55 @@ Add Secondary Indexes to Time Series Collections
 
 To improve query performance for :term:`time series collections <time
 series collection>`, add one or more :term:`secondary indexes <secondary
-index>` to support common time series query patterns. Specifically, we
-recommend that you create one or more :ref:`compound indexes
-<index-type-compound>` on the fields specified as the ``timeField`` and
-the ``metaField``. If the field value for the ``metaField`` field is a
-document, you can create secondary indexes on fields inside that
-document.
+index>` to support common time series query patterns. Starting in
+MongoDB 6.3, MongoDB automatically creates a :ref:`compound index
+<index-type-compound>` on the ``metaField`` and ``timeField`` fields for
+new collections.
 
 .. note::
 
    Not all index types are supported. For a list of unsupported index
    types, see :ref:`Limitations for Secondary Indexes on Time Series
    Collections <timeseries-limitations-secondary-indexes>`.
 
-For example, this command creates a :ref:`compound index
-<index-type-compound>` on the ``metadata.sensorId`` and ``timestamp``
-fields:
+You may wish to create additional secondary indexes. Consider a weather
+data collection with the configuration:
 
 .. code-block:: javascript
 
-   db.weather24h.createIndex( { "metadata.sensorId": 1, "timestamp": 1 } )
+   db.createCollection(
+   "weather",
+   { 
+      timeseries: {
+         timeField: "timestamp",
+         metaField: "metadata"
+   }})
+
+In each weather data document, the ``metadata`` field value is a
+subdocument with fields for the weather sensor ID and type:
+
+.. code-block:: javascript
+
+   {
+      "timestamp": ISODate("2021-05-18T00:00:00.000Z"),
+      "metadata": { 
+        "sensorId": 5578, 
+        "type": "temperature" 
+      },
+      "temp": 12
+   }
+
+The default compound index for the collection indexes the entire
+``metadata`` subdocument, so the index is only used with
+:expression:`$eq` queries. By indexing specific ``metadata`` fields, you
+improve query performance for other query types. 
+
+For example, this :expression:`$in` query benefits from a
+secondary index on ``metadata.type``:
+
+.. code-block:: javascript
+
+   { metadata.type:{ $in: ["temperature", "pressure"] }}
 
 .. see::
 
@@ -48,9 +77,22 @@ fields:
 Use Secondary Indexes to Improve Sort Performance
 -------------------------------------------------
 
-Time Series collections can use indexes to improve sort performance on 
-the ``timeField`` and the ``metaField``. 
- 
+Sort operations on time series collections can use secondary indexes 
+on the ``timeField`` field. Under certain conditions, sort operations can also use compound secondary indexes on the ``metaField`` and 
+``timeField`` fields.
+
+The aggregation pipeline stages :pipeline:`$match` and 
+:pipeline:`$sort` determine which indexes a time series collection can
+use. An index can be used in the following scenarios: 
+
+- Sort on ``{ <timeField>: ±1 }`` uses a secondary index on 
+  ``<timeField>``
+- Sort on ``{ <metaField>: ±1, timeField: ±1 }`` uses the default
+  compound index on ``{ <metaField>: ±1, timeField: ±1 }``
+- Sort on ``{ <timeField>: ±1 }`` uses a secondary index on 
+  ``{ metaField: ±1, timeField: ±1 }`` when there is a point predicate 
+  on ``<metaField>``
+
 For example, the following ``sensorData`` collection contains 
 measurements from weather sensors:
 
@@ -59,14 +101,15 @@ measurements from weather sensors:
    db.sensorData.insertMany( [ {
         "metadata": {
             "sensorId": 5578,
+            "type": "omni",
             "location": {
                 type: "Point",
                 coordinates: [-77.40711, 39.03335]
             }
         },
         "timestamp": ISODate("2022-01-15T00:00:00.000Z"),
         "currentConditions": {
-            "windDirecton": 127.0,
+            "windDirection": 127.0,
             "tempF": 71.0,
             "windSpeed": 2.0,
             "cloudCover": null,
@@ -77,14 +120,15 @@ measurements from weather sensors:
       {
         "metadata": {
             "sensorId": 5578,
+            "type": "omni",
             "location": {
                 type: "Point",
                 coordinates: [-77.40711, 39.03335]
             }
         },
         "timestamp": ISODate("2022-01-15T00:01:00.000Z"),
         "currentConditions": {
-            "windDirecton": 128.0,
+            "windDirection": 128.0,
             "tempF": 69.8,
             "windSpeed": 2.2,
             "cloudCover": null,
@@ -95,14 +139,15 @@ measurements from weather sensors:
       {
         "metadata": {
             "sensorId": 5579,
+            "type": "omni",
             "location": {
                 type: "Point",
                 coordinates: [-80.19773, 25.77481]
             }
         },
         "timestamp": ISODate("2022-01-15T00:01:00.000Z"),
         "currentConditions": {
-            "windDirecton": 115.0,
+            "windDirection": 115.0,
             "tempF": 88.0,
             "windSpeed": 1.0,
             "cloudCover": null,
@@ -113,62 +158,7 @@ measurements from weather sensors:
      ] 
    )
 
-Time Series collections :ref:automatically create an internal 
-:ref:`clustered index <db.createCollection.clusteredIndex>`. The query 
-planner uses this index to improve sort performance. 
-
-.. note::
-   If you insert a document into a collection with a ``timeField`` 
-   value before ``1970-01-01T00:00:00.000Z`` or after 
-   ``2038-01-19T03:14:07.000Z``, 
-   MongoDB logs a warning and prevents some query optimizations from 
-   using the internal index. :ref:`Create a secondary index <timeseries-add-secondary-index>` 
-   on the ``timeField`` to regain query performance and resolve the log 
-   warning.
-
-The following sort operation on the ``timestamp`` field uses the 
-clustered index to improve performance:
-
-.. code-block:: javascript
-  
-   db.sensorData.find().sort( { "timestamp": 1 } )
-
-To confirm that the sort operation used the clustered index, run the 
-operation again with the ``.explain( "executionStats" )`` option:
-
-.. code-block:: javascript
-
-   db.sensorData.find().sort( { "timestamp": 1 } ).explain( "executionStats" )
-
-The ``winningPlan.queryPlan.inputStage.stage`` is ``COLLSCAN`` and an 
-``_internalBoundedSort`` stage is present in the explain plan output.
-The ``interalBoundedSort`` field indicates that the clustered index was 
-used. For more information on explain plan output, see :ref:`explain 
-results <explain-results>`.
-
-Secondary indexes on Time Series collections can improve 
-performance for sort operations and increase the number of scenarios 
-where indexes can be used. 
-
-Sort operations on Time Series collections can use secondary indexes 
-on the ``timeField``. Under certain conditions, sort operations can 
-also use compound secondary indexes on the ``metaField`` and 
-``timeField``.
-
-The Aggregation Pipeline Stages :pipeline:`$match` and 
-:pipeline:`$sort` determine which indexes a Time Series collection can
-use. The following list describes scenarios where an index can be used: 
-
-- Sort on ``{ <timeField:> ±1 }`` uses the clustered index
-- Sort on ``{ <timeField>: ±1 }`` uses a secondary index on 
-  ``<timeField>``
-- Sort on ``{ <metaField>: ±1, timeField: ±1 }`` uses a secondary 
-  index on ``{ <metaField>: ±1, timeField: ±1 }``
-- Sort on ``{ <timeField>: ±1 }`` uses a secondary index on 
-  ``{ metaField: ±1, timeField: ±1 }`` when there is a point predicate 
-  on ``<metaField>``
-
-Create a secondary index on the ``timestamp`` field:
+Create a secondary single-field index on the ``timestamp`` field:
 
 .. code-block:: javascript
 
@@ -195,13 +185,15 @@ operation again with the ``.explain( "executionStats" )`` option:
    ] )
 
 
-"Last Point" Queries on Time Series Collections
+Last Point Queries on Time Series Collections
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-A "last point" query fetches the latest measurement for each unique metadata 
-value. For example, you may want to get the latest temperature reading from all
-sensors. Improve performance on last point queries by creating any of the 
-following indexes:
+In time series data, a last point query returns the data point with the
+latest timestamp for a given field. For time series collections, a last
+point query fetches the latest measurement for each unique metadata
+value. For example, you may want to get the latest temperature reading
+from all sensors. Improve performance on last point queries by creating
+any of the following indexes:
 
 .. code-block:: javascript
 
@@ -283,19 +275,32 @@ a collection, use the :method:`db.collection.getIndexes()` method.
 
 .. _timeseries-add-secondary-index-mongodb-6.0:
 
-Time Series Secondary Indexes in MongoDB 6.0
---------------------------------------------
+Time Series Secondary Indexes in MongoDB 6.0 and Later
+------------------------------------------------------
+
+.. versionadded:: 6.3
+
+Starting in MongoDB 6.3, you can create a partial :ref:`TTL
+index<index-feature-ttl>` for a time series collection. You can only
+filter on the ``metaField``. 
+
+If the collection doesn't use the ``expireAfterSeconds`` option to
+expire documents, creating a partial TTL index sets an
+expiration time for matching documents only. If the collection uses
+``expireAfterSeconds`` for all documents, a partial TTL index lets you expire matching documents sooner.
+
+.. versionadded:: 6.0
 
 Starting in MongoDB 6.0, you can:
 
 - Add a :ref:`compound index <index-type-compound>` on the
   ``timeField``, ``metaField``, or measurement fields.
 
 - Use the :query:`$or`, :query:`$in`, and :query:`$geoWithin` operators
-  with :doc:`partial indexes </core/index-partial>` on a time series
+  with :ref:`partial indexes <index-type-partial>` on a time series
   collection.
 
-- Add :doc:`partial </core/index-partial>` on the ``metaField``.
+- Add a partial filter expression on the ``metaField``.
 
 - Add a :term:`secondary index <secondary index>` to any field or
   subfield.