DOCSP-28718 28723 Time Series Granularity procedures and practices (#3061)

* Initital stashed commit

* Codeblock rendering fix

* Self proofreading

* Clarified restrictions on bucketing parameters

* Switching to procedure directive

* Switching to procedure directive

* Internal PR feedback

* Internal review feedback

* Internal review feedback--amend

* Internal review feedback--amend

* Internal review feedback--amend

* External review feedback

Author: nvillahermosa-mdb

source/core/timeseries/timeseries-best-practices.txt

@@ -213,7 +213,45 @@ ratio.
 Optimize Query Performance
 --------------------------
 
-To improve query performance, 
-:ref:`create one or more secondary indexes <timeseries-add-secondary-index>` 
-on your ``timeField`` and ``metaField`` to support common query 
-patterns. 
+Set Appropriate Bucket Granularity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+When you create a time series collection, MongoDB groups incoming time
+series data into buckets. By accurately setting granularity, you control
+how frequently data is bucketed based on the ingestion rate of your data.
+
+Starting in MongoDB 6.3, you can use the custom bucketing parameters
+``bucketMaxSpanSeconds`` and ``bucketRoundingSeconds`` to specify bucket
+boundaries and more precisely control how time series data is bucketed.
+
+You can improve performance by setting the ``granularity`` or custom
+bucketing parameters to the best match for the time span between
+incoming measurements from the same data source. For example, if you are
+recording weather data from thousands of sensors but only record data
+from each sensor once per 5 minutes, you can either set ``granularity``
+to ``"minutes"`` or set the custom bucketing parameters to ``300``
+(seconds).
+
+In this case, setting the ``granularity`` to ``hours`` groups up to a 
+month's worth of data ingest events into a single bucket, resulting in 
+longer traversal times and slower queries. Setting it to ``seconds`` 
+leads to multiple buckets per polling interval, many of which 
+might contain only a single document.
+
+The following table shows the maximum time interval included in one 
+bucket of data when using a given ``granularity`` value:
+
+.. include:: /includes/table-timeseries-granularity-intervals.rst
+
+.. seealso::
+
+   :ref:`Timing of Automatic Removal
+   <timeseries-collection-delete-operations-timing>`
+
+Create Secondary Indexes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+To improve query performance, :ref:`create one or more secondary indexes
+<timeseries-add-secondary-index>` on your ``timeField`` and
+``metaField`` to support common query  patterns. In versions 6.3 and
+higher, MongoDB creates a secondary index on the ``timeField`` and
+``metaField`` automatically.

source/core/timeseries/timeseries-granularity.txt

@@ -16,10 +16,15 @@ Set Granularity for Time Series Data
    :description: Time series, granularity, IOT
    :keywords: Time series, granularity, IOT
 
-When you create a :ref:`time series collection 
-<manual-timeseries-collection>`, MongoDB automatically creates a ``system.buckets`` 
-:ref:`system collection <metadata-system-collections>` based on the ``granularity`` value you specify, and groups 
-documents into those buckets.
+When you create a time series collection, MongoDB automatically creates
+a ``system.buckets`` :ref:`system collection 
+<metadata-system-collections>` and groups incoming time series data 
+into buckets. By setting granularity, you control how 
+frequently data is bucketed based on the ingestion rate of your data.
+
+Starting in MongoDB 6.3, you can use the custom bucketing parameters
+``bucketMaxSpanSeconds`` and ``bucketRoundingSeconds`` to specify bucket
+boundaries and more accurately control how time series data is bucketed.
 
 .. note::
 
@@ -28,83 +33,18 @@ documents into those buckets.
    created. See :ref:`MongoDB 5.0 known issues
    <5.0-known-issue-granularity>`.
 
-When you create a :ref:`time series collection
-<manual-timeseries-collection>`, you can specify the
-``granularity`` parameter. The ``granularity`` determines how data 
-in the time series collection is stored internally. For details about 
-how MongoDB stores time series data, see :ref:`flexible-bucketing`.
- 
-By default, ``granularity`` is set to ``"seconds"``. To optimize  
-data storage, set the ``granularity`` to a value that is closest 
-to the ingestion rate for your data source as specified by
-the ``metaField`` field. The ingestion rate is the time span between 
-consecutive incoming measurements that have the same unique value for 
-the ``metaField``.
-
-Consider the following example:
-
-.. code-block:: javascript
-
-   db.createCollection(
-       "weather24h",
-       {
-          timeseries: {
-             timeField: "timestamp",
-             metaField: "metadata",
-             granularity: "minutes"
-          },
-          expireAfterSeconds: 86400
-       }
-   )
-
-If your ``metaField`` data identifies weather sensors and
-you ingest data from each individual sensor once every 5 minutes, you
-should choose ``"minutes"``. Even if you have thousands of sensors and
-the data coming in from different sensors is only seconds apart, the
-``granularity`` should still be based on the ingestion rate for one
-sensor that is uniquely identified by its metadata. 
-
-In the following table, you can see the max time span of data that is
-stored together for each ``granularity`` value:
-
-.. list-table::
-  :header-rows: 1
-  :widths: 40 60
-
-  * - ``granularity``
+Retrieve the Current Bucketing Parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-    - Covered Time Span
-
-  * - ``"seconds"`` (default)
-
-    - one hour
-
-  * - ``"minutes"``
-
-    - 24 hours
-
-  * - ``"hours"``
-
-    - 30 days
-
-
-.. seealso::
-
-   :ref:`Timing of Automatic Removal
-   <timeseries-collection-delete-operations-timing>`
-
-Retrieve the ``granularity`` of a Time Series Collection
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To retrieve the current value of ``granularity``, use the
+To retrieve current collection values, use the
 :dbcommand:`listCollections` command:
 
 .. code-block:: javascript
 
    db.runCommand( { listCollections: 1 } )
 
-The result document contains a document for the time series collection
-which contains the ``options.timeseries.granularity`` field.
+For time series collections, the output contains
+``granularity``, ``bucketMaxSpanSeconds``, and ``bucketRoundingSeconds`` fields, if present.
 
 .. code-block:: javascript
   :copyable: false
@@ -123,7 +63,8 @@ which contains the ``options.timeseries.granularity`` field.
                     timeField: <string>,
                     metaField: <string>,
                     granularity: <string>,
-                    bucketMaxSpanSeconds: <number>
+                    bucketMaxSpanSeconds: <number>,
+                    bucketRoundingSeconds: <number>
                  }
               },
               ...
@@ -133,55 +74,70 @@ which contains the ``options.timeseries.granularity`` field.
       }
    }
 
-Change the ``granularity`` of a Time Series Collection
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To change the ``granularity`` value, issue the following
-:dbcommand:`collMod` command: 
+Using the "granularity" Field
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. code-block:: javascript
+The following table shows the maximum time interval included in one 
+bucket of data when using a given ``granularity`` value:
 
-   db.runCommand({
-      collMod: "weather24h",
-      timeseries: { granularity: "hours" }
-   })
+.. include:: /includes/table-timeseries-granularity-intervals.rst
 
-Once set, you cannot decrease granularity. For example, if granularity
-is set to ``minutes``, you can increase it to ``hours``, but you cannot
-decrease it to ``seconds``.
+By default, ``granularity`` is set to ``seconds``. You can improve performance by setting the ``granularity`` value to the 
+closest match to the time span between incoming measurements from the 
+same data source. For example, if you are recording weather data from 
+thousands of sensors but only record data from each sensor once per 5 
+minutes, set ``granularity`` to ``"minutes"``.
 
-.. note::
+.. code-block:: javascript
 
-   You cannot modify the ``granularity`` of a sharded time series
-   collection.
+   db.createCollection(
+       "weather24h",
+       {
+          timeseries: {
+             timeField: "timestamp",
+             metaField: "metadata",
+             granularity: "minutes"
+          },
+          expireAfterSeconds: 86400
+       }
+   )
 
-.. _flexible-bucketing:
+Setting the ``granularity`` to ``hours`` groups up to a month's 
+worth of data ingest events into a single bucket, resulting in longer 
+traversal times and slower queries. Setting it to ``seconds`` 
+leads to multiple buckets per polling interval, many of which 
+might contain only a single document.
 
-Flexible Bucketing
-~~~~~~~~~~~~~~~~~~
+.. seealso::
 
-MongoDB groups incoming time series data into buckets. By setting the 
-``granularity`` parameter, you control how frequently data is bucketed 
-based on the ingestion rate of your data.
+   :ref:`Timing of Automatic Removal
+   <timeseries-collection-delete-operations-timing>`
+
+.. _flexible-bucketing:
 
-Starting in MongoDB 6.3, you can specify the bucket boundaries to 
-more accurately control how time series data is bucketed. Use the 
-following parameters:
+Using Custom Bucketing Parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- ``bucketMaxSpanSeconds``, the maximum time span between measurements
-  in a bucket.
-- ``bucketRoundingSeconds``, the time interval that determines the 
-  starting timestamp for a new bucket.
+In MongoDB 6.3 and higher, instead of ``granularity``, you can set
+bucket boundaries manually using the two custom bucketing parameters.
+Consider this approach if you need the additional precision to optimize 
+a high volume of queries and :dbcommand:`insert` operations.
 
-.. note::
+To use custom bucketing parameters, set both parameters to the same 
+value, and do not set ``granularity``:
 
-   If you want to specify the bucket boundaries:
+- ``bucketMaxSpanSeconds`` sets the maximum time between timestamps 
+  in the same bucket. Possible values are 1-31536000.
 
-   - You must set both ``bucketMaxSpanSeconds`` and ``bucketRoundingSeconds``.
-   - Both parameters must have the same value.
-   - You can't additionally set the ``granularity`` parameter.
+- ``bucketRoundingSeconds`` sets the time interval that determines the 
+  starting timestamp for a new bucket. When a document requires a new
+  bucket, MongoDB rounds down the document's timestamp value by this
+  interval to set the minimum time for the bucket.
 
-Consider the following time series collection:
+For the weather station example with 5 minute sensor intervals, you
+could fine tune bucketing by setting the custom bucketing parameters to
+300 seconds, instead of using a ``granularity`` of ``"minutes"``:
 
 .. code-block:: javascript
 
@@ -191,20 +147,48 @@ Consider the following time series collection:
          timeseries: {
             timeField: "timestamp",
             metaField: "metadata",
-            bucketMaxSpanSeconds: 60,
-            bucketRoundingSeconds: 60
+            bucketMaxSpanSeconds: 300,
+            bucketRoundingSeconds: 300
          }
       }   
    )
 
-Each bucket has a maximum time span of 60 seconds. When MongoDB opens a 
-new bucket, the starting timestamp is rounded down to the nearest 60-second 
-interval.
+If a document with a time of ``2023-03-27T18:24:35Z`` does not fit an
+existing bucket, MongoDB creates a new bucket with a minimum time of
+``2023-03-27T18:20:00Z`` and a maximum time of ``2023-03-27T18:24:59Z``.
 
-For example, if you insert a new measurement into the collection with a
-timestamp of ``11:59:59``, the measurement is added to the bucket 
-with boundaries between ``11:59:00`` and ``12:00:00`` (non-inclusive).
-If the bucket doesn't exist, the starting timestamp of the new bucket 
-is determined by rounding ``11:59:59`` down to ``11:59:00``.
+Change Time Series Granularity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can increase ``timeseries.granularity`` from a shorter unit of time
+to a longer one using a :dbcommand:`collMod` command.
+
+.. code-block:: javascript
+
+   db.runCommand({
+      collMod: "weather24h",
+      timeseries: { granularity: "seconds" || "minutes" || "hours" }
+   })
+
+If you are using the custom bucketing parameters
+``bucketRoundingSeconds`` and ``bucketMaxSpanSeconds`` instead of
+``granularity``, include both custom parameters in the ``collMod``
+command and set them to the same value:
+
+.. code-block:: javascript
+
+   db.runCommand({
+      collMod: "weather24h",
+      timeseries: { 
+         bucketRoundingSeconds: "86400",
+         bucketMaxSpanSeconds: "86400" 
+      }
+   })
+
+You cannot decrease the granularity interval or the custom bucketing
+values.
+
+.. note::
 
-For more information on time series parameters, see :ref:`time-series-fields`.
+   To modify the granularity of a **sharded** time series collection,
+   you must be running MongoDB 6.0 or later.

source/core/timeseries/timeseries-limitations.txt

@@ -154,24 +154,13 @@ parameters later.
 
 .. _timeseries-limitations-granularity:
 
-Modification of ``granularity``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-After you set the ``granularity``, you can only increase it one level at
-a time. The ``granularity`` can change from ``"seconds"`` to
-``"minutes"`` or from ``"minutes"`` to ``"hours"``. Other changes are
-not allowed. 
-
-To change the ``granularity`` from ``"seconds"`` to ``"hours"``, first
-increase the ``granularity`` to ``"minutes"`` and then to ``"hours"``.
-
-Modification of Bucket Parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Modifying Bucket Parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Once you set a collection's ``bucketMaxSpanSeconds`` and 
-``bucketRoundingSeconds`` parameters, they can only be increased. Use 
-the :dbcommand:`collMod` command to modify the ``bucketMaxSpanSeconds`` 
-and ``bucketRoundingSeconds`` parameters. For example:
+Once you set a collection's ``granularity`` or custom bucketing
+parameters ``bucketMaxSpanSeconds`` and ``bucketRoundingSeconds``, you
+can increase them, but not decrease them.
+Use the :dbcommand:`collMod` command to modify the parameters. For example:
 
 .. code-block:: javascript
 
@@ -182,8 +171,9 @@ and ``bucketRoundingSeconds`` parameters. For example:
 
 .. note::
 
-   The ``bucketMaxSpanSeconds`` and ``bucketRoundingSeconds`` parameters must be
-   equal. If you modify one parameter, you must also modify the other.
+   ``bucketMaxSpanSeconds`` and ``bucketRoundingSeconds`` must be
+   equal. If you modify one parameter, you must also set the other to
+   the same value.
 
 .. _time-series-limitations-sharding: