Skip to content

Commit 85e81db

Browse files
(DOCSP-20957): Pushdown lookup to SBE (#1028)
* WIP * (DOCSP-20957): Pushdown lookup to SBE * wording * tweaks * wording * fix heading level * wording * wording * wording * address review comments * formatting * address Irina review * address review comment * address final comments
1 parent e80dcd3 commit 85e81db

File tree

8 files changed

+128
-50
lines changed

8 files changed

+128
-50
lines changed

source/core/aggregation-pipeline-optimization.txt

Lines changed: 64 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -387,24 +387,82 @@ option, the ``explain`` output shows the coalesced stage:
387387
}
388388
}
389389

390+
.. _sbe-pipeline-optimizations:
391+
392+
|sbe-title| Pipeline Optimizations
393+
----------------------------------
394+
395+
MongoDB can use the :ref:`slot-based execution query engine
396+
<5.1-rel-notes-sbe>` to execute certain pipeline stages when specific
397+
conditions are met. In most cases, the |sbe-short| provides improved
398+
performance and lower CPU and memory costs compared to the classic query
399+
engine.
400+
401+
To verify that the |sbe-short| is used, run the aggregation with the
402+
``explain`` option. This option outputs information on the
403+
aggregation's query plan. For more information on using ``explain``
404+
with aggregations, see :ref:`example-aggregate-method-explain-option`.
405+
406+
The following sections describe:
407+
408+
- The conditions when the |sbe-short| is used for aggregation.
409+
410+
- How to verify if the |sbe-short| was used.
411+
390412
.. _agg-group-optimization-sbe:
391413

392414
``$group`` Optimization
393-
-----------------------
415+
~~~~~~~~~~~~~~~~~~~~~~~
394416

395417
.. versionadded:: 5.2
396418

397419
.. include:: /includes/fact-sbe-group-overview.rst
398420

399-
To verify that the |sbe-short| is used, run the aggregation with the
400-
``.explain()`` option. This option outputs information on the
401-
aggregation's query plan.
402-
403421
When the |sbe| is used for :pipeline:`$group`, the :ref:`explain results
404422
<explain-results>` include:
405423

406424
- ``explain.explainVersion: '2'``
407-
- ``explain.queryPlanner.winningPlan.queryPlan.stage: "GROUP"``
425+
- ``queryPlanner.winningPlan.queryPlan.stage: "GROUP"``
426+
427+
The location of the ``queryPlanner`` object depends on whether the
428+
pipeline contains stages after the ``$group`` stage which cannot be
429+
executed using the |sbe-short|.
430+
431+
- If ``$group`` is the last stage or all stages after ``$group`` can
432+
be executed using the |sbe-short|, the ``queryPlanner`` object is in
433+
the top-level ``explain`` output object (``explain.queryPlanner``).
434+
435+
- If the pipeline contains stages after ``$group`` which cannot be
436+
executed using the |sbe-short|, the ``queryPlanner`` object is in
437+
``explain.stages[0].$cursor.queryPlanner``.
438+
439+
.. _agg-lookup-optimization-sbe:
440+
441+
``$lookup`` Optimization
442+
~~~~~~~~~~~~~~~~~~~~~~~~
443+
444+
.. versionadded:: 6.0
445+
446+
.. include:: /includes/fact-sbe-lookup-overview.rst
447+
448+
When the |sbe| is used for :pipeline:`$lookup`, the :ref:`explain
449+
results <explain-results>` include:
450+
451+
- ``explain.explainVersion: '2'``
452+
- ``queryPlanner.winningPlan.queryPlan.stage: "EQ_LOOKUP"``.
453+
``EQ_LOOKUP`` means "equality lookup".
454+
455+
The location of the ``queryPlanner`` object depends on whether the
456+
pipeline contains stages after the ``$lookup`` stage which cannot be
457+
executed using the |sbe-short|.
458+
459+
- If ``$lookup`` is the last stage or all stages after ``$lookup`` can
460+
be executed using the |sbe-short|, the ``queryPlanner`` object is in
461+
the top-level ``explain`` output object (``explain.queryPlanner``).
462+
463+
- If the pipeline contains stages after ``$lookup`` which cannot be
464+
executed using the |sbe-short|, the ``queryPlanner`` object is in
465+
``explain.stages[0].$cursor.queryPlanner``.
408466

409467
.. _aggregation-pipeline-optimization-indexes-and-filters:
410468

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,9 @@
1-
Starting in MongoDB 5.2, MongoDB uses the
2-
:ref:`slot-based execution query engine <5.1-rel-notes-sbe>` to execute
3-
:pipeline:`$group` stages when :pipeline:`$group` is either:
1+
Starting in version 5.2, MongoDB uses the :ref:`slot-based execution
2+
query engine <5.1-rel-notes-sbe>` to execute :pipeline:`$group` stages
3+
if either:
44

5-
- The first stage in the pipeline.
5+
- ``$group`` is the first stage in the pipeline.
66

7-
- Part of a series of stages executed by the |sbe-short| that
8-
occurs at the beginning of the pipeline. For example, if a pipeline
9-
begins with :pipeline:`$match` followed by :pipeline:`$group`, the
10-
:pipeline:`$match` and :pipeline:`$group` stages are executed by the
7+
- All preceding stages in the pipeline can also be executed by the
118
|sbe-short|.
129

13-
In most cases, the |sbe-short| provides improved performance and lower
14-
CPU and memory costs compared to the classic query engine.
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Starting in version 6.0, MongoDB can use the :ref:`slot-based execution
2+
query engine <5.1-rel-notes-sbe>` to execute :pipeline:`$lookup` stages
3+
if *all* preceding stages in the pipeline can also be executed by the
4+
|sbe-short| and none of the following conditions are true:
5+
6+
- The ``$lookup`` operation executes a pipeline on a joined collection.
7+
To see an example of this kind of operation, see
8+
:ref:`lookup-syntax-let-pipeline`.
9+
10+
- The ``$lookup``'s ``localField`` or ``foreignField`` specify numeric
11+
components. For example: ``{ localField: "restaurant.0.review" }``.
12+
13+
- The ``from`` field of any ``$lookup`` in the pipeline specifies a view
14+
or sharded collection.

source/reference/explain-results.txt

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -301,10 +301,10 @@ representative. Your output may differ significantly.
301301

302302
A string that denotes the name of the stage.
303303

304-
Each stage consists of information specific to the stage. For
305-
example, an ``IXSCAN`` stage will include the index bounds along
306-
with other data specific to the index scan. If a stage has a child
307-
stage or multiple child stages, the stage will have an
304+
Each stage consists of information specific to the stage. For
305+
example, an ``IXSCAN`` stage includes the index bounds along with
306+
other data specific to the index scan. If a stage has a child
307+
stage or multiple child stages, the stage will have an
308308
``inputStage`` or ``inputStages``.
309309

310310
This field appears if the operation used the classic query
@@ -333,7 +333,7 @@ representative. Your output may differ significantly.
333333

334334
A document that details the plan selected by the :doc:`query
335335
optimizer </core/query-plans>`. MongoDB presents the plan as a tree
336-
of stages.
336+
of stages.
337337

338338
This document appears if the query used the :ref:`slot-based
339339
execution query engine <5.1-rel-notes-sbe>`.
@@ -345,13 +345,8 @@ representative. Your output may differ significantly.
345345
A string that denotes the name of the stage.
346346

347347
Each stage consists of information specific to the stage. For
348-
instance, an ``IXSCAN`` stage will include the index bounds
349-
along with other data specific to the index scan.
350-
351-
Starting in MongoDB 5.2, MongoDB can execute some
352-
:pipeline:`$group` stages using the :ref:`slot-based execution
353-
query engine <5.1-rel-notes-sbe>`. When the |sbe-short| is used
354-
for :pipeline:`$group`, this field value is ``GROUP``.
348+
example, an ``IXSCAN`` stage includes the index bounds along
349+
with other data specific to the index scan.
355350

356351
.. data:: explain.queryPlanner.winningPlan.queryPlan.planNodeId
357352

source/reference/operator/aggregation/group.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ the first document of each group.
131131

132132
.. include:: /includes/fact-sbe-group-overview.rst
133133

134-
See :ref:`agg-group-optimization-sbe`.
134+
For more information, see :ref:`agg-group-optimization-sbe`.
135135

136136

137137
Examples

source/reference/operator/aggregation/lookup.txt

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -367,8 +367,8 @@ See this example:
367367

368368
- :ref:`lookup-concise-correlated-subquery`
369369

370-
Considerations
371-
--------------
370+
Behavior
371+
--------
372372

373373
Views and Collation
374374
~~~~~~~~~~~~~~~~~~~
@@ -378,26 +378,26 @@ Views and Collation
378378
Restrictions
379379
~~~~~~~~~~~~
380380

381-
- .. versionchanged:: 4.2
381+
.. versionchanged:: 4.2
382382

383-
You cannot include the :pipeline:`$out` or the :pipeline:`$merge`
384-
stage in the :pipeline:`$lookup` stage. That is, when specifying a
385-
:ref:`pipeline for the joined collection
386-
<lookup-syntax-let-pipeline>`, you cannot include either stage in
387-
the ``pipeline`` field.
383+
You cannot include the :pipeline:`$out` or the :pipeline:`$merge`
384+
stage in the :pipeline:`$lookup` stage. That is, when specifying a
385+
:ref:`pipeline for the joined collection
386+
<lookup-syntax-let-pipeline>`, you cannot include either stage in
387+
the ``pipeline`` field.
388388

389-
.. code-block:: javascript
390-
:emphasize-lines: 6
389+
.. code-block:: javascript
390+
:emphasize-lines: 6
391391

392-
{
393-
$lookup:
394-
{
395-
from: <collection to join>,
396-
let: { <var_1>: <expression>, …, <var_n>: <expression> },
397-
pipeline: [ <pipeline to execute on the joined collection> ], // Cannot include $out or $merge
398-
as: <output array field>
399-
}
400-
}
392+
{
393+
$lookup:
394+
{
395+
from: <collection to join>,
396+
let: { <var_1>: <expression>, …, <var_n>: <expression> },
397+
pipeline: [ <pipeline to execute on the joined collection> ], // Cannot include $out or $merge
398+
as: <output array field>
399+
}
400+
}
401401

402402
.. _lookup-sharded-collections:
403403

@@ -407,6 +407,13 @@ Sharded Collections
407407
Starting in MongoDB 5.1, you can specify :doc:`sharded collections </sharding>`
408408
in the ``from`` parameter of :pipeline:`$lookup` stages.
409409

410+
|sbe-title|
411+
~~~~~~~~~~~
412+
413+
.. include:: /includes/fact-sbe-lookup-overview.rst
414+
415+
For more information, see :ref:`agg-lookup-optimization-sbe`.
416+
410417
Examples
411418
--------
412419

@@ -1039,4 +1046,3 @@ The previous examples correspond to this pseudo-SQL statement:
10391046
WHERE restaurants.name = orders.restaurant_name
10401047
AND restaurants.beverages = orders.drink
10411048
);
1042-

source/release-notes/5.2.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ processing does not inflate the counters.
155155

156156
.. include:: /includes/fact-sbe-group-overview.rst
157157

158-
See :ref:`agg-group-optimization-sbe`.
158+
For more information, see :ref:`agg-group-optimization-sbe`.
159159

160160
Time Series Collections
161161
-----------------------

source/release-notes/6.0.txt

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,16 @@ the changes introduced in these releases, see the following pages:
2929
To learn more about the differences between |lts| and Rapid releases,
3030
see :ref:`release-version-numbers`.
3131

32+
Aggregation
33+
-----------
34+
35+
|sbe-title| Can Execute :pipeline:`$lookup` Stages
36+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
37+
38+
.. include:: /includes/fact-sbe-lookup-overview.rst
39+
40+
For more information, see :ref:`agg-lookup-optimization-sbe`.
41+
3242
Installation
3343
------------
3444

0 commit comments

Comments
 (0)