Skip to content

DOCSP-41769: Improved bulk write API #590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/vale-tdbx.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ jobs:
- name: checkout
uses: actions/checkout@master

- name: Install docutils
run: sudo apt-get install -y docutils

- id: files
uses: masesgroup/retrieve-changed-files@v2
with:
Expand Down
283 changes: 257 additions & 26 deletions source/fundamentals/crud/write-operations/bulk.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,51 @@ Bulk Operations
:depth: 2
:class: singlecol

.. meta::
:keywords: insert, update, replace, code example, efficiency

Overview
--------

In this guide, you can learn how to use bulk operations in the
MongoDB Java Driver.

To perform a create, replace, update, or delete operation,
use its corresponding method. For example, to insert one document,
update multiple documents, and delete one document in your collection,
use the ``insertOne()``, ``updateMany()`` and ``deleteOne()`` methods.
To perform a single create, replace, update, or delete operation, you can use
the corresponding method. For example, to insert one document and replace one
document, you can use the ``insertOne()`` and ``replaceOne()`` methods. In this
case, the ``MongoClient`` makes a call to the database for each operation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: clarify

Suggested change
document, you can use the ``insertOne()`` and ``replaceOne()`` methods. In this
case, the ``MongoClient`` makes a call to the database for each operation.
document, you can use the ``insertOne()`` and ``replaceOne()`` methods. When you
use these methods, the ``MongoClient`` makes one call to the database for each operation.


Generally, you can reduce the number of calls to the database by using bulk write
operations. You can perform bulk write operations at the following levels:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: 'generally' is a little vague. we can just state a fact instead

Suggested change
Generally, you can reduce the number of calls to the database by using bulk write
operations. You can perform bulk write operations at the following levels:
By using a bulk write operation, you can perform multiple write operations
in a single database call. You can perform bulk write operations at the following levels:


- :ref:`Collection Level <java-sync-coll-bulk-write>`: You can use the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- :ref:`Collection Level <java-sync-coll-bulk-write>`: You can use the
- :ref:`Collection <java-sync-coll-bulk-write>`: You can use the

``MongoCollection.bulkWrite()`` method to perform bulk write operations on a
single collection. This method groups write operations together by the kind
of operation. For example, ``MongoCollection.bulkWrite()`` puts multiple update
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S:

Suggested change
single collection. This method groups write operations together by the kind
of operation. For example, ``MongoCollection.bulkWrite()`` puts multiple update
single collection. In this method, each kind of write operation requires one
database call. For example, ``MongoCollection.bulkWrite()`` puts multiple update

operations in the same call, but makes two calls to the database for an insert
operation and a replace operation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: maybe a little clearer

Suggested change
of operation. For example, ``MongoCollection.bulkWrite()`` puts multiple update
operations in the same call, but makes two calls to the database for an insert
operation and a replace operation.
of operation. For example, to perform two update operations, two insert
operations, and two replace operations, the client performs three database
calls.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left mostly as is because I wasn't sure if this made it clearer - if you feel strongly about this I can change, also happy to discuss further


- :ref:`Client Level <java-sync-client-bulk-write>`: When running {+mdb-server+}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- :ref:`Client Level <java-sync-client-bulk-write>`: When running {+mdb-server+}
- :ref:`Client <java-sync-client-bulk-write>`: When running {+mdb-server+}

version 8.0 or later, you can use the ``MongoClient.bulkWrite()`` method to perform
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- :ref:`Client Level <java-sync-client-bulk-write>`: When running {+mdb-server+}
version 8.0 or later, you can use the ``MongoClient.bulkWrite()`` method to perform
- :ref:`Client Level <java-sync-client-bulk-write>`: If your application connects to
{+mdb-server+} version 8.0 or later, you can use the ``MongoClient.bulkWrite()``
method to perform

bulk write operations on multiple collections and databases in the same cluster.
This method groups multiple kinds of write operations into one call.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This method groups multiple kinds of write operations into one call.
This method performs all write operations in one database call.


The ``MongoClient`` performs these operations by making a call for each
operation to the database. You can reduce the number of calls to the
database to one by using bulk operations.
.. _java-sync-coll-bulk-write:

Performing Bulk Operations
--------------------------
Collection Bulk Write
---------------------

Bulk operations consist of a large number of write operations. To perform
a bulk operation, pass a ``List`` of ``WriteModel`` documents to the
``bulkWrite()`` method. A ``WriteModel`` is a model that represents any
of the write operations.
Bulk operations consist of many write operations. To perform a bulk operation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: specify bulk write

Suggested change
Bulk operations consist of many write operations. To perform a bulk operation
Bulk write operations contain one or more write operations. To perform a bulk write operation

at the collection level, pass a ``List`` of ``WriteModel`` documents to the
``MongoCollection.bulkWrite()`` method. A ``WriteModel`` is a model that
represents any of the write operations.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
represents any of the write operations.
represents a write operation.


The ``MongoCollection.bulkWrite()`` method splits operations of different kinds into
different batches. For example, when you pass ``DeleteOneModel``,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S:

Suggested change
The ``MongoCollection.bulkWrite()`` method splits operations of different kinds into
different batches. For example, when you pass ``DeleteOneModel``,
The ``MongoCollection.bulkWrite()`` method performs each kind of write operation
in a separate batch. For example, when you pass ``DeleteOneModel``,

``DeleteManyModel``, and ``ReplaceOneModel`` operations to the method, the method
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
``DeleteManyModel``, and ``ReplaceOneModel`` operations to the method, the method
``DeleteManyModel``, and ``ReplaceOneModel`` objects to the method, the method

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
``DeleteManyModel``, and ``ReplaceOneModel`` operations to the method, the method
``DeleteManyModel``, and ``ReplaceOneModel`` operations to the method, it

splits the delete operations into one batch and the replace operation
into another. During this process, the client might reorder operations for
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S:

Suggested change
splits the delete operations into one batch and the replace operation
into another. During this process, the client might reorder operations for
creates two batches: one for the delete operations and one for the replace operation.
During this process, the client might reorder operations for

between these two sentences, maybe mention that a batch = one database call (unless that's somewhere else on the page)

efficiency if the bulk operation is not ordered.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: put this last line in a note admonition and link to the section on ordering


The following sections show how to create and use each ``WriteModel``
document. The examples in each section use the following documents in the
Expand All @@ -44,7 +67,7 @@ document. The examples in each section use the following documents in the
{ "_id": 8, "name": "Shayla Ray", "age": 20 }

For more information about the methods and classes mentioned in this section,
see the following API Documentation:
see the following API documentation:

- `bulkWrite() <{+api+}/apidocs/mongodb-driver-sync/com/mongodb/client/MongoCollection.html#bulkWrite(java.util.List,com.mongodb.client.model.BulkWriteOptions)>`__
- `WriteModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/WriteModel.html>`__
Expand Down Expand Up @@ -100,7 +123,7 @@ describing people:
For more information about the methods and classes mentioned in this section,
see the `InsertOneModel
<{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/InsertOneModel.html>`__
API Documentation.
API documentation.

Replace Operation
~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -132,7 +155,7 @@ contains an added ``location`` field:
For more information about the methods and classes mentioned in this section,
see the following resources:

- `ReplaceOneModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/ReplaceOneModel.html>`__ API Documentation
- `ReplaceOneModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/ReplaceOneModel.html>`__ API documentation
- :manual:`Unique indexes </core/index-unique/>` Server Manual Explanation

Update Operation
Expand Down Expand Up @@ -168,8 +191,8 @@ the ``age`` field in a document where the ``_id`` is ``2``:
For more information about the methods and classes mentioned in this section,
see the following resources:

- `UpdateOneModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/UpdateOneModel.html>`__ API Documentation
- `UpdateManyModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/UpdateManyModel.html>`__ API Documentation
- `UpdateOneModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/UpdateOneModel.html>`__ API documentation
- `UpdateManyModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/UpdateManyModel.html>`__ API documentation
- :manual:`unique indexes </core/index-unique/>` Server Manual Explanation

Delete Operation
Expand Down Expand Up @@ -202,22 +225,22 @@ a document where the ``_id`` is ``1``:
:end-before: end deleteDocumentsExample

For more information about the methods and classes mentioned in this section,
see the following API Documentation:
see the following API documentation:

- `DeleteOneModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/DeleteOneModel.html>`__
- `DeleteManyModel <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/DeleteManyModel.html>`__

.. _orderOfExecution:

Order of Execution
------------------
~~~~~~~~~~~~~~~~~~

The ``bulkWrite()`` method accepts an optional ``BulkWriteOptions`` as a
second parameter to specify whether the execution of the bulk operations is
ordered or unordered.
ordered or unordered.

Ordered Execution
~~~~~~~~~~~~~~~~~
+++++++++++++++++

By default, the ``bulkWrite()`` method executes bulk operations in
order. This means that the bulk operations execute in the order you
Expand Down Expand Up @@ -253,7 +276,7 @@ document:
{ "_id": 6, "name": "Zaynab Hassan", "age": 37 }

Unordered Execution
~~~~~~~~~~~~~~~~~~~
+++++++++++++++++++

You can also execute bulk operations in any order by specifying "false"
to the ``order()`` method on ``BulkWriteOptions``. This means that
Expand Down Expand Up @@ -288,18 +311,200 @@ operations to execute in any order:
{ "_id": 6, "name": "Zaynab Omar", "age": 37 }

For more information about the methods and classes mentioned in this section,
see the following API Documentation:
see the following API documentation:

- `BulkWriteOptions <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/BulkWriteOptions.html>`__
- `ordered() <{+api+}/apidocs/mongodb-driver-core/com/mongodb/client/model/BulkWriteOptions.html#ordered(boolean)>`__

.. _java-sync-client-bulk-write:

Client Bulk Write
-----------------

When connecting to a deployment running {+mdb-server+} 8.0 or later,
you can use the ``MongoClient.bulkWrite()`` method to write
to multiple databases and collections in the same cluster. The ``MongoClient.bulkWrite()``
method does not split different kinds of operations into different batches.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: if someone jumps straight to this section, they might not see that this line is intended to compare client bulk writes to collection bulk writes.

Suggested change
to multiple databases and collections in the same cluster. The ``MongoClient.bulkWrite()``
method does not split different kinds of operations into different batches.
to multiple databases and collections in the same cluster. The ``MongoClient.bulkWrite()``
method performs all write operation in a single batch.


In the examples in preceding sections of this page, the ``MongoCollection.bulkWrite()`` method
takes a list of ``WriteModel`` instances in which the specified subclass of
``WriteModel`` represents the corresponding write operation. For example, an
instance of ``InsertOneModel`` represents an operation to insert one document.

Similarly, the ``MongoClient.bulkWrite()`` method takes a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: unfortunately, i think it's worth repeating the info from earlier in the page just to users don't have to read earlier sections if they don't need to. if you have a lot of repeated content, you can always put it in an includes directive 😎

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

honestly, I'm not sure how much this information is relevant so I will remove it 👍

list of ``ClientNamespacedWriteModel`` instances to represent different write operations.
For example, an instance of ``ClientNamespacedInsertOneModel`` represents an
operation to insert one document.

You can construct instances of ``ClientNamespacedWriteModel`` using the following
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can construct instances of ``ClientNamespacedWriteModel`` using the following
You can construct instances of the ``ClientNamespacedWriteModel`` class by using the following

methods:

- ``insertOne()``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S: would use this pattern for each method and delete the following paragraph

Suggested change
- ``insertOne()``
- ``ClientNamespacedWriteModel.insertOne()``: Constructs a ``ClientNamespacedInsertOneModel`` instance.


- ``updateOne()``

- ``updateMany()``

- ``replaceOne()``

- ``deleteOne()``

- ``deleteMany()``

These methods are used to construct corresponding write models.
For example, ``ClientNamespacedWriteModel.updateOne()`` is used to
construct a ``ClientNamespacedUpdateOneModel`` instance, which represents an
update operation.

These methods take a ``MongoNamespace`` object that defines which
database and collection to write to. Some, such as ``insertOne()``, can take a
``Document`` instance that defines information about the write operation.
Some other methods, such as ``updateOne()`` and ``replaceOne()``, take a filter
object that defines the subset of documents to be updated or replaced.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for comprehensiveness and clarity, we probably want to specify which methods take what. you could give each method its own subsection, or add a parameter list/table for each one


The following sections provide examples of how to use the client ``bulkWrite()``
method.

Insert Example
~~~~~~~~~~~~~~

This example shows how to use the ``bulkWrite()`` method to insert
two documents. One document is inserted into the ``db.people`` collection, while
the other document is inserted into the ``db.things`` collection.
The ``MongoNamespace`` instance defines the databases and collections that
each write operation applies to.

.. code-block:: java

MongoNamespace peopleNamespace = new MongoNamespace("db", "people");
MongoNamespace thingsNamespace = new MongoNamespace("db", "things");

List<ClientNamespacedWriteModel> bulkOperations = new ArrayList<>();

ClientNamespacedInsertOneModel insertDocument1 = new ClientNamespacedWriteModel
.insertOne(
peopleNamespace,
new Document("name", "Julia Smith")
);

ClientNamespacedInsertOneModel insertDocument2 = new ClientNamespacedWriteModel
.insertOne(
thingsNamespace,
new Document("object", "washing machine")
);

bulkOperations.add(insertDocument1);
bulkOperations.add(insertDocument2);

ClientBulkWriteResult result = mongoClient.bulkWrite(bulkOperations);

.. TODO: link documentation

Replace Example
~~~~~~~~~~~~~~~

The following example shows how to use the ``bulkWrite()`` method to replace
existing documents in the ``db.people`` and ``db.things`` collections.

.. code-block:: java

MongoNamespace peopleNamespace = new MongoNamespace("db", "people");
MongoNamespace thingsNamespace = new MongoNamespace("db", "things");

List<ClientNamespacedWriteModel> bulkOperations = new ArrayList<>();

bulkOperations.add(ClientNamespacedWriteModel.replaceOne(
peopleNamespace,
Filters.eq("_id", 1),
new Document("name", "Frederic Hilbert")
)
);

bulkOperations.add(ClientNamespacedWriteModel.replaceOne(
thingsNamespace,
Filters.eq("_id", 1),
new Document("object", "potato")
)
);

ClientBulkWriteResult result = mongoClient.bulkWrite(bulkOperations);

After this example runs successfully, the document that has an ``_id`` value of ``1``
in the ``people`` collection is replaced with a new document. The document in
the ``things`` collection that has an ``_id`` value of ``1``
is replaced with a new document.

.. _java-sync-client-bulk-write-options:

Bulk Write Options
~~~~~~~~~~~~~~~~~~

You can pass an instance of ``ClientBulkWriteOptions`` to the ``bulkWrite()``
method to specify options when running the bulk write operations.

Order of Execution Example
``````````````````````````

By default, the individual operations in a bulk operation are executed in the
order that you specify them until an error occurs, or until they execute
successfully. However, you can pass ``false`` to the ``ordered()`` method on
the ``ClientBulkWriteOptions`` interface to perform write operations in an
unordered way. When using the unordered option, an error-producing operation
does not prevent execution of other write operations in the call to the
``bulkWrite()`` method.

The following code sets the ``ordered()`` method on an
instance of ``ClientBulkWriteOptions`` and performs a bulk write operation to
insert multiple documents.

.. code-block:: java

MongoNamespace namespace = new MongoNamespace("db", "people");

ClientBulkWriteOptions options = new ClientBulkWriteOptions().ordered(false);

List<ClientNamespacedWriteModel> bulkOperations = new ArrayList<>();

bulkOperations.add(
ClientNamespacedWriteModel.insertOne(
namespace,
new Document("_id", 1).append("name", "Rudra Suraj")
)
);

// Causes a duplicate key error
bulkOperations.add(
ClientNamespacedWriteModel.insertOne(
namespace,
new Document("_id", 1).append("name", "Mario Bianchi")
)
);

bulkOperations.add(
ClientNamespacedWriteModel.insertOne(
namespace,
new Document("name", "Wendy Zhang")
)
);

ClientBulkWriteResult result = mongoClient.bulkWrite(bulkOperations, options);

Even though the write operation inserting a document with a duplicate key results
in an error, the other operations are executed because the write operation is
unordered.

.. TODO: link documentation

Summary
-------

``MongoCollection.bulkWrite()``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To perform a bulk operation, you create and pass a list of
``WriteModel`` documents to the ``bulkWrite()`` method.
``WriteModel`` instances to the ``bulkWrite()`` method.

There are 6 different ``WriteModel`` documents: ``InsertOneModel``,
There are 6 different ``WriteModel`` subtypes: ``InsertOneModel``,
``ReplaceOneModel``, ``UpdateOneModel``, ``UpdateManyModel``,
``DeleteOneModel`` and ``DeleteManyModel``.

Expand All @@ -308,3 +513,29 @@ There are two ways to execute the ``bulkWrite()`` method:
- Ordered, which performs the bulk operations in order until an error occurs, if any
- Unordered, which performs all the bulk operations in any order and reports errors
at the end, if any

``MongoClient.bulkWrite()``
~~~~~~~~~~~~~~~~~~~~~~~~~~~

When connecting to a deployment running {+mdb-server+} version 8.0 or later, you
can use the ``MongoClient.bulkWrite()`` method to perform bulk operations on multiple
databases and collections at once.

To perform a client bulk operation, you create an pass a list of
``ClientNamespacedWriteModel`` instances to this method.

There are six subtypes of ``ClientNamespacedWriteModel`` that are used to
represent write operations. To construct these write models, you can use the
corresponding ``ClientNamespacedWriteModel`` methods ``insertOne()``, ``updateOne()``,
``updateMany()``, ``replaceOne()``, ``deleteOne()``, and ``deleteMany()``. These
methods take a ``MongoNamespace`` object that defines which
database and collection to write to.

The ``MongoClient.bulkWrite()`` method can also take a ``ClientBulkWriteOptions``
object to specify different options for how the command is executed.

To learn more about the client ``bulkWrite`` command, see the
:manual:`bulkWrite() <reference/command/bulkWrite/>` method reference in the {+mdb-server+}
Manual.

.. TODO: Add API Documentation
Loading
Loading