Skip to content

Commit 8d6c63d

Browse files
authored
DOCSP-36972: Copy data from multiple sources (#199)
* DOCSP-36972: Copy data from multiple sources * edits * edits * admonition * JV feedback * fix * CN feedback
1 parent 8d4c934 commit 8d6c63d

File tree

4 files changed

+82
-8
lines changed

4 files changed

+82
-8
lines changed
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"_id": { "_id": 1, "copyingData": true },
3+
"operationType": "insert",
4+
"documentKey": { "_id": 1 },
5+
"fullDocument": {
6+
"_id": 1,
7+
"item_name": "lipstick",
8+
"department": "cosmetics",
9+
"quantity": 45
10+
},
11+
"ns": { "db": "shopping", "coll": "products" }
12+
}
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
connector.class=com.mongodb.kafka.connect.MongoSourceConnector
2+
connection.uri=<your production MongoDB connection uri>
3+
database=shopping
4+
startup.mode=copy_existing
5+
startup.mode.copy.existing.namespace.regex=^shopping\.(customers|products)$

source/introduction/connect.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ To learn more about the connection URI format, see
3434
Prerequisites
3535
-------------
3636

37-
To connect the {+conenctor+} to MongoDB, you must grant the database user either
37+
To connect the {+connector+} to MongoDB, you must grant the database user either
3838
of the following:
3939

4040
- the ``readWrite`` role

source/source-connector/usage-examples/copy-existing-data.txt

Lines changed: 64 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,30 @@
44
Copy Existing Data
55
==================
66

7-
This usage example demonstrates how to copy data from a MongoDB collection to an
7+
.. facet::
8+
:name: genre
9+
:values: reference
10+
11+
.. meta::
12+
:keywords: code example, copy, filter, configuration setting
13+
14+
.. contents:: On this page
15+
:local:
16+
:backlinks: none
17+
:depth: 2
18+
:class: singlecol
19+
20+
These usage examples demonstrate how to copy data from MongoDB to an
821
{+kafka+} topic using the {+source-connector+}.
922

10-
Example
11-
-------
23+
Examples
24+
--------
25+
26+
The following examples show how to configure your source connector to
27+
copy existing data from a single collection or from multiple collections.
28+
29+
Copy and Filter Collection Data
30+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1231

1332
Suppose you want to copy a MongoDB collection to {+kafka+} and filter some data.
1433

@@ -51,7 +70,7 @@ The ``customers`` collection contains the following documents:
5170
.. _source-usage-example-copy-existing-data-copy-data:
5271

5372
Copy Data
54-
~~~~~~~~~
73+
`````````
5574

5675
Copy the contents of the ``customers`` collection of the ``shopping`` database by
5776
specifying the following configuration options in your source connector:
@@ -78,7 +97,7 @@ To learn more about the ``startup.mode`` option, see
7897
.. _source-usage-example-copy-existing-data-mask-data:
7998

8099
Filter Data
81-
~~~~~~~~~~~
100+
```````````
82101

83102
You can filter data by specifying an aggregation pipeline in the
84103
``startup.mode.copy.existing.pipeline`` option of your source connector configuration. The
@@ -97,9 +116,10 @@ To learn more about aggregation pipelines, see the following resources:
97116
- :ref:`<source-usage-example-custom-pipeline>` Usage Example
98117
- :manual:`Aggregation </aggregation>` in the MongoDB manual.
99118

119+
.. _source-usage-example-copy-existing-data-config:
100120

101121
Specify the Configuration
102-
~~~~~~~~~~~~~~~~~~~~~~~~~
122+
`````````````````````````
103123

104124
Your final source connector configuration to copy the ``customers`` collection should
105125
look like this:
@@ -116,10 +136,47 @@ in the ``shopping.customers`` {+kafka+} topic:
116136
.. literalinclude:: /includes/usage-examples/copy/payload.json
117137
:language: json
118138
:copyable: false
119-
:emphasize-lines: 6,7
120139

121140
.. note:: Write the Data in your Topic into a Collection
122141

123142
Use a change data capture handler to convert change event documents in an
124143
{+kafka+} topic into MongoDB write operations. To learn more, see the
125144
:ref:`Change Data Capture Handlers <sink-fundamentals-cdc-handler>` guide.
145+
146+
Copy Data From Multiple Sources
147+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
148+
149+
Suppose you want to copy data from another collection in the ``shopping``
150+
database named ``products``, which contains the following document:
151+
152+
.. code-block:: json
153+
:copyable: false
154+
155+
{
156+
"_id": 1,
157+
"item_name": "lipstick",
158+
"department": "cosmetics",
159+
"quantity": 45
160+
}
161+
162+
You can copy from both the ``customers`` and ``products`` collections
163+
by using the ``startup.mode.copy.existing.namespace.regex`` configuration
164+
setting, as shown in the following code:
165+
166+
.. literalinclude:: /includes/usage-examples/copy/multisource.properties
167+
:language: properties
168+
:emphasize-lines: 5
169+
170+
In addition to the change event document in the ``shopping.customers`` {+kafka+} topic,
171+
described in the :ref:`preceding section <source-usage-example-copy-existing-data-config>`,
172+
you can see the following document in the ``shopping.products`` topic:
173+
174+
.. literalinclude:: /includes/usage-examples/copy/multisource.json
175+
:language: json
176+
:copyable: false
177+
178+
.. tip::
179+
180+
To learn more about the ``startup.mode.copy.existing.namespace.regex`` setting,
181+
see the :ref:`Settings table <source-configuration-startup-table-start>` in
182+
the Startup Properties guide.

0 commit comments

Comments
 (0)