4
4
Copy Existing Data
5
5
==================
6
6
7
- This usage example demonstrates how to copy data from a MongoDB collection to an
7
+ .. facet::
8
+ :name: genre
9
+ :values: reference
10
+
11
+ .. meta::
12
+ :keywords: code example, copy, filter, configuration setting
13
+
14
+ .. contents:: On this page
15
+ :local:
16
+ :backlinks: none
17
+ :depth: 2
18
+ :class: singlecol
19
+
20
+ These usage examples demonstrate how to copy data from MongoDB to an
8
21
{+kafka+} topic using the {+source-connector+}.
9
22
10
- Example
11
- -------
23
+ Examples
24
+ --------
25
+
26
+ The following examples show how to configure your source connector to
27
+ copy existing data from a single collection or from multiple collections.
28
+
29
+ Copy and Filter Collection Data
30
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
12
31
13
32
Suppose you want to copy a MongoDB collection to {+kafka+} and filter some data.
14
33
@@ -51,7 +70,7 @@ The ``customers`` collection contains the following documents:
51
70
.. _source-usage-example-copy-existing-data-copy-data:
52
71
53
72
Copy Data
54
- ~~~~~~~~~
73
+ `````````
55
74
56
75
Copy the contents of the ``customers`` collection of the ``shopping`` database by
57
76
specifying the following configuration options in your source connector:
@@ -78,7 +97,7 @@ To learn more about the ``startup.mode`` option, see
78
97
.. _source-usage-example-copy-existing-data-mask-data:
79
98
80
99
Filter Data
81
- ~~~~~~~~~~~
100
+ ```````````
82
101
83
102
You can filter data by specifying an aggregation pipeline in the
84
103
``startup.mode.copy.existing.pipeline`` option of your source connector configuration. The
@@ -97,9 +116,10 @@ To learn more about aggregation pipelines, see the following resources:
97
116
- :ref:`<source-usage-example-custom-pipeline>` Usage Example
98
117
- :manual:`Aggregation </aggregation>` in the MongoDB manual.
99
118
119
+ .. _source-usage-example-copy-existing-data-config:
100
120
101
121
Specify the Configuration
102
- ~~~~~~~~~~~~~~~~~~~~~~~~~
122
+ `````````````````````````
103
123
104
124
Your final source connector configuration to copy the ``customers`` collection should
105
125
look like this:
@@ -116,10 +136,47 @@ in the ``shopping.customers`` {+kafka+} topic:
116
136
.. literalinclude:: /includes/usage-examples/copy/payload.json
117
137
:language: json
118
138
:copyable: false
119
- :emphasize-lines: 6,7
120
139
121
140
.. note:: Write the Data in your Topic into a Collection
122
141
123
142
Use a change data capture handler to convert change event documents in an
124
143
{+kafka+} topic into MongoDB write operations. To learn more, see the
125
144
:ref:`Change Data Capture Handlers <sink-fundamentals-cdc-handler>` guide.
145
+
146
+ Copy Data From Multiple Sources
147
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
148
+
149
+ Suppose you want to copy data from another collection in the ``shopping``
150
+ database named ``products``, which contains the following document:
151
+
152
+ .. code-block:: json
153
+ :copyable: false
154
+
155
+ {
156
+ "_id": 1,
157
+ "item_name": "lipstick",
158
+ "department": "cosmetics",
159
+ "quantity": 45
160
+ }
161
+
162
+ You can copy from both the ``customers`` and ``products`` collections
163
+ by using the ``startup.mode.copy.existing.namespace.regex`` configuration
164
+ setting, as shown in the following code:
165
+
166
+ .. literalinclude:: /includes/usage-examples/copy/multisource.properties
167
+ :language: properties
168
+ :emphasize-lines: 5
169
+
170
+ In addition to the change event document in the ``shopping.customers`` {+kafka+} topic,
171
+ described in the :ref:`preceding section <source-usage-example-copy-existing-data-config>`,
172
+ you can see the following document in the ``shopping.products`` topic:
173
+
174
+ .. literalinclude:: /includes/usage-examples/copy/multisource.json
175
+ :language: json
176
+ :copyable: false
177
+
178
+ .. tip::
179
+
180
+ To learn more about the ``startup.mode.copy.existing.namespace.regex`` setting,
181
+ see the :ref:`Settings table <source-configuration-startup-table-start>` in
182
+ the Startup Properties guide.
0 commit comments