Skip to content

Commit 6c8ca4b

Browse files
Chris Choschmalliso
authored andcommitted
DOCSP-12667: add new copy.existing settings to example (#73)
* DOCSP-12667: add new copy.existing settings to example
1 parent 922a7bb commit 6c8ca4b

File tree

1 file changed

+42
-10
lines changed

1 file changed

+42
-10
lines changed

source/kafka-source.txt

Lines changed: 42 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -196,9 +196,9 @@ an example source connector configuration file, see
196196

197197
* - output.schema.key
198198
- string
199-
- The `Avro schema <https://avro.apache.org/docs/current/spec.html#schemas>`__
199+
- The `Avro schema <https://avro.apache.org/docs/current/spec.html#schemas>`__
200200
definition for the key document of the SourceRecord.
201-
201+
202202
**Default**:
203203

204204
.. code-block:: json
@@ -213,13 +213,13 @@ an example source connector configuration file, see
213213

214214
* - output.schema.value
215215
- string
216-
- The `Avro schema <https://avro.apache.org/docs/current/spec.html#schemas>`__
216+
- The `Avro schema <https://avro.apache.org/docs/current/spec.html#schemas>`__
217217
definition for the value document of the SourceRecord.
218218

219219
**Default**:
220220

221221
.. code-block:: json
222-
222+
223223
{
224224
"name": "ChangeStream",
225225
"type": "record",
@@ -300,8 +300,8 @@ an example source connector configuration file, see
300300
- string
301301
- | Prefix to prepend to database & collection names to generate the name of the Kafka topic to publish data to.
302302

303-
.. seealso::
304-
303+
.. seealso::
304+
305305
:ref:`Topic naming example <topic-naming-example>`.
306306

307307
| **Default**: ""
@@ -329,6 +329,11 @@ an example source connector configuration file, see
329329

330330
copy.existing.namespace.regex=stats\.page.*
331331

332+
Note that in the example above, the "\\" character in the example
333+
escapes the following "." character in the regular expression.
334+
For more information on how to build regular expressions, see the
335+
`Java SE documentation on Patterns <https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html>`__.
336+
332337
| **Default**: ""
333338
| **Accepted Values**: A valid regular expression
334339

@@ -409,9 +414,9 @@ an example source connector configuration file, see
409414

410415
* - heartbeat.interval.ms
411416
- int
412-
- The length of time in milliseconds between sending heartbeat messages to
413-
record a post batch resume token when no source records have been published.
414-
This can improve the resumability of the connector for low volume namespaces.
417+
- The length of time in milliseconds between sending heartbeat messages to
418+
record a post batch resume token when no source records have been published.
419+
This can improve the resumability of the connector for low volume namespaces.
415420
Use ``0`` to disable.
416421

417422
| **Default**: ``0``
@@ -420,7 +425,7 @@ an example source connector configuration file, see
420425
* - heartbeat.topic.name
421426
- string
422427
- The name of the topic to write heartbeat messages to.
423-
428+
424429
| **Default**: ``__mongodb_heartbeats``
425430
| **Accepted Values**: A valid Kafka topic name
426431

@@ -557,6 +562,33 @@ events, and broadcast changes after any collection data is updated.
557562
database=example
558563
copy.existing=true
559564

565+
You can use the ``copy.existing.namespace.regex`` setting to provide
566+
a regular expression that matches specific collections by their namespace
567+
(database and collection name, separated with a "." character).
568+
569+
.. code-block:: properties
570+
571+
copy.existing.namespace.regex=stats\.page.*
572+
copy.existing=true
573+
574+
The configuration shown above matches and copies existing data from all
575+
collections within the ``stats`` database. E.g. it matches
576+
``stats.pageViews`` or ``stats.pageAverageRevenue``, but not
577+
``stat.pageViews`` or ``stats.uniquePages``.
578+
579+
You can use the ``copy.existing.pipeline`` setting to provide a MongoDB
580+
aggregation pipeline expression that matches specific documents to include
581+
in the copy operation.
582+
583+
.. code-block:: properties
584+
585+
copy.existing.pipeline=[ { $match: { totalUniqueViews: { $gte: 5000 }} ]
586+
copy.existing=true
587+
588+
The configuration shown above matches and copies existing documents that
589+
contain a field named ``totalUniqueViews`` with a value of ``5000`` or
590+
greater.
591+
560592
How to Recover from An Invalid Resume Token
561593
-------------------------------------------
562594

0 commit comments

Comments
 (0)