Skip to content

Commit c0cffa7

Browse files
Chris Choschmalliso
authored andcommitted
DOCSP-14392: add topic naming strategies (#82)
* DOCSP-14392: add topic naming strategies
1 parent c5d6fc6 commit c0cffa7

File tree

1 file changed

+123
-12
lines changed

1 file changed

+123
-12
lines changed

source/kafka-source.txt

Lines changed: 123 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ an example source connector configuration file, see
117117
* - connection.uri
118118
- string
119119
- | A :manual:`MongoDB connection URI string </reference/connection-string/#standard-connection-string-format>`.
120-
120+
121121
.. code-block:: none
122122

123123
mongodb://username:password@localhost/
@@ -192,7 +192,7 @@ an example source connector configuration file, see
192192

193193
* - output.json.formatter
194194
- string
195-
- | Full class name of the JSON formatter. You can also provide your own custom JSON formatter.
195+
- | Full class name of the JSON formatter. You can also provide your own custom JSON formatter.
196196
|
197197
| **Default**: ``com.mongodb.kafka.connect.source.json.formatter.DefaultJson``
198198
| **Accepted Values**:
@@ -311,15 +311,72 @@ an example source connector configuration file, see
311311

312312
* - topic.prefix
313313
- string
314-
- | Prefix to prepend to database & collection names to generate the name of the Kafka topic to publish data to.
314+
- | Prefix to prepend to database and collection names to generate the name of the Kafka topic to publish data to.
315+
316+
.. seealso::
317+
318+
:ref:`Topic naming example <topic-naming-prefix-example>`.
319+
320+
| **Default**: ""
321+
| **Accepted Values**: A string
322+
323+
* - topic.suffix
324+
- string
325+
- | Suffix to append to database and collection names to generate the name of the Kafka topic to publish data to.
315326

316327
.. seealso::
317328

318-
:ref:`Topic naming example <topic-naming-example>`.
329+
:ref:`Topic naming example <topic-naming-suffix-example>`.
319330

320331
| **Default**: ""
321332
| **Accepted Values**: A string
322333

334+
* - topic.namespace.map
335+
- string
336+
- | JSON object that maps change stream document namespaces to
337+
topics.
338+
339+
.. example::
340+
341+
The following configuration specifies the following two mappings:
342+
343+
- All change documents in the ``myDb.myColl`` are sent to the ``topicTwo`` topic
344+
- All other change documents in the ``myDb`` database are sent to the ``topicOne`` topic
345+
346+
.. code-block:: properties
347+
:copyable: false
348+
349+
topic.namespace.map={"myDb": "topicOne", "myDb.myColl\": "topicTwo"}
350+
351+
352+
You can also use the "*" wildcard character to match namespaces.
353+
354+
.. example::
355+
356+
The following configuration specifies a mapping from all of the
357+
change stream document namespaces to the ``topicThree`` topic:
358+
359+
.. code-block:: properties
360+
:copyable: false
361+
362+
topic.namespace.map={"*": "topicThree"}
363+
364+
.. seealso::
365+
366+
:ref:`Topic naming example <topic-naming-namespace-map-example>`.
367+
368+
| **Default**: ""
369+
| **Accepted Values**: A valid JSON object
370+
371+
* - topic.mapper
372+
- string
373+
- | Full class name of the class that specifies custom topic mapping logic.
374+
|
375+
| **Default**: ``com.mongodb.kafka.connect.source.topic.mapping.DefaultTopicMapper``
376+
| **Accepted Values**: Valid full class name of an implementation
377+
of the `TopicMapper <https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/source/topic/mapping/TopicMapper.java>`__
378+
class.
379+
323380
* - copy.existing
324381
- boolean
325382
- | Copy existing data from source collections and convert them to Change Stream events on their respective topics. Any changes to the data that occur during the copy process are applied once the copy is completed.
@@ -535,19 +592,28 @@ ones in the collection named "customers":
535592
For more information on how to build regular expressions, see the
536593
`Java SE documentation on Patterns <https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html>`__.
537594

538-
.. _topic-naming-example:
539595

540-
Topic Naming Example
541-
--------------------
542596

543-
The MongoDB Kafka Source connector publishes the changed data events to
544-
a Kafka topic that consists of the database and collection name from which
545-
the change originated. For example, if an insert was performed on the
597+
Topic Naming Examples
598+
---------------------
599+
600+
By default, the MongoDB Kafka Source connector publishes the change data
601+
events to a Kafka topic that consists of the database and collection name --
602+
also known as a :manual:`namespace </reference/limits/#faq-dev-namespace>` --
603+
from which the change originated. For example, if an insert was performed on the
546604
``test`` database and ``data`` collection, the connector will publish the
547605
data to a topic named ``test.data``.
548606

549-
If the ``topic.prefix`` configuration is set to **true**, the Kafka topic
550-
name will be prepended with the specified value. For example:
607+
The following examples show how you can configure the topic name for
608+
change data events.
609+
610+
.. _topic-naming-prefix-example:
611+
612+
Topic Prefix Example
613+
~~~~~~~~~~~~~~~~~~~~
614+
615+
If you specify a value in the ``topic.prefix`` configuration setting, the
616+
connector prepends that value to the Kafka topic name. For example:
551617

552618
.. code-block:: properties
553619

@@ -556,6 +622,51 @@ name will be prepended with the specified value. For example:
556622
Once set, any data changes to the ``data`` collection in the ``test`` database
557623
are published to a topic named ``mongo.test.data``.
558624

625+
.. _topic-naming-suffix-example:
626+
627+
Topic Suffix Example
628+
~~~~~~~~~~~~~~~~~~~~
629+
630+
If you specify a value in the ``topic.suffix`` configuration setting, the
631+
connector appends that value to the Kafka topic name. For example:
632+
633+
.. code-block:: properties
634+
635+
topic.suffix=mongo
636+
637+
Once set, any data changes to the ``data`` collection in the ``test`` database
638+
are published to a topic named ``test.data.mongo``.
639+
640+
.. _topic-naming-namespace-map-example:
641+
642+
Topic Namespace Map Example
643+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
644+
645+
If you specify mappings in the ``topic.namespace.map`` configuration
646+
setting, the connector publishes using the default topic naming scheme
647+
unless otherwise specified by the mapping.
648+
649+
Any mapping that includes both database and collection takes precedence
650+
over mappings that only specify the source database name. The following
651+
configuration example shows mappings for the ``carDb`` database as well as the
652+
``carDb.ev`` namespace:
653+
654+
.. code-block:: properties
655+
656+
topic.namespace.map={"carDb": "automobiles", "carDb.ev": "electricVehicles"}
657+
658+
Since the ``carDb.ev`` takes precedence over the ``carDb`` mapping, the
659+
connector performs the following:
660+
661+
- If the change document is from the database ``carDb`` and collection ``ev``,
662+
change documents are sent to the ``electricVehicles`` topic.
663+
- Any change documents from the database ``carDb`` that are *not* from the
664+
collection ``ev`` are sent to the ``automobiles`` topic.
665+
- If the change document is from any database other than ``carDb``, the
666+
connector sends it to the topic determined by the default namespace naming
667+
scheme which includes any value specified in the ``topic.prefix`` or
668+
``topic.suffix`` setting.
669+
559670
Existing Data Copy Example
560671
--------------------------
561672

0 commit comments

Comments
 (0)