Skip to content

Commit 4842819

Browse files
Chris Choschmalliso
authored andcommitted
DOCSP-15811: topic naming usage examples (#171)
* DOCSP-15811: topic naming usage examples
1 parent c9e4eaf commit 4842819

File tree

1 file changed

+132
-1
lines changed

1 file changed

+132
-1
lines changed

source/source-connector/usage-examples/topic-naming.txt

Lines changed: 132 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,135 @@
44
Topic Naming
55
============
66

7-
asdf
7+
.. default-domain:: mongodb
8+
9+
The examples on this page show how to configure your MongoDB Kafka source
10+
connector to customize the name of the topic to which it publishes records.
11+
12+
By default, the MongoDB Kafka source connector publishes change event data
13+
to a Kafka topic with the same name as the MongoDB **namespace** from which
14+
the change events originated. A namespace is a string that's composed of the
15+
database and collection name concatenated with a dot "." character.
16+
17+
The following examples show different ways that you can customize the
18+
Kafka topics to which the connector publishes change event data:
19+
20+
- :ref:`Topic Prefix <topic-naming-prefix-example>`
21+
- :ref:`Topic Suffix <topic-naming-suffix-example>`
22+
- :ref:`Topic Namespace Map <topic-naming-namespace-map-example>`
23+
- :ref:`Topic Namespace Map with Wildcard <topic-naming-namespace-map-wildcard-example>`
24+
25+
.. _topic-naming-prefix-example:
26+
27+
Topic Prefix Example
28+
~~~~~~~~~~~~~~~~~~~~
29+
30+
You can configure your source connector to prepend a string to the
31+
namespace of the change event data, and publish records to that Kafka
32+
topic. This setting automatically concatenates your prefix with your
33+
namespace with the "." character.
34+
35+
To specify the topic prefix, use the ``topic.prefix`` configuration
36+
setting as shown in the following example:
37+
38+
.. code-block:: ini
39+
:emphasize-lines: 1
40+
41+
topic.prefix=myPrefix
42+
database=test
43+
collection=data
44+
45+
Once set, your connector publishes any changes to the ``data`` collection
46+
in the ``test`` database to the Kafka topic named ``myPrefix.test.data``.
47+
48+
.. _topic-naming-suffix-example:
49+
50+
Topic Suffix Example
51+
~~~~~~~~~~~~~~~~~~~~
52+
53+
You can configure your source connector to append a string to the
54+
namespace of the change event data, and publish records to that Kafka
55+
topic. This setting automatically concatenates your namespace with your
56+
suffix with the "." character.
57+
58+
To specify the topic suffix, use the ``topic.suffix`` configuration
59+
setting as shown in the following example:
60+
61+
.. code-block:: ini
62+
:emphasize-lines: 1
63+
64+
topic.suffix=mySuffix
65+
database=test
66+
collection=data
67+
68+
Once set, your connector publishes any changes to the ``data`` collection
69+
in the ``test`` database to the Kafka topic named ``test.data.mySuffix``.
70+
71+
.. _topic-naming-namespace-map-example:
72+
73+
Topic Namespace Map Example
74+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
75+
76+
You can configure your source connector to map namespace values to Kafka
77+
topic names for incoming change event data.
78+
79+
If the database name or namespace of the change event matches one of the
80+
fields in the map, the connector publishes the record to the value that
81+
corresponds to that mapping.
82+
83+
If the database name or namespace of the change event do not match any
84+
mapping, the connector publishes the record using the default topic naming
85+
scheme unless otherwise specified by a different topic naming setting.
86+
87+
Any mapping that includes both database and collection takes precedence
88+
over mappings that only specify the source database name.
89+
90+
.. important::
91+
92+
The namespace map matching occurs before the connector applies any other
93+
topic naming setting. If defined, the connector applies the
94+
``topic.prefix`` and the ``topic.suffix`` settings to the topic name
95+
after the mapping.
96+
97+
The following example shows how to specify the ``topic.namespace.map``
98+
setting to define a topic namespace mappings from the ``carDb`` database
99+
to the ``automobiles`` topic and the ``carDb.ev`` namespace to the
100+
``electricVehicles`` topic:
101+
102+
.. code-block:: ini
103+
104+
topic.namespace.map={"carDb": "automobiles", "carDb.ev": "electricVehicles"}
105+
106+
Since the ``carDb.ev`` namespace mapping takes precedence over the ``carDb``
107+
mapping, the connector performs the following actions:
108+
109+
- If the change event came from the database ``carDb`` and collection ``ev``,
110+
the connector sets the destination to the ``electricVehicles`` topic.
111+
- If the change event came from the database ``carDb`` and a collection
112+
other than ``ev``, the connector sets the destination to the
113+
``automobiles`` topic.
114+
- If the change document came from any database other than ``carDb``, the
115+
connector sets the destination topic to the default namespace naming
116+
scheme.
117+
- If defined, the connector applies the ``topic.prefix`` and
118+
``topic.suffix`` settings to the destination topic name after it
119+
performs namespace mapping.
120+
121+
.. _topic-naming-namespace-map-wildcard-example:
122+
123+
Topic Namespace Map with Wildcard Example
124+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
125+
126+
In addition to specifying database name and namespace in your topic
127+
namespace map as shown in :ref:`<topic-naming-namespace-map-example>`,
128+
you can use a wildcard ``*`` to match change events from all databases and
129+
namespaces without mappings.
130+
131+
.. code-block:: ini
132+
133+
topic.namespace.map={"carDb": "automobiles", "carDb.ev": "electricVehicles", "*": "otherVehicles"}
134+
135+
In the preceding wildcard example, the connector publishes change documents
136+
that originated from all databases other than ``carDb`` to the
137+
``otherVehicles`` topic.
138+

0 commit comments

Comments
 (0)