Skip to content

Commit aaaea09

Browse files
Chris Choschmalliso
Chris Cho
authored and
schmalliso
committed
Docsp 17833 sink configuration id postprocessor (#123)
* DOCSP-17833: sink properties for id strategy and post-processors
1 parent d1b9c2b commit aaaea09

File tree

6 files changed

+238
-7
lines changed

6 files changed

+238
-7
lines changed

‎source/includes/sink-config-link.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
For a list of sink connector configuration settings organized by category, see
2+
the guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.

‎source/sink-connector/configuration-properties.txt

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ Overview
1717

1818
TODO
1919

20+
For an example sink connector configuration file, see
21+
`MongoSinkConnector.properties
22+
<https://github.com/mongodb/mongo-kafka/blob/master/config/MongoSinkConnector.properties>`__.
23+
2024
.. _sink-settings:
2125

2226
Settings
@@ -32,12 +36,16 @@ See the following categories for a list of related configuration properties:
3236
- Description
3337

3438
* - :doc:`MongoDB Connection Properties </sink-connector/configuration-properties/mongodb-connection>`
35-
- How to connect to your MongoDB cluster.
39+
- Specify how to connect to your MongoDB cluster.
3640

3741
* - :doc:`MongoDB Namespace Mapping Properties </sink-connector/configuration-properties/mongodb-namespace>`
3842
- Specify where to sink your data.
3943

40-
TODO: add additional configuration properties pages
44+
* - :doc:`Connector Id Strategy Properties </sink-connector/configuration-properties/id-strategy>`
45+
- Specify how the connector generates document ids.
46+
47+
* - :doc:`Connector Post-processor Properties </sink-connector/configuration-properties/post-processors>`
48+
- Specify transformations of Kafka topic data.
4149

4250
See the `Confluent Sink Connector documentation <https://docs.confluent.io/current/installation/configuration/connect/sink-connect-configs.html>`__
4351
for more information on these settings.
@@ -47,4 +55,6 @@ for more information on these settings.
4755

4856
MongoDB Connection </sink-connector/configuration-properties/mongodb-connection>
4957
MongoDB Namespace </sink-connector/configuration-properties/mongodb-namespace>
58+
Id Strategy </sink-connector/configuration-properties/id-strategy>
59+
Post-processors <sink-connector/configuration-properties/post-processors>
5060

‎source/sink-connector/configuration-properties/id-strategy.txt

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
=====================================
2+
Sink Connector Id Strategy Properties
3+
=====================================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify how the sink connector
17+
should determine the ``_id`` value for each document it writes to MongoDB.
18+
19+
.. include:: /includes/sink-config-link.rst
20+
21+
Settings
22+
--------
23+
24+
.. list-table::
25+
:header-rows: 1
26+
:widths: 45 55
27+
28+
* - Name
29+
- Description
30+
31+
* - | **document.id.strategy**
32+
- | **Type:** string
33+
|
34+
| **Description:**
35+
| The class the connector should use to generate a unique ``_id`` field.
36+
|
37+
| **Default**:
38+
39+
.. code-block:: none
40+
41+
com.mongodb.kafka.connect.sink.processor.id.strategy.BsonOidStrategy
42+
43+
| **Accepted Values**: An empty string or a fully qualified Java class name
44+
45+
* - | **document.id.strategy.overwrite.existing**
46+
- | **Type:** boolean
47+
|
48+
| **Description:**
49+
| Whether the connector should overwrite existing values in the ``_id``
50+
field when it applies the strategy defined by the
51+
``document.id.strategy`` property.
52+
|
53+
| **Default**: ``false``
54+
| **Accepted Values**: ``true`` or ``false``
55+
56+
* - | **document.id.strategy.uuid.format**
57+
- | **Type:** string
58+
|
59+
| **Description:**
60+
| Whether the connector should output the UUID in the ``_id`` field
61+
in string format or in
62+
`BsonBinary <https://mongodb.github.io/mongo-java-driver/4.1/apidocs/bson/org/bson/BsonBinary.html>`__
63+
format.
64+
|
65+
| **Default**: ``string``
66+
| **Accepted Values**: ``string`` or ``binary``
67+
68+
* - | **delete.on.null.values**
69+
- | **Type:** boolean
70+
|
71+
| **Description:**
72+
| Whether the connector should delete documents when the key value
73+
matches a document in MongoDB and the value field is null. This
74+
setting applies when you specify an id generation strategy that
75+
operates on the key document such as ``FullKeyStrategy``,
76+
``PartialKeyStrategy``, and ``ProvidedInKeyStrategy``.
77+
|
78+
| **Default**: ``false``
79+
| **Accepted Values**: ``true`` or ``false``
80+

‎source/sink-connector/configuration-properties/mongodb-connection.txt

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,7 @@ Overview
1616
Use the following configuration settings to specify how your sink
1717
connector connects and communicates with your MongoDB cluster.
1818

19-
For a list of all sink connector configuration settings, see the
20-
guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
19+
.. include:: /includes/sink-config-link.rst
2120

2221
Settings
2322
--------

‎source/sink-connector/configuration-properties/mongodb-namespace.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,7 @@ Use the following configuration settings to specify which MongoDB database
1717
and collection that your sink connector writes data to. You can use the
1818
default ``DefaultNamespaceMapper`` or specify a custom class.
1919

20-
For a list of categories of sink connector configuration settings, see the
21-
guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
20+
.. include:: /includes/sink-config-link.rst
2221

2322
Settings
2423
--------
@@ -51,6 +50,7 @@ Settings
5150
.. code-block:: none
5251

5352
com.mongodb.kafka.connect.sink.namespace.mapping.DefaultNamespaceMapper
53+
5454
| **Accepted Values**: A fully qualified Java class name of a class that implements the ``NamespaceMapper`` interface.
5555

5656
* - **database**
@@ -90,7 +90,7 @@ configuration property to the fully-qualified class name as shown below:
9090

9191
.. code-block:: properties
9292

93-
namespace.mapper=com.mongodb.kafka.connect.sink.topic.mapping.FieldPathNamespaceMapper
93+
namespace.mapper=com.mongodb.kafka.connect.sink.namespace.mapping.FieldPathNamespaceMapper
9494

9595
The ``FieldPathNamespaceMapper`` requires you to specify the following
9696
settings:

‎source/sink-connector/configuration-properties/post-processors.txt

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
========================================
2+
Sink Connector Post-processor Properties
3+
========================================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify how the sink connector
17+
should transform Kafka data before inserting it into MongoDB.
18+
19+
.. include:: /includes/sink-config-link.rst
20+
21+
Settings
22+
--------
23+
24+
.. list-table::
25+
:header-rows: 1
26+
:widths: 25 75
27+
28+
* - Name
29+
- Description
30+
31+
* - **post.processor.chain**
32+
- | **Type:** list
33+
|
34+
| **Description:**
35+
| A list of post-processor classes the connector should apply to
36+
process the data before saving it to MongoDB.
37+
38+
.. seealso::
39+
40+
For more information on post-processors and examples of
41+
their usage, see the section on
42+
:doc:`Post-processors </sink-connector/fundamentals/post-processors>`.
43+
44+
|
45+
| **Default**:
46+
47+
.. code-block:: none
48+
49+
com.mongodb.kafka.connect.sink.processor.DocumentIdAdder
50+
51+
| **Accepted Values**: A comma-separated list of fully qualified Java class names
52+
53+
* - | **field.renamer.mapping**
54+
- | **Type:** string
55+
|
56+
| **Description:**
57+
| A list of field name mappings for key and value fields. Define
58+
the mappings in an inline JSON array in the following format:
59+
60+
.. code-block:: none
61+
62+
[ { "oldName":"key.fieldA", "newName":"field1" }, { "oldName":"value.xyz", "newName":"abc" } ]
63+
64+
| **Default**: ``[]``
65+
| **Accepted Values**: A valid JSON array
66+
67+
* - | **field.renamer.regexp**
68+
- | **Type:** string
69+
|
70+
| **Description:**
71+
| A list of field name mappings for key and value fields using
72+
regular expressions. Define the mappings in an inline JSON array
73+
in the following format:
74+
75+
.. code-block:: none
76+
77+
[ {"regexp":"^key\\\\..*my.*$", "pattern":"my", "replace":""}, {"regexp":"^value\\\\..*$", "pattern":"\\\\.", "replace":"_"} ]
78+
79+
| **Default**: ``[]``
80+
| **Accepted Values**: A valid JSON array
81+
82+
* - | **key.projection.list**
83+
- | **Type:** string
84+
|
85+
| **Description:**
86+
| A list of field names the connector should include in the key
87+
projection.
88+
|
89+
| **Default**: ``""``
90+
| **Accepted Values**: A comma-separated list of field names
91+
92+
* - | **key.projection.type**
93+
- | **Type:** string
94+
|
95+
| **Description:**
96+
| The key projection type the connector should use.
97+
|
98+
| **Default**: ``none``
99+
| **Accepted Values**: ``none``, ``BlockList``, or ``AllowList`` (*Deprecated: blacklist, whitelist*)
100+
101+
* - | **value.projection.list**
102+
- | **Type:** string
103+
|
104+
| **Description:**
105+
| A list of field names the connector should include in the value
106+
projection.
107+
|
108+
| **Default**: ``""``
109+
| **Accepted Values**: A comma-separated list of field names
110+
111+
* - | **value.projection.type**
112+
- | **Type:** string
113+
|
114+
| **Description:**
115+
| The type of value projection the connector should use.
116+
|
117+
| **Default**: ``none``
118+
| **Accepted Values**: ``none``, ``BlockList``, or ``AllowList`` (*Deprecated: blacklist, whitelist*)
119+
120+
* - | **writemodel.strategy**
121+
- | **Type:** string
122+
|
123+
| **Description:**
124+
| The class that specifies the ``WriteModel`` the connector should
125+
use for :manual:`Bulk Writes </core/bulk-write-operations/index.html>`.
126+
127+
.. seealso::
128+
129+
For information on how to create your own strategy, see the tutorial
130+
on :doc:`Write Strategies </tutorials/write-strategies/>`.
131+
132+
|
133+
| **Default**:
134+
135+
.. code-block:: none
136+
137+
com.mongodb.kafka.connect.sink.writemodel.strategy.DefaultWriteModelStrategy
138+
139+
| **Accepted Values**: A fully qualified Java class name
140+

0 commit comments

Comments
 (0)