Skip to content

Commit 5ce3b5d

Browse files
Chris Choschmalliso
Chris Cho
authored and
schmalliso
committed
DOCSP-17832: sink connector topic, message, error handling properties (#120)
* DOCSP-17832: sink connector topic, message, error handling properties
1 parent aaaea09 commit 5ce3b5d

File tree

4 files changed

+312
-0
lines changed

4 files changed

+312
-0
lines changed

‎source/sink-connector/configuration-properties.txt

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,15 @@ See the following categories for a list of related configuration properties:
4141
* - :doc:`MongoDB Namespace Mapping Properties </sink-connector/configuration-properties/mongodb-namespace>`
4242
- Specify where to sink your data.
4343

44+
* - :doc:`Kafka Topic Properties </sink-connector/configuration-properties/kafka-topic>`
45+
- Specify the Kafka topics to which the connector should subscribe.
46+
47+
* - :doc:`Sink Connector Message Processing Properties </sink-connector/configuration-properties/connector-message>`
48+
- Set batch size, rate limiting, and number of parallel tasks.
49+
50+
* - :doc:`Sink Connector Error Handling Properties </sink-connector/configuration-properties/error-handling>`
51+
- Specify how to respond to errors and configure the dead letter queue.
52+
4453
* - :doc:`Connector Id Strategy Properties </sink-connector/configuration-properties/id-strategy>`
4554
- Specify how the connector generates document ids.
4655

@@ -55,6 +64,9 @@ for more information on these settings.
5564

5665
MongoDB Connection </sink-connector/configuration-properties/mongodb-connection>
5766
MongoDB Namespace </sink-connector/configuration-properties/mongodb-namespace>
67+
Connector Topic </sink-connector/configuration-properties/kafka-topic>
68+
Connector Message Processing </sink-connector/configuration-properties/connector-message>
69+
Connector Error Handling </sink-connector/configuration-properties/error-handling>
5870
Id Strategy </sink-connector/configuration-properties/id-strategy>
5971
Post-processors <sink-connector/configuration-properties/post-processors>
6072

‎source/sink-connector/configuration-properties/connector-message.txt

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
=======================================
2+
Connector Message Processing Properties
3+
=======================================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the settings on this page to configure the message processing behavior of
17+
the sink connector including the following:
18+
19+
- Message batch size
20+
- Rate limits
21+
- Number of parallel tasks
22+
23+
For a list of categories of sink connector configuration settings, see our
24+
guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
25+
26+
Settings
27+
--------
28+
29+
.. list-table::
30+
:header-rows: 1
31+
:widths: 25 75
32+
33+
* - Name
34+
- Description
35+
36+
* - | **max.batch.size**
37+
- | **Type:** int
38+
|
39+
| **Description:**
40+
| Maximum number of sink records to batch together for processing.
41+
|
42+
| **Default**: ``0``
43+
| **Accepted Values**: An integer
44+
45+
* - | **rate.limiting.every.n**
46+
- | **Type:** int
47+
|
48+
| **Description:**
49+
| Number of batches of records the sink connector processes in
50+
order to trigger the rate limiting timeout. A value of 0 means no
51+
rate limiting.
52+
|
53+
| **Default**: ``0``
54+
| **Accepted Values**: An integer
55+
56+
* - | **rate.limiting.timeout**
57+
- | **Type:** int
58+
|
59+
| **Description:**
60+
| How long (in milliseconds) to wait before the sink connector
61+
should resume processing after reaching the rate limiting
62+
threshold.
63+
|
64+
| **Default**: ``0``
65+
| **Accepted Values**: An integer
66+
67+
* - | **tasks.max**
68+
- | **Type:** int
69+
|
70+
| **Description:**
71+
| The maximum number of tasks to create for this connector. The
72+
connector may create fewer than the maximum tasks specified if it
73+
cannot handle the level of parallelism you specify.
74+
75+
.. important:: Multiple Tasks May Process Messages Out of Order
76+
77+
If you specify a value greater than ``1``, the connector enables
78+
parallel processing of the tasks. If your topic has multiple
79+
partition logs, which enables the connector to read from the
80+
topic in parallel, the tasks may process the messages out of
81+
order.
82+
83+
| **Default**: ``1``
84+
| **Accepted Values**: An integer.
85+

‎source/sink-connector/configuration-properties/error-handling.txt

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
===================================
2+
Connector Error Handling Properties
3+
===================================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify how the sink connector
17+
handles errors and to configure the dead letter queue.
18+
19+
For a list of categories of sink connector configuration settings, see the
20+
guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
21+
22+
Settings
23+
--------
24+
25+
.. list-table::
26+
:header-rows: 1
27+
:widths: 50 50
28+
29+
* - Name
30+
- Description
31+
32+
* - | **mongo.errors.tolerance**
33+
- | **Type:** string
34+
|
35+
| **Description:**
36+
| Whether to continue processing messages if the connector encounters
37+
an error.
38+
| When set to ``none``, the connector reports any error and
39+
blocks further processing of the rest of the messages.
40+
| When set to ``all``, the connector ignores any problematic messages.
41+
42+
.. note::
43+
44+
This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#sinkconnectorconfigs_errors.tolerance>`__
45+
property of the Connect Framework.
46+
47+
| **Default:** ``"none"``
48+
| **Accepted Values**: ``"none"`` or ``"all"``
49+
50+
* - | **mongo.errors.log.enable**
51+
- | **Type:** boolean
52+
|
53+
| **Description:**
54+
| Whether the connector should write details of errors including
55+
failed operations to the log file. The connector classifies
56+
errors as "tolerated" or "not tolerated" using the
57+
``errors.tolerance`` or ``mongo.errors.tolerance`` settings.
58+
59+
| When set to ``true``, the connector logs both "tolerated" and
60+
"not tolerated" errors.
61+
| When set to ``false``, the connector logs "not tolertaed" errors.
62+
63+
.. note::
64+
65+
This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#sinkconnectorconfigs_errors.log.enable>`__
66+
property of the Connect Framework.
67+
68+
| **Default:** ``false``
69+
| **Accepted Values**: ``true`` or ``false``
70+
71+
* - | **errors.log.include.messages**
72+
- | **Type:** boolean
73+
|
74+
| **Description:**
75+
| Whether the connector should include the invalid message when
76+
logging an error. An invalid message includes data such as record
77+
keys, values, and headers.
78+
|
79+
| **Default:** ``false``
80+
| **Accepted Values**: ``true`` or ``false``
81+
82+
* - | **errors.deadletterqueue.topic.name**
83+
- | **Type:** string
84+
|
85+
| **Description:**
86+
| Name of topic to use as the dead letter queue. If blank, the
87+
connector does not send any invalid messages to the dead letter
88+
queue.
89+
| For more information about the dead letter queue, see the
90+
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
91+
|
92+
| **Default:** ``""``
93+
| **Accepted Values**: A valid Kafka topic name
94+
95+
* - | **errors.deadletterqueue.context.headers.enable**
96+
- | **Type:** boolean
97+
|
98+
| **Description:**
99+
| Whether the connector should include context headers when it
100+
writes messages to the dead letter queue.
101+
| For more information about the dead letter queue, see the
102+
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
103+
|
104+
| **Default:** ``false``
105+
| **Accepted Values**: ``true`` or ``false``
106+
107+
* - | **errors.deadletterqueue.topic.replication.factor**
108+
- | **Type:** integer
109+
|
110+
| **Description:**
111+
| The number of nodes on which to replicate the dead letter queue
112+
topic. If you are running a single-node Kafka cluster, you must
113+
set this to ``1``.
114+
| For more information about the dead letter queue, see the
115+
:ref:`Dead Letter Queue Configuration Example <sink-dead-letter-queue-configuration-example>`.
116+
|
117+
| **Default:** ``3``
118+
| **Accepted Values**: A valid number of nodes
119+
120+
121+
.. _sink-dead-letter-queue-configuration-example:
122+
123+
Dead Letter Queue Configuration Example
124+
---------------------------------------
125+
126+
Apache Kafka version 2.6 added support for handling errant records. The
127+
Kafka connector automatically sends messages that it cannot process to the
128+
**dead letter queue**. Once on the dead letter queue, you can inspect the
129+
errant records, update them, and resubmit them for processing.
130+
131+
The following is an example configuration for enabling the dead letter queue
132+
topic ``example.deadletterqueue``. This configuration specifies that the
133+
dead letter queue and log file should record invalid messages, and that
134+
the dead letter queue messages should include context headers.
135+
136+
.. code-block:: properties
137+
138+
mongo.errors.tolerance=all
139+
mongo.errors.log.enable=true
140+
errors.log.include.messages=true
141+
errors.deadletterqueue.topic.name=example.deadletterqueue
142+
errors.deadletterqueue.context.headers.enable=true
143+

‎source/sink-connector/configuration-properties/kafka-topic.txt

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
======================
2+
Kafka Topic Properties
3+
======================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify which Kafka topics the
17+
sink connector should watch for data.
18+
19+
For a list of categories of sink connector configuration settings, see the
20+
section on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
21+
22+
Settings
23+
--------
24+
25+
.. list-table::
26+
:header-rows: 1
27+
:widths: 25 75
28+
29+
* - Name
30+
- Description
31+
32+
* - | **topics**
33+
- | *Required*
34+
|
35+
| **Type:** list
36+
|
37+
| **Description:**
38+
| A list of Kafka topics that the sink connector watches.
39+
40+
.. note::
41+
42+
You can define either the ``topics`` or the ``topics.regex``
43+
setting, but not both.
44+
45+
| **Accepted Values**: A comma-separated list of valid Kafka topics
46+
47+
* - | **topics.regex**
48+
- | *Required*
49+
|
50+
| **Type:** string
51+
|
52+
| **Description:**
53+
| A regular expression that matches the Kafka topics that the sink
54+
connector watches.
55+
56+
.. example::
57+
58+
.. code-block:: properties
59+
60+
topics.regex=activity\\.\\w+\\.clicks$
61+
62+
This regex matches topic names such as "activity.landing.clicks"
63+
and "activity.support.clicks". It does not match the topic names
64+
"activity.landing.views" and "activity.clicks".
65+
66+
.. note::
67+
68+
You can define either the ``topics`` or the ``topics.regex``
69+
setting, but not both.
70+
71+
| **Accepted Values**: A valid regular expression pattern using ``java.util.regex.Pattern``.
72+

0 commit comments

Comments
 (0)