Skip to content

Commit d712d9c

Browse files
Chris Choschmalliso
Chris Cho
authored and
schmalliso
committed
DOCSP-18344 source error config (#142)
* DOCSP-18344: source connector configuration - error handling
1 parent eb18e7b commit d712d9c

File tree

3 files changed

+137
-2
lines changed

3 files changed

+137
-2
lines changed

‎source/source-connector/configuration-properties.txt

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,9 @@ See the following categories for a list of related configuration properties:
4141
* - :ref:`Change Stream Properties <source-configuration-change-stream>`
4242
- Specify your change stream pipelines and cursor settings.
4343

44+
* - :ref:`Error Handling and Resume Properties <source-configuration-error-handling>`
45+
- Specify how the connector handles errors and resumes reading after an
46+
interruption.
4447

4548
See the `Confluent Source Connector configuration documentation <https://docs.confluent.io/platform/current/installation/configuration/connect/source-connect-configs.html>`__
4649
for more information on these settings.
@@ -51,7 +54,7 @@ See the following categories for a list of related configuration properties:
5154
MongoDB Connection </source-connector/configuration-properties/mongodb-connection>
5255
Kafka Topic </source-connector/configuration-properties/kafka-topic>
5356
Change Stream </source-connector/configuration-properties/change-stream>
57+
Error Handling </source-connector/configuration-properties/error-handling>
5458

5559
..
5660
- Document/Data Format
57-
- Error Handling and Resume Options

‎source/source-connector/configuration-properties/error-handling.txt

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
.. _source-configuration-error-handling:
2+
3+
====================================
4+
Error Handling and Resume Properties
5+
====================================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
Overview
16+
--------
17+
18+
Use the following configuration settings to specify how the source connector
19+
behaves when it encounters errors and to specify settings related to resuming
20+
interrupted reads.
21+
22+
Settings
23+
--------
24+
25+
.. list-table::
26+
:header-rows: 1
27+
:widths: 45 55
28+
29+
* - Name
30+
- Description
31+
32+
* - | **mongo.errors.tolerance**
33+
- | **Type:** string
34+
|
35+
| **Description:**
36+
| Whether to continue processing messages when the connector encounters
37+
an error.
38+
| Set this to ``"none"`` if you want the connector to stop
39+
processing messages and report the issue if it encounters an
40+
error.
41+
| Set this to ``"all"`` if you want the connector to continue
42+
processing messages and ignore any errors it encounters.
43+
44+
.. important::
45+
46+
This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/source-connect-configs.html#sourceconnectorconfigs_errors.tolerance>`__
47+
Connect Framework property.
48+
49+
|
50+
| **Default:** ``"none"``
51+
| **Accepted Values**: ``"none"`` or ``"all"``
52+
53+
* - | **mongo.errors.log.enable**
54+
- | **Type:** boolean
55+
|
56+
| **Description:**
57+
| Whether the connector should report errors in the log file.
58+
| Set this to ``true`` to log all errors the connector encounters.
59+
| Set this to ``false`` to log errors that are not tolerated by the
60+
connector. You can specify which errors the connector should
61+
tolerate using the ``errors.tolerance`` or ``mongo.errors.tolerance``
62+
setting.
63+
64+
.. important::
65+
66+
This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/source-connect-configs.html#sourceconnectorconfigs_errors.log.enable>`__
67+
Connect Framework property.
68+
69+
|
70+
| **Default:** ``false``
71+
| **Accepted Values**: ``true`` or ``false``
72+
73+
* - | **mongo.errors.deadletterqueue.topic.name**
74+
- | **Type:** string
75+
|
76+
| **Description:**
77+
| The name of topic to use as the dead letter queue.
78+
| If you specify a value, the connector writes invalid messages to the
79+
dead letter queue topic as :manual:`extended JSON strings </reference/mongodb-extended-json/>`.
80+
| If you leave this setting blank, the connector does not write
81+
invalid messages to any topic.
82+
83+
.. important::
84+
85+
You must set ``errors.tolerance`` or ``mongo.errors.tolerance``
86+
setting to ``"all"`` to enable this property.
87+
88+
| **Default:** ``""``
89+
| **Accepted Values**: A valid Kafka topic name
90+
91+
* - | **offset.partition.name**
92+
- | **Type:** string
93+
|
94+
| **Description:**
95+
| The custom offset partition name to use. You can use this option
96+
to instruct the connector to start a new change stream when an
97+
existing offset contains an invalid resume token. If you leave
98+
this setting blank, the connector uses the default partition name
99+
based on the connection details.
100+
101+
| **Default:** ""
102+
| **Accepted Values**: A valid partition name
103+
104+
* - | **heartbeat.interval.ms**
105+
- | **Type:** long
106+
|
107+
| **Description:**
108+
| The number of milliseconds the connector waits between sending
109+
heartbeat messages. The connector sends heartbeat messages when
110+
source records are not published in the specified interval.
111+
| Heartbeat messages contain a ``postBatchResumeToken`` data field.
112+
The value of this field contains the MongoDB server oplog entry that
113+
the connector last read from the change stream.
114+
| This mechanism improves resumability of the connector for low volume
115+
namespaces. See the (TODO: link to Troubleshooting > Invalid Resume
116+
Token page) for more information on this feature.
117+
118+
| Set this to ``0`` to disable heartbeat messages.
119+
| **Default**: ``0``
120+
| **Accepted Values**: An integer
121+
122+
* - | **heartbeat.topic.name**
123+
- | **Type:** string
124+
|
125+
| **Description:**
126+
| The name of the topic on which the connector should publish
127+
heartbeat messages. You must provide a positive value in the
128+
``heartbeat.interval.ms`` setting to enable this feature.
129+
|
130+
| **Default**: ``__mongodb_heartbeats``
131+
| **Accepted Values**: A valid Kafka topic name
132+

‎source/source-connector/configuration-properties/mongodb-connection.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ MongoDB Source Connection Properties
1515
Overview
1616
--------
1717

18-
Use the following configuration settings to specify how your sink
18+
Use the following configuration settings to specify how your source
1919
connector establishes a connection and communicates with your MongoDB cluster.
2020

2121
.. include:: /includes/source-config-link.rst

0 commit comments

Comments
 (0)