Skip to content

Commit 3951008

Browse files
Chris Choschmalliso
Chris Cho
authored and
schmalliso
committed
Revert "DOCSP-18679: ia-v2 v1.5 branch DOT NOT MERGE (#198)"
This reverts commit 7d55b1e0ece9db33d00ae2c870b2761a266bde07.
1 parent 9cc41b4 commit 3951008

File tree

14 files changed

+645
-20
lines changed

14 files changed

+645
-20
lines changed

‎config/redirects

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,3 @@ raw: kafka-connector/ -> ${base}/current/
1818
[*-master]: kafka-connector/${version}/kafka-sink-postprocessors/ -> ${base}/${version}/fundamentals/post-processors/
1919
[*-master]: kafka-connector/${version}/kafka-sink-time-series-example/ -> ${base}/${version}/tutorials/migrate-time-series/
2020
[*-master]: kafka-connector/${version}/kafka-connect-migration/ -> ${base}/${version}/
21-
22-
[*-v1.5]: kafka-connector/${version}/tutorials/migrate-time-series/ -> ${base}/${version}/
23-
[*-v1.5]: kafka-connector/${version}/sink-connector/configuration-properties/time-series/ -> ${base}/${version}/
24-

‎snooty.toml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,9 @@ kc = "Kafka Connect"
1111
avro = "Apache Avro"
1212
avro-short = "Avro"
1313
kafka-connect = "Confluent Kafka Connect"
14-
connector_version="1.5"
15-
connector_driver_version="4.2"
14+
connector_version="1.6"
15+
connector_driver_version="4.3"
1616
sink-connector="MongoDB Kafka sink connector"
1717
source-connector="MongoDB Kafka source connector"
18+
connector_driver_url_base="https://docs.mongodb.com/drivers/java/sync/v{+connector_driver_version+}/"
1819
pipeline-size = "2.4 GB"

‎source/introduction/connect.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ to interact with MongoDB.
6161
{+connector_driver_version+} of the MongoDB Java driver.
6262

6363
To learn what connection URI options are available in the {+mkc+}, see
64-
`the MongoDB Java driver Connection tutorial <https://mongodb.github.io/mongo-java-driver/4.2/driver/tutorials/connect-to-mongodb/>`__.
64+
`the MongoDB Java driver Connection guide <{+connector_driver_url_base+}fundamentals/connection/#connection-options>`__.
6565

6666
Authentication
6767
~~~~~~~~~~~~~~
@@ -84,7 +84,7 @@ The following is an example of a connection URI that authenticates with MongoDB
8484
`the Externalize Secrets guide from Confluent <https://docs.confluent.io/platform/current/connect/security.html#externalizing-secrets>`__.
8585

8686
To learn what authentication mechanisms are available, see
87-
`the MongoDB Java driver Authentication tutorial <https://mongodb.github.io/mongo-java-driver/4.2/driver/tutorials/authentication/>`__.
87+
`the MongoDB Java driver Authentication Mechanisms guide <{+connector_driver_url_base+}fundamentals/auth/#mechanisms>`__.
8888

8989
To learn more about authentication in the {+mkc+}, see the
9090
:doc:`Security and Authentication guide </security-and-authentication>`.

‎source/sink-connector/configuration-properties.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,10 @@ See the following categories for a list of related configuration properties:
6767
* - :doc:`Change Data Capture Properties </sink-connector/configuration-properties/cdc>`
6868
- Specify how the connector captures CDC events from a Kafka topic.
6969

70+
* - :doc:`Time Series Properties </sink-connector/configuration-properties/time-series>`
71+
- Configure the connector to sink data to a MongoDB time series
72+
collection.
73+
7074
See the `Confluent Sink Connector documentation <https://docs.confluent.io/current/installation/configuration/connect/sink-connect-configs.html>`__
7175
for more information on these settings.
7276

@@ -83,4 +87,5 @@ for more information on these settings.
8387
Write Model Strategy <sink-connector/configuration-properties/write-strategies>
8488
Topic Override <sink-connector/configuration-properties/topic-override>
8589
Change Data Capture <sink-connector/configuration-properties/cdc>
90+
Time Series <sink-connector/configuration-properties/time-series>
8691

‎source/sink-connector/configuration-properties/error-handling.txt

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,45 @@ Settings
3030
* - Name
3131
- Description
3232

33+
* - | **mongo.errors.tolerance**
34+
- | **Type:** string
35+
|
36+
| **Description:**
37+
| Whether to continue processing messages if the connector encounters
38+
an error.
39+
| When set to ``none``, the connector reports any error and
40+
blocks further processing of the rest of the messages.
41+
| When set to ``all``, the connector ignores any problematic messages.
42+
43+
.. note::
44+
45+
This property overrides the `errors.tolerance <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-tolerance>`__
46+
property of the Connect Framework.
47+
48+
| **Default:** ``"none"``
49+
| **Accepted Values**: ``"none"`` or ``"all"``
50+
51+
* - | **mongo.errors.log.enable**
52+
- | **Type:** boolean
53+
|
54+
| **Description:**
55+
| Whether the connector should write details of errors including
56+
failed operations to the log file. The connector classifies
57+
errors as "tolerated" or "not tolerated" using the
58+
``errors.tolerance`` or ``mongo.errors.tolerance`` settings.
59+
60+
| When set to ``true``, the connector logs both "tolerated" and
61+
"not tolerated" errors.
62+
| When set to ``false``, the connector logs "not tolertaed" errors.
63+
64+
.. note::
65+
66+
This property overrides the `errors.log.enable <https://docs.confluent.io/platform/current/installation/configuration/connect/sink-connect-configs.html#errors-log-enable>`__
67+
property of the Connect Framework.
68+
69+
| **Default:** ``false``
70+
| **Accepted Values**: ``true`` or ``false``
71+
3372
* - | **errors.log.include.messages**
3473
- | **Type:** boolean
3574
|
@@ -97,8 +136,8 @@ the dead letter queue messages should include context headers.
97136

98137
.. code-block:: properties
99138

100-
errors.tolerance=all
101-
errors.log.enable=true
139+
mongo.errors.tolerance=all
140+
mongo.errors.log.enable=true
102141
errors.log.include.messages=true
103142
errors.deadletterqueue.topic.name=example.deadletterqueue
104143
errors.deadletterqueue.context.headers.enable=true

‎source/sink-connector/configuration-properties/mongodb-connection.txt

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,45 @@ Settings
6161
| **Default**: ``5000``
6262
| **Accepted Values**: An integer
6363

64+
* - **server.api.version**
65+
- | **Type:** string
66+
|
67+
| **Description:**
68+
| The Versioned API version you want to use with your MongoDB
69+
server. For more information on the Versioned API and versions of
70+
the server that support it, see the :manual:`Versioned API </reference/versioned-api/>`
71+
MongoDB server manual guide.
72+
|
73+
| **Default**: ``""``
74+
| **Accepted Values**: An empty string or a valid Versioned API version.
75+
76+
* - **server.api.deprecationErrors**
77+
- | **Type:** boolean
78+
|
79+
| **Description:**
80+
| When set to ``true``, if the connector calls a command on your
81+
MongoDB instance that's deprecated in the declared Versioned API
82+
version, it raises an exception.
83+
| You can set the API version with the ``server.api.version``
84+
configuration option. For more information on the Versioned API, see
85+
the MongoDB manual entry on the
86+
:manual:`Versioned API </reference/versioned-api/>`.
87+
|
88+
| **Default**: ``false``
89+
| **Accepted Values**: ``true`` or ``false``
90+
91+
* - **server.api.strict**
92+
- | **Type:** boolean
93+
|
94+
| **Description:**
95+
| When set to ``true``, if the connector calls a command on your
96+
MongoDB instance that's not covered in the declared Versioned API
97+
version, it raises an exception.
98+
| You can set the API version with the ``server.api.version``
99+
configuration option. For more information on the Versioned API, see
100+
the MongoDB manual entry on the
101+
:manual:`Versioned API </reference/versioned-api/>`.
102+
|
103+
| **Default**: ``false``
104+
| **Accepted Values**: ``true`` or ``false``
105+

‎source/sink-connector/configuration-properties/time-series.txt

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
.. _sink-configuration-time-series:
2+
3+
============================
4+
Kafka Time Series Properties
5+
============================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
Overview
16+
--------
17+
18+
Use the following configuration settings to specify how the connector
19+
should sink data to a MongoDB time series collection.
20+
21+
.. include:: /includes/sink-config-link.rst
22+
23+
Settings
24+
--------
25+
26+
.. list-table::
27+
:header-rows: 1
28+
:widths: 45 55
29+
30+
* - Name
31+
- Description
32+
33+
* - | **timeseries.timefield**
34+
- | **Type:** string
35+
|
36+
| **Description:**
37+
| The name of the top-level field in the source data that contains time
38+
information that you want to associate with the new document in the
39+
time series collection.
40+
|
41+
| **Default**: ``""``
42+
| **Accepted Values**: An empty string or the name of a field
43+
that contains a BSON ``DateTime`` value
44+
45+
* - | **timeseries.timefield.auto.convert.date.format**
46+
- | **Type:** string
47+
|
48+
| **Description:**
49+
| The date format pattern the connector should use to convert the
50+
source data contained in the field specified by the
51+
``timeseries.timefield`` setting.
52+
The connector passes the date format pattern to the Java
53+
`DateTimeFormatter.ofPattern(pattern, locale) <https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ofPattern-java.lang.String-java.util.Locale->`__
54+
method to perform date and time conversions on the time field.
55+
| If the date value from the source data only contains date information,
56+
the connector sets the time information to the start of the specified
57+
day. If the date value does not contain the timezone offset, the
58+
connector sets the offset to UTC.
59+
|
60+
| **Default**:
61+
62+
.. code-block:: none
63+
64+
yyyy-MM-dd[['T'][ ]][HH:mm:ss[[.][SSSSSS][SSS]][ ]VV[ ]'['VV']'][HH:mm:ss[[.][SSSSSS][SSS]][ ]X][HH:mm:ss[[.][SSSSSS][SSS]]]
65+
66+
| **Accepted Values**: A valid ``DateTimeFormatter`` format
67+
68+
* - | **timeseries.timefield.auto.convert**
69+
- | **Type:** boolean
70+
|
71+
| **Description:**
72+
| Whether to convert the data in the field into the BSON ``Date``
73+
format.
74+
| When set to ``true``, the connector uses the milliseconds
75+
after epoch and discards fractional parts if the value is
76+
a number. If the value is a string, the connector uses the
77+
setting in the following configuration to parse the date:
78+
79+
.. code-block:: none
80+
81+
timeseries.timefield.auto.convert.date.format
82+
83+
| If the connector fails to convert the value, it sends the
84+
original value to the time series collection.
85+
|
86+
| **Default**: ``false``
87+
| **Accepted Values**: ``true`` or ``false``
88+
89+
* - | **timeseries.timefield.auto.convert.locale.language.tag**
90+
- | **Type:** string
91+
|
92+
| **Description:**
93+
| Which ``DateTimeFormatter`` locale language tag to use with the date
94+
format pattern (e.g. ``"en-US"``). For more information on
95+
locales, see the Java SE documentation of `Locale <https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html>`__.
96+
|
97+
| **Default**: ``ROOT``
98+
| **Accepted Values**: A valid ``Locale`` language tag format
99+
100+
* - | **timeseries.metafield**
101+
- | **Type:** string
102+
|
103+
| **Description:**
104+
| Which top-level field to read from the source data to describe
105+
a group of related time series documents.
106+
107+
.. important::
108+
109+
This field must not be the ``_id`` field nor the field you specified
110+
in the ``timeseries.timefield`` setting.
111+
112+
| **Default**: ``""``
113+
| **Accepted Values**: An empty string or the name of a field
114+
that contains any BSON type except ``BsonArray``.
115+
116+
* - | **timeseries.expire.after.seconds**
117+
- | **Type:** int
118+
|
119+
| **Description:**
120+
| The number of seconds MongoDB should wait before automatically
121+
removing the time series collection data. The connector disables
122+
timed expiry when the setting value is less than ``1``.
123+
For more information on this collection setting, see the MongoDB
124+
Server Manual page on :manual:`Automatic Removal for Time Series Collections </core/timeseries/timeseries-automatic-removal/>`.
125+
|
126+
| **Default**: ``0``
127+
| **Accepted Values**: An integer
128+
129+
130+
* - | **timeseries.granularity**
131+
- | **Type:** string
132+
|
133+
| **Description:**
134+
| The expected interval between subsequent measurements of your
135+
source data. For more information on this setting, see the
136+
MongoDB Server Manual page on :manual:`Granularity for Time
137+
Series Data </core/timeseries/timeseries-granularity/>`.
138+
|
139+
| *Optional*
140+
| **Default**: ``""``
141+
| **Accepted Values**: ``""``, ``"seconds"``, ``"minutes"``, ``"hours"``
142+
143+
For an example on how to convert an existing collection to a time series
144+
collection, see the tutorial on how to :ref:`<tutorial-migrate-time-series>`.
145+

‎source/sink-connector/fundamentals/error-handling-strategies.txt

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ when it encounters an error. This is a good option for you if any error in
5252
your sink connector indicates a serious problem.
5353

5454
When your sink connector crashes, you must do one of the
55-
following actions and then restart your connector to resume processing messages:
55+
following actions and then restart your connector to resume processing messages:
5656

5757
- Allow your sink connector to temporarily :ref:`tolerate errors <kafka-sink-tolerate-errors>`
5858
- Update your sink connector's configuration to allow it to process the message
@@ -73,7 +73,7 @@ Tolerate All Errors
7373

7474
You can configure your sink connector to tolerate all errors and never stop
7575
processing messages. This is a good option for getting your sink connector up and
76-
running quickly, but you run the risk of missing problems in your connector
76+
running quickly, but you run the risk of missing problems in your connector
7777
as you do not receive any feedback if something goes wrong.
7878

7979
You can have your sink connector tolerate all errors by specifying the following
@@ -88,7 +88,7 @@ option:
8888
Write Errors and Errant Messages to a Topic
8989
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9090

91-
You can configure your sink connector to write errors and errant messages to a
91+
You can configure your sink connector to write errors and errant messages to a
9292
topic, called a **dead letter queue**, for you to inspect or process further.
9393
A dead letter queue is a location in message queueing
9494
systems such as {+ak+} where the system routes errant messages instead of
@@ -111,12 +111,24 @@ errant message, use the following option:
111111

112112
errors.deadletterqueue.context.headers.enable=true
113113

114-
.. warning:: Bulk Write Errors
114+
.. important:: Bulk Write Errors
115115

116-
You cannot configure the connector to write bulk write errors to the
117-
dead letter queue.
116+
If an error occurs while performing a bulk write to MongoDB, you must specify
117+
the following to log any information about the error to a dead letter queue:
118118

119-
For more information, see Confluent's guide on
119+
.. code-block:: properties
120+
121+
errors.tolerance=all
122+
mongo.errors.tolerance=none
123+
errors.deadletterqueue.topic.name=<name of topic to use as dead letter queue>
124+
125+
We will fix this special case soon. To track the status of this issue,
126+
see :issue:`this JIRA ticket <KAFKA-223>`.
127+
128+
For more information on the ``mongo.errors.tolerance`` option, see our
129+
section on :ref:`connector level options <kakfa-sink-connector-level>`.
130+
131+
For more information, see Confluent's guide on
120132
`Dead Letter Queues <https://docs.confluent.io/cloud/current/connectors/dead-letter-queue.html#dead-letter-queue>`__.
121133

122134
To view another dead letter queue configuration example, see :ref:`<sink-dead-letter-queue-configuration-example>`.
@@ -157,6 +169,34 @@ topic and offset, use the following option:
157169

158170
errors.log.include.messages=true
159171

160-
For more information, see Confluent's guide on
172+
For more information, see Confluent's guide on
161173
`logging with Kafka Connect <https://docs.confluent.io/platform/current/connect/logging.html>`__.
162174

175+
.. _kakfa-sink-connector-level:
176+
177+
Handle Errors at the Connector Level
178+
------------------------------------
179+
180+
The {+mkc+} provides options that allow you to configure error
181+
handling at the connector level. The options are as follows:
182+
183+
.. list-table::
184+
:header-rows: 1
185+
:widths: 50 50
186+
187+
* - Kafka Connect Option
188+
- {+mkc+} Option
189+
190+
* - ``errors.tolerance``
191+
- ``mongo.errors.tolerance``
192+
193+
* - ``errors.log.enable``
194+
- ``mongo.errors.log.enable``
195+
196+
You want to use these options if you want your connector to respond differently
197+
to errors related to MongoDB than to errors related to the {+kc+} framework.
198+
199+
For more information, see the following resources:
200+
201+
- :ref:`<sink-configuration-error-handling>`
202+
- :issue:`New Names for Error Tolerance Options JIRA Ticket <KAFKA-215>`

0 commit comments

Comments
 (0)