Skip to content

Commit d1b9c2b

Browse files
Chris Choschmalliso
Chris Cho
authored and
schmalliso
committed
DOCSP-17831: sink connector MongoDB properties (#119)
* DOCSP-17831: sink connector MongoDB properties
1 parent 7de4bfe commit d1b9c2b

File tree

4 files changed

+313
-5
lines changed

4 files changed

+313
-5
lines changed

‎snooty.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ name = "kafka-connector"
22
title = "MongoDB Kafka Connector"
33
intersphinx = ["https://docs.mongodb.com/manual/objects.inv"]
44

5-
toc_landing_pages = ["/kafka-sink"]
5+
toc_landing_pages = ["/sink-connector", "/sink-connector/configuration-properties"]

‎source/sink-connector/configuration-properties.txt

Lines changed: 49 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,50 @@
1-
========================
2-
Configuration Properties
3-
========================
1+
.. _kafka-sink-configuration-properties:
2+
3+
=======================================
4+
Sink Connector Configuration Properties
5+
=======================================
6+
7+
.. default-domain:: mongodb
8+
9+
.. contents:: On this page
10+
:local:
11+
:backlinks: none
12+
:depth: 2
13+
:class: singlecol
14+
15+
Overview
16+
--------
17+
18+
TODO
19+
20+
.. _sink-settings:
21+
22+
Settings
23+
--------
24+
25+
See the following categories for a list of related configuration properties:
26+
27+
.. list-table::
28+
:header-rows: 1
29+
:widths: 35 65
30+
31+
* - Category
32+
- Description
33+
34+
* - :doc:`MongoDB Connection Properties </sink-connector/configuration-properties/mongodb-connection>`
35+
- How to connect to your MongoDB cluster.
36+
37+
* - :doc:`MongoDB Namespace Mapping Properties </sink-connector/configuration-properties/mongodb-namespace>`
38+
- Specify where to sink your data.
39+
40+
TODO: add additional configuration properties pages
41+
42+
See the `Confluent Sink Connector documentation <https://docs.confluent.io/current/installation/configuration/connect/sink-connect-configs.html>`__
43+
for more information on these settings.
44+
45+
.. toctree::
46+
:caption: Sink Connector Configuration Properties
47+
48+
MongoDB Connection </sink-connector/configuration-properties/mongodb-connection>
49+
MongoDB Namespace </sink-connector/configuration-properties/mongodb-namespace>
450

5-
asdf

‎source/sink-connector/configuration-properties/mongodb-connection.txt

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
===========================================
2+
MongoDB Connection Configuration Properties
3+
===========================================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify how your sink
17+
connector connects and communicates with your MongoDB cluster.
18+
19+
For a list of all sink connector configuration settings, see the
20+
guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
21+
22+
Settings
23+
--------
24+
25+
.. list-table::
26+
:header-rows: 1
27+
:widths: 30 70
28+
29+
* - Name
30+
- Description
31+
32+
* - **connection.uri**
33+
- | *Required*
34+
|
35+
| **Type:** string
36+
|
37+
| **Description:**
38+
| The :manual:`MongoDB connection URI string </reference/connection-string/#standard-connection-string-format>`
39+
to connect to your MongoDB instance or cluster.
40+
| For more information, see the (TODO: Connect your Kafka Connector
41+
to MongoDB page link) guide
42+
43+
.. include:: /includes/externalize-secrets.rst
44+
45+
| **Default**: ``mongodb://localhost:27017``
46+
| **Accepted Values**: A MongoDB connection URI string
47+
48+
* - | **max.num.retries**
49+
- | **Type:** int
50+
|
51+
| **Description:**
52+
| The number of retries to attempt when encountering write errors to MongoDB.
53+
|
54+
| **Default**: ``1``
55+
| **Accepted Values**: An integer
56+
57+
* - | **retries.defer.timeout**
58+
- | **Type:** int
59+
|
60+
| **Description:**
61+
| Amount of time (in milliseconds) to defer a retry attempt.
62+
|
63+
| **Default**: ``5000``
64+
| **Accepted Values**: An integer
65+
66+
* - **server.api.version**
67+
- | **Type:** string
68+
|
69+
| **Description:**
70+
| The Versioned API version you want to use with your MongoDB
71+
server. For more information on the Versioned API and versions of
72+
the server that support it, see the :manual:`Versioned API </reference/versioned-api/>`
73+
MongoDB server manual guide.
74+
|
75+
| **Default**: ``""``
76+
| **Accepted Values**: An empty string or a valid Versioned API version.
77+
78+
* - **server.api.deprecationErrors**
79+
- | **Type:** boolean
80+
|
81+
| **Description:**
82+
| When set to ``true``, if the connector calls a command on your
83+
MongoDB instance that's deprecated in the declared Versioned API
84+
version, it raises an exception.
85+
| You can set the API version with the ``server.api.version``
86+
configuration option. For more information on the Versioned API, see
87+
the MongoDB manual entry on the
88+
:manual:`Versioned API </reference/versioned-api/>`.
89+
|
90+
| **Default**: ``false``
91+
| **Accepted Values**: ``true`` or ``false``
92+
93+
* - **server.api.strict**
94+
- | **Type:** boolean
95+
|
96+
| **Description:**
97+
| When set to ``true``, if the connector calls a command on your
98+
MongoDB instance that's not covered in the declared Versioned API
99+
version, it raises an exception.
100+
| You can set the API version with the ``server.api.version``
101+
configuration option. For more information on the Versioned API, see
102+
the MongoDB manual entry on the
103+
:manual:`Versioned API </reference/versioned-api/>`.
104+
|
105+
| **Default**: ``false``
106+
| **Accepted Values**: ``true`` or ``false``
107+

‎source/sink-connector/configuration-properties/mongodb-namespace.txt

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
==================================================
2+
MongoDB Namespace Mapping Configuration Properties
3+
==================================================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify which MongoDB database
17+
and collection that your sink connector writes data to. You can use the
18+
default ``DefaultNamespaceMapper`` or specify a custom class.
19+
20+
For a list of categories of sink connector configuration settings, see the
21+
guide on :doc:`Sink Connector Configuration Properties </sink-connector/configuration-properties>`.
22+
23+
Settings
24+
--------
25+
26+
.. list-table::
27+
:header-rows: 1
28+
:widths: 25 75
29+
30+
* - Name
31+
- Description
32+
33+
* - | **namespace.mapper**
34+
- | **Type:** string
35+
|
36+
| **Description:**
37+
| The fully-qualified class name of the class that specifies which
38+
database or collection in which to sink the data. The default
39+
``DefaultNamespaceMapper`` uses values specified in the
40+
``database`` and ``collection`` properties.
41+
42+
.. seealso::
43+
44+
The connector includes an alternative class for specifying the
45+
database and collection called ``FieldPathNamespaceMapper``. See
46+
the :ref:`FieldPathNamespaceMapper settings <fieldpathnamespacemapper-settings>`
47+
for more information.
48+
49+
| **Default**:
50+
51+
.. code-block:: none
52+
53+
com.mongodb.kafka.connect.sink.namespace.mapping.DefaultNamespaceMapper
54+
| **Accepted Values**: A fully qualified Java class name of a class that implements the ``NamespaceMapper`` interface.
55+
56+
* - **database**
57+
- | *Required*
58+
|
59+
| **Type:** string
60+
|
61+
| **Description:**
62+
| The name of the MongoDB database to which the sink connector writes.
63+
|
64+
| **Accepted Values**: A MongoDB database name
65+
66+
* - **collection**
67+
- | *Required*
68+
|
69+
| **Type:** string
70+
|
71+
| **Description:**
72+
| The name of the MongoDB collection to which the sink connector
73+
writes. If your sink connector follows multiple topics, this
74+
is the default collection for any writes that are not otherwise
75+
specified.
76+
|
77+
| **Accepted Values**: A MongoDB collection name
78+
79+
.. _fieldpathnamespacemapper-settings:
80+
81+
FieldPathNamespaceMapper Settings
82+
---------------------------------
83+
84+
If you configure your sink connector to use the ``FieldPathNamespaceMapper``,
85+
you can specify which database and collection to sink a document based on the
86+
data's field values.
87+
88+
To enable this mapping behavior, set your sink connector ``namespace.mapper``
89+
configuration property to the fully-qualified class name as shown below:
90+
91+
.. code-block:: properties
92+
93+
namespace.mapper=com.mongodb.kafka.connect.sink.topic.mapping.FieldPathNamespaceMapper
94+
95+
The ``FieldPathNamespaceMapper`` requires you to specify the following
96+
settings:
97+
98+
- One or both mapping properties to a database and collection
99+
- One of the ``key`` or ``value`` mappings to a database
100+
- One of the ``key`` or ``value`` mappings to a collection
101+
102+
You can use the following settings to customize the behavior of the
103+
``FieldPathNamespaceMapper``:
104+
105+
.. list-table::
106+
:header-rows: 1
107+
:widths: 40 60
108+
109+
* - Name
110+
- Description
111+
112+
* - | **namespace.mapper.key.database.field**
113+
- | **Type:** string
114+
|
115+
| **Description:**
116+
| The name of the key document field that specifies the name of the
117+
database in which to write.
118+
119+
* - | **namespace.mapper.key.collection.field**
120+
- | **Type:** string
121+
|
122+
| **Description:**
123+
| The name of the key document field that specifies the name of the
124+
collection in which to write.
125+
126+
* - | **namespace.mapper.value.database.field**
127+
- | **Type:** string
128+
|
129+
| **Description:**
130+
| The name of the value document field that specifies the name of the
131+
database in which to write.
132+
133+
* - | **namespace.mapper.value.collection.field**
134+
- | **Type:** string
135+
|
136+
| **Description:**
137+
| The name of the value document field that specifies the name of the
138+
collection in which to write.
139+
140+
* - | **namespace.mapper.error.if.invalid**
141+
- | **Type:** boolean
142+
|
143+
| **Description:**
144+
| Whether to throw an exception when either the document is missing the
145+
mapped field or it has an invalid BSON type.
146+
| When set to ``true``, the connector does not process documents
147+
missing the mapped field or that contain an invalid BSON type.
148+
The connector may halt or skip processing depending on the related
149+
error-handling configuration settings.
150+
| When set to ``false``, if a document is missing the mapped field or
151+
if it has an invalid BSON type, the connector defaults to
152+
writing to the specified ``database`` and ``collection`` settings.
153+
|
154+
| **Default**: ``false``
155+
| **Accepted Values**: ``true`` or ``false``
156+

0 commit comments

Comments
 (0)