Skip to content

Commit f8ac40f

Browse files
Chris Choschmalliso
Chris Cho
authored and
schmalliso
committed
DOCSP-18341: source change stream (#139)
* DOCSP-18341: Source Configuration Change Stream
1 parent af245ca commit f8ac40f

File tree

5 files changed

+195
-4
lines changed

5 files changed

+195
-4
lines changed

‎snooty.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@ name = "kafka-connector"
22
title = "MongoDB Kafka Connector"
33
intersphinx = ["https://docs.mongodb.com/manual/objects.inv"]
44

5-
toc_landing_pages = ["/sink-connector", "/sink-connector/configuration-properties"]
5+
toc_landing_pages = ["/sink-connector", "/sink-connector/configuration-properties", "/source-connector/configuration-properties"]

‎source/includes/source-config-link.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
For a list of source connector configuration settings organized by category, see
2+
the guide on :doc:`Source Connector Configuration Properties </source-connector/configuration-properties>`.

‎source/sink-connector/configuration-properties/topic-override.txt

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,5 @@ the following for data consumed from ``topicA``:
7878
``BlockList`` projection type.
7979

8080
For an example of how to configure the Block List Projector, see the
81-
:ref:`Sink Post-Processors guide
82-
<https://docs.mongodb.com/kafka-connector/current/kafka-sink-postprocessors/#block-list---allow-list-projector>`.
81+
(TODO: link to Sink Post-Processors Block List Projector section).
8382

‎source/source-connector/configuration-properties.txt

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,48 @@
22
Source Connector Configuration Properties
33
=========================================
44

5-
asdf
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
TODO
17+
18+
For an example source connector configuration file, see
19+
`MongoSourceConnector.properties <https://github.com/mongodb/mongo-kafka/blob/master/config/MongoSourceConnector.properties>`__.
20+
21+
Settings
22+
--------
23+
24+
See the following categories for a list of related configuration properties:
25+
26+
.. list-table::
27+
:header-rows: 1
28+
:widths: 35 65
29+
30+
* - Category
31+
- Description
32+
33+
* - :doc:`Change Stream Properties </source-connector/configuration-properties/change-stream>`
34+
- Specify your change stream pipelines and cursor settings.
35+
36+
See the `Confluent Source Connector configuration documentation <https://docs.confluent.io/platform/current/installation/configuration/connect/source-connect-configs.html>`__
37+
for more information on these settings.
38+
39+
.. toctree::
40+
:caption: Sink Connector Configuration Properties
41+
42+
Change Stream Properties </source-connector/configuration-properties/change-stream>
43+
44+
..
45+
- Connection
46+
- Document/Data Format
47+
- Kafka Topic
48+
- Error Handling and Resume Options
49+

‎source/source-connector/configuration-properties/change-stream.txt

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
========================
2+
Change Stream Properties
3+
========================
4+
5+
.. default-domain:: mongodb
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
Overview
14+
--------
15+
16+
Use the following configuration settings to specify aggregation pipelines
17+
for change streams and read preferences for change stream cursors.
18+
19+
.. include:: /includes/source-config-link.rst
20+
21+
Settings
22+
--------
23+
24+
.. list-table::
25+
:header-rows: 1
26+
:widths: 35 65
27+
28+
* - Name
29+
- Description
30+
31+
* - | **pipeline**
32+
- | **Type:** string
33+
|
34+
| **Description:**
35+
| An array of aggregation pipelines to run in your change stream.
36+
37+
.. example::
38+
39+
.. code-block:: none
40+
41+
[{"$match": {"operationType": "insert"}}, {"$addFields": {"Kafka": "Rules!"}}]
42+
43+
.. tip:: Additional Examples
44+
45+
- (TODO: link to Custom pipeline example)
46+
- (TODO: link to Multiple source example)
47+
48+
| **Default**: ``[]``
49+
| **Accepted Values**: Valid aggregation pipeline stage
50+
51+
* - | **change.stream.full.document**
52+
- | **Type:** string
53+
|
54+
| **Description:**
55+
| Determines what values your change stream returns on update
56+
operations.
57+
| The ``default`` setting returns the differences between the original
58+
document and the updated document.
59+
| The ``updateLookup`` setting returns the differences between the
60+
original document and updated document as well as a copy of the
61+
entire updated document at a *point in time* after the update.
62+
63+
.. tip::
64+
65+
For more information on how this change stream option works, see
66+
the MongoDB server manual guide on :manual:`Lookup Full Document for
67+
Update Operations </changeStreams/#lookup-full-document-for-update-operations>`.
68+
|
69+
| **Default**: ``""``
70+
| **Accepted Values**: ``""`` or ``"default"`` or ``"updateLookup"``
71+
72+
* - | **publish.full.document.only**
73+
- | **Type:** boolean
74+
|
75+
| **Description:**
76+
| Whether to publish the changed document instead of the full change
77+
stream document. When set to ``true``, the connector
78+
automatically sets the ``change.stream.full.document`` property to
79+
``updateLookup`` to receive the updated documents.
80+
|
81+
| **Default**: ``false``
82+
| **Accepted Values**: ``true`` or ``false``
83+
84+
* - | **collation**
85+
- | **Type:** string
86+
|
87+
| **Description:**
88+
| A JSON :manual:`collation document </reference/collation/#collation-document>`
89+
that specifies language-specific ordering rules that MongoDB
90+
applies to the documents returned by the change stream.
91+
|
92+
| **Default**: ``""``
93+
| **Accepted Values**: A valid collation JSON document
94+
95+
* - | **copy.existing.pipeline**
96+
- | **Type:** string
97+
|
98+
| **Description:**
99+
| An array of :manual:`pipeline operations </core/aggregation-pipeline/>`
100+
to run when copying existing data. You can use this setting to
101+
improve the use of indexes by the copying manager to make the copying
102+
process more efficient.
103+
104+
.. example::
105+
106+
The following example shows how you can use the :manual:`$match </reference/operator/aggregation/match/>`
107+
aggregation operator to copy only documents that contain a
108+
``closed`` field with a value of ``false``.
109+
110+
.. code-block:: none
111+
112+
copy.existing.pipeline=[ { "$match": { "closed": "false" } } ]
113+
114+
| **Default**: ``[]``
115+
| **Accepted Values**: Valid aggregation pipeline stages
116+
117+
* - | **batch.size**
118+
- | **Type:** int
119+
|
120+
| **Description:**
121+
| The change stream cursor batch size.
122+
|
123+
| **Default**: ``0``
124+
| **Accepted Values**: An integer
125+
126+
* - | **poll.await.time.ms**
127+
- | **Type:** long
128+
|
129+
| **Description:**
130+
| The amount of time in milliseconds to wait before checking the change
131+
stream cursor for new results.
132+
|
133+
| **Default**: ``5000``
134+
| **Accepted Values**: An integer
135+
136+
* - | **poll.max.batch.size**
137+
- | **Type:** int
138+
|
139+
| **Description:**
140+
| Maximum number of documents to read in a single batch when polling
141+
a change stream cursor for new data. You can use this setting to
142+
limit the amount of data buffered internally in the connector.
143+
|
144+
| **Default**: ``1000``
145+
| **Accepted Values**: An integer
146+

0 commit comments

Comments
 (0)