(DOSCP-15813) Change Streams (#138)

Co-authored-by: David Cuthbert <[email protected]>

Author: 2 people

snooty.toml

@@ -1,5 +1,7 @@
 name = "kafka-connector"
 title = "MongoDB Kafka Connector"
 intersphinx = ["https://docs.mongodb.com/manual/objects.inv"]
-
 toc_landing_pages = ["/sink-connector", "/sink-connector/configuration-properties", "/source-connector/configuration-properties"]
+
+[constants]
+mkc = "MongoDB Kafka Connector"

source/sink-connector/configuration-properties/topic-override.txt

@@ -77,5 +77,6 @@ the following for data consumed from ``topicA``:
 - Omit fields ``k2`` and ``k4`` from the value projection using the
   ``BlockList`` projection type.
 
+
 For an example of how to configure the Block List Projector, see the
 (TODO: Sink Post-Processors guide).

source/source-connector/fundamentals/change-streams.txt

@@ -2,4 +2,92 @@
 Change Streams
 ==============
 
-TODO
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 2
+   :class: singlecol
+
+Overview
+--------
+
+In this guide, you can learn about **change streams** and how they are
+used in a {+mkc+} source connector. 
+
+Change Streams
+--------------
+
+Change streams are a feature of MongoDB that allow you to receive real-time
+updates on data changes. Change streams return
+**change event documents**. A change event document is a document in your 
+**oplog** that contains idempotent instructions to recreate a change that
+occurred in your MongoDB deployment as well as the metadata related to that
+change.
+
+The oplog is a special collection in MongoDB that keeps track of all changes
+within a MongoDB replica set. Change streams help you use the change event data
+stored in the oplog without you having to learn details about how the oplog
+works.
+
+.. important:: You must have a replica set or a sharded cluster
+
+   Change streams are available for replica sets and sharded clusters. A
+   standalone MongoDB instance cannot produce a change stream.
+
+For more information on change streams, see the following MongoDB manual entries:
+
+- :manual:`Change streams </changeStreams/>`
+- :manual:`Oplog </core/replica-set-oplog/>`
+
+Change Event Structure
+~~~~~~~~~~~~~~~~~~~~~~
+
+You can find the complete structure of change event documents, including
+descriptions of all fields, 
+:ref:`in the MongoDB manual <change-stream-output>`. 
+
+.. note:: The Full Document Option
+
+   If you want Kafka Connect to receive just the document created or modified
+   from your change operation, use the ``publish.full.document.only=true``
+   option. For more information see our
+   guide on <TODO: Link to source connector properties>.
+
+Source Connectors
+-----------------
+
+A {+mkc+} source connector works by opening a single change stream with
+MongoDB and sending data from that change stream to Kafka Connect. Your source
+connector maintains its change stream for the duration of its runtime, and your
+connector closes its change stream when you stop it.
+
+.. note:: MongoDB Java driver
+
+   Your source connector uses the MongoDB Java driver to create a change
+   stream. For more information, see this guide on change streams in the MongoDB
+   java driver. <TODO: Link to Watch For Changes Usage Example>
+
+Resume Tokens
+~~~~~~~~~~~~~
+
+Your connector stores a **resume token** to keep track of what changes
+it has processed. A resume token is a piece of data that references
+the ``_id`` field of a change event document in your MongoDB oplog.
+Your connector only processes relevant change event documents written to the oplog after the
+document referenced by its resume token. 
+
+If your source connector does not have a resume token, such as when you start
+the connector for the first time, your connector processes relevant change
+events written to the oplog after it first connects to MongoDB.
+
+.. <TODO for Ross>: Confirm if this is after the source connector first connects
+   to MongoDB, or after it first starts. Pretty sure it is after it first
+   connects to MongoDB. I think this is getting in the weeds of how change streams are
+   implemented in the Java driver and I'm not sure if this is a valuable distinction.
+
+If your source connector's resume token does not correspond to any entry in your
+oplog, your connector has an invalid resume token. To learn how to recover from an
+invalid resume token, 
+:ref:`see our troubleshooting guide <kafka-troubleshoot-recover-invalid-resume-token>`.

source/troubleshooting/recover-from-invalid-resume-token.txt

@@ -10,6 +10,9 @@ Invalid Resume Token
    :depth: 2
    :class: singlecol
 
+
+.. _kafka-troubleshoot-recover-invalid-resume-token:
+
 Overview
 --------