(DOCSP-18212) Avro Schema (#133)

Co-authored-by: corryroot <[email protected]>

Author: 2 people

source/includes/avro-customers.avro

@@ -1,27 +1,24 @@
 {
-   "type":"record",
-   "name":"Customer",
-   "fields":[
-      {
-         "name":"name",
-         "type":"string"
-      },
-      {
-         "name":"visits",
-         "type":{
-            "type":"array",
-            "items":{
-               "type":"long",
-               "logicalType":"timestamp-millis"
-            }
-         }
-      },
-      {
-         "name":"total_purchased",
-         "type":{
-            "type":"map",
-            "values":"int"
-         }
+  "type": "record",
+  "name": "Customer",
+  "fields": [{
+      "name": "name",
+      "type": "string"
+    },{
+      "name": "visits",
+      "type": {
+        "type": "array",
+        "items": {
+          "type": "long",
+          "logicalType": "timestamp-millis"
+        }
       }
-   ]
+    },{
+      "name": "total_purchased",
+      "type": {
+        "type": "map",
+        "values": "int"
+      }
+    }
+  ]
 }

source/includes/properties-files/avro-schema/avro-source.properties

@@ -0,0 +1,10 @@
+connector.class="com.mongodb.kafka.connect.MongoSourceConnector",
+connection.uri=<your mongodb uri>
+database=<your database name>
+collection="customers"
+publish.full.document.only=true
+output.format.value="schema"
+output.schema.value="{\"type\": \"record\", \"name\": \"Customer\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}, {\"name\": \"visits\", \"type\": {\"type\": \"array\", \"items\": {\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}}}, {\"name\": \"total_purchased\", \"type\": {\"type\": \"map\", \"values\": \"int\"}}]}"
+key.converter="org.apache.kafka.connect.storage.StringConverter"
+value.converter="io.confluent.connect.avro.AvroConverter"
+value.converter.schema.registry.url=<your schema registry uri>

source/introduction/data-formats.txt

@@ -6,4 +6,4 @@ Data Formats
     :titlesonly:
     :maxdepth: 1
 
-
+    Avro Schema </introduction/data-formats/avro-schema>

source/introduction/data-formats/avro-schema.txt

@@ -0,0 +1,122 @@
+===========
+Avro Schema
+===========
+
+.. default-domain:: mongodb
+
+.. contents:: On this page
+   :local:
+   :backlinks: none
+   :depth: 2
+   :class: singlecol
+
+Overview
+--------
+    
+In this guide, you can learn about Avro schema and how to use it in the
+{+mkc+}. 
+
+Avro Schema
+-----------
+
+<TODO: Group this with Raw JSON and JSON Data formats pages>
+
+Avro schema is a JSON-based schema definition syntax. Avro schema supports the
+specification of the following groups of data types:
+
+- `Primitive Types <https://avro.apache.org/docs/current/spec.html#schema_primitive>`__
+- `Complex Types <https://avro.apache.org/docs/current/spec.html#schema_complex>`__
+- `Logical Types <https://avro.apache.org/docs/current/spec.html#Logical+Types>`__
+
+.. important:: Sink Connectors and Logical Types
+
+   {+mkc+} sink connectors support all Avro schema primitive and complex types, however sink
+   connectors support only the following logical types:   
+
+   - ``decimal``
+   - ``date``
+   - ``time-millis``
+   - ``time-micros``
+   - ``timestamp-millis``
+   - ``timestamp-micros``
+
+For more information on Apache Avro, the open-source project that specifies Avro
+schema, see the 
+`Apache Avro Documentation <https://avro.apache.org/docs/current/index.html>`__.
+
+For a list of all Avro schema types, see the
+`Apache Avro specification <https://avro.apache.org/docs/current/spec.html>`__.
+
+Construct and Apply a Schema
+----------------------------
+
+<TODO: Move this Content to somewhere in the Source Connector section>
+
+In this section, you can learn how to perform the following actions with Avro
+schema and the {+mkc+}:
+
+- :ref:`Construct a schema for a MongoDB collection <avro-schema-construct-schema>`
+- :ref:`Apply the schema in a source connector <avro-schema-apply-schema>`
+
+The example in this section references the fictional ``customers`` collection
+that contains documents with the following structure:
+
+.. literalinclude:: /includes/avro-customers.json
+   :language: json
+
+.. _avro-schema-construct-schema:
+
+Construct a Schema for your Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following table shows how to convert the BSON types of documents in the ``customers``
+collection into Avro schema types:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 33 33 34
+
+   * - Field Name
+     - BSON Type
+     - Avro Schema Type
+
+   * - ``name`` 
+     - ``String`` type
+     - `string <https://avro.apache.org/docs/current/spec.html#schema_primitive>`__ primitive type
+
+   * - ``visits``
+     - ``Array`` type holding ``Date`` type values
+     -  `array <https://avro.apache.org/docs/current/spec.html#Arrays>`__
+        complex type holding
+        `timestamp-millis
+        <https://avro.apache.org/docs/current/spec.html#Timestamp+%28millisecond+precision%29>`__
+        logical type values
+
+   * - ``total_purchased``
+     - ``Object`` type holding ``Int32`` type values
+     - `map <https://avro.apache.org/docs/current/spec.html#Maps>`__ complex
+       type with `int <https://avro.apache.org/docs/current/spec.html#schema_primitive>`__ 
+       primitive type values.
+
+The complete schema for the ``customers`` collection looks like this:
+
+.. literalinclude:: /includes/avro-customers.avro
+   :language: json
+
+.. _avro-schema-apply-schema:
+
+Apply the Schema in your Source Connector
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following properties file configures your {+mkc+} source connector to apply
+your schema to incoming documents:
+
+.. literalinclude:: /includes/properties-files/avro-schema/avro-source.properties
+   :language: java
+
+For more information on applying schemas in the {+mkc+}, see our guide on
+applying schemas.
+<TODO: Link to Schema Guide>
+
+For more information on the Avro converter, see our guide on converters.
+<TODO: Link to Avro Converter>