DOCSP-27245: improved schema inference for nested docs (#85)

* DOCSP-27245: improved schema inference for nested docs

* RW comment

* DB PR fixes 1

Author: rustagir

source/source-connector/fundamentals/specify-schema.txt

@@ -86,7 +86,7 @@ schema for your incoming documents, you must specify a key schema
 
 You can enable the default key schema with the following option:
 
-.. code-block:: java
+.. code-block:: properties
 
    output.format.key=schema
 
@@ -182,6 +182,23 @@ following options:
    output.format.value=schema
    output.schema.infer.value=true
 
+The source connector can infer schemas for incoming documents that
+contain nested documents stored in arrays. Starting in Version 1.9 of the
+{+connector+}, schema inference will gather the appropriate data type
+for fields instead of defaulting to a ``string`` type assignment if there are
+differences between nested documents described by the following cases:
+
+- A field is present in one document but missing in another.
+- A field is present in one document but ``null`` in another.
+- A field is an array with elements of any type in one document but
+  has additional elements or elements of other data types in another.
+- A field is an array with elements of any type in one document but an
+  empty array in another.
+
+If field types conflict between nested documents, the connector
+pushes the conflict down to the schema for the field and defaults to a
+``string`` type assignment.
+
 .. note:: Cannot Infer Key Schema
 
    The {+connector+} does not support key schema inference. If you want to use a key

source/whats-new.txt

@@ -40,6 +40,8 @@ What's New in 1.9
 - Introduced the ``change.stream.full.document.before.change`` setting that
   allows you to access and configure the pre-image of an update
   operation in the change stream event document.
+- Improved :ref:`schema inference <source-infer-a-schema>` for nested
+  documents contained in arrays.
 
 .. _kafka-connector-whats-new-1.8.1: