Skip to content

Commit d850d1f

Browse files
rustagirrustagir
authored
DOCSP-29507: json formatters (#112)
* DOCSP-29507: json formatters * add to tree * staging fixes * fixes * formatting * formatting and wording fixes * remove custom class content * DB PR fixes 1 * small fixes * DB grammar fixes
1 parent 5eed27e commit d850d1f

File tree

3 files changed

+202
-2
lines changed

3 files changed

+202
-2
lines changed

‎source/source-connector/configuration-properties/output-format.txt

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,15 +78,16 @@ Settings
7878
com.mongodb.kafka.connect.source.json.formatter.DefaultJson
7979

8080
| **Accepted Values**:
81-
| One of the following full class names:
81+
| Your custom JSON formatter full class name or one of the
82+
following built-in formatter class names:
8283

8384
.. code-block:: none
8485

8586
com.mongodb.kafka.connect.source.json.formatter.DefaultJson
8687
com.mongodb.kafka.connect.source.json.formatter.ExtendedJson
8788
com.mongodb.kafka.connect.source.json.formatter.SimplifiedJson
8889

89-
| Or your custom JSON formatter full class name.
90+
| To learn more about these output formats, see :ref:`kafka-source-json-formatters`.
9091

9192
* - | **output.schema.key**
9293
- | **Type:** string

‎source/source-connector/fundamentals.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,11 @@ Fundamentals
1010

1111
Change Streams </source-connector/fundamentals/change-streams>
1212
Apply Schemas </source-connector/fundamentals/specify-schema>
13+
JSON Formatters </source-connector/fundamentals/json-formatters>
1314

1415
Read the following sections to learn how {+source-connector+} features work and
1516
how to configure them:
1617

1718
- :doc:`Receive Real-time Updates on Data Changes in MongoDB </source-connector/fundamentals/change-streams>`
1819
- :doc:`Apply Schemas to Documents </source-connector/fundamentals/specify-schema>`
20+
- :doc:`Specify a JSON Formatter for Output </source-connector/fundamentals/json-formatters>`

‎source/source-connector/fundamentals/json-formatters.txt

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
.. _kafka-source-json-formatters:
2+
3+
===============
4+
JSON Formatters
5+
===============
6+
7+
.. contents:: On this page
8+
:local:
9+
:backlinks: none
10+
:depth: 2
11+
:class: singlecol
12+
13+
14+
Overview
15+
--------
16+
17+
In this guide, you can learn how to specify built-in JSON formatter
18+
classes to use in a {+source-connector+}.
19+
20+
JSON Formatters
21+
---------------
22+
23+
**JSON formatters** specify how JSON data appears. They provide instructions
24+
for how the connector represents different types when it outputs data.
25+
26+
Built-In Formatters
27+
-------------------
28+
29+
The following table describes the formatter classes available in the
30+
source connector:
31+
32+
.. list-table::
33+
:header-rows: 1
34+
:widths: 60 40
35+
36+
* - Class
37+
- Description
38+
39+
* - ``com.mongodb.kafka.connect.source.json.formatter.DefaultJson``
40+
- The legacy strict JSON formatter.
41+
42+
* - ``com.mongodb.kafka.connect.source.json.formatter.ExtendedJson``
43+
- The fully type-safe extended JSON formatter. This formatter
44+
emphasizes type preservation and represents most values with
45+
their BSON type.
46+
47+
* - ``com.mongodb.kafka.connect.source.json.formatter.SimplifiedJson``
48+
- The simplified JSON formatter. This formatter represents
49+
``ObjectId``, ``Decimal``, ``Date``, and ``Binary`` values as
50+
strings.
51+
52+
Examples
53+
--------
54+
55+
The following table describes the fields of the sample document that the
56+
examples in this guide use to demonstrate each output format. The
57+
columns describe the field name, value, and type for each field or
58+
nested field.
59+
60+
.. list-table::
61+
:header-rows: 1
62+
:widths: 15 55 30
63+
64+
* - Name
65+
- Value
66+
- Type
67+
68+
* - ``_id``
69+
- .. code-block:: none
70+
:copyable: false
71+
72+
"5f15aab12435743f9bd126a4"
73+
74+
- ``ObjectID`` (``$oid``)
75+
76+
* - ``w``
77+
- .. code-block:: none
78+
:copyable: false
79+
80+
[ 12345.6789, 23.53 ]
81+
82+
- | Array of:
83+
| - ``Decimal128`` (``$numberDecimal``)
84+
| - ``Double`` (``$numberDouble``)
85+
86+
* - ``x``
87+
- .. code-block:: none
88+
:copyable: false
89+
90+
"SSBsb3ZlIGZvcm1hdHRpbmch" of type "00"
91+
92+
- ``Binary`` (``$binary``)
93+
94+
* - ``y``
95+
- .. code-block:: none
96+
:copyable: false
97+
98+
"2023-05-11T08:27:07.000Z"
99+
100+
- ``Date`` (``$date``)
101+
102+
* - ``z``
103+
- .. code-block:: none
104+
:copyable: false
105+
106+
{ a: false, b: 87, c: "hello world" }
107+
108+
- | Document with fields:
109+
| - ``a``: ``Boolean``
110+
| - ``b``: ``Int32`` (``$numberInt``)
111+
| - ``c``: ``String``
112+
113+
Default JSON Formatter
114+
~~~~~~~~~~~~~~~~~~~~~~
115+
116+
In the configuration properties for your source connector, set the
117+
following property to specify the default JSON formatter:
118+
119+
.. code-block:: json
120+
121+
"output.json.formatter": "com.mongodb.kafka.connect.source.json.formatter.DefaultJson"
122+
123+
When you output the sample document contents from your Kafka topic, the
124+
output shows the following type representations:
125+
126+
- ``ObjectId`` value with its BSON type
127+
- ``Decimal`` value with its BSON type
128+
- ``Binary`` value with its buffer string and binary type
129+
- ``Date`` value as milliseconds since the UNIX epoch
130+
131+
.. code-block:: json
132+
133+
{
134+
"_id": {"$oid": "5f15aab12435743f9bd126a4"},
135+
"w": [{"$numberDecimal": "12345.6789"}, 23.53],
136+
"x": {"$binary": "SSBsb3ZlIGZvcm1hdHRpbmch", "$type": "00"},
137+
"y": {"$date": 1683793627000},
138+
"z": {"a": false, "b": 87, "c": "hello world"}
139+
}
140+
141+
Extended JSON Formatter
142+
~~~~~~~~~~~~~~~~~~~~~~~
143+
144+
In the configuration properties for your source connector, set the
145+
following property to specify the extended JSON formatter:
146+
147+
.. code-block:: json
148+
149+
"output.json.formatter": "com.mongodb.kafka.connect.source.json.formatter.ExtendedJson"
150+
151+
When you output the sample document contents from your Kafka topic, the
152+
output shows the following type representations:
153+
154+
- ``ObjectId`` value with its BSON type
155+
- ``Decimal`` value with its BSON type
156+
- ``Double`` value with its BSON type
157+
- ``Binary`` value with its buffer string and binary type
158+
- ``Date`` value as milliseconds since the UNIX epoch
159+
- ``Int32`` value with its BSON type
160+
161+
.. code-block:: json
162+
163+
{
164+
"_id": {"$oid": "5f15aab12435743f9bd126a4"},
165+
"w": [{"$numberDecimal": "12345.6789"}, {"$numberDouble": "23.53"}],
166+
"x": {"$binary": "SSBsb3ZlIGZvcm1hdHRpbmch", "$type": "00"},
167+
"y": {"$date": 1683793627000},
168+
"z": {"a": false, "b": {"$numberInt": "87"}, "c": "hello world"}
169+
}
170+
171+
Simplified JSON Formatter
172+
~~~~~~~~~~~~~~~~~~~~~~~~~
173+
174+
In the configuration properties for your source connector, set the
175+
following property to specify the simplified JSON formatter:
176+
177+
.. code-block:: json
178+
179+
"output.json.formatter": "com.mongodb.kafka.connect.source.json.formatter.SimplifiedJson"
180+
181+
When you output the sample document contents from your Kafka topic, the
182+
output shows the following type representations:
183+
184+
- ``ObjectId`` value as its hexadecimal string
185+
- ``Decimal`` value as a string
186+
- ``Binary`` value as its buffer string
187+
- ``Date`` value as a string
188+
189+
.. code-block:: json
190+
191+
{
192+
"_id": "5f15aab12435743f9bd126a4",
193+
"w": ["12345.6789", 23.53],
194+
"x": "SSBsb3ZlIGZvcm1hdHRpbmch",
195+
"y": "2023-05-11T08:27:07Z",
196+
"z": {"a": false, "b": 87, "c": "hello world"}
197+
}

0 commit comments

Comments
 (0)