Skip to content

Commit 26f84cf

Browse files
viragtripathiviragtripathi
committed
sample stage writers and demo updates
1 parent f263b9c commit 26f84cf

File tree

9 files changed

+251
-81
lines changed

9 files changed

+251
-81
lines changed

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# Redis Connect (Continuous Replication and Initial Load to Redis Enterprise)
44

55
- [Redis Connect Overview](#technical-overview)
6-
* [Using redis-connect-cassandra](#connectors/cassandra#redis-connect-cassandra)
6+
* [Using redis-connect-cassandra](connectors/cassandra#redis-connect-cassandra)
77
* [Using redis-connect-db2](connectors/db2#redis-connect-db2)
88
* [Using redis-connect-gemfire](connectors/gemfire#redis-connect-gemfire)
99
* [Using redis-connect-mysql](connectors/mysql#redis-connect-mysql)

‎connectors/cassandra/README.md

Lines changed: 113 additions & 74 deletions
Large diffs are not rendered by default.

‎connectors/gemfire/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ Copy the _sample_ directory and it's contents i.e. _yml_ files, _mappers_ and te
106106
<p>
107107

108108
#### logging configuration file.
109+
109110
### Sample logback.xml under redis-connect-gemfire/config folder
110111
```xml
111112
<configuration debug="true" scan="true" scanPeriod="15 seconds">
@@ -276,7 +277,8 @@ metricsReporter:
276277
<details><summary>Configure JobConfig.yml</summary>
277278
<p>
278279
279-
#### Job level details.
280+
#### Job level details. Please see [writers](../../docs/writers) for other write stage usages.
281+
280282
### Sample JobConfig.yml under redis-connect-gemfire/config/samples/gemfire2redis folder
281283
You can have one or more JobConfig.yml (or with any name e.g. JobConfig-<region_type>.yml) and specify them in the Setup.yml under jobConfig: tag. If specifying more than one table (as below) then make sure maxNumberOfJobs: tag under JobManager.yml is set accordingly e.g. if maxNumberOfJobs: tag is set to 2 then Redis Connect will start 2 cdc jobs under the same JVM instance. If the workload is more and you want to spread out (scale) the cdc jobs then create multiple JobConfig's and specify them in the Setup.yml under jobConfig: tag.
282284
```yml

‎connectors/mssql/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -292,7 +292,7 @@ metricsReporter:
292292
<details><summary>Configure JobConfig.yml</summary>
293293
<p>
294294
295-
#### Job level details.
295+
#### Job level details. Please see [writers](../../docs/writers) for other write stage usages.
296296
297297
### Sample JobConfig.yml under redis-connect-sqlserver/config/samples/sqlserver folder
298298

‎connectors/mysql/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -279,7 +279,7 @@ metricsReporter:
279279
<details><summary>Configure JobConfig.yml</summary>
280280
<p>
281281
282-
#### Job level details.
282+
#### Job level details. Please see [writers](../../docs/writers) for other write stage usages.
283283
284284
### Sample JobConfig.yml under redis-connect-mysql/config/samples/mysql folder
285285

‎connectors/oracle/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -289,7 +289,7 @@ metricsReporter:
289289
<details><summary>Configure JobConfig.yml</summary>
290290
<p>
291291
292-
#### Job level details.
292+
#### Job level details. Please see [writers](../../docs/writers) for other write stage usages.
293293
294294
### Sample JobConfig.yml under redis-connect-oracle/config/samples/oracle folder
295295

‎connectors/postgres/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -284,7 +284,7 @@ metricsReporter:
284284
<details><summary>Configure JobConfig.yml</summary>
285285
<p>
286286
287-
#### Job level details.
287+
#### Job level details. Please see [writers](../../docs/writers) for other write stage usages.
288288
289289
### Sample JobConfig.yml under redis-connect-postgres/config/samples/postgres folder
290290

‎docs/images/README.md

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎docs/writers.md

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
## Supported Redis data structures
2+
3+
You can have one or more write stages in any Redis Connect job pipeline config. Here are the supported write stages, and it's usage examples within JobConfig.yml.
4+
5+
<details><summary>StringWriteStage</summary>
6+
<p>
7+
8+
```yml
9+
StringWriteStage:
10+
handlerId: REDIS_STRING_WRITER
11+
connectionId: targetConnection
12+
metricsEnabled: false
13+
deleteOnKeyUpdate: true
14+
async: true
15+
```
16+
17+
</p>
18+
</details>
19+
20+
<details><summary>HashWriteStage</summary>
21+
<p>
22+
23+
```yml
24+
HashWriteStage:
25+
handlerId: REDIS_HASH_WRITER
26+
connectionId: targetConnection
27+
metricsEnabled: false
28+
prependTableNameToKeys: true
29+
deleteOnKeyUpdate: true
30+
async: true
31+
```
32+
33+
</p>
34+
</details>
35+
36+
<details><summary>SetWriteStage</summary>
37+
<p>
38+
39+
```yml
40+
Set1WriteStage:
41+
handlerId: REDIS_SET_WRITER
42+
connectionId: targetConnection
43+
metricsEnabled: false
44+
deleteOnKeyUpdate: true
45+
async: true
46+
delimiter: "|"
47+
keyPrefix: "empNumSet:"
48+
keyColumns:
49+
- empno
50+
valueColumns:
51+
- empno
52+
Set2WriteStage:
53+
handlerId: REDIS_SET_WRITER
54+
connectionId: targetConnection
55+
metricsEnabled: false
56+
deleteOnKeyUpdate: true
57+
async: true
58+
delimiter: "|"
59+
keyPrefix: "empNumSet:"
60+
keyColumns:
61+
- empno
62+
valueColumns:
63+
- fname
64+
Set3WriteStage:
65+
handlerId: REDIS_SET_WRITER
66+
connectionId: targetConnection
67+
metricsEnabled: false
68+
deleteOnKeyUpdate: true
69+
async: true
70+
delimiter: "|"
71+
keyPrefix: "empNumSet:"
72+
keyColumns:
73+
- empno
74+
valueColumns:
75+
- lname
76+
```
77+
78+
</p>
79+
</details>
80+
81+
<details><summary>SortedSetWriteStage</summary>
82+
<p>
83+
84+
```yml
85+
SortedSetWriteStage:
86+
handlerId: REDIS_SORTEDSET_WRITER
87+
connectionId: targetConnection
88+
metricsEnabled: false
89+
deleteOnKeyUpdate: true
90+
async: true
91+
keyPrefix: "Z:DataSet"
92+
valueColumns:
93+
- empno
94+
- fname
95+
- lname
96+
```
97+
98+
</p>
99+
</details>
100+
101+
<details><summary>StreamWriteStage</summary>
102+
<p>
103+
104+
```yml
105+
StreamWriteStage:
106+
handlerId: REDIS_STREAM_WRITER
107+
connectionId: targetConnection
108+
metricsEnabled: false
109+
deleteOnKeyUpdate: true
110+
async: true
111+
```
112+
113+
</p>
114+
</details>
115+
116+
<details><summary>JSONWriteStage</summary>
117+
<p>
118+
119+
```yml
120+
JSONWriteStage:
121+
handlerId: REDIS_JSON_WRITER
122+
connectionId: targetConnection
123+
metricsEnabled: false
124+
prependTableNameToKeys: true
125+
deleteOnKeyUpdate: true
126+
async: true
127+
```
128+
129+
</p>
130+
</details>

0 commit comments

Comments
 (0)