Skip to content

Commit a827bdd

Browse files
viragtripathiviragtripathi
committed
Gemfire demo docs
1 parent 10cffce commit a827bdd

24 files changed

+1369
-3
lines changed

‎connectors/gemfire/README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ The functionality of the connector is based upon [Durable Client/Server Messagin
99

1010
The connector is also tolerant of failures. As the connector reads changes and produces events, it records the Checkpoint i.e. <i>DURABLE_CLIENT_ID</i> in the target Redis Enterprise database that is associated with _CDC_ record with each event.
1111
If the connector stops for any reason (including communication failures, network problems, or crashes), upon restart it simply continues reading the Regions where it last left off.
12-
The connector then produces a _change event_ for every row-level insert, update, and delete operation that was published via the _CDC API_, while recording all the change events for each table in a Redis Enterprise Database with a choice of your data structure such as [Hashes](https://redis.io/topics/data-types#hashes). Please see a list of supported data structures [here](../../docs/writers.md), and it's usage examples.
12+
The connector then produces a _change event_ for every row-level insert, update, and delete operation that was published via the _CDC API_, while recording all the change events for each table in a Redis Enterprise Database.
1313

1414
## Architecture
1515

@@ -164,7 +164,7 @@ Copy the _sample_ directory and it's contents i.e. _yml_ files, _mappers_ and te
164164
</logger>
165165

166166

167-
<logger name="com.redislabs" level="INFO" additivity="false">
167+
<logger name="com.redis" level="INFO" additivity="false">
168168
<appender-ref ref="REDISCONNECT"/>
169169
<appender-ref ref="CONSOLE" />
170170
</logger>
@@ -469,4 +469,9 @@ Once staging is done, execute the same script with <i>start</i> option to start
469469

470470
```bash
471471
redis-connect-gemfire/bin$ ./redisconnect.sh start
472-
```
472+
```
473+
474+
| ℹ️ |
475+
|:-------------------------------------------|
476+
| Quick Start: Follow the [demo](demo) |
477+
| K8s Setup: Follow the [k8s-docs](k8s-docs) |

‎connectors/gemfire/demo/README.md

Lines changed: 336 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,337 @@
1+
# Prerequisites
2+
Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed.
3+
<br>Please have 8 vCPU*, 8 GB RAM and 50 GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect Connector instances deployed on separate hosts.</br>
4+
<br>Execute the following commands (copy & paste) to download and setup Redis Connect Gemfire Connector and demo scripts.
5+
i.e.</br>
16

7+
```bash
8+
wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
9+
mkdir -p redis-connect-gemfire/demo && \
10+
mkdir -p redis-connect-gemfire/k8s-docs && \
11+
unzip main.zip "redis-connect-dist-main/connectors/gemfire/*" -d redis-connect-gemfire && \
12+
cp -R redis-connect-gemfire/redis-connect-dist-main/connectors/gemfire/demo/* redis-connect-gemfire/demo && \
13+
cp -R redis-connect-gemfire/redis-connect-dist-main/connectors/gemfire/k8s-docs/* redis-connect-gemfire/k8s-docs && \
14+
rm -rf main.zip redis-connect-gemfire/redis-connect-dist-main && \
15+
cd redis-connect-gemfire && \
16+
chmod a+x demo/*.sh
17+
```
18+
19+
Expected output:
20+
```bash
21+
redis-connect-gemfire$ ls
22+
config demo
23+
```
24+
25+
## Setup Gemfire (Source)
26+
Please refer to the installation guide and [Insall and Setup Gemfire](https://gemfire.docs.pivotal.io/910/gemfire/getting_started/installation/install_intro.html).
27+
28+
Here is an example with the included cache config files in the `redis-connect-gemfire/config/samples/gemfire2redis` folder.
29+
30+
```bash
31+
~/pivotal-gemfire-9.10.4/bin$ ./gfsh
32+
_________________________ __
33+
/ _____/ ______/ ______/ /____/ /
34+
/ / __/ /___ /_____ / _____ /
35+
/ /__/ / ____/ _____/ / / / /
36+
/______/_/ /______/_/ /_/ 9.10.4
37+
38+
Monitor and Manage VMware Tanzu GemFire
39+
Start locator
40+
gfsh>start locator --name=locator1 --bind-address=127.0.0.1
41+
42+
Start server1
43+
gfsh>start server --name=server1 --bind-address=127.0.0.1 --cache-xml-file=~/redis-connect-gemfire/config/samples/cdc/gemfire2redis/cache.xml
44+
45+
Start server2
46+
gfsh>start server --name=server2 --bind-address=127.0.0.1 --cache-xml-file=~/redis-connect-gemfire/config/samples/cdc/gemfire2redis/cache1.xml
47+
48+
Deploy jar for the initial loader process
49+
gfsh>deploy --jar=~/redis-connect-gemfire/lib/connector-gemfire-fn-0.8.0.jar
50+
```
51+
52+
## Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target)
53+
<br>Execute [setup_re.sh](setup_re.sh)</br>
54+
```bash
55+
demo$ ./setup_re.sh
56+
```
57+
**NOTE**
58+
59+
The above script will create a 1-node Redis Enterprise cluster in a docker container, [Create a target database with RediSearch module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a job management and metrics database with RedisTimeSeries module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a RediSearch index for emp Hash](https://redislabs.com/blog/getting-started-with-redisearch-2-0/), [Start a docker instance of grafana with Redis Data Source](https://redisgrafana.github.io/) and [Start an instance of RedisInsight](https://docs.redislabs.com/latest/ri/installing/install-docker/).
60+
61+
## Start Redis Connect Gemfire Connector
62+
63+
<details><summary>Run Redis Connect Gemfire Connector docker container to see all the options</summary>
64+
<p>
65+
66+
```bash
67+
docker run \
68+
-it --rm --privileged=true \
69+
--name redis-connect-gemfire \
70+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
71+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/cdc/gemfire2redis \
72+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx256m" \
73+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
74+
--net host \
75+
redislabs/redis-connect-gemfire:latest
76+
```
77+
78+
</p>
79+
</details>
80+
81+
<details><summary>Expected output:</summary>
82+
<p>
83+
84+
```bash
85+
-------------------------------
86+
Redis Connect startup script.
87+
*******************************
88+
Please ensure that the values of environment variables in /opt/redislabs/redis-connect-gemfire/bin/redisconnect.conf are correctly mapped before executing any of the options below
89+
*******************************
90+
Usage: [-h|cli|stage|start]
91+
options:
92+
-h: Print this help message and exit.
93+
cli: starts redis-connect-cli.
94+
stage: clean and stage redis database with cdc or initial loader job configurations.
95+
start: start Redis Connect instance with provided cdc or initial loader job configurations.
96+
-------------------------------
97+
```
98+
99+
</p>
100+
</details>
101+
102+
-------------------------------
103+
104+
### Initial Loader Steps
105+
<details><summary><b>INSERT few records into gemfire table (source)</b></summary>
106+
<p>
107+
108+
```bash
109+
110+
```
111+
112+
</p>
113+
</details>
114+
115+
<details><summary><b>Stage pre-configured loader job</b></summary>
116+
<p>
117+
118+
```bash
119+
docker run \
120+
-it --rm --privileged=true \
121+
--name redis-connect-gemfire \
122+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
123+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/loader \
124+
-e REDISCONNECT_SOURCE_USERNAME= \
125+
-e REDISCONNECT_SOURCE_PASSWORD= \
126+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx256m" \
127+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
128+
--net host \
129+
redislabs/redis-connect-gemfire:latest stage
130+
```
131+
132+
</p>
133+
</details>
134+
135+
<details><summary>Expected output:</summary>
136+
<p>
137+
138+
```bash
139+
-------------------------------
140+
Staging Redis Connect redis-connect-gemfire v1.0.2.151 job using Java 11.0.12 on 16229e5715a1 started by root in /opt/redislabs/redis-connect-gemfire/bin
141+
Loading Redis Connect redis-connect-gemfire Configurations from /opt/redislabs/redis-connect-gemfire/config/samples/loader
142+
.....
143+
.....
144+
12:31:38.726 [main] INFO startup - Setup Completed.
145+
-------------------------------
146+
```
147+
148+
</p>
149+
</details>
150+
151+
<details><summary><b>Start pre configured loader job</b></summary>
152+
<p>
153+
154+
```bash
155+
docker run \
156+
-it --rm --privileged=true \
157+
--name redis-connect-gemfire \
158+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
159+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/loader \
160+
-e REDISCONNECT_REST_API_ENABLED=false \
161+
-e REDISCONNECT_REST_API_PORT=8282 \
162+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx1g" \
163+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
164+
--net host \
165+
redislabs/redis-connect-gemfire:latest start
166+
```
167+
168+
</p>
169+
</details>
170+
171+
<details><summary>Expected output:</summary>
172+
<p>
173+
174+
```bash
175+
176+
```
177+
178+
</p>
179+
</details>
180+
181+
<details><summary><b>Query for the above inserted record in Redis (target)</b></summary>
182+
<p>
183+
184+
```bash
185+
demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000'
186+
187+
```
188+
189+
</p>
190+
</details>
191+
192+
-------------------------------
193+
194+
### CDC Steps
195+
<details><summary><b>Stage pre-configured cdc job</b></summary>
196+
<p>
197+
198+
```bash
199+
docker run \
200+
-it --rm --privileged=true \
201+
--name redis-connect-gemfire \
202+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
203+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/gemfire2redis \
204+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx256m" \
205+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
206+
--net host \
207+
redislabs/redis-connect-gemfire:latest stage
208+
```
209+
210+
</p>
211+
</details>
212+
213+
<details><summary>Expected output:</summary>
214+
<p>
215+
216+
```bash
217+
-------------------------------
218+
Staging Redis Connect redis-connect-gemfire v1.0.2.151 job using Java 11.0.12 on virag-cdc started by root in /opt/redislabs/redis-connect-gemfire/bin.
219+
Loading Redis Connect redis-connect-gemfire Configurations from /opt/redislabs/redis-connect-gemfire/config/samples/gemfire.
220+
.....
221+
.....
222+
20:15:06.819 [main] INFO startup - Setup Completed.
223+
-------------------------------
224+
```
225+
226+
</p>
227+
</details>
228+
229+
<details><summary><b>Start pre-configured cdc job</b></summary>
230+
<p>
231+
232+
```bash
233+
docker run \
234+
-it --rm --privileged=true \
235+
--name redis-connect-gemfire \
236+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
237+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/gemfire2redis \
238+
-e REDISCONNECT_REST_API_ENABLED=true \
239+
-e REDISCONNECT_REST_API_PORT=8282 \
240+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx1g" \
241+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
242+
--net host \
243+
redislabs/redis-connect-gemfire:latest start
244+
```
245+
246+
</p>
247+
</details>
248+
249+
<details><summary>Expected output:</summary>
250+
<p>
251+
252+
```bash
253+
```
254+
255+
</p>
256+
</details>
257+
258+
<details><summary><b>INSERT a record into gemfire table (source)</b></summary>
259+
<p>
260+
261+
```bash
262+
263+
```
264+
265+
</p>
266+
</details>
267+
268+
<details><summary><b>Query for the above inserted record in Redis (target)</b></summary>
269+
<p>
270+
271+
```bash
272+
demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000'
273+
274+
```
275+
276+
</p>
277+
</details>
278+
279+
Similarly `UPDATE` and `DELETE` records on Gemfire source and see Redis target getting updated in near real-time.
280+
281+
-------------------------------
282+
283+
### [_Custom Stage_](https://github.com/redis-field-engineering/redis-connect-custom-stage-demo)
284+
285+
Review the Custom Stage Demo then use the pre-built CustomStage function by passing it as an external library then follow the same [Initial Loader Steps](#initial-loader-steps) and [CDC Steps](#cdc-steps).
286+
287+
Add the `CustomStage` `handlerId` in JobConfig.yml as explained in the Custom Stage Demo i.e.
288+
```yml
289+
stages:
290+
CustomStage:
291+
handlerId: TO_UPPER_CASE
292+
```
293+
<details><summary><b>Stage pre-configured loader job with Custom Stage</b></summary>
294+
<p>
295+
296+
```bash
297+
docker run \
298+
-it --rm --privileged=true \
299+
--name redis-connect-gemfire \
300+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
301+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/loader \
302+
-e REDISCONNECT_SOURCE_USERNAME=redisconnect \
303+
-e REDISCONNECT_SOURCE_PASSWORD=Redis@123 \
304+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx256m" \
305+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
306+
-v $(pwd)/extlib:/opt/redislabs/redis-connect-gemfire/extlib \
307+
--net host \
308+
redislabs/redis-connect-gemfire:latest stage
309+
```
310+
311+
</p>
312+
</details>
313+
314+
<details><summary><b>Start pre-configured loader job with Custom Stage</b></summary>
315+
<p>
316+
317+
```bash
318+
docker run \
319+
-it --rm --privileged=true \
320+
--name redis-connect-gemfire \
321+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
322+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/loader \
323+
-e REDISCONNECT_REST_API_ENABLED=false \
324+
-e REDISCONNECT_REST_API_PORT=8282 \
325+
-e REDISCONNECT_SOURCE_USERNAME= \
326+
-e REDISCONNECT_SOURCE_PASSWORD= \
327+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx1g" \
328+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
329+
-v $(pwd)/extlib:/opt/redislabs/redis-connect-gemfire/extlib \
330+
--net host \
331+
redislabs/redis-connect-gemfire:latest start
332+
```
333+
334+
</p>
335+
</details>
336+
337+
Validate the output after CustomStage run and make sure that `fname` and `lname` values in Redis has been updated to UPPER CASE.

0 commit comments

Comments
 (0)