Skip to content

Commit f744081

Browse files
viragtripathiviragtripathi
committed
updated gemfire demo with initial loader steps
1 parent ae740db commit f744081

File tree

2 files changed

+222
-0
lines changed

2 files changed

+222
-0
lines changed

‎connectors/gemfire/demo/README.md

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,228 @@ start: start Redis Connect instance with provided cdc or initial loader job conf
101101

102102
----
103103

104+
### Initial Loader Steps
105+
<details><summary><b>INSERT records into gemfire region(s) (source)</b></summary>
106+
107+
We are going to leverage a sample [Gemfire Producer](https://github.com/redis-field-engineering/redisconnect-gemfire-producer) for this task. Download and un-tar the latest [release](https://github.com/redis-field-engineering/redisconnect-gemfire-producer/releases) then run the sample gemfire producer
108+
109+
````bash
110+
java -jar redisconnect-gemfire-producer-1.0-SNAPSHOT.jar --help
111+
Usage: com.redis.connect.gemfire.producer.GemfireProducer [--help] [-h=<host>] [-i=<iter>] [-o=<opType>] [-p=<port>] [-r=<regionName>]
112+
Gemfire producer load generator.
113+
-h, --host=<host> Gemfire locator host (default: 127.0.0.1)
114+
--help Show this help message and exit
115+
-i, --iter=<iter> Iterations to run (default: 1)
116+
-o, --operation=<opType> Name of the operation i.e. I (put), U (put) and D (remove) (default: I)
117+
-p, --port=<port> Gemfire locator port (default: 10334)
118+
-r, --region=<regionName> Name of the region (default: session)
119+
````
120+
121+
````bash
122+
redisconnect-gemfire-producer$ java -jar redisconnect-gemfire-producer-1.0-SNAPSHOT.jar -i 100 -o I
123+
````
124+
125+
</details>
126+
127+
<details><summary><b>Stage pre-configured loader job</b></summary>
128+
<p>
129+
130+
```bash
131+
docker run \
132+
-it --rm --privileged=true \
133+
--name redis-connect-gemfire \
134+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
135+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/loader \
136+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx256m" \
137+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
138+
--net host \
139+
redislabs/redis-connect-gemfire:latest stage
140+
```
141+
142+
</p>
143+
</details>
144+
145+
<details><summary>Expected output:</summary>
146+
<p>
147+
148+
```bash
149+
-------------------------------
150+
Staging Redis Connect redis-connect-gemfire v0.5.0.139 job using Java 11.0.13 on virag-cdc started by root in /opt/redislabs/redis-connect-gemfire/bin
151+
Loading Redis Connect redis-connect-gemfire Configurations from /opt/redislabs/redis-connect-gemfire/config/samples/loader
152+
04:05:17,742 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [/opt/redislabs/redis-connect-gemfire/config/logback.xml] at [file:/opt/redislabs/redis-connect-gemfire/config/logback.xml]
153+
....
154+
04:05:17.981 [main] INFO startup - ##################################################################
155+
04:05:17.983 [main] INFO startup -
156+
04:05:17.983 [main] INFO startup - REDIS CONNECT SETUP CLEAN - Deletes metadata related to Redis Connect from Job Management Database
157+
158+
04:05:17.983 [main] INFO startup -
159+
04:05:17.983 [main] INFO startup - ##################################################################
160+
....
161+
04:05:20.212 [main] INFO startup - ##################################################################
162+
04:05:20.215 [main] INFO startup -
163+
04:05:20.215 [main] INFO startup - REDIS CONNECT SETUP CREATE - Seed metadata related to Redis Connect to Job Management Database
164+
04:05:20.215 [main] INFO startup -
165+
04:05:20.215 [main] INFO startup - ##################################################################
166+
....
167+
04:05:21.567 [main] INFO startup - Instance: 97@virag-cdc successfully established Redis connection for INIT service
168+
04:05:21.570 [main] INFO startup - Instance: 97@virag-cdc successfully created Job Claim Assignment Stream and Consumer Group
169+
04:05:21.587 [main] INFO startup - Instance: 97@virag-cdc successfully seeded Job related metadata
170+
04:05:21.587 [main] INFO startup - Instance: 97@virag-cdc successfully seeded Metrics related metadata
171+
04:05:21.587 [main] INFO startup - Instance: 97@virag-cdc successfully staged Job Management Database (Redis) with all the configurations and scripts, if applicable, needed to execute jobs
172+
-------------------------------
173+
```
174+
175+
</p>
176+
</details>
177+
178+
<details><summary><b>Start pre-configured loader job</b></summary>
179+
<p>
180+
181+
```bash
182+
docker run \
183+
-it --rm --privileged=true \
184+
--name redis-connect-gemfire \
185+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect-gemfire/config/logback.xml \
186+
-e REDISCONNECT_CONFIG=/opt/redislabs/redis-connect-gemfire/config/samples/loader \
187+
-e REDISCONNECT_REST_API_ENABLED=false \
188+
-e REDISCONNECT_REST_API_PORT=8282 \
189+
-e REDISCONNECT_JAVA_OPTIONS="-Xms256m -Xmx1g" \
190+
-v $(pwd)/config:/opt/redislabs/redis-connect-gemfire/config \
191+
--net host \
192+
redislabs/redis-connect-gemfire:latest start
193+
```
194+
195+
</p>
196+
</details>
197+
198+
<details><summary>Expected output:</summary>
199+
<p>
200+
201+
```bash
202+
-------------------------------
203+
Starting Redis Connect redis-connect-gemfire v0.5.0.139 instance using Java 11.0.13 on virag-cdc started by root in /opt/redislabs/redis-connect-gemfire/bin
204+
Loading Redis Connect redis-connect-gemfire Configurations from /opt/redislabs/redis-connect-gemfire/config/samples/loader
205+
04:06:16,684 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [/opt/redislabs/redis-connect-gemfire/config/logback.xml] at [file:/opt/redislabs/redis-connect-gemfire/config/logback.xml]
206+
....
207+
04:06:16.959 [main] INFO startup -
208+
04:06:16.962 [main] INFO startup - /$$$$$$$ /$$ /$$ /$$$$$$ /$$
209+
04:06:16.962 [main] INFO startup - | $$__ $$ | $$|__/ /$$__ $$ | $$
210+
04:06:16.963 [main] INFO startup - | $$ \ $$ /$$$$$$ /$$$$$$$ /$$ /$$$$$$$ | $$ \__/ /$$$$$$ /$$$$$$$ /$$$$$$$ /$$$$$$ /$$$$$$$ /$$$$$$
211+
04:06:16.963 [main] INFO startup - | $$$$$$$/ /$$__ $$ /$$__ $$| $$ /$$_____/ | $$ /$$__ $$| $$__ $$| $$__ $$ /$$__ $$ /$$_____/|_ $$_/
212+
04:06:16.963 [main] INFO startup - | $$__ $$| $$$$$$$$| $$ | $$| $$| $$$$$$ | $$ | $$ \ $$| $$ \ $$| $$ \ $$| $$$$$$$$| $$ | $$
213+
04:06:16.963 [main] INFO startup - | $$ \ $$| $$_____/| $$ | $$| $$ \____ $$ | $$ $$| $$ | $$| $$ | $$| $$ | $$| $$_____/| $$ | $$ /$$
214+
04:06:16.964 [main] INFO startup - | $$ | $$| $$$$$$$| $$$$$$$| $$ /$$$$$$$/ | $$$$$$/| $$$$$$/| $$ | $$| $$ | $$| $$$$$$$| $$$$$$$ | $$$$/
215+
04:06:16.964 [main] INFO startup - |__/ |__/ \_______/ \_______/|__/|_______/ \______/ \______/ |__/ |__/|__/ |__/ \_______/ \_______/ \___/
216+
04:06:16.964 [main] INFO startup -
217+
04:06:16.964 [main] INFO startup - ##################################################################
218+
04:06:16.964 [main] INFO startup -
219+
04:06:16.964 [main] INFO startup - Initializing Redis Connect Instance
220+
04:06:16.964 [main] INFO startup -
221+
04:06:16.964 [main] INFO startup - ##################################################################
222+
....
223+
04:06:23.290 [main] INFO startup - Instance: 30@virag-cdc successfully established Redis connection for JobManager service
224+
04:06:23.411 [main] INFO startup - Instance: 30@virag-cdc successfully established PUB/SUB Redis connection
225+
04:06:23.451 [main] INFO startup - Instance: 30@virag-cdc successfully established PUB/SUB Redis connection
226+
04:06:23.644 [main] INFO startup - Instance: 30@virag-cdc successfully started JobManager service
227+
04:06:23.683 [main] INFO startup - Instance: 30@virag-cdc successfully established Redis connection for JobReaper service
228+
04:06:23.684 [main] INFO startup - Instance: 30@virag-cdc successfully started JobReaper service
229+
04:06:23.720 [main] INFO startup - Instance: 30@virag-cdc successfully established Redis connection for JobClaimer service
230+
04:06:23.721 [main] INFO startup - Instance: 30@virag-cdc successfully started JobClaimer service
231+
04:06:23.769 [main] INFO startup - Instance: 30@virag-cdc successfully subscribed to Channel: REDIS.CONNECT.JOB.CLAIM.TRANSITION.EVENTS
232+
04:06:23.769 [main] INFO startup - Instance: 30@virag-cdc did not enable embedded REST API server
233+
04:06:33.683 [JobManager-1] INFO startup - Instance: 30@virag-cdc successfully established Redis connection for HeartbeatManager service
234+
04:06:33.684 [JobManager-1] INFO startup - Instance: 30@virag-cdc was successfully elected Redis Connect cluster leader
235+
04:06:44.814 [JobManager-1] INFO startup - Instance: 30@virag-cdc successfully started job execution for JobId: {connect}:job:initial_load
236+
04:06:44.814 [JobManager-1] INFO startup - Instance: 30@virag-cdc has successfully claimed ownership of JobId: {connect}:job:initial_load
237+
04:06:44.814 [JobManager-1] INFO startup - Instance: 30@virag-cdc has claimed 1 job(s) from its 2 max allowable capacity
238+
04:06:44.827 [JobManager-1] INFO startup - JobId: {connect}:job:initial_load claim request with ID: 1641528321578-0 has been fully processed and all metadata has been updated
239+
04:06:44.831 [JobManager-1] INFO startup - Instance: 30@virag-cdc published Job Claim Transition Event to Channel: REDIS.CONNECT.JOB.CLAIM.TRANSITION.EVENTS Message: {"jobId":"{connect}:job:initial_load","instanceName":"30@virag-cdc","transitionEvent":"CLAIMED","serviceName":"JobClaimer"}
240+
04:06:44.831 [lettuce-nioEventLoop-4-3] INFO startup - Instance: 30@virag-cdc consumed Job Claim Transition Event on Channel: REDIS.CONNECT.JOB.CLAIM.TRANSITION.EVENTS Message: {"jobId":"{connect}:job:initial_load","instanceName":"30@virag-cdc","transitionEvent":"CLAIMED","serviceName":"JobClaimer"}
241+
04:06:45.185 [EventProducer-1] WARN startup - Instance: 30@virag-cdc did not find entry in its executor threads local cache during stop process for JobId: {connect}:job:initial_load
242+
04:06:45.185 [EventProducer-1] WARN startup - Instance: 30@virag-cdc could not cancel executor thread future for JobId: {connect}:job:initial_load
243+
04:06:45.185 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully cancelled heartbeat for JobId: {connect}:job:initial_load
244+
04:06:45.185 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully stopped replication pipeline for JobId: {connect}:job:initial_load
245+
04:06:45.185 [EventProducer-1] INFO startup - Instance: 30@virag-cdc now owns 0 job(s) from its 2 max allowable capacity
246+
04:06:45.186 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully stopped JobId: {connect}:job:initial_load and added it to {connect}:jobs:stopped
247+
....
248+
04:07:13.919 [EventProducer-2] INFO redisconnect - In Load
249+
04:07:13.920 [JobManager-1] INFO startup - Instance: 30@virag-cdc has claimed 1 job(s) from its 2 max allowable capacity
250+
04:07:13.920 [EventProducer-2] INFO redisconnect - Processing LoadSegment for Region : session , Bucket : 0
251+
04:07:13.924 [JobManager-1] INFO startup - JobId: session-0 claim request with ID: 1641528405159-0 has been fully processed and all metadata has been updated
252+
....
253+
04:07:14.099 [EventProducer-2] INFO redisconnect - Completed Results For bucket : 22
254+
04:07:14.106 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully cancelled heartbeat for JobId: session-0
255+
04:07:14.106 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully stopped replication pipeline for JobId: session-0
256+
04:07:14.106 [EventProducer-2] INFO startup - Instance: 30@virag-cdc now owns 0 job(s) from its 2 max allowable capacity
257+
04:07:14.106 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully stopped JobId: session-0 and added it to {connect}:jobs:stopped
258+
....
259+
04:07:43.763 [JobManager-2] INFO startup - Instance: 30@virag-cdc successfully started job execution for JobId: session-1
260+
04:07:43.763 [EventProducer-1] INFO redisconnect - In Load
261+
04:07:43.764 [EventProducer-1] INFO redisconnect - Processing LoadSegment for Region : session , Bucket : 23
262+
....
263+
04:07:43.827 [EventProducer-1] INFO redisconnect - Completed Results For bucket : 45
264+
04:07:43.832 [EventProducer-1] WARN startup - Instance: 30@virag-cdc could not cancel executor thread future for JobId: session-1
265+
04:07:43.832 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully cancelled heartbeat for JobId: session-1
266+
04:07:43.832 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully stopped replication pipeline for JobId: session-1
267+
04:07:43.832 [EventProducer-1] INFO startup - Instance: 30@virag-cdc now owns 0 job(s) from its 2 max allowable capacity
268+
04:07:43.832 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully stopped JobId: session-1 and added it to {connect}:jobs:stopped
269+
....
270+
04:08:13.742 [EventProducer-2] INFO redisconnect - In Load
271+
04:08:13.742 [JobManager-1] INFO startup - Instance: 30@virag-cdc has claimed 1 job(s) from its 2 max allowable capacity
272+
04:08:13.742 [EventProducer-2] INFO redisconnect - Processing LoadSegment for Region : session , Bucket : 46
273+
....
274+
04:08:13.797 [EventProducer-2] INFO redisconnect - Completed Results For bucket : 68
275+
04:08:13.801 [EventProducer-2] WARN startup - Instance: 30@virag-cdc could not cancel executor thread future for JobId: session-2
276+
04:08:13.801 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully cancelled heartbeat for JobId: session-2
277+
04:08:13.802 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully stopped replication pipeline for JobId: session-2
278+
04:08:13.802 [EventProducer-2] INFO startup - Instance: 30@virag-cdc now owns 0 job(s) from its 2 max allowable capacity
279+
04:08:13.802 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully stopped JobId: session-2 and added it to {connect}:jobs:stopped
280+
....
281+
04:08:43.733 [JobManager-1] INFO startup - Instance: 30@virag-cdc successfully started job execution for JobId: session-3
282+
04:08:43.733 [EventProducer-1] INFO redisconnect - In Load
283+
04:08:43.733 [JobManager-1] INFO startup - Instance: 30@virag-cdc has successfully claimed ownership of JobId: session-3
284+
04:08:43.733 [EventProducer-1] INFO redisconnect - Processing LoadSegment for Region : session , Bucket : 69
285+
....
286+
04:08:43.787 [EventProducer-1] INFO redisconnect - Completed Results For bucket : 90
287+
04:08:43.791 [EventProducer-1] WARN startup - Instance: 30@virag-cdc could not cancel executor thread future for JobId: session-3
288+
04:08:43.791 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully cancelled heartbeat for JobId: session-3
289+
04:08:43.791 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully stopped replication pipeline for JobId: session-3
290+
04:08:43.791 [EventProducer-1] INFO startup - Instance: 30@virag-cdc now owns 0 job(s) from its 2 max allowable capacity
291+
04:08:43.791 [EventProducer-1] INFO startup - Instance: 30@virag-cdc successfully stopped JobId: session-3 and added it to {connect}:jobs:stopped
292+
....
293+
04:09:13.735 [JobManager-1] INFO startup - Instance: 30@virag-cdc successfully started job execution for JobId: session-4
294+
04:09:13.735 [EventProducer-2] INFO redisconnect - In Load
295+
04:09:13.736 [JobManager-1] INFO startup - Instance: 30@virag-cdc has successfully claimed ownership of JobId: session-4
296+
04:09:13.736 [EventProducer-2] INFO redisconnect - Processing LoadSegment for Region : session , Bucket : 91
297+
....
298+
04:09:13.785 [EventProducer-2] INFO redisconnect - Processing LoadSegment for Region : session , Bucket : 112
299+
04:09:13.786 [Function Execution Thread-1] INFO redisconnect - Publishing data for Region : session , BucketId - 112 , Batch - 1 - num records in batch : 1
300+
04:09:13.786 [Function Execution Thread-1] INFO redisconnect - Total Records read for Region : session , BucketId - 112 , Total Records read : 1 ,
301+
04:09:13.786 [EventProducer-2] INFO redisconnect - Completed Results For bucket : 112
302+
04:09:13.791 [EventProducer-2] WARN startup - Instance: 30@virag-cdc could not cancel executor thread future for JobId: session-4
303+
04:09:13.792 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully cancelled heartbeat for JobId: session-4
304+
04:09:13.792 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully stopped replication pipeline for JobId: session-4
305+
04:09:13.792 [EventProducer-2] INFO startup - Instance: 30@virag-cdc now owns 0 job(s) from its 2 max allowable capacity
306+
04:09:13.792 [EventProducer-2] INFO startup - Instance: 30@virag-cdc successfully stopped JobId: session-4 and added it to {connect}:jobs:stopped
307+
....
308+
```
309+
310+
</p>
311+
</details>
312+
313+
<details><summary><b>Query for the above inserted record in Redis (target)</b></summary>
314+
<p>
315+
316+
```bash
317+
demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000'
318+
319+
```
320+
321+
</p>
322+
</details>
323+
324+
----
325+
104326
### CDC Steps
105327
<details><summary><b>Stage pre-configured cdc job</b></summary>
106328
<p>

‎connectors/gemfire/demo/extlib/connector-gemfire-fn-0.8.0.jar

-109 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)