Skip to content

Commit d09b11f

Browse files
viragtripathiviragtripathi
committed
demo updates
1 parent edc192f commit d09b11f

23 files changed

+779
-11
lines changed

‎README.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -149,11 +149,12 @@ redis-connect\bin> redisconnect.bat start
149149

150150
<table>
151151
<tr>
152-
<td><a href="/examples/db2/demo/README.md"><img src="/images/icons/DB2 Icon.png" style="float: center;" width="125" height="100"/></a></td>
153-
<td><a href="/examples/mysql/demo/README.md"><img src="/images/icons/MySQL Icon.png" style="float: center;" width="125" height="100"/></a></td>
154-
<td><a href="/examples/oracle/demo/README.md"><img src="/images/icons/Oracle Icon.png" style="float: center;" width="125" height="100"/></a></td>
155-
<td><a href="/examples/postgres/demo/README.md"><img src="/images/icons/Postgres Icon.png" style="float: center;" width="125" height="100"/></a></td>
156-
<td><a href="/examples/mssql/demo/README.md"><img src="/images/icons/SQL Server Icon.png" style="float: center;" width="125" height="100"/></a></td>
152+
<td><a href="/examples/db2/demo/README.md"><img src="/images/icons/DB2 Icon.png" style="float: center;" width="125" height="100"/></a></td>
153+
<td><a href="/examples/mongodb/demo/README.md"><img src="/images/icons/MongoDB Icon.png" style="float: center;" width="125" height="100"/></a></td>
154+
<td><a href="/examples/mysql/demo/README.md"><img src="/images/icons/MySQL Icon.png" style="float: center;" width="125" height="100"/></a></td>
155+
<td><a href="/examples/oracle/demo/README.md"><img src="/images/icons/Oracle Icon.png" style="float: center;" width="125" height="100"/></a></td>
156+
<td><a href="/examples/postgres/demo/README.md"><img src="/images/icons/Postgres Icon.png" style="float: center;" width="125" height="100"/></a></td>
157+
<td><a href="/examples/mssql/demo/README.md"><img src="/images/icons/SQL Server Icon.png" style="float: center;" width="125" height="100"/></a></td>
157158
<td><a href="/examples/vertica/demo/README.md"><img src="/images/icons/Vertica Icon.png" style="float: center;" width="125" height="100"/></a></td>
158159
</tr>
159160
</table>

‎examples/db2/demo/config/jobmanager.properties

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
#job.claim.attempt.interval=5000
1717
#job.claim.batch.size.per.attempt=4
1818
#job.claim.max.capacity=4
19-
#job.claim.heartbeat.lease.renewal.interval=5000
19+
#job.claim.heartbeat.lease.renewal.interval=10000
2020

2121
##### REST properties
2222
#rest.api.enabled=true

‎examples/mongodb/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
## Setting up MongoDB (Source)
2+
3+
Please see <a href="https://debezium.io/documentation/reference/stable/connectors/mongodb.html#setting-up-mongodb" target="_blank">MongoDB Setup</a> for reference.
4+
5+
## Setting up Redis Enterprise Databases (Target)
6+
7+
Before using Redis Connect to capture the changes committed on SQL Server into Redis Enterprise Database, first create a database for the metadata management and metrics provided by Redis Connect by creating a database with [RedisTimeSeries](https://redislabs.com/modules/redis-timeseries/) module enabled, see [Create Redis Enterprise Database](https://docs.redislabs.com/latest/rs/administering/creating-databases/#creating-a-new-redis-database) for reference. Then, create (or use an existing) another Redis Enterprise database (Target) to store the changes coming from SQL Server. Additionally, you can enable [RediSearch 2.0](https://redislabs.com/blog/introducing-redisearch-2-0/) module on the target database to enable secondary index with full-text search capabilities on the existing hashes where SQL Server changed events are being written at then [create an index, and start querying](https://oss.redislabs.com/redisearch/Commands/) the document in hashes.
8+
9+
| ℹ️ |
10+
|:--------------------------------------------|
11+
| Docker demo: Follow the [Docker demo](demo) |
12+
| K8s Setup: Follow the [k8s-docs](k8s-docs) |

‎examples/mongodb/demo/README.md

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Demo Outline
2+
:white_check_mark: Setup MongoDB database (Source)<br>
3+
:white_check_mark: Setup and start Redis Enterprise database (Target)<br>
4+
:white_check_mark: Setup and start Redis Connect<br>
5+
:white_check_mark: Perform Initial load and CDC with Redis Connect<br>
6+
7+
# Prerequisites
8+
Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed.
9+
<br>Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect Connector instances deployed on separate hosts.</br>
10+
<br>Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts.
11+
i.e.</br>
12+
13+
```bash
14+
wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
15+
mkdir -p redis-connect/demo && \
16+
mkdir -p redis-connect/k8s-docs && \
17+
unzip main.zip "redis-connect-dist-main/examples/mongodb/*" -d redis-connect && \
18+
cp -R redis-connect/redis-connect-dist-main/examples/mongodb/demo/* redis-connect/demo && \
19+
cp -R redis-connect/redis-connect-dist-main/examples/mongodb/k8s-docs/* redis-connect/k8s-docs && \
20+
rm -rf main.zip redis-connect/redis-connect-dist-main && \
21+
cd redis-connect && \
22+
chmod a+x demo/*.sh && \
23+
cd demo
24+
```
25+
26+
## Setup MongoDB database (Source)
27+
28+
https://www.mongodb.com/atlas/database
29+
30+
## Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target)
31+
<br>Execute [setup_re.sh](setup_re.sh)</br>
32+
```bash
33+
demo$ ./setup_re.sh
34+
```
35+
**NOTE**
36+
37+
The above script will create a 1-node Redis Enterprise cluster in a docker container, [Create a target database with RediSearch module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a job management and metrics database with RedisTimeSeries module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a RediSearch index for emp Hash](https://redislabs.com/blog/getting-started-with-redisearch-2-0/), [Start a docker instance of grafana with Redis Data Source](https://redisgrafana.github.io/) and [Start an instance of RedisInsight](https://docs.redislabs.com/latest/ri/installing/install-docker/).
38+
39+
## Start Redis Connect
40+
41+
<details><summary>Review options by running Redis Connect docker container </summary>
42+
<p>
43+
44+
```bash
45+
demo$ docker run \
46+
-it --rm --privileged=true \
47+
--name redis-connect-$(hostname) \
48+
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
49+
-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \
50+
--net host \
51+
redislabs/redis-connect
52+
```
53+
54+
</p>
55+
</details>
56+
57+
<details><summary>Expected output:</summary>
58+
<p>
59+
60+
```bash
61+
-------------------------------
62+
Redis Connect startup script.
63+
*******************************
64+
Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf
65+
Example environment variables and volume mapping for docker based deployments
66+
-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL]
67+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL]
68+
-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL]
69+
-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL]
70+
-v <HOST_PATH_TO_JOB_MANAGER_PROPERTIES>:/opt/redislabs/redis-connect/config
71+
-v <HOST_PATH_TO_CREDENTIALS>:/opt/redislabs/redis-connect/config/samples/credentials
72+
-v <HOST_PATH_TO_EXTLIB>:/opt/redislabs/redis-connect/extlib [OPTIONAL]
73+
-p 8282:8282
74+
75+
Usage: [-h|cli|start]
76+
options:
77+
-h: Print this help message and exit.
78+
-v: Print version.
79+
cli: init Redis Connect CLI
80+
start: init Redis Connect Instance (Cluster Member)
81+
-------------------------------
82+
```
83+
84+
</p>
85+
</details>
86+
87+
<details><summary><b>Start Redis Connect Instance</b></summary>
88+
<p>
89+
90+
```bash
91+
docker run \
92+
-it --rm --privileged=true \
93+
--name redis-connect-$(hostname) \
94+
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
95+
-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \
96+
--net host \
97+
redislabs/redis-connect start
98+
```
99+
100+
</p>
101+
</details>
102+
103+
<details><summary>Expected output:</summary>
104+
<p>
105+
106+
```bash
107+
108+
```
109+
110+
</p>
111+
</details>
112+
113+
**Open browser to access Swagger UI -** [http://localhost:8282/swagger-ui/index.html]()
114+
<br>_For quick start, use '**cdc_job**' as **jobName**_
115+
<br><br><img src="/images/quick-start/Redis Connect Swagger Front Page.jpg" style="float: right;" width = 700px height = 425px/>
116+
117+
**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`
118+
<br>_For quick start, use the sample `cdc-job.json` configuration:_ <a href="/examples/mongodb/demo/config/samples/payloads/cdc-job.json">SQL Server</a>
119+
<br><br><img src="/images/quick-start/Redis Connect Save Job Config.png" style="float: right;" width = 700px height = 375px/>
120+
<br>
121+
122+
**Or Use `curl` to create the `cdc-job` configuration** <br>
123+
`demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"`
124+
125+
-------------------------------
126+
127+
### Initial Loader Steps
128+
129+
**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}`
130+
<br>Use '**load**' as _**jobType**_
131+
<br><br><img src="/images/quick-start/Redis Connect Start Job.png" style="float: right;" width = 700px height = 375px/>
132+
133+
**Or Use `curl` to start the initial load for `cdc-job`** <br>
134+
`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/load" -H "accept: */*"`
135+
136+
<details><summary><b>Query for the above inserted record in Redis (target)</b></summary>
137+
<p>
138+
139+
```bash
140+
demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 ft.search idx_emp "@empno:[1 11]"'
141+
142+
```
143+
144+
</p>
145+
</details>
146+
147+
-------------------------------
148+
149+
### CDC Steps
150+
151+
**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}`
152+
<br>Use '**stream**' as _**jobType**_
153+
<br><br><img src="/images/quick-start/Redis Connect Start Job.png" style="float: right;" width = 700px height = 375px/>
154+
155+
**Or Use `curl` to start the stream for `cdc-job`** <br>
156+
`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/stream" -H "accept: */*"`
157+
158+
**Confirm Job Claim -** `/connect/api/vi/jobs/claim/{jobStatus}`
159+
<br>_For quick start, use '**all**' as **jobStatus**_
160+
<br><br><img src="/images/quick-start/Redis Connect Get Claims.png" style="float: right;" width = 700px height = 250px/>
161+
162+
**Or Use `curl` to query the `cdc-job` status** <br>
163+
`demo$ curl -X GET "http://localhost:8282/connect/api/v1/cluster/jobs/claim/all" -H "accept: */*"`
164+
165+
Expected output: `[{"jobId":"{connect}:job:cdc-job","jobName":"cdc-job","jobStatus":"CLAIMED","jobOwner":"30@virag-cdc","jobType":"STREAM"}]`
166+
167+
<b>INSERT/UPDATE a document in MongoDB (source)</b>
168+
<p>
169+
170+
<details><summary><b>Query for the above inserted record in Redis (target)</b></summary>
171+
<p>
172+
173+
```bash
174+
demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 ft.search idx_emp "@empno:[1002 1002]"'
175+
176+
```
177+
178+
</p>
179+
</details>

‎examples/mongodb/demo/config/jobmanager.properties

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
##################################################################################################
2+
##### Properties to start a Redis Connect cluster. Default values are commented out. #####
3+
##################################################################################################
4+
5+
##### Cluster properties
6+
#cluster.name=default
7+
#cluster.leader.heartbeat.lease.renewal.interval=5000
8+
#cluster.election.attempt.interval=5000
9+
#cluster.timeseries.metrics.enabled=false
10+
11+
##### Job Manager Services properties
12+
#job.manager.services.enabled=true
13+
#job.manager.services.threadpool.size=2
14+
#job.reap.attempt.interval=7000
15+
#job.claim.service.enabled=true
16+
#job.claim.attempt.interval=5000
17+
#job.claim.batch.size.per.attempt=4
18+
#job.claim.max.capacity=4
19+
#job.claim.heartbeat.lease.renewal.interval=10000
20+
21+
##### REST properties
22+
#rest.api.enabled=true
23+
#rest.api.port=8282
24+
25+
##### Job Manager Database properties
26+
redis.connection.url=redis://127.0.0.1:14001
27+
#redis.connection.sslEnabled=false
28+
#redis.truststore=<path_to_JKS>
29+
#redis.keystore=<path_to_PKCS>
30+
#redis.connection.insecure=false
31+
#redis.connection.timeout.duration=5
32+
#redis.connection.auto.reconnect=true
33+
#redis.connection.suspend.reconnect.on.protocol.failure=true
34+
35+
##### Credentials properties
36+
credentials.file.path=/opt/redislabs/redis-connect/config/samples/credentials
37+
#credentials.rotation.event.listener.enabled=false
38+
#credentials.rotation.event.listener.interval=60000

‎examples/mongodb/demo/config/logback-cli.xml

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
<configuration debug="false" scan="true" scanPeriod="15 seconds">
2+
3+
<property name="LOG_REDIS_CONNECT_PATH" value="logs/redis-connect"/>
4+
<property name="LOG_REDIS_CONNECT_MANAGER_PATH" value="logs/redis-connect-manager.log"/>
5+
<property name="LOG_REDIS_CONNECT_HEARTBEAT_PATH" value="logs/redis-connect-heartbeat.log"/>
6+
7+
<appender name="REDIS_CONNECT_HEARTBEAT" class="ch.qos.logback.core.rolling.RollingFileAppender">
8+
<file>${LOG_REDIS_CONNECT_HEARTBEAT_PATH}</file>
9+
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
10+
<fileNamePattern>logs/archived/redis-connect-heartbeat.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
11+
<!-- each archived file, size max 10MB -->
12+
<maxFileSize>10MB</maxFileSize>
13+
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
14+
<totalSizeCap>20GB</totalSizeCap>
15+
<!-- 60 days to keep -->
16+
<maxHistory>60</maxHistory>
17+
</rollingPolicy>
18+
<encoder>
19+
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
20+
</encoder>
21+
</appender>
22+
<appender name="REDIS_CONNECT_MANAGER" class="ch.qos.logback.core.rolling.RollingFileAppender">
23+
<file>${LOG_REDIS_CONNECT_MANAGER_PATH}</file>
24+
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
25+
<fileNamePattern>logs/archived/redis-connect-manager.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
26+
<!-- each archived file, size max 10MB -->
27+
<maxFileSize>10MB</maxFileSize>
28+
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
29+
<totalSizeCap>20GB</totalSizeCap>
30+
<!-- 60 days to keep -->
31+
<maxHistory>60</maxHistory>
32+
</rollingPolicy>
33+
<encoder>
34+
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
35+
</encoder>
36+
</appender>
37+
<appender name="REDIS_CONNECT" class="ch.qos.logback.core.rolling.RollingFileAppender">
38+
<file>${LOG_REDIS_CONNECT_PATH}-${PID}.log</file>
39+
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
40+
<fileNamePattern>logs/archived/redis-connect-${PID}.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
41+
<!-- each archived file, size max 10MB -->
42+
<maxFileSize>10MB</maxFileSize>
43+
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
44+
<totalSizeCap>20GB</totalSizeCap>
45+
<!-- 60 days to keep -->
46+
<maxHistory>60</maxHistory>
47+
</rollingPolicy>
48+
<encoder>
49+
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
50+
</encoder>
51+
</appender>
52+
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
53+
<encoder>
54+
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
55+
</encoder>
56+
</appender>
57+
58+
<logger name="redis-connect-heartbeat" level="OFF" additivity="false">
59+
<appender-ref ref="REDIS_CONNECT_HEARTBEAT"/>
60+
<appender-ref ref="CONSOLE" />
61+
</logger>
62+
<logger name="redis-connect-manager" level="OFF" additivity="false">
63+
<appender-ref ref="REDIS_CONNECT_MANAGER"/>
64+
<appender-ref ref="CONSOLE" />
65+
</logger>
66+
<logger name="redis-connect" level="OFF" additivity="false">
67+
<appender-ref ref="REDIS_CONNECT"/>
68+
<appender-ref ref="CONSOLE" />
69+
</logger>
70+
<logger name="io.netty" level="OFF" additivity="false">
71+
<appender-ref ref="REDIS_CONNECT"/>
72+
<appender-ref ref="CONSOLE" />
73+
</logger>
74+
<logger name="io.lettuce" level="OFF" additivity="false">
75+
<appender-ref ref="REDIS_CONNECT"/>
76+
<appender-ref ref="CONSOLE" />
77+
</logger>
78+
<logger name="com.zaxxer" level="OFF" additivity="false">
79+
<appender-ref ref="REDIS_CONNECT"/>
80+
<appender-ref ref="CONSOLE"/>
81+
</logger>
82+
<logger name="io.debezium" level="OFF" additivity="false">
83+
<appender-ref ref="REDIS_CONNECT"/>
84+
<appender-ref ref="CONSOLE"/>
85+
</logger>
86+
<logger name="org.apache.kafka" level="OFF" additivity="false">
87+
<appender-ref ref="REDIS_CONNECT"/>
88+
<appender-ref ref="CONSOLE"/>
89+
</logger>
90+
<logger name="org.springframework" level="OFF" additivity="false">
91+
<appender-ref ref="REDIS_CONNECT"/>
92+
<appender-ref ref="CONSOLE"/>
93+
</logger>
94+
<logger name="springfox" level="OFF" additivity="false">
95+
<appender-ref ref="REDIS_CONNECT"/>
96+
<appender-ref ref="CONSOLE"/>
97+
</logger>
98+
<logger name="org" level="OFF" additivity="false">
99+
<appender-ref ref="REDIS_CONNECT"/>
100+
<appender-ref ref="CONSOLE"/>
101+
</logger>
102+
<logger name="Validator" level="OFF" additivity="false"/>
103+
<logger name="_org.springframework.web.servlet.HandlerMapping" level="OFF" additivity="false"/>
104+
<logger name="com.redis.connect.cluster.RedisConnectSpringBootApplication" level="OFF" additivity="false"/>
105+
<logger name="reactor.util" level="OFF" additivity="false"/>
106+
107+
<root>
108+
<appender-ref ref="REDIS_CONNECT"/>
109+
<appender-ref ref="REDIS_CONNECT_MANAGER"/>
110+
<appender-ref ref="REDIS_CONNECT_HEARTBEAT"/>
111+
</root>
112+
113+
</configuration>

0 commit comments

Comments
 (0)