Skip to content

Commit f78387e

Browse files
viragtripathiviragtripathi
committed
MySQL demo initial updates
1 parent 6f4c1bb commit f78387e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+893
-58
lines changed

‎connectors/db2/demo/config/samples/loader/JobConfig.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,28 @@
11
jobId: ${jobId}
22
producerConfig:
3-
producerId: RDB_BATCH_EVENT_PRODUCER
3+
producerId: RDB_INITIAL_LOAD_PRODUCER
44
connectionId: RDBConnection
55
batchSize: 5000
66
startRecord: ${startRecord}
77
endRecord: ${endRecord}
88
loadQuery: ${loadQuery}
99
table: ${table}
1010
pipelineConfig:
11-
bufferSize: 1024
1211
eventTranslator: "RDB_RS_TO_OP_TRANSLATOR"
1312
checkpointConfig:
14-
providerId: HASH_CHECKPOINT_READER
13+
providerId: REDIS_HASH_CHECKPOINT_READER
1514
connectionId: targetConnection
1615
checkpoint: "${jobId}"
1716
stages:
1817
HashWriteStage:
1918
handlerId: REDIS_HASH_WRITER
2019
connectionId: targetConnection
20+
metricsEnabled: false
2121
prependTableNameToKeys: true
2222
deleteOnKeyUpdate: true
2323
async: true
2424
CheckpointStage:
25-
handlerId: REDIS_OP_CP_WRITER
25+
handlerId: REDIS_HASH_CHECKPOINT_WRITER
2626
connectionId: targetConnection
2727
metricEnabled: false
2828
async: true

‎connectors/db2/demo/config/samples/loader/JobManager.yml

Lines changed: 2 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,3 @@
11
connectionId: jobConfigConnection
2-
jobTypeId: empLoader
3-
jobStream: loaderJobStream
4-
jobConfigSet: loaderJobConfigs
5-
completedJobsSet: loaderCompletedJobs
6-
initialDelay: 10000
7-
numManagementThreads: 2
8-
heartBeatConfig:
9-
key: hb-loaderJobManager
10-
expiry: 30000
11-
jobHeartBeatKeyPrefix: "hb-job:"
12-
jobHeartbeatCheckInterval: 45000
13-
jobClaimerConfig:
14-
initialDelay: 10000
15-
claimInterval: 30000
16-
heartBeatConfig:
17-
key: "hb-job:"
18-
expiry: 30000
19-
maxNumberOfJobs: 5
20-
consumerGroup: loaderJobGroup
21-
batchSize: 3
2+
metricsReporter:
3+
- REDIS_TS_METRICS_REPORTER

‎connectors/db2/demo/config/samples/loader/Setup.yml

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,5 @@
11
connectionId: jobConfigConnection
22
job:
3-
stream: loaderJobStream
4-
configSet: loaderJobConfigs
5-
completedJobsSet: loadercompletedJobs
6-
consumerGroup: loaderJobGroup
73
jobConfig:
8-
- name: batchtaskcreator
4+
- name: initial_load
95
config: TaskCreator.yml

‎connectors/db2/demo/config/samples/loader/TaskCreator.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ producerConfig:
55
connectionId: RDBConnection
66
jobs:
77
- table: jjd47182.emp # schema.table
8-
jobPrefix: empLoader-
98
numJobs: 5
109
config: JobConfig.yml
1110
loadQuery: "select * from (select ROW_NUMBER() OVER (ORDER BY empno) AS RowIndex, * from jjd47182.emp) AS emps Where emps.RowIndex >= ${start} and emps.RowIndex <= ${end}"

‎connectors/db2/demo/config/samples/loader/env.yml

Lines changed: 18 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
connections:
2-
jobConfigConnection:
3-
redisUrl: redis://${REDISCONNECT_TARGET_USERNAME}:${REDISCONNECT_TARGET_PASSWORD}@127.0.0.1:14001
4-
targetConnection:
5-
redisUrl: redis://${REDISCONNECT_TARGET_USERNAME}:${REDISCONNECT_TARGET_PASSWORD}@127.0.0.1:14000
6-
metricsConnection:
7-
redisUrl: redis://${REDISCONNECT_TARGET_USERNAME}:${REDISCONNECT_TARGET_PASSWORD}@127.0.0.1:14001
8-
RDBConnection:
9-
database:
10-
name: loaderdb #database pool name
11-
db: BLUDB #database
12-
hostname: dashdb-txn-sbox-yp-dal09-11.services.dal.bluemix.net
13-
port: 50000
14-
username: ${REDISCONNECT_SOURCE_USERNAME}
15-
password: ${REDISCONNECT_SOURCE_PASSWORD}
16-
jdbcUrl: "jdbc:db2://dashdb-txn-sbox-yp-dal09-11.services.dal.bluemix.net:50000/BLUDB"
17-
maximumPoolSize: 10
18-
minimumIdle: 2
2+
- id: jobConfigConnection
3+
type: Redis
4+
url: redis://${REDISCONNECT_TARGET_USERNAME}:${REDISCONNECT_TARGET_PASSWORD}@127.0.0.1:14001
5+
- id: targetConnection
6+
type: Redis
7+
url: redis://${REDISCONNECT_TARGET_USERNAME}:${REDISCONNECT_TARGET_PASSWORD}@127.0.0.1:14000
8+
- id: metricsConnection
9+
type: Redis
10+
url: redis://${REDISCONNECT_TARGET_USERNAME}:${REDISCONNECT_TARGET_PASSWORD}@127.0.0.1:14001
11+
- id: RDBConnection
12+
type: RDB
13+
name: RedisConnect #database pool name
14+
database: BLUDB #database
15+
url: "jdbc:db2://dashdb-txn-sbox-yp-dal09-11.services.dal.bluemix.net:50000/BLUDB"
16+
host: dashdb-txn-sbox-yp-dal09-11.services.dal.bluemix.net
17+
port: 50000
18+
username: ${REDISCONNECT_SOURCE_USERNAME}
19+
password: ${REDISCONNECT_SOURCE_PASSWORD}

‎connectors/mssql/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -418,8 +418,8 @@ cli: starts redis-connect-cli.
418418
stage: clean and stage redis database with cdc or initial loader job configurations.
419419
start: start Redis Connect instance with provided cdc or initial loader job configurations.
420420
-------------------------------
421-
422421
```
422+
423423
</p>
424424
</details>
425425

‎connectors/mssql/demo/README.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -97,9 +97,6 @@ The above script will create a 1-node Redis Enterprise cluster in a docker conta
9797

9898
---
9999

100-
## Setup RedisInsight
101-
Open a web browser and navigate to http://127.0.0.1:18001/ and [add both job config & metrics and target Redis databbases](https://docs.redislabs.com/latest/ri/using-redisinsight/add-instance/) (use redisUrl's from env.yml) to RedisInsight UI. Use Redis database endpoints for job management and target databasees, use the `Internal IP` instead of `127.0.0.1` on cloud machines.
102-
103100
## Start Redis Connect SQL Server Connector
104101

105102
<details><summary>Run Redis Connect SQL Server Connector docker container to see all the options</summary>

‎connectors/mssql/demo/config/samples/loader/Setup.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,3 @@ job:
33
jobConfig:
44
- name: initial_load
55
config: TaskCreator.yml
6-
7-

‎connectors/mssql/demo/config/samples/sqlserver/JobConfig.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@ producerConfig:
44
connectionId: RDBConnection
55
tables:
66
- dbo.emp #schema.table
7-
metricsKey: RedisConnect-emp
87
metricsEnabled: false
98
pipelineConfig:
109
eventTranslator: "${sourceValueTranslator}"

‎connectors/mssql/demo/config/samples/sqlserver/Setup.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ job:
99
labels:
1010
schema: dbo
1111
table: emp
12-
op: I
12+
op: C
1313
- key: "dbo:emp:U:Throughput"
1414
retentionInHours: 4
1515
labels:

‎connectors/mysql/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# redis-connect-mysql
2+
3+
redis-connect-mysql is a Redis Connect connector for capturing changes (INSERT, UPDATE and DELETE) from MySQL (source) and writing them to a Redis Enterprise database (Target). redis-connect-mysql cdc connector implementation is based on [Debezium](https://debezium.io/documentation/reference/stable/connectors/mysql.html), which is an open source distributed platform for change data capture.
4+
5+
<p>
6+
The first time redis-connect-mysql connects to a MySQL database, it reads a consistent snapshot of all of the schemas.
7+
When that snapshot is complete, the connector continuously streams the changes that were committed to MySQL and generates a corresponding insert, update or delete event.
8+
All of the events for each tables are recorded in a separate Redis data structure or module of your choice, where they can be easily consumed by applications and services.

0 commit comments

Comments
 (0)