Skip to content

add version in datalake table name #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

divanshu-aggarwal
Copy link

use this config to create table with suffix cdcv2

datalake-KCL-test-dynamodb-connector-cdcv2

{
    "name": "dynamodb-source",
    "config": {
        "connector.class": "com.trustpilot.connector.dynamodb.DynamoDBSourceConnector",
        "tasks.max": "1",
        "aws.region": "fakeRegion",
        "dynamodb.table.whitelist": "test-dynamodb-connector",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "key.converter.schemas.enable": "false",
        "value.converter.schemas.enable": "false",
        "aws.credentials.provider.class": "com.amazonaws.auth.profile.ProfileCredentialsProvider",
        "aws.credentials.provider.profile": "dynamo",
        "aws.access.key.id": "dummy",
        "aws.secret.key": "dummy",  
        "dynamodb.service.endpoint": "http://localhost:8000",
        "dynamodb.table.version": "cdcv2",
        "kafka.topic.prefix": "dynamodb-",
        "init.sync.delay.period": "60"
    }
}

@hardik-devrev
Copy link

@Nandy-006 Can you take a look as well?

Copy link
Collaborator

@Nandy-006 Nandy-006 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@divanshu-aggarwal divanshu-aggarwal merged commit ea3d032 into old-doc-and-topic-map Jan 3, 2025
@divanshu-aggarwal divanshu-aggarwal deleted the Divanshu/Use_version_as_connector branch January 3, 2025 06:39
dariogavranovic added a commit that referenced this pull request Mar 20, 2025
* Add the oldDocument in field value when present

* Add EventID and Key inside value

* Make logs debug to reduce noise (#3)

* move some info logs to debug

* make info

* use jackson instead of gson (#5)

* delay log warning to 60 minutes (#6)

* changes to version (#7)

* Add flag to disable init sync (#8)

* add initial value (#9)

---------

Co-authored-by: Rabi Shanker Guha <[email protected]>
Co-authored-by: Nandan H R <[email protected]>
Co-authored-by: hardik-devrev <[email protected]>
Co-authored-by: Divanshu Aggarwal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants