Skip to content

Add flag to disable init sync #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 20, 2025
Merged

Add flag to disable init sync #8

merged 1 commit into from
Jan 20, 2025

Conversation

divanshu-aggarwal
Copy link

No description provided.

codeon
codeon approved these changes Jan 20, 2025
View reviewed changes
source/src/main/java/com/trustpilot/connector/dynamodb/DynamoDBSourceConnectorConfig.java
@@ -223,16 +228,25 @@ public static ConfigDef baseConfigDef() {
SRC_INIT_SYNC_DELAY_DEFAULT,
ConfigDef.Importance.LOW,
SRC_INIT_SYNC_DELAY_DOC,
CONNECTOR_GROUP, 2,
CONNECTOR_GROUP, 4,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this imply?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it ordering of params?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, ordering of params in a group. higher order takes more priority. if the same config, higher order is preferred over lower order.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameter of apache config.

@codeon
Copy link

codeon commented Jan 20, 2025

how do we test this?

@divanshu-aggarwal
Copy link
Author

Checkpointing happening:

`{
    "Items": [
        {
            "checkpoint": {
                "S": "000000000000000000652"
            },
            "parentShardId": {
                "SS": [
                    "shardId-00000001735044449263-ee924fd8"
                ]
            },
            "leaseCounter": {
                "N": "388"
            },
            "ownerSwitchesSinceCheckpoint": {
                "N": "0"
            },
            "checkpointSubSequenceNumber": {
                "N": "0"
            },
            "leaseKey": {
                "S": "shardId-00000001737361835982-5b944b07"
            },
            "leaseOwner": {
                "S": "datalake-KCL-test-dynamodb-connector-cdcv2-worker-task-1"
            }
        }`

Tested in local.

@divanshu-aggarwal divanshu-aggarwal merged commit 11a0d79 into old-doc-and-topic-map Jan 20, 2025
@divanshu-aggarwal divanshu-aggarwal deleted the Divanshu/Disable_init_sync branch January 20, 2025 10:19
dariogavranovic added a commit that referenced this pull request Mar 20, 2025
@dariogavranovic @Nandy-006
@hardik-devrev @divanshu-aggarwal
feat: merge all devrev features into main (#11)
255dca5 
* Add the oldDocument in field value when present

* Add EventID and Key inside value

* Make logs debug to reduce noise (#3)

* move some info logs to debug

* make info

* use jackson instead of gson (#5)

* delay log warning to 60 minutes (#6)

* changes to version (#7)

* Add flag to disable init sync (#8)

* add initial value (#9)

---------

Co-authored-by: Rabi Shanker Guha <[email protected]>
Co-authored-by: Nandan H R <[email protected]>
Co-authored-by: hardik-devrev <[email protected]>
Co-authored-by: Divanshu Aggarwal <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@codeon codeon codeon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@divanshu-aggarwal @codeon