[CDCSDK] Introduce a boolean flag to enable / disable dynamic table addition #22406
Assignees
Labels
area/cdcsdk
CDC SDK
jira-originated
kind/enhancement
This is an enhancement of an existing feature
priority/high
High Priority
Comments
yugabyte-ci
added
area/cdcsdk
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
May 30, 2024
…dition in virtual wal Summary: This diff introduces a boolean runtime flag called `cdcsdk_enable_dynamic_table_addition`, which can be used to enable / disable dynamic tables addition in replication slot consumption model. The default value of this flag is false, keeping dynamic table addition disabled by default. In order to implement runtime enabling and disabling the dynamic table addition, a new filed called `last_decided_pub_refresh_time` has been introduced in the data map for the slot entry in cdc_state table. This field holds the commit time of the last pub refresh record pushed into the pub refresh queue, along with the value of the boolean flag corresponding to the record. In other words, this holds the last decided pub refresh time along with the decision whether to perform pub refresh at that time or not. Inorder to protect LSN determinism, publication refresh will be performed at same points in time across restarts. To ensure this, the field `pub_refresh_times` will contain the list of hybrid times at which the publication was refreshed. This list will be trimmed as and when we get suitable acknowledgements. The `last_decided_pub_refresh_time` will give us the threshold upto which the pub refresh times have been already decided. In case of restarts, the publication should not be refreshed at times other than `pub_refresh_times` which are less than or equal to `last_decided_pub_refresh_time`. From the implementation perspective, the pub refresh record message will be populated with a dummy transaction id if a publication refresh is to be performed corresponding to the pub refresh record. The decision to populate transaction id in a particular pub refresh record will be made based on the value of cdcsdk_enable_dynamic_table_addition` at the time when the record is pushed to the pub refresh queue for the very first time. Jira: DB-11309 Test Plan: Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.*|.*ReplicationSlot.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestDynamicTablesSwitch Reviewers: asrinivasan, siddharth.shah, stiwary, skumar Reviewed By: asrinivasan Subscribers: ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35339
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
May 30, 2024
…dynamic tables addition in virtual wal Summary: Original commit: be5e8f1 / D35339 This diff introduces a boolean runtime flag called `cdcsdk_enable_dynamic_table_addition`, which can be used to enable / disable dynamic tables addition in replication slot consumption model. The default value of this flag is false, keeping dynamic table addition disabled by default. In order to implement runtime enabling and disabling the dynamic table addition, a new filed called `last_decided_pub_refresh_time` has been introduced in the data map for the slot entry in cdc_state table. This field holds the commit time of the last pub refresh record pushed into the pub refresh queue, along with the value of the boolean flag corresponding to the record. In other words, this holds the last decided pub refresh time along with the decision whether to perform pub refresh at that time or not. Inorder to protect LSN determinism, publication refresh will be performed at same points in time across restarts. To ensure this, the field `pub_refresh_times` will contain the list of hybrid times at which the publication was refreshed. This list will be trimmed as and when we get suitable acknowledgements. The `last_decided_pub_refresh_time` will give us the threshold upto which the pub refresh times have been already decided. In case of restarts, the publication should not be refreshed at times other than `pub_refresh_times` which are less than or equal to `last_decided_pub_refresh_time`. From the implementation perspective, the pub refresh record message will be populated with a dummy transaction id if a publication refresh is to be performed corresponding to the pub refresh record. The decision to populate transaction id in a particular pub refresh record will be made based on the value of cdcsdk_enable_dynamic_table_addition` at the time when the record is pushed to the pub refresh queue for the very first time. #####Backport Description Minor merge conflicts were encountered in TestPgReplicationSlot. Jira: DB-11309 Test Plan: Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.*|.*ReplicationSlot.* ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestDynamicTablesSwitch Reviewers: asrinivasan, siddharth.shah, stiwary, skumar Reviewed By: asrinivasan Subscribers: ycdcxcluster, ybase Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35420
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-11309
The text was updated successfully, but these errors were encountered: