How many rows of data can be copied ?

[sandeepraghu]

How many rows of data can be copied ?

[MichalisDBA]

Hi @sandeepraghu Theoretically unlimited. I have copied from a MSSQL to Azure MSSQL aprox 50million rows at no problem. Try to use the java binary instead of docker.

[sandeepraghu]

Hi @MichalisDBA can you please share the steps for setup java binary and run , i have tried but getting errors..

[MichalisDBA]

I have a Linux VM running AlmaLinux (RHEL Clone)

Let's say you a have a user sandeepraghu, the home dir path is:

User home path

/home/sandeepraghu

Install JAVA

sudo dnf install java-1.8.0-openjdk.x86_64

Create a directory

mkdir replicadb && cd replicadb

Download ReplicaDB

wget https://github.com/osalvador/ReplicaDB/releases/download/v0.15.1/ReplicaDB-0.15.1.tar.gz

Extract it

tar -xvzf ReplicaDB-0.15.1.tar.gz

Create a symlink for the replicadb binary

sudo ln -s /home/sandeepraghu/replicadb/bin/replicadb /usr/local/bin/replicadb

Add JAVA to to user environment

nano ~/.bashrc or vi ~/.bashrc

Write this somewhere inside .bashrc

### Export JAVA
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.402.b06-2.el9.x86_64/jre

The java-1.8.0-openjdk-1.8.0.402.b06-2.el9.x86_64 part may be different depending on the version you installed

Finally read the changes you made to bashrc or logoff from the terminal and login again.
source ~/.bashrc

Now you can run replicadb in whatever path you have. Test it by running:
replicadb --version

replicadb --version
INFO  ReplicaDB:63 Running ReplicaDB version: 0.15.1
INFO  ReplicaDB:66 Setting verbose mode INFO
INFO  ReplicaDB:54 Total process time: 13ms

[sandeepraghu]

Hi @MichalisDBA
sandeepraghuvanshi@Sandeeps-MacBook-Pro replicadb % replicadb --version
2024-03-15 18:37:41,540 INFO ReplicaDB:63 Running ReplicaDB version: 0.15.1
2024-03-15 18:37:41,544 INFO ReplicaDB:66 Setting verbose mode INFO
2024-03-15 18:37:41,544 INFO ReplicaDB:54 Total process time: 11ms
sandeepraghuvanshi@Sandeeps-MacBook-Pro replicadb %

successfully installed , now to run and replicate/copy data which command i should use..

[MichalisDBA]

Create the appropriate conf file and run

replicadb --options-file replicadb.conf

also read the docs are very comprehensive

https://osalvador.github.io/ReplicaDB/index.html#full-documentation

and try the configuration wizard

https://osalvador.github.io/ReplicaDB/wizard/index.html

[sandeepraghu]

Hi @MichalisDBA

yes it working fine and copying data
just one more thing can we also have a cron/schedule in this method and also how to run it for multiple tables my making multiple conf file..
please share the steps for this.
Thanks!!!

[MichalisDBA]

Yes you can create cron schedules.
Type cron -e and edit the cron file. Don't forget to add the
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.402.b06-2.el9.x86_64/jre
inside the cron file at the top so that cron background process find the java environment to use.

For multiple tables you have to create multiple .conf files for the time being.

[sandeepraghu]

HI @MichalisDBA
I have created multiple .conf files , can all these .conf files can be executed one by one with single command ?

[MichalisDBA]

Do something like this.
Go to the directory that you have all replicadb .conf files and run:

for file in *.conf; do replicadb --options-file "$file"; done

[sandeepraghu]

okay will do likw this.
Thanks @MichalisDBA for the help.

[sandeepraghu]

Hi @MichalisDBA
getting deadlock issue again and again , how can this be resolve.. ?

2024-03-16 12:25:53,142 ERROR SQLServerManager:113 Error while performing BulkCopy into dbo.Job
com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:258) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:104) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:26) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.doInsertBulk(SQLServerBulkCopy.java:1517) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.access$300(SQLServerBulkCopy.java:65) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy$1InsertBulk.doExecute(SQLServerBulkCopy.java:649) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2935) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.sendBulkLoadBCP(SQLServerBulkCopy.java:683) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:1581) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeResultSet(SQLServerBulkCopy.java:582) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:520) ~[mssql-jdbc-7.2.2.jre8.jar:?]
at org.replicadb.manager.SQLServerManager.insertDataToTable(SQLServerManager.java:108) [ReplicaDB-0.15.1.jar:0.15.1]
at org.replicadb.ReplicaTask.call(ReplicaTask.java:65) [ReplicaDB-0.15.1.jar:0.15.1]
at org.replicadb.ReplicaTask.call(ReplicaTask.java:15) [ReplicaDB-0.15.1.jar:0.15.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_401]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_401]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_401]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_401]
2024-03-16 12:25:53,157 ERROR ReplicaTask:69 ERROR in TaskId-0 inserting data to sink table: [Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.]

[MichalisDBA]

@sandeepraghu i think this does not have to do with replicadb.

Is your sink table being access by other processes or users? Do anyone else insert, update or delete data to the sink table while you run replicadb targeting the sink table?

[sandeepraghu]

Hi @MichalisDBA , please check the command is executing successful , also it showing correct total number of task process i.e.
2024-03-31 16:46:00,041 INFO ReplicaTask:67 A total of 82190 rows processed by task 0
2024-03-31 16:46:07,240 INFO ReplicaTask:67 A total of 111370 rows processed by task 3
2024-03-31 16:46:16,639 INFO ReplicaTask:67 A total of 122369 rows processed by task 2
2024-03-31 16:46:22,455 INFO ReplicaTask:67 A total of 151059 rows processed by task 1
but when i am seeing in my destination the total number is not exact to this , sometimes it is less and sometime it is greater , how to resolve this.
for your info: i am on MSSQL.

ubuntu@ip-172-16-1-106:~/replica_data/replicadb$ replicadb --options-file conf/tableWorkCenter.conf
2024-03-31 16:45:24,549 INFO ReplicaDB:63 Running ReplicaDB version: 0.15.1
2024-03-31 16:45:24,558 INFO ReplicaDB:66 Setting verbose mode INFO
2024-03-31 16:45:25,192 INFO SQLServerManager:278 Creating staging table with this command: SELECT * INTO dbo.Jobrepdb4410 FROM dbo.Job WHERE 0 = 1
2024-03-31 16:45:25,218 INFO SqlManager:373 Truncating sink table with this command: TRUNCATE TABLE dbo.Jobrepdb4410
2024-03-31 16:45:25,223 INFO ReplicaTask:35 Starting TaskId-2
2024-03-31 16:45:25,223 INFO ReplicaTask:35 Starting TaskId-0
2024-03-31 16:45:25,224 INFO ReplicaTask:35 Starting TaskId-1
2024-03-31 16:45:25,225 INFO ReplicaTask:35 Starting TaskId-3
2024-03-31 16:45:25,411 INFO SqlManager:128 TaskId-3: Executing SQL statement: SELECT * FROM dbo.Job where ABS(CHECKSUM(%% physloc %%)) % 4 = ?
2024-03-31 16:45:25,446 INFO SqlManager:128 TaskId-0: Executing SQL statement: SELECT * FROM dbo.Job where ABS(CHECKSUM(%% physloc %%)) % 4 = ?
2024-03-31 16:45:25,446 INFO SqlManager:128 TaskId-1: Executing SQL statement: SELECT * FROM dbo.Job where ABS(CHECKSUM(%% physloc %%)) % 4 = ?
2024-03-31 16:45:25,447 INFO SqlManager:128 TaskId-2: Executing SQL statement: SELECT * FROM dbo.Job where ABS(CHECKSUM(%% physloc %%)) % 4 = ?
2024-03-31 16:45:25,502 INFO SqlManager:148 TaskId-1: With args: 1,
2024-03-31 16:45:25,503 INFO SqlManager:148 TaskId-2: With args: 2,
2024-03-31 16:45:25,503 INFO SqlManager:148 TaskId-3: With args: 3,
2024-03-31 16:45:25,503 INFO SqlManager:148 TaskId-0: With args: 0,
2024-03-31 16:45:59,791 WARN ConnManager:188 Options source-columns and sink-columns are null, getting from Source ResultSetMetaData: Job,Sales_Rep,Customer,Ship_To,User_Values,Quote,Contact,Ship_Via,Terms,Tax_Code,Sales_Code,Top_Lvl_Job,Type,Order_Date,Status,Status_Date,Part_Number,Rev,Description,Ext_Description,Drawing,Build_To_Stock,Order_Quantity,Extra_Quantity,Pick_Quantity,Make_Quantity,Split_Quantity,Completed_Quantity,Shipped_Quantity,FG_Transfer_Qty,Returned_Quantity,In_Production_Quantity,Assembly_Level,Certs_Required,Time_And_Materials,Open_Operations,Scrap_Pct,Est_Scrap_Qty,Est_Rem_Hrs,Est_Total_Hrs,Est_Labor,Est_Material,Est_Service,Est_Labor_Burden,Est_Machine_Burden,Est_GA_Burden,Act_Revenue,Act_Scrap_Quantity,Act_Total_Hrs,Act_Labor,Act_Material,Act_Service,Act_Labor_Burden,Act_Machine_Burden,Act_GA_Burden,Priority,Unit_Price,Total_Price,Price_UofM,Currency_Conv_Rate,Trade_Currency,Fixed_Rate,Trade_Date,Commission_Pct,Customer_PO,Customer_PO_LN,Sched_End,Sched_Start,Quantity_Per,Profit_Pct,Labor_Markup_Pct,Mat_Markup_Pct,Serv_Markup_Pct,Labor_Burden_Markup_Pct,Machine_Burden_Markup_Pct,GA_Burden_Markup_Pct,Lead_Days,Profit_Markup,Prepaid_Amt,Split_To_Job,Note_Text,Comment,Last_Updated,Order_Unit,Price_Unit_Conv,Holder_ID,Source,Order_Taken_By,Plan_Modified,ObjectID,Released_Date,Prepaid_Tax_Amount,Prepaid_Trade_Amt,Last_Updated_By,CommissionIncluded
2024-03-31 16:46:00,041 INFO ReplicaTask:67 A total of 82190 rows processed by task 0
2024-03-31 16:46:07,240 INFO ReplicaTask:67 A total of 111370 rows processed by task 3
2024-03-31 16:46:16,639 INFO ReplicaTask:67 A total of 122369 rows processed by task 2
2024-03-31 16:46:22,455 INFO ReplicaTask:67 A total of 151059 rows processed by task 1
2024-03-31 16:46:22,456 INFO SQLServerManager:55 IF OBJECTPROPERTY(OBJECT_ID('dbo.Job'), 'TableHasIdentity') = 1 SET IDENTITY_INSERT dbo.Job ON
2024-03-31 16:46:25,089 INFO SqlManager:315 Getting PKs for schema: dbo and table: Job. Found.
2024-03-31 16:46:25,111 INFO SQLServerManager:346 Merging staging table and sink table with this command: MERGE INTO dbo.Job trg USING (SELECT Job,Sales_Rep,Customer,Ship_To,User_Values,Quote,Contact,Ship_Via,Terms,Tax_Code,Sales_Code,Top_Lvl_Job,Type,Order_Date,Status,Status_Date,Part_Number,Rev,Description,Ext_Description,Drawing,Build_To_Stock,Order_Quantity,Extra_Quantity,Pick_Quantity,Make_Quantity,Split_Quantity,Completed_Quantity,Shipped_Quantity,FG_Transfer_Qty,Returned_Quantity,In_Production_Quantity,Assembly_Level,Certs_Required,Time_And_Materials,Open_Operations,Scrap_Pct,Est_Scrap_Qty,Est_Rem_Hrs,Est_Total_Hrs,Est_Labor,Est_Material,Est_Service,Est_Labor_Burden,Est_Machine_Burden,Est_GA_Burden,Act_Revenue,Act_Scrap_Quantity,Act_Total_Hrs,Act_Labor,Act_Material,Act_Service,Act_Labor_Burden,Act_Machine_Burden,Act_GA_Burden,Priority,Unit_Price,Total_Price,Price_UofM,Currency_Conv_Rate,Trade_Currency,Fixed_Rate,Trade_Date,Commission_Pct,Customer_PO,Customer_PO_LN,Sched_End,Sched_Start,Quantity_Per,Profit_Pct,Labor_Markup_Pct,Mat_Markup_Pct,Serv_Markup_Pct,Labor_Burden_Markup_Pct,Machine_Burden_Markup_Pct,GA_Burden_Markup_Pct,Lead_Days,Profit_Markup,Prepaid_Amt,Split_To_Job,Note_Text,Comment,Last_Updated,Order_Unit,Price_Unit_Conv,Holder_ID,Source,Order_Taken_By,Plan_Modified,ObjectID,Released_Date,Prepaid_Tax_Amount,Prepaid_Trade_Amt,Last_Updated_By,CommissionIncluded FROM dbo.Jobrepdb4410 ) src ON (src.Job= trg.Job ) WHEN MATCHED THEN UPDATE SET trg.Sales_Rep = src.Sales_Rep, trg.Customer = src.Customer, trg.Ship_To = src.Ship_To, trg.User_Values = src.User_Values, trg.Quote = src.Quote, trg.Contact = src.Contact, trg.Ship_Via = src.Ship_Via, trg.Terms = src.Terms, trg.Tax_Code = src.Tax_Code, trg.Sales_Code = src.Sales_Code, trg.Top_Lvl_Job = src.Top_Lvl_Job, trg.Type = src.Type, trg.Order_Date = src.Order_Date, trg.Status = src.Status, trg.Status_Date = src.Status_Date, trg.Part_Number = src.Part_Number, trg.Rev = src.Rev, trg.Description = src.Description, trg.Ext_Description = src.Ext_Description, trg.Drawing = src.Drawing, trg.Build_To_Stock = src.Build_To_Stock, trg.Order_Quantity = src.Order_Quantity, trg.Extra_Quantity = src.Extra_Quantity, trg.Pick_Quantity = src.Pick_Quantity, trg.Make_Quantity = src.Make_Quantity, trg.Split_Quantity = src.Split_Quantity, trg.Completed_Quantity = src.Completed_Quantity, trg.Shipped_Quantity = src.Shipped_Quantity, trg.FG_Transfer_Qty = src.FG_Transfer_Qty, trg.Returned_Quantity = src.Returned_Quantity, trg.In_Production_Quantity = src.In_Production_Quantity, trg.Assembly_Level = src.Assembly_Level, trg.Certs_Required = src.Certs_Required, trg.Time_And_Materials = src.Time_And_Materials, trg.Open_Operations = src.Open_Operations, trg.Scrap_Pct = src.Scrap_Pct, trg.Est_Scrap_Qty = src.Est_Scrap_Qty, trg.Est_Rem_Hrs = src.Est_Rem_Hrs, trg.Est_Total_Hrs = src.Est_Total_Hrs, trg.Est_Labor = src.Est_Labor, trg.Est_Material = src.Est_Material, trg.Est_Service = src.Est_Service, trg.Est_Labor_Burden = src.Est_Labor_Burden, trg.Est_Machine_Burden = src.Est_Machine_Burden, trg.Est_GA_Burden = src.Est_GA_Burden, trg.Act_Revenue = src.Act_Revenue, trg.Act_Scrap_Quantity = src.Act_Scrap_Quantity, trg.Act_Total_Hrs = src.Act_Total_Hrs, trg.Act_Labor = src.Act_Labor, trg.Act_Material = src.Act_Material, trg.Act_Service = src.Act_Service, trg.Act_Labor_Burden = src.Act_Labor_Burden, trg.Act_Machine_Burden = src.Act_Machine_Burden, trg.Act_GA_Burden = src.Act_GA_Burden, trg.Priority = src.Priority, trg.Unit_Price = src.Unit_Price, trg.Total_Price = src.Total_Price, trg.Price_UofM = src.Price_UofM, trg.Currency_Conv_Rate = src.Currency_Conv_Rate, trg.Trade_Currency = src.Trade_Currency, trg.Fixed_Rate = src.Fixed_Rate, trg.Trade_Date = src.Trade_Date, trg.Commission_Pct = src.Commission_Pct, trg.Customer_PO = src.Customer_PO, trg.Customer_PO_LN = src.Customer_PO_LN, trg.Sched_End = src.Sched_End, trg.Sched_Start = src.Sched_Start, trg.Quantity_Per = src.Quantity_Per, trg.Profit_Pct = src.Profit_Pct, trg.Labor_Markup_Pct = src.Labor_Markup_Pct, trg.Mat_Markup_Pct = src.Mat_Markup_Pct, trg.Serv_Markup_Pct = src.Serv_Markup_Pct, trg.Labor_Burden_Markup_Pct = src.Labor_Burden_Markup_Pct, trg.Machine_Burden_Markup_Pct = src.Machine_Burden_Markup_Pct, trg.GA_Burden_Markup_Pct = src.GA_Burden_Markup_Pct, trg.Lead_Days = src.Lead_Days, trg.Profit_Markup = src.Profit_Markup, trg.Prepaid_Amt = src.Prepaid_Amt, trg.Split_To_Job = src.Split_To_Job, trg.Note_Text = src.Note_Text, trg.Comment = src.Comment, trg.Last_Updated = src.Last_Updated, trg.Order_Unit = src.Order_Unit, trg.Price_Unit_Conv = src.Price_Unit_Conv, trg.Holder_ID = src.Holder_ID, trg.Source = src.Source, trg.Order_Taken_By = src.Order_Taken_By, trg.Plan_Modified = src.Plan_Modified, trg.ObjectID = src.ObjectID, trg.Released_Date = src.Released_Date, trg.Prepaid_Tax_Amount = src.Prepaid_Tax_Amount, trg.Prepaid_Trade_Amt = src.Prepaid_Trade_Amt, trg.Last_Updated_By = src.Last_Updated_By, trg.CommissionIncluded = src.CommissionIncluded WHEN NOT MATCHED THEN INSERT ( Job,Sales_Rep,Customer,Ship_To,User_Values,Quote,Contact,Ship_Via,Terms,Tax_Code,Sales_Code,Top_Lvl_Job,Type,Order_Date,Status,Status_Date,Part_Number,Rev,Description,Ext_Description,Drawing,Build_To_Stock,Order_Quantity,Extra_Quantity,Pick_Quantity,Make_Quantity,Split_Quantity,Completed_Quantity,Shipped_Quantity,FG_Transfer_Qty,Returned_Quantity,In_Production_Quantity,Assembly_Level,Certs_Required,Time_And_Materials,Open_Operations,Scrap_Pct,Est_Scrap_Qty,Est_Rem_Hrs,Est_Total_Hrs,Est_Labor,Est_Material,Est_Service,Est_Labor_Burden,Est_Machine_Burden,Est_GA_Burden,Act_Revenue,Act_Scrap_Quantity,Act_Total_Hrs,Act_Labor,Act_Material,Act_Service,Act_Labor_Burden,Act_Machine_Burden,Act_GA_Burden,Priority,Unit_Price,Total_Price,Price_UofM,Currency_Conv_Rate,Trade_Currency,Fixed_Rate,Trade_Date,Commission_Pct,Customer_PO,Customer_PO_LN,Sched_End,Sched_Start,Quantity_Per,Profit_Pct,Labor_Markup_Pct,Mat_Markup_Pct,Serv_Markup_Pct,Labor_Burden_Markup_Pct,Machine_Burden_Markup_Pct,GA_Burden_Markup_Pct,Lead_Days,Profit_Markup,Prepaid_Amt,Split_To_Job,Note_Text,Comment,Last_Updated,Order_Unit,Price_Unit_Conv,Holder_ID,Source,Order_Taken_By,Plan_Modified,ObjectID,Released_Date,Prepaid_Tax_Amount,Prepaid_Trade_Amt,Last_Updated_By,CommissionIncluded ) VALUES ( src.Job , src.Sales_Rep , src.Customer , src.Ship_To , src.User_Values , src.Quote , src.Contact , src.Ship_Via , src.Terms , src.Tax_Code , src.Sales_Code , src.Top_Lvl_Job , src.Type , src.Order_Date , src.Status , src.Status_Date , src.Part_Number , src.Rev , src.Description , src.Ext_Description , src.Drawing , src.Build_To_Stock , src.Order_Quantity , src.Extra_Quantity , src.Pick_Quantity , src.Make_Quantity , src.Split_Quantity , src.Completed_Quantity , src.Shipped_Quantity , src.FG_Transfer_Qty , src.Returned_Quantity , src.In_Production_Quantity , src.Assembly_Level , src.Certs_Required , src.Time_And_Materials , src.Open_Operations , src.Scrap_Pct , src.Est_Scrap_Qty , src.Est_Rem_Hrs , src.Est_Total_Hrs , src.Est_Labor , src.Est_Material , src.Est_Service , src.Est_Labor_Burden , src.Est_Machine_Burden , src.Est_GA_Burden , src.Act_Revenue , src.Act_Scrap_Quantity , src.Act_Total_Hrs , src.Act_Labor , src.Act_Material , src.Act_Service , src.Act_Labor_Burden , src.Act_Machine_Burden , src.Act_GA_Burden , src.Priority , src.Unit_Price , src.Total_Price , src.Price_UofM , src.Currency_Conv_Rate , src.Trade_Currency , src.Fixed_Rate , src.Trade_Date , src.Commission_Pct , src.Customer_PO , src.Customer_PO_LN , src.Sched_End , src.Sched_Start , src.Quantity_Per , src.Profit_Pct , src.Labor_Markup_Pct , src.Mat_Markup_Pct , src.Serv_Markup_Pct , src.Labor_Burden_Markup_Pct , src.Machine_Burden_Markup_Pct , src.GA_Burden_Markup_Pct , src.Lead_Days , src.Profit_Markup , src.Prepaid_Amt , src.Split_To_Job , src.Note_Text , src.Comment , src.Last_Updated , src.Order_Unit , src.Price_Unit_Conv , src.Holder_ID , src.Source , src.Order_Taken_By , src.Plan_Modified , src.ObjectID , src.Released_Date , src.Prepaid_Tax_Amount , src.Prepaid_Trade_Amt , src.Last_Updated_By , src.CommissionIncluded );
2024-03-31 16:48:42,205 INFO SQLServerManager:55 IF OBJECTPROPERTY(OBJECT_ID('dbo.Job'), 'TableHasIdentity') = 1 SET IDENTITY_INSERT dbo.Job OFF
2024-03-31 16:48:42,210 INFO SqlManager:462 Dropping staging table with this command: DROP TABLE dbo.Jobrepdb4410
2024-03-31 16:48:42,306 INFO ReplicaDB:54 Total process time: 197773ms

[MichalisDBA]

@sandeepraghu Are your destination tables being accessed and data are written from other proccesses? As you mentioned replicadb works. You have to investigate that.

To test and see that actually replicadb works fine i would do something like that.

SourceDB -> run the select query that you use in replicadb conf but count only the rows -> select count(*) from table where... -> for example 10.000 rows.

Run replicadb with that query but for the sink table create a new one so that no other app or proccess know about this table.

Check SinkDB table if the count of rows are matched.

For me replicadb always transfers the exact rows no matter what and we are talking about million of rows being transfered that i tested.

[osalvador]

Hi @sandeepraghu,

In the INCREMENTAL mode, a total of exact rows are replicated, but then a MERGE is performed to the sink table. In a MERGE if the primary key matches an UPDATE will be performed and if it does not exist an INSERT will be performed.

Maybe you are counting rows that have been UPDATED?