CODEWITHSUNDEEP
ssis
Arslan Aslam
Arslan Aslam

Reputation: 131

Data Validation and Reconciliation in SSIS

I have to migrate data from one Non-SQL Server-database to SQL-Server database using ssis.

Data contains millions of rows.

However I want to make sure that data in the source and data in the destination remains same.

One of the answer that i followed is suggesting to use Staging Tables.

In addition to above technique What could be the best technique for doing this.

Any Thought/Suggestion would be appreciated.

Thanks

Upvotes: 1

Views: 561

Answers (1)

Carbon 4horse
Carbon 4horse

Reputation: 328

The staging area in the world of Data warehouse is the place where you just copy the data from the source for multiple reasons :

  • To only execute bulk copy from production server and then avoid to use too much ressources from production servers.
  • To keep the data unmodified during your calculation
  • To apply filter and other aggregation to prepare your queries that fill the DWH.

In your case, the staging area is a good idea to make the first step from non-sql to relationnal Database. Moreover the staging is just a copy then you won't alter the integrity of the data during this step.

Because of this you can run some "integrity tests" after your migration by running count on the staging table and you final structure or by summing data and compare the global result to identify differences.

Upvotes: 1

Related Questions