CODEWITHSUNDEEP
ssis
null
null

Reputation: 747

How to log duplicate rows removed using Sort function in SSIS?

Is there a way to log the rows to a flat file that were removed using the sort functions in SSIS?

Upvotes: 1

Views: 59

Answers (1)

KeithL
KeithL

Reputation: 5594

Maybe you could use aggregate instead of sort. Add all columns into the aggregate and use an aggregation of count.

Run through a multicast and have one flow go normally and the 2nd flow can go through a conditional split splitting out where counts are >1. This will give you the duplicates to handle.

EDIT:

The default aggregation operation is group by. The easiest way to implement this is to check the box next to name in the editor which will select all (group by) and also add in a count of each group.

EDIT2:

You said you have a SQL Source, consider using 

row_Number() over (partition by [primary key] order by [EnteredDate] desc)

This column will give you something to key off of for splitting.

Actually, it would have to be foreign key to have duplication in the table.

Upvotes: 1

Related Questions