CODEWITHSUNDEEP
powershellpowershell-3.0
Fiddle Freak
Fiddle Freak

Reputation: 2041

Only remove duplicates records from csv file

I've tried googling different things, but I can't seem to find an answer.

Csv Contents

"ColumnOne","ColumnTwo","ColumnThree"
"C1V1","C2V1","C3V1"
"C1V2","C2V2","C3V2"
"C1V2","C2V2","C3V2"
"C1V3","C2V3","C3V3"

Code

$input = 'C:\Temp\test.csv'
$inputCsv = Import-Csv $input | Select-Object -Unique

$inputCsv | Export-Csv "$input-temp.csv" -NoTypeInformation
#Move-Item "$input-temp.csv" $input -Force

Output

"ColumnOne","ColumnTwo","ColumnThree"
"C1V2","C2V2","C3V2"

Expected output

"ColumnOne","ColumnTwo","ColumnThree"
"C1V1","C2V1","C3V1"
"C1V2","C2V2","C3V2"
"C1V3","C2V3","C3V3"

Is there a way to do this?

Upvotes: 5

Views: 18577

Answers (1)

Aman Sharma
Aman Sharma

Reputation: 1990

You need to specify the column name with either Select-Object or Sort-Object to get unique results. Try the below code if you want to sort as well. If you don't want then use Select-Object. 

$input = 'C:\Temp\test.csv'
$inputCsv = Import-Csv $input | Sort-Object * -Unique

$inputCsv | Export-Csv "$input-temp.csv" -NoTypeInformation

You can also refer the blog HERE.

Upvotes: 8

Related Questions