Removing duplicate values from a PowerShell array

Question

How can I remove duplicates from a PowerShell array?

$a = @(1,2,3,4,5,5,6,7,8,9,0,0)

Answer 1

I was searching for an answer to this question that would still work when the array contains objects AND when order matters, but couldn’t find any.

• Select-Object doesn’t preserve the properties on which de-duplication is not sought.
• Sort-Object doesn’t preserve order.

So I’ve tried a few things in the console and came up with the function below, that matches these needs and still works with simple arrays.

function Select-Uniques {
    param (
        # The input Array to get unique objects from
        [Parameter(
            Mandatory = $true,
            Position = 0)]
        [PSCustomObject[]]$Array,
        # The objects’ properties to de-duplicate on. 
        #   If missing, all properties are used.
        [Parameter(
            Mandatory = $false,
            Position = 1)]
        [string[]]$properties,
        # Whether case matters (default to case sensitive)
        [Parameter(
            Mandatory = $false,
            Position = 2)]
        [switch]$caseInsensitive,
        # Whether to return the first or last (default) duplicate
        [Parameter(
            Mandatory = $false,
            Position = 3)]
        [switch]$first
    )
    if (($Array -isnot [array]) -or ($Array.Count -lt 2)) { return $Array }

    $AllProperties = $Array[0].PSObject.Properties.Name
    $properties = if (-not $PSBoundParameters.ContainsKey('properties')) { 
        $AllProperties 
    } else {
        $properties | Where-Object { $_ -in $AllProperties}
    }

    $CS = !$caseInsensitive.IsPresent
    $CI = $caseInsensitive.IsPresent
    $Position = @{ ($first.IsPresent ? "First" : "Last") = 1 }

    # Object comparison function
    #   Returns $true if no difference is found between the 2 objects
    function Test-Equal($obj1, $obj2) {
        $props1 = $obj1.PSObject.Properties
        $props2 = $obj2.PSObject.Properties
        $Equal = -not (Compare-Object $props1 $props2 -CaseSensitive:$CS)
        return $Equal
    }

    # For a Unique Set of the selected Properties’ values, returns the 
    #   First or Last (default) matching row from the input Array
    function Select-OneUnique($UniqueValues) { 
        $UniqueRow = $Array 
        | Where-Object { 
            $RowValues = $_ | Select-Object -property $properties
            Test-Equal $RowValues $UniqueValues
        } | Select-Object @Position 
        return $UniqueRow
    }
    
    $Uniques = if ($null -eq $properties) { $Array 
        # Elements aren’t objects → just get the unique values
        | Select-Object -unique -CaseInsensitive:$CI
    } else { $Array 
        # Elements are objects → 
        # 1) Get the unique values for the selected properties
        | Select-Object -unique -property $properties -CaseInsensitive:$CI
        # 2) For each unique row, get the matching elements in the input Array
        | ForEach-Object { Select-OneUnique $_ }
    }

    return $Uniques
}

If you’re interested in trying it, here are examples:

$a=ConvertFrom-Csv @(
    "c 5 3", 
    "B 4 3", 
    "b 1 2", 
    "a 1 2") -delimiter " " -header "x","y","z"

Select-Uniques $a "z"
<# output
x y z
- - -
B 4 3
a 1 2             #>

Select-Uniques $a "z" -first
<# output
x y z
- - -
c 5 3
b 1 2             #>

Select-Uniques $a "y", "z"
<# output
x y z
- - -
c 5 3
B 4 3
a 1 2             #>

Select-Uniques $a "x"
<# output
x y z
- - -
c 5 3
B 4 3
b 1 2
a 1 2             #>

Select-Uniques $a "x" -caseinsensitive
<# output
x y z
- - -
c 5 3
b 1 2
a 1 2             #>

# and this also works:
Select-Uniques @(3,3,3,2,2,1)
<# output
3
2
1                 #>

Answer 2

For an "array" aka string with multiple lines, you can use this:

$a.Split("`n") | select -unique | sort

Answer 3

I do this:

$data = Get-Content $file |
    ConvertFrom-Json |
    Select-Object -Property id, name, streetName1, zipCode, city |
    Sort-Object -Property id -Unique

Answer 4

$a = @('D:/', 'two', 'three')
$s = "D:/"
$null -ne ($a | ? { $s -match $_ })  # Returns $true

another solution can be used here , u're welcome

Answer 5

In case you want to be fully bombproof, this is what I would advise:

@('Apples', 'Apples ', 'APPLES', 'Banana') | 
    Sort-Object -Property @{Expression={$_.Trim()}} -Unique

Output:

Apples
Banana

This uses the Property parameter to first Trim() the strings, so extra spaces are removed and then selects only the -Unique values.

More info on Sort-Object:

Get-Help Sort-Object -ShowWindow

Answer 6

To get unique items from an array and preserve their order, you can use .NET HashSet:

$Array = @(1, 3, 1, 2)
$Set = New-Object -TypeName 'System.Collections.Generic.HashSet[int]' -ArgumentList (,[int[]]$Array)

# PS> $Set
# 1
# 3
# 2

Works best with string arrays that contain both uppercase and lowercase items where you need to preserve first occurrence of each item in case-insensitive manner:

$Array = @("B", "b", "a", "A")
$Set = New-Object -TypeName 'System.Collections.Generic.HashSet[string]' -ArgumentList ([string[]]$Array, [StringComparer]::OrdinalIgnoreCase)

# PS> $Set
# B
# a

Works as expected with other types.

Shortened syntax, compatible with PowerShell 5.1 and newer:

$Array = @("B", "b", "a", "A")
$Set = [Collections.Generic.HashSet[string]]::new([string[]]$Array, [StringComparer]::OrdinalIgnoreCase)

$Array = @(1, 3, 1, 2)
$Set = [Collections.Generic.HashSet[int]]::new([int[]]$Array)

Answer 7

A lot of the provided answers give buggy results. select -Unqiue is not case sensitive. sort -Unique gives you sorted results, which you might want in the original order.

Will gave a great answer, but it's flawed as it discards all duplicate results but forgets to keep one of them.

This is a version I created that seems to work perfectly. It gives unique results back and retains the original sort order.

($properties | Group-Object -NoElement).Name | Get-Unique

Answer 8

This is how you get unique from an array with two or more properties. The sort is vital and the key to getting it to work correctly. Otherwise you just get one item returned.

PowerShell Script:

$objects = @(
    [PSCustomObject] @{ Message = "1"; MachineName = "1" }
    [PSCustomObject] @{ Message = "2"; MachineName = "1" }
    [PSCustomObject] @{ Message = "3"; MachineName = "1" }
    [PSCustomObject] @{ Message = "4"; MachineName = "1" }
    [PSCustomObject] @{ Message = "5"; MachineName = "1" }
    [PSCustomObject] @{ Message = "1"; MachineName = "2" }
    [PSCustomObject] @{ Message = "2"; MachineName = "2" }
    [PSCustomObject] @{ Message = "3"; MachineName = "2" }
    [PSCustomObject] @{ Message = "4"; MachineName = "2" }
    [PSCustomObject] @{ Message = "5"; MachineName = "2" }
    [PSCustomObject] @{ Message = "1"; MachineName = "1" }
    [PSCustomObject] @{ Message = "2"; MachineName = "1" }
    [PSCustomObject] @{ Message = "3"; MachineName = "1" }
    [PSCustomObject] @{ Message = "4"; MachineName = "1" }
    [PSCustomObject] @{ Message = "5"; MachineName = "1" }
    [PSCustomObject] @{ Message = "1"; MachineName = "2" }
    [PSCustomObject] @{ Message = "2"; MachineName = "2" }
    [PSCustomObject] @{ Message = "3"; MachineName = "2" }
    [PSCustomObject] @{ Message = "4"; MachineName = "2" }
    [PSCustomObject] @{ Message = "5"; MachineName = "2" }
)

Write-Host "Sorted on both properties with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property Message,MachineName -Unique | Out-Host

Write-Host "Sorted on just Message with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property Message -Unique | Out-Host

Write-Host "Sorted on just MachineName with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property MachineName -Unique | Out-Host

Output:

Sorted on both properties with -Unique

Message MachineName
------- -----------
1       1          
1       2          
2       1          
2       2          
3       1          
3       2          
4       1          
4       2          
5       1          
5       2          


Sorted on just Message with -Unique

Message MachineName
------- -----------
1       1          
2       1          
3       1          
4       1          
5       2          


Sorted on just MachineName with -Unique

Message MachineName
------- -----------
1       1          
3       2  

Source: https://powershell.org/forums/topic/need-to-unique-based-on-multiple-properties/

Answer 9

Whether you're using SORT -UNIQUE, SELECT -UNIQUE or GET-UNIQUE from Powershell 2.0 to 5.1, all the examples given are on single Column arrays. I have yet to get this to function across Arrays with multiple Columns to REMOVE Duplicate Rows to leave single occurrences of a Row across said Columns, or develop an alternative script solution. Instead these cmdlets have only returned Rows in an Array that occurred ONCE with singular occurrence and dumped everything that had a duplicate. Typically I have to Remove Duplicates manually from the final CSV output in Excel to finish the report, but sometimes I would like to continue working with said data within Powershell after removing the duplicates.

Answer 10

$a | sort -unique

This works with case-insensitive, therefore removing duplicates strings with differing cases. Solved my problem.

$ServerList = @(
    "FS3",
    "HQ2",
    "hq2"
) | sort -Unique

$ServerList

The above outputs:

FS3
HQ2

Answer 11

Another option is to use Sort-Object (whose alias is sort, but only on Windows) with the
-Unique switch, which combines sorting with removal of duplicates:

$a | sort -unique

Answer 12

Use Select-Object (whose alias is select) with the -Unique switch; e.g.:

$a = @(1,2,3,4,5,5,6,7,8,9,0,0)
$a = $a | select -Unique

Answer 13

With my method you can completely remove duplicate values, leaving you with values from the array that only had a count of 1. It was not clear if this is what the OP actually wanted however I was unable to find an example of this solution online so here it is.

$array=@'
Bananna
Apple
Carrot
Pear
Apricot
Pear
Bananna
'@ -split '\r\n'

($array | Group-Object -NoElement | ?{$_.count -eq 1}).Name

Answer 14

If the list is sorted, you can use the Get-Unique cmdlet:

 $a | Get-Unique