Remove Duplicates from Array in PowerShell

Using Select-Object with the -Unique Parameter

To remove duplicates from array in PowerShell:

• Use the array operator to create an array having duplicate values.
• Use the Select-Object with the -Unique parameter to select unique values from the array we created in the previous step.

For this code, we used the array operator (@()) to create an array having duplicate values and saved the array in the $array variable using the assignment operator (=). Note that, in PowerShell, the variables names are prefixed with a $ sign, for instance, $VariableName.

Next, we used the Select-Object cmdlet (it is pronounced as command-let, a command used in Windows PowerShell) with the -Unique parameter to select unique values from the given array. How? Let’s break down the $array = $array | Select-Object -Unique command and understand it.

First, we specified the input array ($array) and used pipe (|) to send its results to the Select-Object cmdlet. Here, the pipe was used to send the results of the preceding command to the following command.

Then, the Select-Object cmdlet chooses the given properties of the object/set of objects; it can select the specified objects in a specific index in the array, a certain number of objects and unique objects.

We used this cmdlet with the -Unique parameter to select unique values from the given array. After that, we used the assignment operator to update the original array.

Remember, the -Unique parameter is case-sensitive, which means Hello, and hello would be considered unique values. Let’s see it in the following example containing string-type elements in the array.

Did you see, we got Hello and hello, both. It is because the difference in character casing is considered as unique values for the -Unique parameter with the Select-Object cmdlet (whose alias is Select).

Using Sort-Object with -Unique Parameter

To remove duplicates from an array in PowerShell:

• Use the array operator to create an array with duplicate values.
• Use the Sort-Object with the -Unique parameter to remove duplicate values from the array we created in the first step.

We’ve already learned about array creation using the array operator, variable declaration and initialization, assignment operator, pipe sign, selecting unique values and the execution flow of the cmdlet while using Select-Object.

This code example is similar to the one we used while learning Select-Object; here, we used the Sort-Object cmdlet, which can sort the objects in descending or ascending order depending on the object property values. What does that mean?

It means if we do not specify the sort property with the Sort-Object, then it will use the default sort property of the first input object. Remember, if there is no default sort property for the input object’s type, then PowerShell will compare the objects themselves. You can read more about it here

Here, properties will be sorted as case-insensitive or case-sensitive while working with string-type array elements. How? Let’s understand it using the -Unique parameter with the Sort-Object cmdlet below.

The above code fence showed that the -Unique parameter will consider Hello and hello as unique values if the -CaseSensitive parameter is given. Also, the elements will be sorted in alphabetical order considering their first letter due by using the Sort-Object cmdlet.

If you want case-insensitive results, omit the -CaseSensitive parameter and use the following command.

We can also use the Sort as follows, an alias of the Sort-Object cmdlet.

Further reading:

Check if array is empty in PowerShell

Read more →

Add Object to Property in PowerShell

Read more →

Using Get-Unique with Sort-Object

To remove duplicates from an array in PowerShell:

• Use an array operator to create an array which contains duplicate values.
• Use Sort-Object to sort the given array and Get-Unique to remove duplicate values from the specified array.

We have already learned about Sort-Object in the previous section. Here, we used the Get-Unique cmdlet with the Sort-Object cmdlet to remove duplicate values from the given array.

You might be wondering why we are supposed to use the Get-Unique cmdlet with the Sort-Object cmdlet. It is because the Get-Unique cmdlet selects unique values if the array is sorted. So, how the command $array = $array | Sort-Object | Get-Unique was executed?

First, we specified the input array ($array), and its results were forwarded to the Sort-Object cmdlet using a pipe sign. Then, the Sort-Object cmdlet sorted the input array in ascending order which was the default sort property of $array. Afterwards, we used another pipe to forward the sorted array to the Get-Unique cmdlet.

Now, the Get-Unique cmdlet compared every element in the provided sorted array to the next, removed the identical values and returned one instance only for every element. Finally, sorted unique values were assigned to the $array using the assignment operator (=).

Just like the -Unique parameter, the Get-Unique is also case-sensitive, meaning strings with a difference in character casing will be considered unique values. See the following example for a demonstration.

That’s all about how to remove duplicates from Array in PowerShell.