How can I remove duplicate lines in Visual Studio Code?

Question

Say you have the following text:

abc
123
abc
456
789
abc
abc

I want to remove all "abc" lines and just keep one. I don't mind sorting. The result should be like this:

abc
123
456
789

Answer 1

Sort Unique - native VS Code

For assigning <shift>-F5 keybinding for sorting uniquely selected text, add the following to the keybindings.json:

  {
    "key": "shift+f5",
    "command": "runCommands",
    "args": {"commands": ["editor.action.removeDuplicateLines",
                          "editor.action.sortLinesAscending"]}
  }

Answer 2

As of Visual Studio Code 1.62 (released October 2021) there is a new command Delete Duplicate Lines. It deletes duplicate lines within a selection or the whole document.

There is no default keybinding for this command. If you want to set one, the command name is editor.action.removeDuplicateLines.

---

Here is a very interesting extension: Transformer

Features:

• Unique Lines As New Document
• Unique Lines

• Align CSV
• Align To Cursor
• Compact CSV
• Copy To New Document
• Count Duplicate Lines As New Document
• Encode / Decode
• Filter Lines As New Document
• Filter Lines
• Join Lines
• JSON String As Text
• Lines As JSON String Array
• Normalize Diacritical Marks
• Randomize Lines
• Randomize Selections
• Reverse Lines
• Reverse Selections
• Rotate Backward Selections
• Rotate Forward Selections
• Select Highlights
• Select Lines
• Selection As JSON String
• Sort Lines By Length
• Sort Lines
• Sort Selections
• Split Lines After
• Split Lines Before
• Split Lines
• Trim Lines
• Trim Selections

Unique Lines

Removes duplicate lines from the document Operates on selection or current block if no selection

Unique Lines As New Document

Unique lines are opened in a new document Operates on selection or current block if no selection

I haven't played with it much besides the "Unique Lines" command but it seems quite nicely done (including attempting a macro recorder!).

Answer 3

To remove the duplicate lines in Visual Studio Code:

1. Select entire the text.

2. Press: Ctrl + Shift + P on Windows and Linux

   Command + Shift + P on macOS

3. Type Delete Duplicate Lines and select the option. It will filtered the duplicate line and give it a unique text.

Answer 4

To add to @Marc.2377 's reply.

If the order is important and you don't care that you just keep the last of the duplicate lines, simply search for the following regexp if you want to only remove duplicte non-empty lines

^(.+)\n(?=(?:.*\n)*?\1$)

If you also want to remove duplicate empty lines, use * instead of +

^(.*)\n(?=(?:.*\n)*?\1$)

and replace with nothing.

This will take a line and try to find ahead some more (maybe 0) lines followed by the exact same line taken. It will remove the taken line.

This is just a one-shot regex. No need to spam the replace button.

This now also takes the comment of @awk into account, in where the last line has to have a linefeed in order to be identified as a duplicate. This is no longer the case now by excluding the \n from the line to search and adding a $ to the line found.

Answer 5

If you don't mind some Vim in your VS Code. You can install Vim emulation plugin.

Then you can use vim commands

:sort u

It will sort lines and it will remove duplicates

Answer 6

Sublime Text 3

It has blisteringly fast native permutation functions.

• Edit > Permute Lines > Unique or ⇧⌘U, and
• Edit > Permute Selections > Unique

Visual Studio Code is my daily driver. But, I keep Sublime Text on standby for these situations.

Answer 7

Not actually in Visual Studio Code, but if it works, it works.

1. Open a new Excel spreadsheet
2. Paste the data into a column
3. Go to the Data tab
4. Select the column of data (if you haven't already)
5. Click Remove Duplicates (somewhat in the middle of the bar)
6. Click OK to remove duplicates.

It is not the best answer, as you specified Visual Studio Code, but as I said: If it works, it works :)

Answer 8

Install the DupChecker extension, hit F1, and type "Check Duplicates".

It will check for duplicates and ask if you want to remove them.

Answer 9

Try find and replace with a regular expression.

• Find: ^(.+)((?:\r?\n.*)*)(?:\r?\n\1)$

• Replace: $1$2

It is possible to introduce some variance in the first group.

Answer 10

I just had the same issue and found the Visual Studio Code package "Sort lines". See the Visual Studio Code market place for details (e.g. Sort lines).

This package has the option "Sorting lines (unique)", which did it for me. Take care of any white spaces at the beginning/end of lines. They influence whether lines are considered unique or not.

Answer 11

If the order of lines is not important

Sort lines alphabetically, if they aren't already, and perform these steps:
_{(based on this related question: How do I find and remove duplicate lines from a file using Regular Expressions?)}

1. Control+F

2. Toggle "Replace mode"

3. Toggle "Use Regular Expression" (the icon with the .* symbol)

4. In the search field, type ^(.*)(\n\1)+$

5. In the "replace with" field, type $1

6. Click ("Replace All").

If the order of lines is important so you can't sort

In this case, either resort to a solution outside VS Code (see here), or - if your document is not very large and you don't mind spamming the Replace All button - follow the previous steps, but in steps 4 and 5, enter these:
_{(based on Remove specific duplicate lines without sorting)}

Caution: Blocks for files with too many lines (1000+); may cause VS Code to crash; may introduce blank lines in some cases.

• search: ((^[^\S$]*?(?=\S)(?:.*)+$)[\S\s]*?)^\2$(?:\n)?

• replace with: $1

and then click the "Replace All" button as many times as there are duplicate occurrences.

You'll know it's enough when the line count stops decreasing when you click the button. Navigate to the last line of the document to keep an eye on that.