Remove Duplicate Lines

What this tool does

Remove duplicate lines means: keep the first time a line appears, drop later copies, leave the survivors in that first-seen order. This page does that in the browser. Comparison is case-sensitive unless you turn that off. You can ignore leading and trailing spaces when comparing, and you can drop blank lines instead of treating them as values. The original paste stays in the first box.

That is not a sort. Order is the order of first occurrence. If you also need alphabetical or numeric-aware order, run the result through the Text Sorter. Doing both jobs in one unlabeled button is how people lose the “first wins” rule they thought they still had.

How to use

1. Paste the list, one item per line.
2. Leave case-sensitive on unless “Apple” and “apple” should collapse. Turn on trim if apple and apple should match. Turn on drop blank lines if empty rows should disappear instead of being de-duplicated as one blank.
3. Remove duplicates. Copy writes the result. Example uses mixed case and a blank. Clear empties both boxes.

The status reports how many lines were kept and how many were removed. Removed includes both later duplicates and, when that option is on, blank lines.

Example

Example input:

apple
Banana
apple
banana

pear
pear

With case-sensitive on and blanks kept, the result is apple, Banana, banana, a blank line, pear. The second apple and the second pear go away. Banana and banana stay, because they are different under a case-sensitive compare. Turn case-sensitive off and the later banana is also removed. Turn drop blank lines on and the empty row never appears in the result.

Limits

• First occurrence wins. There is no “keep the last” mode.
• Default comparison is exact, including spaces inside the line. Trailing space only matches if you enable trim.
• Blank lines compare as empty strings. Without “drop blank lines,” the first blank is kept and later blanks are duplicates.
• Windows and Unix line breaks are normalized before the split so a CR/LF list does not create phantom empty lines from leftover \r.
• This is not a CSV column unique. Commas stay part of the line.
• Input over 200,000 characters is refused.

Pasted markup is text. A repeated <script>alert(1)</script> line is de-duplicated as a string and never run. If you needed to replace a phrase inside a line, that is the Find and Replace page, not this one.

The work stays in this tab. Unique lists are often pasted from exports that still contain a header row. The first line is kept even if it later repeats as a label. If that header should not survive, delete it before you run the tool. There is no “skip first line” control, because that would hide whether line 1 was a header or a value.

Very wide lines are still one line. A paragraph wrapped by your editor into several visual rows is several lines here only if it contains actual line breaks. Soft wrap in the textarea is not a break. That is the same rule a sort command on a file would use.