Remove Duplicate Lines
Paste a list, get it back with every repeated line removed. The first occurrence of each line is kept in its original position.
What this tool does
Deduplication sounds trivial until you try it on real data. The two things people usually get wrong are order and near-matches. This tool keeps the first occurrence of every line exactly where it was, rather than sorting the list as a side effect, because in most real lists the order carries meaning — priority, chronology, or simply the order a colleague typed things in.
The two optional toggles handle near-matches. Ignore case treats
Banana and banana as the same entry, which is what you
almost always want for names, tags and email addresses. Ignore leading and
trailing spaces catches the invisible culprit behind most "but these look
identical!" moments: a stray space picked up from a spreadsheet cell or a copied
table column.
Show only duplicates
The second mode inverts the job: instead of removing repeats it shows you only the lines that appear more than once, one entry per repeated value. This is the fast way to answer "which entries are duplicated?" before you decide what to do about them — useful when you are auditing a mailing list, checking for double-booked IDs, or looking for accidentally re-imported rows.
When you would reach for this
- Mailing lists and CRM exports. Merged exports from two sources almost always overlap. Deduplicating before import stops you from emailing the same person twice and, on most platforms, from paying for the same contact twice.
- Log files. A repeated stack trace tells you an error is frequent, but when you are trying to work out how many distinct errors you have, collapsing to unique lines turns thousands of lines into a readable handful.
- Keyword and tag lists. Keyword research tools produce overlapping sets by design; deduplicating with case-insensitivity on is usually the first step before any analysis.
- Config and dependency files. Duplicate entries are often harmless but occasionally shadow each other in confusing ways. Spotting them is easier than debugging them.
A note on very large lists. Because everything runs locally, the practical limit is your browser's memory rather than an upload cap. Lists in the tens of thousands of lines process in well under a second on ordinary hardware.
How it compares to the alternatives
In a spreadsheet you would use Data → Remove duplicates, which works
well but forces your data into a column and often reformats it on the way in and
out — numbers become dates, leading zeros vanish. On the command line
sort -u is one keystroke but it sorts, destroying the original order;
the order-preserving version is awk '!seen[$0]++', which is precise
but not something most people keep in their head. This tool is the middle ground:
order-preserving by default, nothing to remember, nothing to install.
Frequently asked questions
Does removing duplicates change the order of my lines?
No. The first occurrence of each line stays exactly where it was, and later repeats are dropped. If you want a sorted result, run the output through the sort tool afterwards.
Can it treat uppercase and lowercase as the same line?
Yes. Turn on Ignore case and Apple, APPLE and apple collapse into a single entry — the first spelling that appeared is the one kept.
Why do two lines that look identical survive deduplication?
Almost always invisible whitespace: a trailing space, a tab, or a non-breaking space pasted from a web page. Turn on Ignore leading/trailing spaces, or clean the text first with the extra-spaces tool.
Is my data uploaded anywhere?
No. The deduplication runs entirely in your browser using JavaScript. Your text is never sent to a server, which also means the tool keeps working offline once the page has loaded.