Five Ways to Remove Duplicate Lines, and When to Use Each

6 min read · Updated 2026-08-04

Removing duplicates is one of those tasks with a dozen solutions, most of which are subtly wrong for your particular case. The differences come down to three questions: does it preserve the original order, does it handle near-matches, and how much data can it cope with?

The spreadsheet way

Excel and Google Sheets both offer Data → Remove duplicates, and for a one-off job on a few thousand rows it is hard to beat. It preserves order, it works across multiple columns, and you can see what happened immediately.

The catch is what a spreadsheet does to your data on the way in. Paste a column of values and it will helpfully interpret them: leading zeros disappear from postcodes and account numbers, anything resembling a date becomes one, long numeric IDs turn into scientific notation, and a value like +39 02 1234 may be read as a formula. None of these are recoverable after the fact. If your data is purely textual and none of it looks numeric, this is fine. Otherwise, set the column format to Text before pasting, or use something else.

sort -u on the command line

sort -u file.txt is the shortest thing that works, and on large files it is extremely fast — it handles files larger than memory by spilling to disk. Every Unix-like system has it, including macOS and WSL.

The important caveat is in the name: it sorts. The output is alphabetical, and the original order is gone. For a reference list that is fine or even desirable. For a log file, a chronological export, or any list where position carries meaning, it is destructive in a way that is not obvious until later.

awk, when order matters

The order-preserving equivalent is awk '!seen[$0]++' file.txt. It keeps the first occurrence of each line exactly where it was and drops the rest — the same behaviour as a well-behaved browser tool, at command-line speed.

It is worth understanding rather than just copying. seen is an associative array keyed by the whole line; seen[$0]++ returns the current count and then increments it, so the first time a line appears the expression is 0, which awk treats as false. The ! inverts it to true, and a bare true condition in awk means "print this line". Every subsequent occurrence returns a non-zero count, which is true, inverted to false, so nothing is printed.

The limitation is memory: the array holds every distinct line, so a file with tens of millions of unique lines will exhaust RAM where sort -u would not.

SQL, when the data is already in a database

SELECT DISTINCT is the obvious answer and usually the right one for whole-row duplicates. Where it gets more interesting is partial duplicates — rows that are the same in the columns you care about but differ elsewhere, such as an import timestamp.

For that case, window functions are the modern approach:

SELECT * FROM (
  SELECT *, ROW_NUMBER() OVER (
    PARTITION BY email ORDER BY created_at
  ) AS rn
  FROM contacts
) t WHERE rn = 1;

This keeps the earliest row per email address and discards later ones. Changing ORDER BY created_at to created_at DESC keeps the newest instead. The advantage over DISTINCT is that you decide explicitly which copy survives, rather than accepting whichever one the engine happens to return.

A browser tool

The case for doing it in a browser is convenience and privacy: nothing to install, nothing to remember, and with a client-side tool the data never leaves your machine. Order is preserved, and options such as case-insensitive matching are a checkbox rather than a flag you have to look up.

The limit is size. A browser will comfortably handle lists in the tens of thousands of lines and struggle well before the point where sort -u would break a sweat. Use it for the everyday case — a list from an email, a column from a spreadsheet, a set of keywords — and reach for the command line when the file gets large.

Choosing

The problem underneath all of them

Whichever method you pick, exact-match deduplication only removes lines that are byte-for-byte identical. The duplicates that actually cause trouble usually are not: Mario Rossi and Rossi, Mario, or the same address with and without a trailing space.

Two habits help. First, normalise before you deduplicate — trim whitespace, decide on a case convention, and fix encoding problems. Half of all "near duplicates" turn out to be exact duplicates once the invisible characters are gone. Second, for the genuinely fuzzy remainder, accept that no automatic tool will be right every time; extract the candidates, look at them, and decide by hand. That is slower, but it is the only approach that does not quietly delete real data.