Duplicate lines creep into lists for mundane reasons: two exports merged together, a spreadsheet copied twice, a mailing list built from several sources with overlap. Finding them by eye in anything longer than a dozen lines is unreliable, and a spreadsheet's own feature to delete duplicate lines is often buried behind several menus for a task that should take one paste and one click. This tool works as a focused unique lines tool with none of that overhead.
Everything runs locally in your browser using a hash set comparison, so a list of customer emails or internal server logs never leaves your device. There is no line limit beyond what your browser's memory can hold, and no file is uploaded at any point.
Options that control how duplicates are found
A line only counts as a duplicate if it matches another line under the rules you choose, and getting those rules right matters more than the matching logic itself. Two lines that look identical to a human eye can differ by a single trailing space or a capital letter a spreadsheet formula would never notice, and getting the comparison rule wrong either leaves duplicates behind or removes lines that were not actually duplicates.
- Case sensitive: Apple and apple are treated as different lines when enabled, and as the same line when disabled.
- Trim whitespace: strips leading and trailing spaces before comparing, so trailing spaces do not hide an otherwise identical duplicate.
- Keep first: when duplicates are found, the earliest occurrence in the text is kept and later ones are removed.
- Keep last: when duplicates are found, the final occurrence is kept and earlier ones are removed instead.
When to keep the first line and when to keep the last
Keep first is the right default for most lists, since it preserves whatever order you originally built the list in and simply drops the redundant repeats that came later. It is the natural choice for a list of unique URLs or unique tags where the first entry usually carries whatever context came with it.
Keep last matters when later entries in your list are more likely to be correct or more current than earlier ones, for example a log file where a repeated identifier's last line reflects its final status, or a spreadsheet export where a corrected row was appended below an earlier mistaken one rather than replacing it in place.
What counts as an empty line, and whether it gets removed
Blank lines are treated as lines like any other for the purposes of this tool, which means a text with three blank lines in a row will have two of them removed as duplicates of the first, collapsing the gap to a single blank line. If you need to preserve deliberate spacing between paragraphs rather than treat blank lines as duplicates, deduplicate a copy of the text with the blank lines temporarily removed instead, or check the result before relying on it for anything format sensitive.
How this remove duplicate lines tool works
The text is split into an array of lines, and each line is checked against a Set of lines already seen. The Set gives constant time lookups, so even a list of a hundred thousand lines stays fast, running in roughly linear time rather than the quadratic time a naive comparison of every line against every other line would take.
When case sensitivity or trimming is turned off, a normalised version of each line, lowercased and trimmed, is what actually gets compared and stored in the Set, while the original line text, complete with its original casing and spacing, is what gets kept in the output. That distinction is what lets the tool treat Apple and apple as duplicates for matching purposes while still outputting the exact original text of whichever one was kept.
Using this tool to dedupe text lines from a spreadsheet export
A common source of duplicate rows is exporting the same report twice and pasting both into one sheet. Copy the affected column as plain text, paste it here, and the removed line count tells you immediately how many rows were redundant before you paste the cleaned list back, which is faster than sorting the column and scanning for adjacent matches by eye.
How to remove duplicate lines
- 1
Paste your list
Drop a text file or paste any list of lines, one item per line, into the input box.
- 2
Set the matching rules
Choose whether matching is case sensitive and whether whitespace should be trimmed before comparing.
- 3
Choose keep first or keep last
Decide whether the earliest or the most recent occurrence of a repeated line should be the one that survives.
- 4
Remove and review
Press remove duplicates to see the cleaned list along with a count of exactly how many lines were removed.
Related tools worth bookmarking
Sources and further reading
Frequently asked questions
Common questions about the remove duplicate lines.
Only if case sensitive matching is turned off. With it off, Apple and apple are treated as the same line and one is removed as a duplicate. With case sensitive matching turned on, which is the default, those two lines are treated as distinct and both are kept, since a difference in capitalisation can carry meaning in code or identifiers.
Blank lines are compared the same as any other line, so a block of several consecutive blank lines is reduced to a single blank line, with the rest removed as duplicates of it. If preserving intentional blank line spacing matters for your document, check the result afterward, since the tool has no way to distinguish a meaningful blank line from an accidental one.
Yes. This tool processes text entirely inside your browser using JavaScript, so nothing you paste or drop is uploaded to a server at any point. You can confirm this by checking your browser's network activity while running the tool, which will show no outgoing request carrying your text.
No. Trimming only affects how lines are compared to find duplicates. When trimming is enabled, a line with trailing spaces is treated as a duplicate of an otherwise identical line without them, but the surviving line in the output keeps its original spacing exactly as it was typed or pasted, whichever one the keep first or keep last setting selected.
There is no limit built into the tool itself. It uses a Set based comparison that runs in roughly linear time, so lists with tens of thousands of lines process in a fraction of a second on typical hardware. The practical ceiling is the memory available in your browser tab rather than any restriction imposed by the tool.
No. The original line order is always preserved for whichever lines survive the deduplication, regardless of whether you choose keep first or keep last. Only the duplicate lines themselves are removed, the relative order of every remaining line stays exactly as it was in the pasted text.