What does this tool do?
Lists collect duplicates: email addresses exported from two sources, keywords gathered from several tools, product SKUs, log entries, URLs from a crawl or names from a sign-up sheet. This tool removes the repeats in one step. Unlike sorting with “unique” in a spreadsheet, it keeps your original order, so the first time each item appears is the one that stays.
Options let you decide what counts as a duplicate. With ignore case on, “Apple” and “apple” are treated as the same line. With trim spaces on, invisible spaces at the start or end of a line no longer stop two lines from matching. You can also strip empty lines at the same time.
How to use it
- Paste your list into the left box, one item per line.
- Choose whether to ignore case, trim spaces and remove empty lines.
- Copy the cleaned list from the right box. The summary shows how many lines were kept and removed.
How duplicates are detected
The tool reads your text line by line and remembers every line it has already seen. If a line matches one seen before, using your case and whitespace settings, it is skipped. Otherwise it is added to the result. This is a single pass, so even lists with hundreds of thousands of lines are processed almost instantly, and nothing is uploaded.
Example
The list apple, banana, Apple, cherry, banana, (empty), date, cherry, one per line, becomes apple, banana, cherry, date with ignore case and remove empty lines turned on: three duplicates and one empty line removed. With ignore case off, “Apple” would be kept as a separate item.
More worked examples
Email list: merging two exports with 1,200 and 900 addresses often leaves hundreds of repeats; with “ignore case” on, “Ann@Example.com” and “ann@example.com” count as one.
Common mistakes to avoid
- Leaving case sensitivity on for emails. Email addresses are effectively case-insensitive, so turn on “ignore case”.
- Missing near-duplicates with extra spaces. Keep “trim spaces” ticked.
- Expecting it to sort too. It keeps the original order; use sort lines afterwards if you need alphabetical order.
Sources and methodology
The calculations on this page follow these published standards and references. The code is covered by automated tests against known values.
- Single-pass deduplication using a hash set (standard algorithm)