How to Remove Duplicate Lines from Text (Online Tool + Code)

Duplicate lines waste space in logs, CSVs, and data exports. Here's how to remove them.

Command Line

# Sort and deduplicate
sort input.txt | uniq > output.txt

# Preserve original order (awk)
awk '!seen[$0]++' input.txt > output.txt

Python

lines = open('input.txt').readlines()
seen = set()
result = []
for line in lines:
    if line not in seen:
        seen.add(line)
        result.append(line)

Online Tool

Use our duplicate line remover for instant deduplication.