uniq
Collapse adjacent identical lines, count repeats, find duplicates
uniq removes adjacent repeated lines. That's why it almost always comes after sort: sorting brings identical lines together, and uniq collapses or counts them.
Syntax
sort [file] | uniq [options]
uniq [options] [input [output]]
The second argument is the output file, not another input.
Options
What to output
| Option | What it does |
|---|---|
| (no options) | one line from each group of identical adjacent lines |
-c | prefix each line with how many times it repeated in a row |
-d | only repeated lines, one per group |
-D | only repeated lines, all instances |
--all-repeated=separate | like -D, but groups are separated by an empty line (prepend — an empty line before the first group too) |
-u | only lines that were not repeated |
--group | all lines, groups separated by an empty line (=prepend, =append, =both — where to put the separator) |
How to compare
| Option | What it does |
|---|---|
-i | case-insensitive |
-f N | skip the first N fields (a field is blanks followed by non-blank characters) |
-s N | skip the first N characters (after the fields from -f) |
-w N | compare at most N characters |
-z | lines are separated by a null byte, not \n |
-c output format
The counter is padded with spaces to a width of 7 characters, then a space and the line: 3 b. If you need to parse the lines further, use awk (awk '{print $1}') — leading spaces don't bother it.
Examples
printf 'b\na\nb\nc\nb\n' > items.txt
# unique values (same as sort -u)
sort items.txt | uniq
# a frequency table: how many times each value occurs
sort items.txt | uniq -c
# most frequent on top, first three
sort items.txt | uniq -c | sort -rn | head -n 3
# only values that occur more than once
sort items.txt | uniq -d
# all instances of repeats
sort items.txt | uniq -D
# only those that occur exactly once
sort items.txt | uniq -u
# compare without the first field (e.g. without a number or timestamp at the start)
printf '1 x\n2 x\n3 y\n' | uniq -f1
# case-insensitive
printf 'Apple\napple\nbanana\n' | uniq -i
# by the first 3 characters
printf 'abc-1\nabc-2\nxyz-1\n' | uniq -w3
# split a sorted list into groups with empty lines
printf 'a\na\nb\nc\nc\n' | uniq --group
Pitfalls
- Without sorting, duplicates stay.
uniqcompares a line only with the previous one: ina b abothas remain. Firstsort, thenuniq. uniq -cpads the counter with spaces (2 a). If you then cut the output withcut -d' ', the fields shift — use awk orsort -rn, which aren't bothered by leading spaces.- Trailing spaces make lines different.
aanda(with a space) are different lines. Remove them beforehand, e.g. with sed. uniqdoesn't see columns. To find duplicates by one field, cut it out first (cut, awk) or use-f/-w. For "the first line for each field value",awk '!seen[$1]++'is handy.uniq a.txt b.txtoverwritesb.txt— the second argument is the output.-fsplits fields on spaces and tabs; no other separator can be set.- Sort order depends on the locale, but that doesn't matter for
uniq: what matters is that identical lines end up next to each other. Just sort and compare in the same locale, especially withsort -fanduniq -i.