← Full reference

uniq

Collapse adjacent identical lines, count repeats, find duplicates

uniq removes adjacent repeated lines. That's why it almost always comes after sort: sorting brings identical lines together, and uniq collapses or counts them.

Syntax

sort [file] | uniq [options]
uniq [options] [input [output]]

The second argument is the output file, not another input.

Options

What to output

OptionWhat it does
(no options)one line from each group of identical adjacent lines
-cprefix each line with how many times it repeated in a row
-donly repeated lines, one per group
-Donly repeated lines, all instances
--all-repeated=separatelike -D, but groups are separated by an empty line (prepend — an empty line before the first group too)
-uonly lines that were not repeated
--groupall lines, groups separated by an empty line (=prepend, =append, =both — where to put the separator)

How to compare

OptionWhat it does
-icase-insensitive
-f Nskip the first N fields (a field is blanks followed by non-blank characters)
-s Nskip the first N characters (after the fields from -f)
-w Ncompare at most N characters
-zlines are separated by a null byte, not \n

-c output format

The counter is padded with spaces to a width of 7 characters, then a space and the line: 3 b. If you need to parse the lines further, use awk (awk '{print $1}') — leading spaces don't bother it.

Examples

printf 'b\na\nb\nc\nb\n' > items.txt

# unique values (same as sort -u)
sort items.txt | uniq

# a frequency table: how many times each value occurs
sort items.txt | uniq -c

# most frequent on top, first three
sort items.txt | uniq -c | sort -rn | head -n 3

# only values that occur more than once
sort items.txt | uniq -d

# all instances of repeats
sort items.txt | uniq -D

# only those that occur exactly once
sort items.txt | uniq -u

# compare without the first field (e.g. without a number or timestamp at the start)
printf '1 x\n2 x\n3 y\n' | uniq -f1

# case-insensitive
printf 'Apple\napple\nbanana\n' | uniq -i

# by the first 3 characters
printf 'abc-1\nabc-2\nxyz-1\n' | uniq -w3

# split a sorted list into groups with empty lines
printf 'a\na\nb\nc\nc\n' | uniq --group

Pitfalls

  • Without sorting, duplicates stay. uniq compares a line only with the previous one: in a b a both as remain. First sort, then uniq.
  • uniq -c pads the counter with spaces ( 2 a). If you then cut the output with cut -d' ', the fields shift — use awk or sort -rn, which aren't bothered by leading spaces.
  • Trailing spaces make lines different. a and a (with a space) are different lines. Remove them beforehand, e.g. with sed.
  • uniq doesn't see columns. To find duplicates by one field, cut it out first (cut, awk) or use -f/-w. For "the first line for each field value", awk '!seen[$1]++' is handy.
  • uniq a.txt b.txt overwrites b.txt — the second argument is the output.
  • -f splits fields on spaces and tabs; no other separator can be set.
  • Sort order depends on the locale, but that doesn't matter for uniq: what matters is that identical lines end up next to each other. Just sort and compare in the same locale, especially with sort -f and uniq -i.

See also

sort · wc · awk · datamash