Regular expressions
BRE, ERE and PCRE in practice — classes, anchors, quantifiers, groups, backreferences, greediness
A regular expression is a pattern describing a set of strings. grep, sed, awk and many other programs understand them, but in several dialects. The basics are shared; the differences are mostly in which special characters need (or don't need) escaping.
Three dialects
- BRE (basic) —
grepandsedwithout options. The characters+ ? | ( ) { }are ordinary here; backslashes give them special meaning:\+ \? \| \( \) \{ \}(\+ \? \|are a GNU extension). - ERE (extended) —
grep -E,sed -E,awk. The same+ ? | ( ) { }work without backslashes, while\+and\(on the contrary mean ordinary characters. - PCRE (Perl) —
grep -P. Everything from ERE plus\d \s \w, lazy quantifiers*?, lookarounds(?=…)and(?<=…),\K, modifiers(?i).
The same thing in three dialects — "one or more digits, then a dot":
echo 'v12.3' | grep -o '[0-9]\+\.' # BRE
echo 'v12.3' | grep -oE '[0-9]+\.' # ERE
echo 'v12.3' | grep -oP '\d+\.' # PCRE
Characters and sets
| Element | Meaning |
|---|---|
a, 1, - | an ordinary character matches itself |
. | any single character |
\. \* \[ \\ \$ \^ | a special character as an ordinary one |
[abc] | one character from the set |
[^abc] | any character except those listed |
[a-z] [0-9A-F] | ranges inside a set |
[]a] / [a-] / [.*] | ] — first, - — first or last; . and * inside brackets are ordinary characters |
POSIX classes
Written inside square brackets: [[:digit:]], [^[:space:]], [[:alpha:]_].
| Class | What it includes |
|---|---|
[:digit:] / [:xdigit:] | digits / hexadecimal digits |
[:alpha:] / [:alnum:] | letters / letters and digits |
[:upper:] / [:lower:] | upper case / lower case |
[:space:] / [:blank:] | all whitespace (space, \t, \n, \r …) / only space and tab |
[:punct:] | punctuation |
[:print:] / [:graph:] / [:cntrl:] | printable including space / printable without space / control characters |
Shorthands
| Element | Meaning | Where it works |
|---|---|---|
\w / \W | word character (letter, digit, _) / not one | GNU grep, sed, gawk, PCRE |
\s / \S | whitespace / non-whitespace | GNU grep, sed, gawk, PCRE |
\d / \D | digit / non-digit | PCRE only (grep -P) |
\t \n | tab, newline | PCRE, gawk; in sed \n and \t too |
Anchors
| Element | Meaning |
|---|---|
^ / $ | start / end of line |
\< / \> | start / end of a word (GNU grep, sed, gawk) |
\b / \B | word boundary / not a boundary (GNU grep, sed, PCRE; in gawk the boundary is \y) |
Quantifiers
They apply to the preceding character, set or group.
| ERE / PCRE | BRE | How many times |
|---|---|---|
* | * | 0 or more |
+ | \+ | 1 or more |
? | \? | 0 or 1 |
{3} | \{3\} | exactly 3 |
{2,5} | \{2,5\} | from 2 to 5 |
{2,} / {,5} | \{2,\} / \{,5\} | 2 or more / at most 5 |
All quantifiers are greedy: they take as much as possible and then back off if a match can't be found otherwise. PCRE has lazy *?, +?, ??, {2,5}? — they take as little as possible.
Groups and alternation
| Element | Meaning |
|---|---|
(…) (in BRE — \(…\)) | group: a quantifier applies to all of its content, the matched text is remembered |
\1 … \9 | backreference — the same text the group matched. In grep and sed patterns, in sed replacements; awk doesn't have them |
a|b | a or b; inside a group — (jpg|png). In BRE the bar needs a backslash |
(?:…) | non-capturing group (PCRE) |
(?=…) / (?!…) | followed / not followed by … (PCRE, not part of the match) |
(?<=…) / (?<!…) | preceded / not preceded by … (PCRE) |
\K | forget everything matched up to this point (PCRE) |
(?i) | case-insensitive from here on (PCRE) |
Examples
# a line consisting entirely of digits
printf '123\n12a\n' | grep -E '^[0-9]+$'
# empty lines and lines of only spaces
printf 'a\n\n \nb\n' | grep -c '^[[:space:]]*$'
# lines whose first character is neither a space nor #
printf '# comment\n indented\nvalue\n' | grep '^[^[:space:]#]'
# a whole word: cat, but not catalog
echo 'cat catalog' | grep -o '\<cat\>'
echo 'cat catalog' | grep -oE '\bcat\b'
# an optional letter and an exact number of repeats
echo 'color colour' | grep -oE 'colou?r'
echo '2024-01-05' | grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}$'
# alternation and a group with a quantifier
printf 'ab\nabab\naba\n' | grep -xE '(ab)+'
printf 'a.jpg\nb.png\nc.txt\n' | grep -E '\.(jpg|png)$'
# backreference: two identical characters in a row
echo 'aa bc dd' | grep -oE '(.)\1'
# groups in a replacement: rearrange the parts of a date
echo '2024-01-05' | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3.\2.\1/'
# greediness: .* grabs the maximum
echo '<a><b>' | grep -oE '<.*>'
# the lazy variant exists only in PCRE
echo '<a><b>' | grep -oP '<.*?>'
# a portable substitute for laziness — a negated class
echo '<a><b>' | grep -oE '<[^>]*>'
# PCRE: lookbehind and \K — a match without the prefix
echo 'id=42' | grep -oP '(?<=id=)\d+'
echo 'id=42' | grep -oP 'id=\K\d+'
# PCRE: a word NOT followed by a digit
echo 'v1 vx' | grep -oP 'v(?!\d)\w'
# a special character as an ordinary one: dot and asterisk
printf '1.5\n105\n' | grep '^1\.5$'
printf 'a*b\naab\n' | grep 'a\*b'
Pitfalls
- A dot is any character.
grep '1.2'also finds1x2. Escape it (1\.2), put it in brackets (1[.]2) or usegrep -Fto search for a literal string. - POSIX classes only inside brackets.
[:digit:]without the second pair of brackets means the set of characters: d i g t; GNU grep stops with the errorcharacter class syntax is [[:space:]], not [:space:]in this case. \doutside-Pis the letterd, and\tin grep is the lettert. Write[0-9]or[[:digit:]], and pass a tab as$'\t'.- No lazy quantifiers in BRE/ERE.
.*?ingrep -Eandsed -Eworks like the usual greedy.*. Use[^…]*orgrep -P. ^and$in BRE are special only at the edges.grep 'a^b'searches for a literala^b.*at the start of a pattern is an ordinary character in BRE:grep '*a'searches for*a. In ERE GNU grep accepts it with the warning* at start of expression.- A regex searches for a substring.
grep '[0-9]'finds the lineabc1. To check the whole line, use^…$orgrep -x. - Quote the pattern. Without single quotes the shell processes
*,?,[…],$and\itself. - Ranges and
.depend on the locale. The sandbox locale isPOSIX:.is one byte,[a-z]is exactly the Latin lower-case letters, and[[:alpha:]]doesn't include accented letters. WithLC_ALL=C.utf8.matches a whole letter, and[[:alpha:]]findsé.