awk
Column-based text processing — selecting fields, filters, counts and sums
awk splits each line into fields and runs rules of the form condition { action } on it. It's a small language with variables, arrays, functions and printf, which makes it handy for selecting columns, filtering lines by a field's value and computing totals. In the sandbox awk is GNU awk (gawk) 5.2.
Syntax
awk [options] 'program' [file…]
awk [options] -f program.awk [file…]
A program is a set of condition { action } rules. For each input line awk checks the rules in order. No condition — the action runs for every line. No action — the whole line is printed ({print}). A BEGIN { … } block runs before input is read, END { … } — after.
Options
| Option | What it does |
|---|---|
-F sep | field separator: a character (-F:), a string or a regex (-F'[,;]'); -F'\t' — tab |
-v name=value | set a variable before the program starts (can be repeated) |
-f file | read the program from a file |
-e 'text' | add a piece of program (together with -f) |
-i library | include a file with functions (e.g. -i inplace) |
-b | treat characters as bytes |
-n | recognize octal (011) and hexadecimal (0x1F) numbers in input data |
-M | arbitrary-precision arithmetic |
-P / -c | strict POSIX mode / old-awk mode without GNU extensions |
-L | warn about dubious and non-portable constructs |
-d | after running, dump all global variables to awkvars.out |
-o | pretty-print the program to awkprof.out instead of running it |
You can also put an assignment among the files: awk '…' x=1 file1 x=2 file2 — it takes effect before the next file is read.
Conditions
| Condition | When it matches |
|---|---|
/re/ | the line matches the regex (ERE) |
$3 ~ /re/ / $3 !~ /re/ | the field matches / doesn't match |
$2 > 10, $1 == "x", NF != 3 | comparison; &&, ||, ! — logic |
NR == 5, NR > 1 | by line number |
cond1, cond2 | range: from the line where cond1 is true to the line where cond2 is true |
BEGIN / END | before input / after input |
Built-in variables
| Variable | Meaning |
|---|---|
$0 | the whole current line |
$1 … $NF | fields; $(NF-1) — the second to last |
NF | number of fields in the line |
NR | line number, counted across all files |
FNR | line number in the current file |
FILENAME | name of the current file |
FS / OFS | input field separator (default — spaces and tabs) / output field separator (a space) |
RS / ORS | input / output record separator (default \n); RS="" — records separated by blank lines |
RSTART / RLENGTH | position and length of the last match() |
ENVIRON["HOME"] | environment variables |
IGNORECASE=1 | case-insensitive regex matching (gawk) |
The language
- Actions:
print a, b(joined withOFS),printf "format", a, b(as in C, no newline at the end), assignments= += -= ++,if (…) … else …,while,for (i = 1; i <= NF; i++),for (k in array),next(go to the next line),exit,delete array[k]. - Arrays are associative:
cnt[$1]++, checking for a key —if ("x" in cnt). - Output can be redirected:
print > "file",print >> "file",print | "sort". - Strings are concatenated by placing them side by side:
x = $1 "-" $2.
Functions
| Function | What it does |
|---|---|
length(s) | string length (no argument — $0); for an array — the number of elements |
substr(s, i, n) | substring starting at position i (from 1), n long |
index(s, t) | position of t in s, 0 if not found |
split(s, a, sep) | split s into array a, return the number of parts |
sub(/re/, "r", s) / gsub(…) | replace the first / all occurrences in s (default $0); & in the replacement is the match |
gensub(/re/, "r", "g", s) | like gsub, but returns the result and understands \\1 (gawk) |
match(s, /re/, a) | find a match, set RSTART, RLENGTH and the groups in a (gawk) |
tolower(s) / toupper(s) | case |
sprintf(format, …) | like printf, but returns the string |
int(x), sqrt(x), rand() | integer part, square root, random number |
system("command") | run a shell command |
strftime("%F", t), systime() | date and current time (gawk) |
asort(a), asorti(a) | sort an array by values / by keys (gawk) |
Examples
printf 'alice 30 dev\nbob 25 ops\ncarol 35 dev\n' > people.txt
# first and last column
awk '{print $1, $NF}' people.txt
# filter by a field value and by a regex
awk '$3 == "dev" {print $1, $2}' people.txt
awk '$2 > 28' people.txt
awk '$1 ~ /^c/' people.txt
# skip the first line (header)
awk 'NR > 1' people.txt
# sum, average, maximum
awk '{s += $2} END {print s, s / NR}' people.txt
awk 'NR == 1 || $2 > max {max = $2} END {print max}' people.txt
# count per group (associative array)
awk '{c[$3]++} END {for (k in c) print k, c[k]}' people.txt
# sum per group
awk '{s[$3] += $2} END {for (k in s) print k, s[k]}' people.txt
# first line for each field value — duplicates are dropped
awk '!seen[$3]++' people.txt
# different separator on input and output
echo 'a:b:c' | awk -F: -v OFS=- '{$1 = $1; print}'
printf 'id,name\n1,alice\n' | awk -F, '{print $2}'
# formatted output
awk '{printf "%-6s %3d\n", $1, $2}' people.txt
# pass a value from the shell
min=30
awk -v min="$min" '$2 >= min {print $1}' people.txt
# swap columns
awk '{print $2, $1}' people.txt
# replace inside a field and print
awk '{gsub(/a/, "A", $1); print $1}' people.txt
# regex group (gawk): pull a number out of a line
echo 'took 125ms' | awk 'match($0, /([0-9]+)ms/, m) {print m[1]}'
# lines that don't have 3 fields — with line numbers
printf 'a b c\nd e\n' | awk 'NF != 3 {print NR": "$0}'
# join two files on the first field: read the first one into an array first
printf 'alice admin\nbob user\n' > roles.txt
awk 'FNR == NR {role[$1] = $2; next} {print $1, role[$1]}' roles.txt people.txt
Pitfalls
- Write the program in single quotes. In double quotes the shell substitutes
$1itself (usually with nothing), andawk "{print $1}"prints whole lines. - Shell variables are not visible inside
'…'. Pass them with-vrather than splicing quotes. - String or number.
$1 > "9"is a string comparison, and10turns out "less" than9. Compare with an unquoted number:$1 > 9. - By default the separator is any run of spaces and tabs, and leading spaces are ignored (
-F' 'means the same). With any other separator empty fields are kept: with-F,,a,,bhas three fields and$3isb; with-F'\t'two tabs in a row also produce an empty field. OFSis applied only when the line is rebuilt. An unmodifiedprintoutputs$0as it was; assign any field ($1 = $1) to rebuild the line with the new separator.print $1 $2concatenates fields without a space. A comma (print $1, $2) insertsOFS.printfdoesn't add a newline — put\nin the format.- The order of
for (k in array)is undefined. If you need an order, sort the output with sort. - awk regexes have no
\1backreferences. For groups usematch(…, …, array)orgensub. - Locale. The sandbox locale is
POSIX, solength()andsubstr()count bytes:éhas length 2, andtoupperdoesn't touch non-ASCII letters. WithLC_ALL=C.utf8characters are counted.