← Full reference

awk

Column-based text processing — selecting fields, filters, counts and sums

awk splits each line into fields and runs rules of the form condition { action } on it. It's a small language with variables, arrays, functions and printf, which makes it handy for selecting columns, filtering lines by a field's value and computing totals. In the sandbox awk is GNU awk (gawk) 5.2.

Syntax

awk [options] 'program' [file…]
awk [options] -f program.awk [file…]

A program is a set of condition { action } rules. For each input line awk checks the rules in order. No condition — the action runs for every line. No action — the whole line is printed ({print}). A BEGIN { … } block runs before input is read, END { … } — after.

Options

OptionWhat it does
-F sepfield separator: a character (-F:), a string or a regex (-F'[,;]'); -F'\t' — tab
-v name=valueset a variable before the program starts (can be repeated)
-f fileread the program from a file
-e 'text'add a piece of program (together with -f)
-i libraryinclude a file with functions (e.g. -i inplace)
-btreat characters as bytes
-nrecognize octal (011) and hexadecimal (0x1F) numbers in input data
-Marbitrary-precision arithmetic
-P / -cstrict POSIX mode / old-awk mode without GNU extensions
-Lwarn about dubious and non-portable constructs
-dafter running, dump all global variables to awkvars.out
-opretty-print the program to awkprof.out instead of running it

You can also put an assignment among the files: awk '…' x=1 file1 x=2 file2 — it takes effect before the next file is read.

Conditions

ConditionWhen it matches
/re/the line matches the regex (ERE)
$3 ~ /re/ / $3 !~ /re/the field matches / doesn't match
$2 > 10, $1 == "x", NF != 3comparison; &&, ||, ! — logic
NR == 5, NR > 1by line number
cond1, cond2range: from the line where cond1 is true to the line where cond2 is true
BEGIN / ENDbefore input / after input

Built-in variables

VariableMeaning
$0the whole current line
$1 … $NFfields; $(NF-1) — the second to last
NFnumber of fields in the line
NRline number, counted across all files
FNRline number in the current file
FILENAMEname of the current file
FS / OFSinput field separator (default — spaces and tabs) / output field separator (a space)
RS / ORSinput / output record separator (default \n); RS="" — records separated by blank lines
RSTART / RLENGTHposition and length of the last match()
ENVIRON["HOME"]environment variables
IGNORECASE=1case-insensitive regex matching (gawk)

The language

  • Actions: print a, b (joined with OFS), printf "format", a, b (as in C, no newline at the end), assignments = += -= ++, if (…) … else …, while, for (i = 1; i <= NF; i++), for (k in array), next (go to the next line), exit, delete array[k].
  • Arrays are associative: cnt[$1]++, checking for a key — if ("x" in cnt).
  • Output can be redirected: print > "file", print >> "file", print | "sort".
  • Strings are concatenated by placing them side by side: x = $1 "-" $2.

Functions

FunctionWhat it does
length(s)string length (no argument — $0); for an array — the number of elements
substr(s, i, n)substring starting at position i (from 1), n long
index(s, t)position of t in s, 0 if not found
split(s, a, sep)split s into array a, return the number of parts
sub(/re/, "r", s) / gsub(…)replace the first / all occurrences in s (default $0); & in the replacement is the match
gensub(/re/, "r", "g", s)like gsub, but returns the result and understands \\1 (gawk)
match(s, /re/, a)find a match, set RSTART, RLENGTH and the groups in a (gawk)
tolower(s) / toupper(s)case
sprintf(format, …)like printf, but returns the string
int(x), sqrt(x), rand()integer part, square root, random number
system("command")run a shell command
strftime("%F", t), systime()date and current time (gawk)
asort(a), asorti(a)sort an array by values / by keys (gawk)

Examples

printf 'alice 30 dev\nbob 25 ops\ncarol 35 dev\n' > people.txt

# first and last column
awk '{print $1, $NF}' people.txt

# filter by a field value and by a regex
awk '$3 == "dev" {print $1, $2}' people.txt
awk '$2 > 28' people.txt
awk '$1 ~ /^c/' people.txt

# skip the first line (header)
awk 'NR > 1' people.txt

# sum, average, maximum
awk '{s += $2} END {print s, s / NR}' people.txt
awk 'NR == 1 || $2 > max {max = $2} END {print max}' people.txt

# count per group (associative array)
awk '{c[$3]++} END {for (k in c) print k, c[k]}' people.txt

# sum per group
awk '{s[$3] += $2} END {for (k in s) print k, s[k]}' people.txt

# first line for each field value — duplicates are dropped
awk '!seen[$3]++' people.txt

# different separator on input and output
echo 'a:b:c' | awk -F: -v OFS=- '{$1 = $1; print}'
printf 'id,name\n1,alice\n' | awk -F, '{print $2}'

# formatted output
awk '{printf "%-6s %3d\n", $1, $2}' people.txt

# pass a value from the shell
min=30
awk -v min="$min" '$2 >= min {print $1}' people.txt

# swap columns
awk '{print $2, $1}' people.txt

# replace inside a field and print
awk '{gsub(/a/, "A", $1); print $1}' people.txt

# regex group (gawk): pull a number out of a line
echo 'took 125ms' | awk 'match($0, /([0-9]+)ms/, m) {print m[1]}'

# lines that don't have 3 fields — with line numbers
printf 'a b c\nd e\n' | awk 'NF != 3 {print NR": "$0}'

# join two files on the first field: read the first one into an array first
printf 'alice admin\nbob user\n' > roles.txt
awk 'FNR == NR {role[$1] = $2; next} {print $1, role[$1]}' roles.txt people.txt

Pitfalls

  • Write the program in single quotes. In double quotes the shell substitutes $1 itself (usually with nothing), and awk "{print $1}" prints whole lines.
  • Shell variables are not visible inside '…'. Pass them with -v rather than splicing quotes.
  • String or number. $1 > "9" is a string comparison, and 10 turns out "less" than 9. Compare with an unquoted number: $1 > 9.
  • By default the separator is any run of spaces and tabs, and leading spaces are ignored (-F' ' means the same). With any other separator empty fields are kept: with -F,, a,,b has three fields and $3 is b; with -F'\t' two tabs in a row also produce an empty field.
  • OFS is applied only when the line is rebuilt. An unmodified print outputs $0 as it was; assign any field ($1 = $1) to rebuild the line with the new separator.
  • print $1 $2 concatenates fields without a space. A comma (print $1, $2) inserts OFS.
  • printf doesn't add a newline — put \n in the format.
  • The order of for (k in array) is undefined. If you need an order, sort the output with sort.
  • awk regexes have no \1 backreferences. For groups use match(…, …, array) or gensub.
  • Locale. The sandbox locale is POSIX, so length() and substr() count bytes: é has length 2, and toupper doesn't touch non-ASCII letters. With LC_ALL=C.utf8 characters are counted.

See also

sed · grep · cut · sort · datamash · Regular expressions