Redirection and pipelines
Where a command's output goes — >, >>, 2>&1, &>, here-doc, the | pipeline and pipefail
Every program has three standard streams: input (stdin, descriptor 0), output (stdout, 1) and errors (stderr, 2). Redirections replace these streams with files, and the | pipeline connects one command's output to another's input.
Syntax
command > file # stdout to a file
command 2> file # stderr to a file
command < file # stdin from a file
command1 | command2 # stdout of the first — to stdin of the second
Redirections are processed left to right, before the command starts. They can appear anywhere on the line: > f echo hi works too.
Output
| Operator | What it does |
|---|---|
> file | stdout to a file; the file is created or truncated to zero |
>> file | stdout is appended to the end of the file |
2> file / 2>> file | the same for stderr |
&> file / &>> file | stdout and stderr into one file: overwrite / append (bash) |
> file 2>&1 | the same, portable form for sh |
>| file | overwrite even if noclobber is set |
> file (no command) | create an empty file or truncate an existing one |
>/dev/null | discard output; 2>/dev/null — discard errors |
Input
| Operator | What it does |
|---|---|
< file | stdin from a file |
<<MARKER … MARKER | here-doc: lines up to the marker go to stdin, $var and $(…) are expanded |
<<'MARKER' | here-doc without substitutions: text as is |
<<-MARKER | here-doc with leading tabs stripped from lines (for indentation in scripts) |
<<< "string" | here-string: a single string (with an appended \n) to stdin |
<(command) | process substitution: a command's output as a "file" to read |
>(command) | the reverse: a "file" whose writes go to the command's stdin |
Descriptors
| Operator | What it does |
|---|---|
2>&1 | stderr to wherever stdout points right now |
>&2 (1>&2) | stdout to stderr — how scripts write error messages |
n>&- | close descriptor n |
exec 3> file | open descriptor 3 for writing for the whole script; echo x >&3 writes to the file |
exec 3< file | open descriptor 3 for reading; read -r line <&3 |
exec 3<> file | open for reading and writing |
exec > file 2>&1 | redirect all further output of the script |
{ …; } > file | redirect the output of a group of commands at once |
Pipelines
| Form | What it does |
|---|---|
a | b | stdout of a goes to stdin of b; both commands run at the same time |
a |& b | stdout and stderr of a to stdin of b (same as a 2>&1 | b) |
${PIPESTATUS[@]} | exit codes of all parts of the last pipeline |
set -o pipefail | the pipeline's code is the code of the last failed part, not just the last part |
! a | b | invert the exit code of the whole pipeline |
By default a pipeline's exit code is that of the last command: false | true counts as success, an error in the middle is lost. pipefail fixes this:
false | true; echo $? # 0
set -o pipefail
false | true; echo $? # 1
Examples
# write output to a file, then append another line
echo one > out.txt
echo two >> out.txt
# errors separately, normal output to the screen
ls /etc/hostname /nope 2> errors.txt
# everything into one file: first stdout to the file, then stderr to the same place
ls /etc/hostname /nope > all.txt 2>&1
ls /etc/hostname /nope &> all.txt # shorter, bash only
# hide permission errors
find / -name '*.conf' 2>/dev/null
# count lines without printing the file name
wc -l < /etc/passwd
# an error message from a script — to stderr
echo "file not found" >&2
# here-doc: variables inside are expanded
cat <<EOF
home directory: $HOME
EOF
# here-doc with quotes around the marker: text goes as is
cat <<'EOF' > script.sh
echo "$HOME"
EOF
# here-string
tr a-z A-Z <<< "hello"
# send errors into the pipeline along with the output
ls /nope 2>&1 | wc -l
# compare the output of two commands without temporary files
diff <(ls dir1) <(ls dir2)
# read a command's output in a loop without losing variables (no subshell)
while read -r line; do n=$((n+1)); done < <(ls /etc)
# exit codes of each part of a pipeline
false | true; echo "${PIPESTATUS[@]}" # 1 0
# your own descriptor for a log
exec 3>> /tmp/script.log
echo "step 1" >&3
exec 3>&-
Pitfalls
- The order of
2>&1matters.cmd > f 2>&1writes both streams to the file.cmd 2>&1 > fsends stderr to where stdout pointed before the redirection (the screen), and only stdout goes to the file. sort f > fdestroys the file. The shell truncatesfbeforesortstarts, and it reads an empty file. Write to a temporary file and rename it (sorthassort -o f ffor this).>overwrites silently.set -o noclobberforbids overwriting existing files with>; to force —>|.- Each part of a pipeline runs in a subshell.
echo "a b" | read x ywon't changexin the current shell, and a counter incat f | while …is empty again after the loop. Useread x y <<< "a b"orwhile … done < file/< <(command). sudo echo text > /root/filewon't work: the redirection is performed by your shell, not bysudo. Useecho text | sudo tee file.<<-strips only tabs, not spaces. The closing marker must be alone on its line, with no spaces after it.&>is bash. In#!/bin/sh(dash)cmd &> fmeans "runcmdin the background and create an emptyf". Write> f 2>&1.
See also
tee · exit-codes · variables · xargs