← Full reference

Background processes: &, nohup, setsid, disown, jobs, wait

Running in the background, bash job control, and how to make a process survive closing the terminal

A command with & at the end runs in the background: the shell doesn't wait for it to finish and returns the prompt right away. Such processes are jobs of the current shell: you can see them in jobs, bring them back to the foreground and wait for them with wait. But when the terminal closes, the shell sends SIGHUP to its jobs and they exit. To make a process survive leaving SSH, you detach it from the terminal: nohup, disown or setsid.

Syntax

command &                    # run in the background
jobs [-l|-p|-r|-s]           # list jobs
fg [%N] / bg [%N]            # to the foreground / continue in the background
wait [-n] [PID|%N…]          # wait for completion
disown [-h] [-a] [%N]        # detach a job from the shell
nohup command [arguments…] [&]
setsid [-f] [-w] command [arguments…]

Job control (built into bash)

Command / keysWhat it does
command &run in the background; prints [1] 4242 — job number and PID
$!PID of the last background process — save it to a variable
Ctrl+Zsuspend the current command (signal SIGTSTP); it becomes Stopped
bg [%N]continue a stopped job in the background
fg [%N]bring a job back to the foreground
jobslist: [1]- Running sleep 100 &; + — current, - — previous
jobs -l / -pwith PIDs / PIDs only
jobs -r / -srunning only / stopped only
kill %Nsend a signal to a job by number

Job references: %1 — by number, %+ or %% — current, %- — previous, %sleep — by the start of the command, %?str — by substring.

wait

FormWhat it does
waitwait for all background processes; exit code 0
wait PID / wait %Nwait for a specific one; the exit code is that process's exit code
wait -nwait for the first one to finish, whichever it is; its exit code
wait -n -p varsame, and store the PID of the finished one in var (bash 5.1+)

If there is no such process among the shell's children — exit code 127.

Surviving terminal close

MethodHow it worksOutput
nohup cmd &runs the command ignoring SIGHUPif stdout is a terminal, writes to ./nohup.out (or ~/nohup.out), mode 600
cmd & disownremoves the job from the shell's table — it won't send it SIGHUPstill attached to the terminal — redirect it yourself
disown -h %Nthe job stays in jobs, but it won't get SIGHUPsame
setsid cmdruns in a new session with no controlling terminal; PID 1 becomes the parentsame — redirect it yourself
tmux / screenthe terminal lives on the server, you can reconnecteverything is visible after you log back in
systemd unitthe proper way for long-running servicesjournalctl

setsid options: -f — always fork a new process (return control immediately), -w — wait for the program and return its exit code, -c — make the current terminal the controlling one.

Examples

# run in the background and remember the PID
sleep 300 &
pid=$!
jobs -l
kill "$pid"

# continue a stopped command in the background
# (you pressed Ctrl+Z during a long cp)
bg
jobs

# several tasks in parallel and wait for all of them, checking codes
for host in web1 web2 web3; do
    ping -c1 -W1 "$host" >/dev/null 2>&1 &
done
wait

# wait for a specific one and get its exit code
(sleep 1; exit 3) &
wait $!
echo "code: $?"        # 3

# survives leaving SSH; output to its own log, stdin closed
nohup ./long-task.sh > task.log 2>&1 < /dev/null &

# already started without nohup — detach after the fact
./long-task.sh > task.log 2>&1 &
disown

# fully detach from the terminal and session
setsid -f ./daemon.sh > /tmp/daemon.log 2>&1 < /dev/null

Pitfalls

  • A plain cmd & dies when SSH closes. An interactive shell that receives SIGHUP forwards it to all its jobs. nohup, disown, disown -h and setsid protect against that.
  • nohup doesn't redirect output if it's not a terminal already. nohup.out appears only when stdout is a terminal. And disown and setsid never redirect: a process writing to a closed terminal gets a write error and may crash. Always add > file 2>&1 < /dev/null.
  • nohup doesn't do & for you. nohup cmd without & holds the terminal until it finishes.
  • wait with no arguments returns 0 even if jobs failed. If you need each exit code, save $! and call wait "$pid" one by one.
  • wait only sees its own children. A process started from another shell or detached with setsid -f can't be waited for this way: "is not a child of this shell", exit code 127. For other processes, loop on kill -0 PID.
  • jobs and %N work only in the shell that started the job. In a new SSH session jobs is empty — find the process with pgrep/ps.
  • A background process that reads from the terminal gets stopped (Stopped (tty input)). Close stdin: < /dev/null.
  • $! is the PID of the last element of a pipeline a | b &, not the first.
  • For real services nohup is a hack: no restart on crash and no proper logs. Use systemd or supervisor.

See also

kill · ps · timeout · tmux · redirection · supervisorctl · exit-codes