← Full reference

supervisor: supervisord and supervisorctl

The supervisor process manager — the [program:…] section, auto-restart, logs and supervisorctl commands

Supervisor keeps long-running programs up: web applications, queue workers, bots. The supervisord daemon starts the programs described in its configuration, restarts the ones that crash and collects their output into logs. supervisorctl is the control panel: show status, stop, start, reread the configuration. Supervisor is often used where there is no systemd — in containers, for example. The supervisor package is written in Python.

Syntax

supervisord [-c file] [-n]                 # start the daemon; -n — don't go to the background
supervisorctl [-c file] command [name…]    # one command
supervisorctl                              # interactive console: supervisor> status

Where the files live (Ubuntu)

PathWhat it is
/etc/supervisor/supervisord.confmain configuration; includes conf.d/*.conf
/etc/supervisor/conf.d/name.confprogram configurations — one file per program
/var/log/supervisor/supervisord.logthe daemon's own log: starts, crashes, restarts
/var/log/supervisor/program logs by default (name-stdout---supervisor-XXXX.log)
/var/run/supervisor.sockthe socket supervisorctl uses to talk to the daemon

The [program:name] section

KeyWhat it setsDefault
commandthe start command; the program must run in the foregroundrequired
directoryworking directory before startingunchanged
userwhich user to run as (the daemon must run as root)the daemon's user
environmentvariables: KEY="value",KEY2="value2"—
autostartstart when supervisord startstrue
autorestartrestart after exit: true, false, unexpected (only on a code not in exitcodes)unexpected
exitcodes"expected" exit codes for autorestart=unexpected0
startsecshow many seconds the program must run for the start to count as successful1
startretrieshow many times to try starting before giving up (FATAL)3
stopsignalsignal to stop: TERM, INT, QUIT, HUP, KILL, USR1, USR2TERM
stopwaitsecshow long to wait after stopsignal before sending SIGKILL10
stopasgroup / killasgroupsend the signal to the whole process group — so no children are left behindfalse
stdout_logfile / stderr_logfilewhere to write output; AUTO — to /var/log/supervisor/, NONE — nowhereAUTO
redirect_stderrwrite stderr to the same log as stdoutfalse
stdout_logfile_maxbytes / _backupslog size before rotation and number of old copies50MB / 10
numprocshow many copies to start; needs a process_name with %(process_num)1
process_nameprocess name, e.g. %(program_name)s_%(process_num)02d%(program_name)s
prioritystart and stop order: lower — starts earlier999

Several programs are combined into a [group:name] with the key programs=a,b — then you manage them as name:*.

supervisorctl commands

CommandWhat it does
status [name…]program state: name, status, PID, uptime
start name / stop namestart / stop; all — all of them, group:* — the whole group
restart namestop and start; does not reread the configuration
rereadreread the configuration and show what changed (available, changed, disappeared) without applying anything
update [group]apply changes: start new programs, restart changed ones, remove deleted ones
tail [-f] [-N] name [stdout|stderr]show the end of a program's log; -f — follow, -1600 — the last 1600 bytes
maintail [-f]the end of supervisord's own log
pid [name]a program's PID; without a name — the daemon's PID
signal SIGNAL namesend a signal to a program: signal HUP web — e.g. to reread its config
availall programs in the configuration and their state
add / remove nameadd / remove a single group after reread
clear nameclear a program's log
fg nameattach to a program in the foreground (exit — Ctrl-C)
reloadrestart supervisord itself — restarting all programs
shutdownstop supervisord and all programs
help [command]help

States in status

StateMeaning
RUNNINGrunning (has lived longer than startsecs)
STARTINGstarting, startsecs hasn't passed yet
STOPPEDstopped manually or never started (autostart=false)
STOPPINGstopping
BACKOFFcrashed on start, supervisor is trying again
EXITEDexited after a successful start and isn't being restarted
FATALcouldn't be started in startretries attempts — check the logs
UNKNOWNan error in supervisor itself

supervisorctl status returns code 3 if at least one program is not RUNNING, and 4 if it can't reach the daemon.

Examples

# describe a program
cat > /etc/supervisor/conf.d/web.conf <<'EOF'
[program:web]
command=python3 -m http.server 8000
directory=/tmp
user=nobody
autostart=true
autorestart=true
environment=APP_ENV="prod"
stdout_logfile=/var/log/web.out.log
redirect_stderr=true
EOF

# start the daemon (where it doesn't start as a service)
supervisord -c /etc/supervisor/supervisord.conf

# apply a new or changed configuration
supervisorctl reread
supervisorctl update

# state of all programs and of one
supervisorctl status
supervisorctl status web

# restart and view the log
supervisorctl restart web
supervisorctl tail -2000 web
supervisorctl tail -f web stderr

# several workers of one program
cat > /etc/supervisor/conf.d/worker.conf <<'EOF'
[program:worker]
command=sleep 1000
numprocs=2
process_name=%(program_name)s_%(process_num)02d
EOF
supervisorctl reread && supervisorctl update
supervisorctl stop 'worker:*'

# why is the program in FATAL
supervisorctl tail web stderr
supervisorctl maintail | tail -20

Pitfalls

  • restart doesn't see config edits. After changing a .conf you need supervisorctl reread and supervisorctl update — update restarts the changed program itself.
  • The program must run in the foreground. If command goes to the background (daemonizes, like nginx without daemon off;), supervisor sees an instant exit, restarts it again and again, and eventually sets FATAL. Look for the program's "don't go to the background" option (-f, --foreground, daemon off;).
  • FATAL with Exited too quickly — the program exited before startsecs. The reason is usually in the program's logs (tail name stderr): a typo in a path, missing permissions, an unset environment variable.
  • supervisorctl doesn't work without the daemon. The error unix:///var/run/supervisor.sock no such file means supervisord isn't running or listens on a different socket.
  • environment doesn't expand $VAR like a shell does. Quote values with commas and special characters; the daemon's environment variables are substituted via %(ENV_NAME)s.
  • tail -N is bytes, not lines. supervisorctl tail -100 web shows the last 100 bytes.
  • Stopping kills only the main process. If command is a wrapper script, its children may stay alive. Use exec in the wrapper or stopasgroup=true and killasgroup=true.
  • A group is addressed with an asterisk: stop 'worker:*'. Quote the asterisk, otherwise the shell may expand it as a file glob.

See also

service · ps · kill · background · logrotate · tmux