supervisor: supervisord and supervisorctl
The supervisor process manager — the [program:…] section, auto-restart, logs and supervisorctl commands
Supervisor keeps long-running programs up: web applications, queue workers, bots. The supervisord daemon starts the programs described in its configuration, restarts the ones that crash and collects their output into logs. supervisorctl is the control panel: show status, stop, start, reread the configuration. Supervisor is often used where there is no systemd — in containers, for example. The supervisor package is written in Python.
Syntax
supervisord [-c file] [-n] # start the daemon; -n — don't go to the background
supervisorctl [-c file] command [name…] # one command
supervisorctl # interactive console: supervisor> status
Where the files live (Ubuntu)
| Path | What it is |
|---|---|
/etc/supervisor/supervisord.conf | main configuration; includes conf.d/*.conf |
/etc/supervisor/conf.d/name.conf | program configurations — one file per program |
/var/log/supervisor/supervisord.log | the daemon's own log: starts, crashes, restarts |
/var/log/supervisor/ | program logs by default (name-stdout---supervisor-XXXX.log) |
/var/run/supervisor.sock | the socket supervisorctl uses to talk to the daemon |
The [program:name] section
| Key | What it sets | Default |
|---|---|---|
command | the start command; the program must run in the foreground | required |
directory | working directory before starting | unchanged |
user | which user to run as (the daemon must run as root) | the daemon's user |
environment | variables: KEY="value",KEY2="value2" | — |
autostart | start when supervisord starts | true |
autorestart | restart after exit: true, false, unexpected (only on a code not in exitcodes) | unexpected |
exitcodes | "expected" exit codes for autorestart=unexpected | 0 |
startsecs | how many seconds the program must run for the start to count as successful | 1 |
startretries | how many times to try starting before giving up (FATAL) | 3 |
stopsignal | signal to stop: TERM, INT, QUIT, HUP, KILL, USR1, USR2 | TERM |
stopwaitsecs | how long to wait after stopsignal before sending SIGKILL | 10 |
stopasgroup / killasgroup | send the signal to the whole process group — so no children are left behind | false |
stdout_logfile / stderr_logfile | where to write output; AUTO — to /var/log/supervisor/, NONE — nowhere | AUTO |
redirect_stderr | write stderr to the same log as stdout | false |
stdout_logfile_maxbytes / _backups | log size before rotation and number of old copies | 50MB / 10 |
numprocs | how many copies to start; needs a process_name with %(process_num) | 1 |
process_name | process name, e.g. %(program_name)s_%(process_num)02d | %(program_name)s |
priority | start and stop order: lower — starts earlier | 999 |
Several programs are combined into a [group:name] with the key programs=a,b — then you manage them as name:*.
supervisorctl commands
| Command | What it does |
|---|---|
status [name…] | program state: name, status, PID, uptime |
start name / stop name | start / stop; all — all of them, group:* — the whole group |
restart name | stop and start; does not reread the configuration |
reread | reread the configuration and show what changed (available, changed, disappeared) without applying anything |
update [group] | apply changes: start new programs, restart changed ones, remove deleted ones |
tail [-f] [-N] name [stdout|stderr] | show the end of a program's log; -f — follow, -1600 — the last 1600 bytes |
maintail [-f] | the end of supervisord's own log |
pid [name] | a program's PID; without a name — the daemon's PID |
signal SIGNAL name | send a signal to a program: signal HUP web — e.g. to reread its config |
avail | all programs in the configuration and their state |
add / remove name | add / remove a single group after reread |
clear name | clear a program's log |
fg name | attach to a program in the foreground (exit — Ctrl-C) |
reload | restart supervisord itself — restarting all programs |
shutdown | stop supervisord and all programs |
help [command] | help |
States in status
| State | Meaning |
|---|---|
RUNNING | running (has lived longer than startsecs) |
STARTING | starting, startsecs hasn't passed yet |
STOPPED | stopped manually or never started (autostart=false) |
STOPPING | stopping |
BACKOFF | crashed on start, supervisor is trying again |
EXITED | exited after a successful start and isn't being restarted |
FATAL | couldn't be started in startretries attempts — check the logs |
UNKNOWN | an error in supervisor itself |
supervisorctl status returns code 3 if at least one program is not RUNNING, and 4 if it can't reach the daemon.
Examples
# describe a program
cat > /etc/supervisor/conf.d/web.conf <<'EOF'
[program:web]
command=python3 -m http.server 8000
directory=/tmp
user=nobody
autostart=true
autorestart=true
environment=APP_ENV="prod"
stdout_logfile=/var/log/web.out.log
redirect_stderr=true
EOF
# start the daemon (where it doesn't start as a service)
supervisord -c /etc/supervisor/supervisord.conf
# apply a new or changed configuration
supervisorctl reread
supervisorctl update
# state of all programs and of one
supervisorctl status
supervisorctl status web
# restart and view the log
supervisorctl restart web
supervisorctl tail -2000 web
supervisorctl tail -f web stderr
# several workers of one program
cat > /etc/supervisor/conf.d/worker.conf <<'EOF'
[program:worker]
command=sleep 1000
numprocs=2
process_name=%(program_name)s_%(process_num)02d
EOF
supervisorctl reread && supervisorctl update
supervisorctl stop 'worker:*'
# why is the program in FATAL
supervisorctl tail web stderr
supervisorctl maintail | tail -20
Pitfalls
restartdoesn't see config edits. After changing a.confyou needsupervisorctl rereadandsupervisorctl update—updaterestarts the changed program itself.- The program must run in the foreground. If
commandgoes to the background (daemonizes, likenginxwithoutdaemon off;), supervisor sees an instant exit, restarts it again and again, and eventually setsFATAL. Look for the program's "don't go to the background" option (-f,--foreground,daemon off;). FATALwithExited too quickly— the program exited beforestartsecs. The reason is usually in the program's logs (tail name stderr): a typo in a path, missing permissions, an unset environment variable.supervisorctldoesn't work without the daemon. The errorunix:///var/run/supervisor.sock no such filemeanssupervisordisn't running or listens on a different socket.environmentdoesn't expand$VARlike a shell does. Quote values with commas and special characters; the daemon's environment variables are substituted via%(ENV_NAME)s.tail -Nis bytes, not lines.supervisorctl tail -100 webshows the last 100 bytes.- Stopping kills only the main process. If
commandis a wrapper script, its children may stay alive. Useexecin the wrapper orstopasgroup=trueandkillasgroup=true. - A group is addressed with an asterisk:
stop 'worker:*'. Quote the asterisk, otherwise the shell may expand it as a file glob.