Supervisor

apt

  • Installation

    sudo apt install supervisor

  • Service

    $ sudo service supervisor start
Starting supervisor: supervisord.
$ sudo service supervisor restart
Restarting supervisor: supervisord.
$ ps -ef | grep supervisor
root     17452     1  0 19:23 ?        00:00:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
$ sudo service supervisor stop
Stopping supervisor: supervisord.
$ ps -ef | grep supervisor

  • /etc/supervisor/supervisord.conf

    ; supervisor config file

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0766                       ; sockef file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor            ; ('AUTO' child log dir, default $TEMP)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /etc/supervisor/conf.d/*.conf

  • /etc/supervisor/conf.d/rlog.conf

    [program:rlog]
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py rlistlog --key=django:logit:tt4it --filename=/tmp/tt4it.logit.log
autostart=true
autorestart=true
startretries=3
exitcodes=0,1,2
stopsignal=QUIT
stdout_logfile=/var/log/supervisor_rlog_access.log
stderr_logfile=/var/log/supervisor_rlog_error.log
user=diors
    • python /home/diors/work/tt4it/manage.py rlistlog --key=django:logit:tt4it --filename=/tmp/tt4it.logit.log cause ImportError: No module named django.core.management

  • Supervisorctl

    $ sudo supervisorctl
supervisor> quit
$ sudo supervisorctl reread
rlog: available
$ sudo supervisorctl update
rlog: added process group
$ sudo supervisorctl
rlog                             FATAL      Exited too quickly (process log may have deta
ils)
supervisor> quit
$ sudo supervisorctl reread
rlog: changed
$ sudo supervisorctl update
rlog: stopped
rlog: updated process group
$ sudo supervisorctl
rlog                             RUNNING    pid 17767, uptime 0:00:07
supervisor> quit

  • Problems

    • [Errno 13] Permission denied

      $ supervisorctl
error: <class 'socket.error'>, [Errno 13] Permission denied: file: /usr/lib/python2.7/soc
ket.py line: 224

yum

  • Installation

    sudo yum install supervisor

  • Service

    $ sudo service supervisord start
Redirecting to /bin/systemctl start  supervisord.service
$ sudo service supervisord restart
Redirecting to /bin/systemctl restart  supervisord.service
$ ps -ef | grep supervisor
root     28383     1  0 16:51 ?        00:00:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
$ sudo service supervisord stop
Redirecting to /bin/systemctl stop  supervisord.service
$ ps -ef | grep supervisor

  • Configuration File

    $ cd /etc/supervisord.
supervisord.conf  supervisord.d/

PIP

  • Installation

    pip install supervisor

  • Creating Configuration File

    # This won’t work if you do not have root access
# If you don’t have root access, or you’d rather not put the supervisord.conf file in /etc/supervisord.conf`
# You can place it in a specify directory (echo_supervisord_conf > /home/diors/supervisord.conf)
# And start supervisord with the -c flag in order to specify the configuration file location
echo_supervisord_conf > /etc/supervisord.conf  # Need root access
echo_supervisord_conf > /home/diors/supervisord.conf  # Specify the configuration file location

  • Modify Configuration File

    ;[include]
;files = relative/directory/*.ini

==>

[include]
files = /home/diors/supervisor/*.ini

    Tips: Supervisor will include files when started

    Tips: Configuration Dir: /home/diors/supervisor/

    Tips: Log Dir: /home/diors/supervisorlog/

    Tips: Create Above Dir if Not Exists

  • Start Supervisor

    supervisord  # Default /etc/supervisord.conf
supervisord -c /home/diors/supervisord.conf  # Specify the configuration file location

  • /home/diors/supervisor/rlog.ini

    [program:rlog]
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py rlistlog --key=django:logit:tt4it --filename=/tmp/tt4it.logit.log
autostart=true
autorestart=true
startretries=3
exitcodes=0,1,2
stopsignal=QUIT
stdout_logfile=/home/diors/supervisorlog/supervisor_rlog_access.log
stderr_logfile=/home/diors/supervisorlog/supervisor_rlog_error.log
user=diors

  • Enter Supervisorctl

    supervisorctl  # Default /etc/supervisord.conf
supervisorctl -c /home/diors/supervisord.conf  # Specify the configuration file location

  • Supervisorctl Commands

    $ supervisorctl -c /home/diors/supervisord.conf reread
rlog: changed
$ supervisorctl -c /home/diors/supervisord.conf update
rlog: stopped
rlog: updated process group
$ supervisorctl -c /home/diors/supervisord.conf reread
No config updates to processes

  • Alias

    • ~/.bashrc

      # Supervisor Start
alias supervisord='supervisord -c /home/diors/supervisord.conf'
alias supervisorctl='supervisorctl -c /home/diors/supervisord.conf'
# Supervisor End

      Tips: If not have effect, exec source ~/.bashrc

    • Commands

      $ supervisorctl reread
rlog: changed
$ supervisorctl update
rlog: stopped
rlog: updated process group
$ supervisorctl status rlog
rlog                             RUNNING   pid 15343, uptime 0:00:15

Configuration

  • [program:nodehook] - Define the program to monitor. We'll call it "nodehook".
  • command - This is the command to run that kicks off the monitored process. We use "node" and run the "http.js" file. If you needed to pass any command line arguments or other data, you could do so here.
  • directory - Set a directory for Supervisord to "cd" into for before running the process, useful for cases where the process assumes a directory structure relative to the location of the executed script.
  • autostart - Setting this "true" means the process will start when Supervisord starts (essentially on system boot).
  • autorestart - If this is "true", the program will be restarted if it exits unexpectedly.
  • startretries - The number of retries to do before the process is considered "failed"
  • stderr_logfile - The file to write any errors output.
  • stdout_logfile - The file to write any regular output.
  • user - The user the process is run as.
  • environment - Environment variables to pass to the process.
  • more detail

Tips: Supervisord won't create a directory for logs if they do not exist

Supervisord

$ supervisord -h
supervisord -- run a set of applications as daemons.

Usage: /usr/bin/supervisord [options]

Options:
-c/--configuration FILENAME -- configuration file
-n/--nodaemon -- run in the foreground (same as 'nodaemon true' in config file)
-h/--help -- print this usage message and exit
-v/--version -- print supervisord version number and exit
-u/--user USER -- run supervisord as this user (or numeric uid)
-m/--umask UMASK -- use this umask for daemon subprocess (default is 022)
-d/--directory DIRECTORY -- directory to chdir to when daemonized
-l/--logfile FILENAME -- use FILENAME as logfile path
-y/--logfile_maxbytes BYTES -- use BYTES to limit the max size of logfile
-z/--logfile_backups NUM -- number of backups to keep when max bytes reached
-e/--loglevel LEVEL -- use LEVEL as log level (debug,info,warn,error,critical)
-j/--pidfile FILENAME -- write a pid file for the daemon process to FILENAME
-i/--identifier STR -- identifier used for this instance of supervisord
-q/--childlogdir DIRECTORY -- the log directory for child process logs
-k/--nocleanup --  prevent the process from performing cleanup (removal of
                   old automatic child log files) at startup.
-a/--minfds NUM -- the minimum number of file descriptors for start success
-t/--strip_ansi -- strip ansi escape codes from process output
--minprocs NUM  -- the minimum number of processes available for start success
--profile_options OPTIONS -- run supervisord under profiler and output
                             results based on OPTIONS, which  is a comma-sep'd
                             list of 'cumulative', 'calls', and/or 'callers',
                             e.g. 'cumulative,callers')

Supervisorctl

  • supervisorctl -h

    $ supervisorctl -h
supervisorctl -- control applications run by supervisord from the cmd line.

Usage: /usr/bin/supervisorctl [options] [action [arguments]]

Options:
-c/--configuration -- configuration file path (default /etc/supervisor.conf)
-h/--help -- print usage message and exit
-i/--interactive -- start an interactive shell after executing commands
-s/--serverurl URL -- URL on which supervisord server is listening
     (default "http://localhost:9001").
-u/--username -- username to use for authentication with server
-p/--password -- password to use for authentication with server
-r/--history-file -- keep a readline history (if readline is available)

action [arguments] -- see below

Actions are commands like "tail" or "stop".  If -i is specified or no action is
specified on the command line, a "shell" interpreting actions typed
interactively is started.  Use the action "help" to find out about available
actions.

  • supervisorctl help

    $ supervisorctl help

default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

Web Interface

[inet_http_server]
port = 9001
username = user # Basic auth username
password = pass # Basic auth password

XML-RPC Interface

Configuration Examples

  • Python manage.py

    # runserver
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py runserver 0.0.0.0:9999
stopsignal=INT  # ``signal SIGQUIT`` can't kill as expect, Use the "fast" shutdown ``signal SIGINT``
stopasgroup=true
killasgroup=true

# rlog
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py rlistlog --key=django:logit:tt4it --filename=/tmp/tt4it.logit.log

# django_q
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py qcluster
stopsignal=INT  # ``signal SIGQUIT`` can't kill as expect, Use the "fast" shutdown ``signal SIGINT``
stopasgroup=true
killasgroup=true

# beanstalk_worker
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py beanstalk_worker -w 5 -l debug
stopsignal=KILL  # ``signal SIGQUIT/SIGINT`` can't kill as expect, Use ``signal SIGKILL``
stopasgroup=true
killasgroup=true

  • uWSGI

    command=/home/diors/env/bin/uwsgi --ini /home/diors/work/tt4it/tt4it/uwsgi/tt4it.ini

  • Gunicorn

    TODO

    Tips: Daemon of Gunicorn should set False

  • Gogs

    directory=/home/diors/gogs/
command=/home/diors/gogs/gogs web
environment=HOME="/home/diors", USER="diors"

Logs

  • Print

  • Logging

    logger = logging.getLogger('xxx')
logger.addHandler(logging.StreamHandler())
logger.info('ooo')

  • Multi Process

    # beanstalk_worker
command=/home/diors/env/bin/python /home/diors/work/tt4it/manage.py beanstalk_worker -w 5 -l debug

# Process
diors    27842 16899  0 18:17 ?        00:00:00 /home/diors/env/bin/python manage.py beanstalk_worker -w 2
diors    27852 27842  0 18:17 ?        00:00:00 /home/diors/env/bin/python manage.py beanstalk_worker -w 2
diors    27853 27842  0 18:17 ?        00:00:00 /home/diors/env/bin/python manage.py beanstalk_worker -w 2

    • Python logging is not process-safe, so when start 5 workers for beanstalk, will miss some logs.

      # Testit ``-w 1``/``-w 2``
for i in range(8): client.call('beanstalkd.background_counting', str(i))

    • When restart, only master process will be killed.

      diors    27852 1  0 18:17 ?        00:00:00 /home/diors/env/bin/python manage.py beanstalk_worker -w 2
diors    27853 1  0 18:17 ?        00:00:00 /home/diors/env/bin/python manage.py beanstalk_worker -w 2
    • Solution
      # stopsignal=INT  # ``signal SIGQUIT`` can't kill as expect, Use the "fast" shutdown ``signal SIGINT``
stopsignal=KILL  # ``signal SIGQUIT/SIGINT`` can't kill as expect, Use ``signal SIGKILL``
stopasgroup=true
killasgroup=true

Supervisord.log

  • exited

    • exit status 0

    • exit status 1; expected

      • Raise Error In Codes

    • exit status 1; not expected

    • terminated by SIGQUIT

      • stop xxx

        waiting ==> stopped

      • restart xxx

        waiting ==> stopped ==> spawned ==> success

      • xxx

    • terminated by SIGKILL; not expected

      • kill -9 xxx

        exited ==> spawned ==> success

      • Other Condition Lead To Process Killed

        • Who “Killed” my process and why?

        • [Ubuntu] sudo vim /var/log/kern.log

          Sep 21 07:36:02 iZ28e73hi9aZ kernel: [37656975.398263] Out of memory: Kill process 18738 (python) score 429 or sacrifice child
Sep 21 07:36:02 iZ28e73hi9aZ kernel: [37656975.399284] Killed process 18738 (python) total-vm:1760008kB, anon-rss:1622680kB, file-rss:1048kB
        • References: Leaking Memory

References

[1] Supervisord@TT4IT, Supervisord — Supervisor process control system for UNIX

[2] DigitalOcean, How To Install and Manage Supervisor on Ubuntu and Debian VPS

[3] Servers for Hackers, Monitoring Processes with Supervisord

results matching ""

    No results matching ""