TJ Holowaychuk

Monitoring processes with mon

mon is an extremely small, simple, and light-weight alternative to monit for monitoring processes. Monit’s approach of using a DSL is at times inflexible and often annoying. Mon’s approach is to monitor only a single process, monitoring of several require a mon instance per process, however reducing single points of failure.

This may sound like it’s a lot of work to manage, but I promise you it’s not! There’s another tool for this called mongroup(1) to tie them together.

Monitoring a single process

Mon instances weigh in at about 400kb per process, so you don’t have to worry about spawning a bunch of them. The simplest use of mon accepts a command to execute, and keep alive:

$ mon "echo hello; sleep 5"

Mon will execute the command with /bin/sh -c, so brace expansion and other shell goodies are completely fine. Once the sub process exits, mon will bring it back to life. By default mon logs to stdout:

mon : child 94136
mon : sh -c "echo hey; sleep 5"
hey
mon : last restart less than one second ago
mon : 10 attempts remaining
mon : child 94138
mon : sh -c "echo hey; sleep 5"
hey
mon : last restart 5 seconds ago
mon : 9 attempts remaining
mon : child 94140
mon : sh -c "echo hey; sleep 5"
hey

Mon will continue to execute the command until mon itself is signalled with SIGQUIT or SIGTERM at which point it will signal the child using the same signal, and gracefully exit.

When a process continues to fail upon restart, cyclic spawning will be detected by mon, it will then exit and log a warning. By default 10 restart attempts within 60 seconds are allowed, however you may tweak this value with --attempts <n>.

Failure alerts

Mon provides two facilities for notification. The first is the --on-restart <cmd> flag, which mon will invoke upon any restart. This is useful for emailing administrators the tail of a log file, notifying your team via IRC, etcetera.

The second, and more crucial facility is the --on-error <cmd> flag, which is invoked only when a cyclic restart is detected, and mon has bailed out. Administrators should be notified immediately as the process is completely down.

Daemonization

Mon’s --daemonize flag may be used to background the process and disassociate with the terminal, at this point stdio will be redirected to the log file specified by --log, otherwise defaulting as “./mon.log”, typically this looks something like:

$ mon -d -l /var/log/app.log "node app.js"

Monitoring multiple processes with mongroup

Using mon with several flags is obviously not something you’d want to be typing all the time, but part of the benefit of using mon is that shell scripts become your DSL. If you’re not interested in doing this yourself but have several processes to manage I recommend checking out a project by jgallen23 called mongroup(1). I’ve forked the project to add some goodies, until merged you can find them here.

Mongroup uses a simple configuration file, defaulting to the name ./mongroup.conf, which simply lists out the pids and files directories, as well as one or more processes to spawn:

 pids = /var/run
 logs = /var/log
 web1 = node server 3000
 web2 = node server 3001
 web3 = node server 3002
 web4 = node server 3003
 redis = redis-server

Mongroup ships with typical init-style commands start, stop, restart, status and so on, without a lot of the boilerplate associated with writing init scripts.

Launching processes

To check the status of your monitoring group, simply invoke mongroup status, or mongroup, as it’s the default sub-command. You’ll see a list of process names, pids, and the status itself:

Firing up the entire suite of processes takes only a single command, mongroup start:

Checking the status again will show the process uptime:

You may also start, stop, or restart single processes at a time via:

$ mongroup stop [name]
$ mongroup start [name]
$ mongroup restart [name]

Since mongroup is working-directory sensitive unless the --config <file> flag is specified, I recommend adding a small snippet to your shell profile similar to the following: