YARB

god is a great ruby-based alternative to monit and other process monitoring tools. While I have had success with monit, I found its configuration syntax tedious. Monit’s configuration syntax makes it painful to do things like decrease/increase the number of mongrels you are monitoring, etc.

Note: the following assumes we’re on some kind of Fedora Core (or similar) system. Replace ”/etc/init.d/mysql stop|start|restart” with however this is done in your distro.

Configuration is where god shines. Let’s get started:

sudo gem install god

In this tutorial, I will only be covering how to monitor your MySQL process using god. The main god website has an excellent tutorial on how to monitor your mongrels.

God works by monitoring pid files. It has other functionality as well, but for MySQL 5.x monitoring, all we need is the location of the MySQL PID file. You can find this on your system with:

locate .pid | grep mysql

On my system, the MySQL pid file was located at:

/var/run/mysqld/mysqld.pid

Next we need to know how to stop/start/restart MySQL. On most systems, this can simply be done with:

cd /etc/init.d
sudo ./mysqld stop|start|restart

It’s a good idea to make sure these commands work by hand, of course, before assuming god should use them to manage MySQL.

Assuming your information matches the above, the following god config file should do the trick:

# God config file.
#
# Documentation: http://god.rubyforge.org/
#
# run with:  god -c /root/monitor.god
#

God.watch do |w|
  w.name = 'mysql-process'
  w.group = 'mysql'
  w.interval = 30.seconds # default      
  w.start = "cd /etc/init.d && ./mysqld start" 
  w.stop = "cd /etc/init.d && ./mysqld start" 
  w.restart = "cd /etc/init.d && ./mysqld restart" 
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = '/var/run/mysqld/mysqld.pid'
  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Place this into a file located at ’/root/monitor.god’. (for the below examples to work)

In order to test god, kick it into non-daemonized mode with:

sudo god -c /root/monitor.god -D

You should see some output like:

I, [2008-01-26 00:30:05 #1841]  INFO -- : Started on drbunix:///tmp/god.17165.sock
I, [2008-01-26 00:30:05 #1841]  INFO -- : mysql-process move 'unmonitored' to 'up'
I, [2008-01-26 00:30:06 #1841]  INFO -- : mysql-process [ok] process is running (ProcessRunning)

In another terminal, stop MySQL by hand:

cd /etc/init.d
sudo ./mysqld stop

This may not replicate exactly what happens when MySQL goes down in the wild, but at least you can test god’s basic functionality.

You should see the command line output of god indicate that it is restarting MySQL:

I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process [trigger] mysql-process God::Conditions::ProcessRunning: no such pid file: /var/run/mysqld/mysqld.pid (ProcessRunning)
I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process move 'up' to 'start'
I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process before_start: no pid file to delete (CleanPidFile)
I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process start: cd /etc/init.d && ./mysqld start
I, [2008-01-26 00:46:12 #18173]  INFO -- : mysql-process [ok] process is running (ProcessRunning)

Next up, you can daemonize god with:

sudo god -c /root/monitor.god

Now check if MySQL is up:

sudo god status mysql

Tail the god’s MySQL status log with:

sudo god log mysql

To add this as a @reboot cronjob, so that god always starts on reboot:

# First su to root:
su
# Edit root's crontab:
crontab -e

Then paste this entry into root’s crontab and save:

@reboot /usr/bin/god -c /root/monitor.god

Of course, execute ‘ps aux | grep god’ to do a sanity check that the above is the same on your system as well. (while god is running)

BTW – I get this warning when starting god, but everything still appears to work fine for me:

***********************************************************************
*
* Event conditions are not available for your installation of god.
* You may still use and write custom conditions using the poll system
*
***********************************************************************

My next question: what program does one use to monitor god itself?