YARB

god is a great ruby-based alternative to monit and other process monitoring tools. While I have had success with monit, I found its configuration syntax tedious. Monit’s configuration syntax makes it painful to do things like decrease/increase the number of mongrels you are monitoring, etc.

Note: the following assumes we’re on some kind of Fedora Core (or similar) system. Replace ”/etc/init.d/mysql stop|start|restart” with however this is done in your distro.

Configuration is where god shines. Let’s get started:

sudo gem install god

In this tutorial, I will only be covering how to monitor your MySQL process using god. The main god website has an excellent tutorial on how to monitor your mongrels.

God works by monitoring pid files. It has other functionality as well, but for MySQL 5.x monitoring, all we need is the location of the MySQL PID file. You can find this on your system with:

locate .pid | grep mysql

On my system, the MySQL pid file was located at:

/var/run/mysqld/mysqld.pid

Next we need to know how to stop/start/restart MySQL. On most systems, this can simply be done with:

cd /etc/init.d
sudo ./mysqld stop|start|restart

It’s a good idea to make sure these commands work by hand, of course, before assuming god should use them to manage MySQL.

Assuming your information matches the above, the following god config file should do the trick:

# God config file.
#
# Documentation: http://god.rubyforge.org/
#
# run with:  god -c /root/monitor.god
#

God.watch do |w|
  w.name = 'mysql-process'
  w.group = 'mysql'
  w.interval = 30.seconds # default      
  w.start = "cd /etc/init.d && ./mysqld start" 
  w.stop = "cd /etc/init.d && ./mysqld start" 
  w.restart = "cd /etc/init.d && ./mysqld restart" 
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = '/var/run/mysqld/mysqld.pid'
  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Place this into a file located at ’/root/monitor.god’. (for the below examples to work)

In order to test god, kick it into non-daemonized mode with:

sudo god -c /root/monitor.god -D

You should see some output like:

I, [2008-01-26 00:30:05 #1841]  INFO -- : Started on drbunix:///tmp/god.17165.sock
I, [2008-01-26 00:30:05 #1841]  INFO -- : mysql-process move 'unmonitored' to 'up'
I, [2008-01-26 00:30:06 #1841]  INFO -- : mysql-process [ok] process is running (ProcessRunning)

In another terminal, stop MySQL by hand:

cd /etc/init.d
sudo ./mysqld stop

This may not replicate exactly what happens when MySQL goes down in the wild, but at least you can test god’s basic functionality.

You should see the command line output of god indicate that it is restarting MySQL:

I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process [trigger] mysql-process God::Conditions::ProcessRunning: no such pid file: /var/run/mysqld/mysqld.pid (ProcessRunning)
I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process move 'up' to 'start'
I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process before_start: no pid file to delete (CleanPidFile)
I, [2008-01-26 00:46:01 #18173]  INFO -- : mysql-process start: cd /etc/init.d && ./mysqld start
I, [2008-01-26 00:46:12 #18173]  INFO -- : mysql-process [ok] process is running (ProcessRunning)

Next up, you can daemonize god with:

sudo god -c /root/monitor.god

Now check if MySQL is up:

sudo god status mysql

Tail the god’s MySQL status log with:

sudo god log mysql

To add this as a @reboot cronjob, so that god always starts on reboot:

# First su to root:
su
# Edit root's crontab:
crontab -e

Then paste this entry into root’s crontab and save:

@reboot /usr/bin/god -c /root/monitor.god

Of course, execute ‘ps aux | grep god’ to do a sanity check that the above is the same on your system as well. (while god is running)

BTW – I get this warning when starting god, but everything still appears to work fine for me:

***********************************************************************
*
* Event conditions are not available for your installation of god.
* You may still use and write custom conditions using the poll system
*
***********************************************************************

My next question: what program does one use to monitor god itself?

Rails by default uses the InnoDB engine, but MySQL full-text indexing is only supported by the MyISAM table type.

So, first we’ll have to convert our target table over to MyISAM. Next we add the full-text index on several columns.

class FullTextSearch < ActiveRecord::Migration
  def self.up
    execute 'ALTER TABLE torrents ENGINE = MyISAM'
    execute 'CREATE FULLTEXT INDEX ft_idx_torrents ON torrents(name,filename,description)'
  end

  def self.down
    execute 'ALTER TABLE torrents DROP INDEX ft_idx_torrents'
    execute 'ALTER TABLE torrents ENGINE = InnoDB'
  end
end

This is the actual migration used in The Hydra Project to add full-text searching support of torrents.

If you have a InnoDB table that you’d like to convert to MyISAM:

class ConvertToMyIsam < ActiveRecord::Migration
  def self.up
    execute 'ALTER TABLE torrents ENGINE = MyISAM'
  end

  def self.down
    execute 'ALTER TABLE torrents ENGINE = InnoDB'
  end
end

You can flip the migrations for the reverse, of course.

To obtain the raw production SQL from which you can then load your own development database, for example:

mysqldump -u username -p database_name > dump.sql

To load this dumpfile into a target database:

mysql database_name -p < dump.sql

You’ll be prompted for a password for both. (omit “-p” if your database does not require a password)

Update 3/24/08: Fark/reddit strikes, resulting in a 920k pageview day (same setup, just 7 mongrels).

Update 1/13/08: The site was on Digg again; this time receiving 450k views over the course of just 12 hours. Brings it up to 12 requests / second. Again this is not a benchmark, just the actual traffic. CPU usage was never a problem; more than likely the app is RAM bound. Oh and the $400 servers handle many other sites. To me, it’s more important not to have to think about servers, rather then pinching pennies in an attempt to squeeze every last ounce of performance out of machines. Just my $.02.

Over the weekend, this New Christian Science Textbook comic made it to the front page of Digg.

The above traffic graph should give you a sense of the kind of traffic that Digg can drive. It was front-paged sometime early Sunday morning. Chances are, if it hit during the middle of a weekday, the stats would be as much as 30-50% higher.

As you can see, roughly 550,000+ pageviews were logged over the course of 24 hours. Had this been over the course of 2-3 hours, the Incredimazing servers surely would’ve been crushed under the load.

Update note: had it been necessary, which it was not, enabling Rails Page Caching (see below) would have rendered the entire scalability question moot. nginx would’ve become the server from stack to stack for many pages—it’s been benchmarked at 250 to 330 requests per second (10M+ per day) which was clearly unnecessary in this case anyway.

Still, people are always (and probably will always) be wondering, can Rails scale?

The answer will always be amorphous, but at least I can give you some cold hard facts as to withstanding a digging of this nature.

The Hardware

The domain is hosted at LayeredTech on a dedicated two-server (DB + Web/App) setup. Several other sites can comfortably be hosted all on the same setup. The total setup runs about $370 per month.

Web/App Server

• Cost: $127 / mo.
• CPU: Intel P4 – 2.8GHz
• Memory: 2GB
• Bandwidth: 1500GB (the box rarely uses more than 25% each month)
• Uplink Port: 10Mbps
• Hard drive: 80GB x2
• OS: Fedora Core 4

DB Server

• Cost: $242 / mo.
• CPU: AMD Single CPU Dual Core Athlon 3800
• Memory: 2GB
• Bandwidth: 2000GB
• Hard drive: 500 GB x2 in SATA RAID 1
• Uplink Port: 100Mbps
• OS: CentOS 4.x X86_64 Bit

Both have dual NIC cards and are connected across a private switch.

Software Versions Used

• Ruby: 1.8.4
• Rails: 2.0.2
• MySQL: 5.0.27 standard
• nginx (web server): 0.4.13 (built by gcc 4.0.2)
• mongrel: 1.0.1
• mongrel_cluster: 0.2.1

nginx – the Little Webserver that Could

nginx is a fantastic little web server developed by Igor Sysoev – and despite taking a hammering like this one, its memory usage rarely got above 10-20MB (total).

If you’re still not convinced (not that I’m trying to win converts here), Ezra Zygmuntowicz, author of Deploying Rails Applications, recommends it highly (and uses it extensively at EngineYard ).

Of course, Apache 2, Lighttpd, etc. would most likely have been able to handle the load similarly, so long as they were setup / configured / etc properly.

Rails 2.0 – Better ActiveRecord, etc. Performance?

I have no scientific evidence supporting this, but since making the switch to Rails 2.0, my sites have seemed a lot more stable and zippy. (this blog excluded – it sits lower on the totem pole!)

I believe some benchmarks of Rails 2.0 have shown some modest improvements over 1.0. My recommendation: update early & often—in about a day several of my sites were converted over to 2.0 with minimal issues.

Mongrel – The Legacy of Zed Shaw

1/24/08: Hmmm… the rest of this article must’ve been nuked somehow while editing it in Mephisto. Sorry!

Creating a new MySQL ‘test_’ database for every Rails app gets to be a pain after a while. SQLite has become stable enough in recent months/years to warrant its due spot at the table next to MySQL, etc. for lightweight database operations.

To use SQLite instead of MySQL for your test environment, drop this into your db/database.yml in place of your existing MySQL ‘test’ environment config:

test:
  adapter: sqlite3
  database: db/test.sqlite3
  timeout: 5000

If you have a ‘has and belongs to many’ lookup table, chances are you don’t want it to have an ‘id’ field.

Unless otherwise specified, the Rails ‘create_table’ method in migrations automatically adds this field for you.

To force Rails not to do this, specify ”:id => false” in the declaration, as such:

    create_table :friends, :id => false, :force => true do |t|
      t.integer  :user_id
      t.integer  :friend_id
      t.datetime :created_at
    end

There are nicer has_many :through ways to do this, but drop this into your user.rb model for a friends list (using the above friends table schema):

class User < ActiveRecord::Base
  ...
  has_and_belongs_to_many :friends,
                          :class_name => 'User',
                          :join_table => 'friends',
                          :association_foreign_key => 'friend_id',
                          :foreign_key => 'user_id'
  ...
end

I used to use a slightly more complicated method of getting random values out of the database, until a friend turned me on to MySQL’s “rand()” operator.

Usage in Rails:

class Product
  def self.get_random
    Product.find(:first, :order => 'RAND()')
  end
end

Will get a random product for you out of your database.