YARB

Update 3/24/08: Fark/reddit strikes, resulting in a 920k pageview day (same setup, just 7 mongrels).

Update 1/13/08: The site was on Digg again; this time receiving 450k views over the course of just 12 hours. Brings it up to 12 requests / second. Again this is not a benchmark, just the actual traffic. CPU usage was never a problem; more than likely the app is RAM bound. Oh and the $400 servers handle many other sites. To me, it’s more important not to have to think about servers, rather then pinching pennies in an attempt to squeeze every last ounce of performance out of machines. Just my $.02.

Over the weekend, this New Christian Science Textbook comic made it to the front page of Digg.

The above traffic graph should give you a sense of the kind of traffic that Digg can drive. It was front-paged sometime early Sunday morning. Chances are, if it hit during the middle of a weekday, the stats would be as much as 30-50% higher.

As you can see, roughly 550,000+ pageviews were logged over the course of 24 hours. Had this been over the course of 2-3 hours, the Incredimazing servers surely would’ve been crushed under the load.

Update note: had it been necessary, which it was not, enabling Rails Page Caching (see below) would have rendered the entire scalability question moot. nginx would’ve become the server from stack to stack for many pages—it’s been benchmarked at 250 to 330 requests per second (10M+ per day) which was clearly unnecessary in this case anyway.

Still, people are always (and probably will always) be wondering, can Rails scale?

The answer will always be amorphous, but at least I can give you some cold hard facts as to withstanding a digging of this nature.

The Hardware

The domain is hosted at LayeredTech on a dedicated two-server (DB + Web/App) setup. Several other sites can comfortably be hosted all on the same setup. The total setup runs about $370 per month.

Web/App Server

• Cost: $127 / mo.
• CPU: Intel P4 – 2.8GHz
• Memory: 2GB
• Bandwidth: 1500GB (the box rarely uses more than 25% each month)
• Uplink Port: 10Mbps
• Hard drive: 80GB x2
• OS: Fedora Core 4

DB Server

• Cost: $242 / mo.
• CPU: AMD Single CPU Dual Core Athlon 3800
• Memory: 2GB
• Bandwidth: 2000GB
• Hard drive: 500 GB x2 in SATA RAID 1
• Uplink Port: 100Mbps
• OS: CentOS 4.x X86_64 Bit

Both have dual NIC cards and are connected across a private switch.

Software Versions Used

• Ruby: 1.8.4
• Rails: 2.0.2
• MySQL: 5.0.27 standard
• nginx (web server): 0.4.13 (built by gcc 4.0.2)
• mongrel: 1.0.1
• mongrel_cluster: 0.2.1

nginx – the Little Webserver that Could

nginx is a fantastic little web server developed by Igor Sysoev – and despite taking a hammering like this one, its memory usage rarely got above 10-20MB (total).

If you’re still not convinced (not that I’m trying to win converts here), Ezra Zygmuntowicz, author of Deploying Rails Applications, recommends it highly (and uses it extensively at EngineYard ).

Of course, Apache 2, Lighttpd, etc. would most likely have been able to handle the load similarly, so long as they were setup / configured / etc properly.

Rails 2.0 – Better ActiveRecord, etc. Performance?

I have no scientific evidence supporting this, but since making the switch to Rails 2.0, my sites have seemed a lot more stable and zippy. (this blog excluded – it sits lower on the totem pole!)

I believe some benchmarks of Rails 2.0 have shown some modest improvements over 1.0. My recommendation: update early & often—in about a day several of my sites were converted over to 2.0 with minimal issues.

Mongrel – The Legacy of Zed Shaw

1/24/08: Hmmm… the rest of this article must’ve been nuked somehow while editing it in Mephisto. Sorry!

This problem was truly maddening. Some kind of rogue tcp process spawned by mongrel was not getting closed properly.

It was truly a ghost in the machine.

Doing a ‘ps aux | grep ruby’ or ‘ps aux | grep 3000’ yielded nothing. So it was darn neigh impossible to find a process to kill that was somehow still holding onto the socket. Attempting to start mongrel yielded the “Address already in use – bind(2) (Errno::EADDRINUSE)” error.

This link provided the answer.

If you normally run your mongrel(s) on port 3000, use the following snippet to kill em dead:

kill -HUP `lsof -t -i TCP:3000`

The following is the complete stack trace:

** Ruby version is not up-to-date; loading cgi_multipart_eof_fix
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails application starting on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
** Starting Mongrel listening at 0.0.0.0:3000
Exiting
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/lib/mongrel/tcphack.rb:12:in `initialize_without_backlog': Address already in use - bind(2) (Errno::EADDRINUSE)
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/lib/mongrel/tcphack.rb:12:in `initialize'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/lib/mongrel.rb:92:in `initialize'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/lib/mongrel/configurator.rb:139:in `listener'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/bin/mongrel_rails:99:in `cloaker_'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/lib/mongrel/configurator.rb:50:in `initialize'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/bin/mongrel_rails:84:in `run'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/lib/mongrel/command.rb:212:in `run'
        from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.1/bin/mongrel_rails:281
         ... 6 levels...

Good luck!

Getting your mongrels to startup on boot is one of the last tricky things to do before your Rails setup is completely streamlined.

Using cron and a special @reboot option, the following steps will show you how to do this on a Fedora Core 5 linux system. (should work on similar systems as well)

About my setup—I deploy using a user account, let’s call it ‘foo’, which has a home directory of ’/home/foo’.

But we need to add the main @reboot cron entry as root.

# su to root:
su
# Now edit the root user's cron jobs:
crontab -e

Add the following to the file:

@reboot /root/start_mongrels

Then save and exit—it should’ve installed your new cron job. (use “crontab -l” to confirm)

Now you need to create a file in /root/start_mongrels that will call a helper script that resides under your ‘foo’ (deployment user) account.

# Still as root:
vi /root/start_mongrels

Paste in the following:

#!/bin/sh
su - foo -c "cd /home/foo && ./restart_all.sh" 

Then ensure the file is executable:

chmod 755 /root/start_mongrels

Now exit as root and return to being logged in as the ‘foo’ user. (whichever user you deploy your Rails apps as)

Create your restart_all.sh Script

Create a new file ’/home/foo/restart_all.sh’ with contents like:

#!/bin/sh

# Restart the 'foojiggly' app
pushd /u/apps/foojiggly/current
mongrel_rails cluster::stop
rm -rf log/*.pid
mongrel_rails cluster::start

# Restart the 'funster' app
pushd /u/apps/foojiggly/current
mongrel_rails cluster::stop
rm -rf log/*.pid
mongrel_rails cluster::start

Add as many entries as necessary. Newer versions of ‘mongrel_rails’ come with a ‘—clean’ option that you may want to use instead of the ‘rm -rf log/*.pid’ command.

Next, make sure the ‘restart_all.sh’ file is executable:

chmod 755 /home/foo/restart_all.sh

Now be sure to do a test run to make sure it all works!

sudo /sbin/restart -r 0