YARB

Jonathan Vaught is the new maintainer of the backup_fu plugin that makes Amazon S3 backups “redonkulous” (I crack myself up sometimes).

You can checkout its new home over on GitHub at:

http://github.com/gravelpup/backup_fu

thanks, Jonathan!

Nothing’s ever safe and stable on the ‘net. You can have the most expensive servers money can buy, the best broadband service on the ‘net, and the most high-end computer in the market but your system is still prone to crashes. That’s why a good knowledge on making backups is a must.

Just a tip, because I’ve finally bumped into an issue with this on one of my servers.

Backup_fu first dumps files to RAILS_ROOT/tmp/backup. This is nice and all to have a local copy of the backup … but it never does any cleaning out.

If you are looking for a solution that handles cleaning up old backups, I believe you can Google around and there are some out there like that.

What I will probably do going forward is create a reminder for myself via Backpack or iCal each month to A) check on the backups in Amazon S3, and B) clean out tmp/backup in the rails apps, especially ones where the backups push 1, 2, 3 gigs a pop.

First grab the source:

wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz

Unarchive it:

tar xvzf ImageMagick.tar.gz

The old ./configure / make / sudo make install ritual:

cd ImageMagick-6.3.8
# Or whichever the current version is, of course.
./configure
make
sudo make install

You should be good to go. Lately I’ve been having luck with MiniMagick (all I need to do is crop/resize for this particular project).

Type this to make sure you can use ImageMagick from the command line at least:

convert -version

I love that ruby (and many scripting languages) make it so easily to “shell out” to scripts (as minimagick does). It really makes ruby performance alarmists look bad when shelling out to time-tested, battle-hardened C-based scripts is so easy, and works so well. (I’ve had success shelling out to the following in many apps: curl, imagemagick, wget, etc)

How to Use MiniMagick in your Rails App

Grab the gem:

sudo gem install mini_magick

Drop this in your config/environment.rb:

require 'rubygems'
gem 'mini_magick'
require 'mini_magick'

Example usage:

class Pic < ActiveRecord::Base

  # Where size is a string like '90x90', '300x200', etc
  def create_perfect_thumbnail(size)

    image = MiniMagick::Image.from_file(self.pic_path)
    height, width = image['height'].to_f, image['width'].to_f

    # FIRST SHAVE off some of the image to make it square
    if width < height
      shave = ((height - width)/2).round
      image.shave("0x#{shave}")
    else
      shave = ((width - height)/2).round
      image.shave("#{shave}x0")
    end

    image.resize(size)
    image.write(self.pic_path(size))

    # I had issues on my linux box with the pic not being readable by the web server,
    #   following the resize.  Set permissions o+r to fix this.    
    if RAILS_ENV == 'production' # Set permissions to o+r
      `chmod o+r #{self.pic_path(size)}`
    end

  end

  def pic_path(size)
    # Just an example -- I normally group pics by user_id under a public static dir.
    File.join(RAILS_ROOT, 'public', 'static', "#{size}_#{self.original_filename}") 
  end

end

That will shave off some of the pic, making a munged square from the original, before proceeding to make square thumbnails from that.

Processing MessagingController#inbox (for 192.168.1.100 at 2008-01-19 11:02:45) [GET]
  Session ID: 17db90c03c03ebfee0c3ae4198900706
  Parameters: {"action"=>"inbox", "controller"=>"messaging"}
/!\ FAILSAFE /!\  Sat Jan 19 11:02:45 MST 2008
  Status: 500 Internal Server Error
  wrong number of arguments (1 for 0)

Chances are you just added a method “send” or similar to your rails app:

class MessagingController < AuthenticatedController
  def send
  end
end

This would, of course, override ruby’s default “send” behavior (an integral part of ruby, and rails).

Hat tip, Justin Ball – thx!

Update 3/24/08: Fark/reddit strikes, resulting in a 920k pageview day (same setup, just 7 mongrels).

Update 1/13/08: The site was on Digg again; this time receiving 450k views over the course of just 12 hours. Brings it up to 12 requests / second. Again this is not a benchmark, just the actual traffic. CPU usage was never a problem; more than likely the app is RAM bound. Oh and the $400 servers handle many other sites. To me, it’s more important not to have to think about servers, rather then pinching pennies in an attempt to squeeze every last ounce of performance out of machines. Just my $.02.

Over the weekend, this New Christian Science Textbook comic made it to the front page of Digg.

The above traffic graph should give you a sense of the kind of traffic that Digg can drive. It was front-paged sometime early Sunday morning. Chances are, if it hit during the middle of a weekday, the stats would be as much as 30-50% higher.

As you can see, roughly 550,000+ pageviews were logged over the course of 24 hours. Had this been over the course of 2-3 hours, the Incredimazing servers surely would’ve been crushed under the load.

Update note: had it been necessary, which it was not, enabling Rails Page Caching (see below) would have rendered the entire scalability question moot. nginx would’ve become the server from stack to stack for many pages—it’s been benchmarked at 250 to 330 requests per second (10M+ per day) which was clearly unnecessary in this case anyway.

Still, people are always (and probably will always) be wondering, can Rails scale?

The answer will always be amorphous, but at least I can give you some cold hard facts as to withstanding a digging of this nature.

The Hardware

The domain is hosted at LayeredTech on a dedicated two-server (DB + Web/App) setup. Several other sites can comfortably be hosted all on the same setup. The total setup runs about $370 per month.

Web/App Server

• Cost: $127 / mo.
• CPU: Intel P4 – 2.8GHz
• Memory: 2GB
• Bandwidth: 1500GB (the box rarely uses more than 25% each month)
• Uplink Port: 10Mbps
• Hard drive: 80GB x2
• OS: Fedora Core 4

DB Server

• Cost: $242 / mo.
• CPU: AMD Single CPU Dual Core Athlon 3800
• Memory: 2GB
• Bandwidth: 2000GB
• Hard drive: 500 GB x2 in SATA RAID 1
• Uplink Port: 100Mbps
• OS: CentOS 4.x X86_64 Bit

Both have dual NIC cards and are connected across a private switch.

Software Versions Used

• Ruby: 1.8.4
• Rails: 2.0.2
• MySQL: 5.0.27 standard
• nginx (web server): 0.4.13 (built by gcc 4.0.2)
• mongrel: 1.0.1
• mongrel_cluster: 0.2.1

nginx – the Little Webserver that Could

nginx is a fantastic little web server developed by Igor Sysoev – and despite taking a hammering like this one, its memory usage rarely got above 10-20MB (total).

If you’re still not convinced (not that I’m trying to win converts here), Ezra Zygmuntowicz, author of Deploying Rails Applications, recommends it highly (and uses it extensively at EngineYard ).

Of course, Apache 2, Lighttpd, etc. would most likely have been able to handle the load similarly, so long as they were setup / configured / etc properly.

Rails 2.0 – Better ActiveRecord, etc. Performance?

I have no scientific evidence supporting this, but since making the switch to Rails 2.0, my sites have seemed a lot more stable and zippy. (this blog excluded – it sits lower on the totem pole!)

I believe some benchmarks of Rails 2.0 have shown some modest improvements over 1.0. My recommendation: update early & often—in about a day several of my sites were converted over to 2.0 with minimal issues.

Mongrel – The Legacy of Zed Shaw

1/24/08: Hmmm… the rest of this article must’ve been nuked somehow while editing it in Mephisto. Sorry!

If you have a ‘has and belongs to many’ lookup table, chances are you don’t want it to have an ‘id’ field.

Unless otherwise specified, the Rails ‘create_table’ method in migrations automatically adds this field for you.

To force Rails not to do this, specify ”:id => false” in the declaration, as such:

    create_table :friends, :id => false, :force => true do |t|
      t.integer  :user_id
      t.integer  :friend_id
      t.datetime :created_at
    end

There are nicer has_many :through ways to do this, but drop this into your user.rb model for a friends list (using the above friends table schema):

class User < ActiveRecord::Base
  ...
  has_and_belongs_to_many :friends,
                          :class_name => 'User',
                          :join_table => 'friends',
                          :association_foreign_key => 'friend_id',
                          :foreign_key => 'user_id'
  ...
end

I had already upgraded my Rails gem to 2.0.2, but for some reason rubygems didn’t install the rest of the updated rails gems (ActiveRecord, etc).

sudo gem update --system
sudo gem update --source http://gems.rubyonrails.org

$ rails foo
/usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:379:in `report_activate_error': RubyGem version error: activesupport(1.2.5 not = 2.0.2) (Gem::LoadError)
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:311:in `activate'
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:337:in `activate'
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:336:in `activate'
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:65:in `active_gem_with_options'
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:50:in `gem'
        from /usr/local/bin/rails:18

Was also seeing it while trying to run ‘rake’ on existing rails projects.

I use the excellent Exception Notifier plugin to receive warnings when exceptions occur in some of my rails apps.

The only problem was that one of the apps received hundreds upon thousands of 404s that were instead showing up as ActionController::RoutingErrors.

Simple enough fix, in vendor/plugins/exception_notification/lib/exception_notifier.rb, around line 38:

def exception_notification(exception, controller, request, data={})
  # Disable notifications for 404s (ActionController::RoutingError)
  return if (exception.message =~ /No route matches/)
  ...

Add the two lines below the function definition to disable these warnings.

Note: the error message changed between Rails 1.x and 2.0. The above works with Rails 2.x.

This was a tricky error to debug because it was only seen sporadically. Eventually we saw that the exception was, generally, only happening when users were uploading large files. (10mb or above)

I’m still not sure as to the root cause of the problem. I suspect it has to do with the combination of plugins we are running (including filecolumn). If this was happening in vanilla Rails it would be a serious bug that the core team would’ve addressed by now.

First I was able to consistently repo the issue by uploading a 10mb file as an attachment. If you don’t have any lying around I found some test files here at cachefly.

The problem occurs when these file data objects (class Tempfile) are left in the params hash for the next session. Clearing out the Tempfile from the params hash prevents this.

What I’ve got now in application.rb:

  after_filter  :clear_filedata_params

  def clear_filedata_params
    params.each_pair do |k, v|
      params.delete(k) if v.is_a?(Tempfile) || v.is_a?(File)
    end
  end
  helper_method :clear_filedata_params

So from now on, if you’re passing a file data object as a root param, i.e. {:file_data => #File<...>}, this will automatically be cleared from params and should work ok.

If you pass it in as, {:foo => {:file_data => #File<...>}}, you’ll need to clear it out manually, since the after_filter won’t catch it. (Or if you know how to flatten a Ruby Hash easily, let me know!) In this case you could simply do a “params.delete(:foo)”.

TypeError (singleton can't be dumped):
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session/active_record_store.rb:83:in `dump'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session/active_record_store.rb:83:in `marshal'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session/active_record_store.rb:137:in `marshal_data!'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:333:in `send'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:333:in `callback'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:330:in `each'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:330:in `callback'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.rb:241:in `create_or_update'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1545:in `save_without_validation'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/validations.rb:752:in `save_without_transactions'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:129:in `save'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:95:in `transaction'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:121:in `transaction'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactions.rb:129:in `save'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session/active_record_store.rb:311:in `update'
    /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:867:in `silence'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session/active_record_store.rb:311:in `update'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session/active_record_store.rb:318:in `close'
    /usr/local/lib/ruby/1.8/cgi/session.rb:324:in `close'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:1184:in `close_session'
    /usr/local/l
  

One of the things I love about Ruby is the ability to easily daisy-chain several functions together. This feature is by no means unique to Ruby, but here I’ll show how one might do this the Ruby way.

You have an array representing days of the week, indexed by 0 – 7, that you get back from a web post as strings.

days = params[:days] # {"5" => "1", "6" => "1", "2" => "1"}

But we want the keys as an array of integers in sorted order.

One way would be:

keys = days.keys
sorted = keys.sort
ints = []
sorted.each { |i| ints << i.to_i }

Of course you wouldn’t actually ever do it this way once you’re familiar with Ruby idioms and are comfortable reading daisy-chained code (it can get ugly if abused!).

In Ruby, the above four lines can be condensed into the following one-liner:

ints = days.keys.sort.map(&:to_i)

Note: Rails monkeypatches collect/map to allow for the map(&:to_i) shortcut.