railsexpress.de

blog maintenance

2009-04-09T10:44:00+02:00

It’s time for an OS upgrade. I’m going to reinstall the server over Easter. Will be back next week.

Goodbye Trac

2009-03-14T06:27:00+01:00

I moved my ticketing system to lighthouse. It just doesn’t make sense anymore to maintain my own system on this server.

Ditching Subversion

2009-03-13T16:35:00+01:00

I’ve converted all my public subversion repositories to git. They’re now hosted on github.

There were several reasons for doing so:

After having used git for some time now, subversion really feels awkward to use, even though no branching and merging was required so far.

It should now be much easier for users to maintain their own extensions of the plugins.

I will need to reinstall my server in the near future. I was not looking forward to reinstall subversion and make proper backups of all the repositories. Having the code in git, I don’t need to make server backups of the repositories anymore.

When the server gets reinstalled, the subversion support will be dropped completely.

railsbench edge version available from github

2009-02-16T13:20:00+01:00

A while ago I moved my railsbench source repository to github. One of the benefits I hoped for was the possibility to have prerelease versions of the railsbench gem available for other people to try out, with minimal effort on my side. But github didn’t manage to build my gem automatically, as promised. I followed all instructions to the letter, but it just didn’t build. But recently the github gem builder received an update and now it does seem to work.

From now on you can install the bleeding edge version of railsbench from github. Do

gem sources -a http://gems.github.com

to add github as a remote gem repository and then do

sudo gem install skaes-railsbench

There a lots of changes in the upcoming release, as I haven’t released a new version for a long time. They are summarized in latest_changes.txt

Regards,

Stefan

How to open textmate links in Emacs on OS X

2009-01-23T21:08:00+01:00

I must admit that I’ve had some jealous feelings lately, watching developers using texmate. It wasn’t the editor which impressed me, of course, but the support they get out of using the footnotes plugin.

However, as a dedicated member of the church of emacs, I muttered to myself: “dammit, it must be possible to invoke the one true editor instead”, even on a Mac.

And indeed, it’s possible. Here’s how to do it.

First, open the the Info.plist file of the Emacs you installed (for me it’s in /Applications/MacPorts/Emacs.app/Contents/Info.plist). Find the lines which look like

<key>CFBundleURLSchemes</key>
  <array>
    <string>mailto</string>
  </array>

and add txmt urls:

<key>CFBundleURLSchemes</key>
  <array>
    <string>mailto</string>
    <string>txmt</string>
  </array>

Second, download textmate-links.el, install it into your Emacs load path and add (require ’textmate-links) to your init.el.

Third, get RcDefaultApp if you haven’t installed it already. This will allow you to associate txmt: urls with Emacs. Open it and select the Applications tab. Navigate to Emacs and check the txtmt checkbox and select it as the default appplication for txmt urls.

Then navigate to the Urls tab and select Emacs as the handler for txmt urls, if this hasn’t happened automatically already.

Next time you click on a txmt: link in the browser of your choice, it should open the file in Emacs.

Enjoy!

Ruby/Rails Consultants wanted

2008-01-23T16:45:00+01:00

I need 2 additional consultants for the Rails CRM project I’m managing for Autoscout24.

It’s a massive Rails application with 198 models and 60 controllers. A large part of the code base consists of business logic.

Requirements:

brightness
ability to get things done
willingness to dive into an existing code base, fix stuff and extend existing functionality
good social skills
ability to speak English (German is a plus)

If you’re interested, send me your credentials.

The contract duration would be 6 months initially.

This is an on site project in Munich, telecommuting is unfortunately not possible.

Update: the positions have been filled.

Writing Efficient Ruby Code - Short Cut

2007-12-28T19:10:00+01:00

This has been a busy year for me, but I have finally managed to complete the first chapter of my book “Performance Rails” and Addison Wesley has published it as a digital short cut.

It’s a collection of performance patterns (and anti patterns) which I have successfully applied to speed up the Rails framework. I hope you will find it useful.

Update: here some comments by readers of the short cut:

Rails Edge Performance

2007-09-18T11:53:00+02:00

Listening to David's RailsConf Europe 2007 keynote, I've learned that he's going to a release a beta version soonish. He's asked me to do some benchmarking, comparing the performance of edge Rails to the latest stable branches. Here's a chart:

stable11 is the current svn version of branch 1-1-stable, stable12 is current version of 1-2-stable branch and edge is the current version of Rails trunk. Numbers are requests per second (measured on my MacBook during David's talk).

If you're a numbers type, these are the numbers:

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
empty	0.94709	1.05049	1055.9	951.9	0.95	1.05	0.90
welcome	1.22878	1.34536	813.8	743.3	1.23	1.35	0.91
recipes	1.37214	1.50364	728.8	665.1	1.37	1.50	0.91
my_recipes	1.36958	1.49912	730.2	667.1	1.37	1.50	0.91
show	3.46944	4.27918	288.2	233.7	3.47	4.28	0.81
cat	3.72675	4.58788	268.3	218.0	3.73	4.59	0.81
cat_page5	3.85624	4.68700	259.3	213.4	3.86	4.69	0.82
letter	3.69970	4.54140	270.3	220.2	3.70	4.54	0.81
all requests	19.66971	23.49406	406.7	340.5	2.46	2.94	0.84
GC statistics	c1 total	c2 total	c1 #gc	c2 #gc	c1 gc%	c2 #gc%	c1/c2
	2.27657	3.09640	21.0	25.0	11.57	13.18	0.74

c1: 1.2-stable, c2: edge, r/s: requests per second, ms/r: milliseconds per request

I haven't had time to analyze the code, but it seems that anything loading a large number of active record objects takes a rather large performance hit.

I hope we can improve performance before 2.0 gets finally released. Unfortunately, I have almost no time to work on this. However, if anyone has a performance patch, I can look into the it.

Rails Consultant wanted

2007-06-06T16:20:00+02:00

I’m looking for an experienced Rails consultant to complement the team working for Autoscout24 on the International Sales Synergy Project which I am managing.

Requirements:

brightness
ability to get things done
willingness to dive into an existing code base, fix stuff and extend existing functionality
good social skills
ability to speak English (German is a plus)

If you’re interested, send me your credentials.

Update: the open position has been filled. Thanks for sending in your emails.

New Railsbench Release (0.9.2)

2007-04-01T12:42:00+02:00

A new version of railsbench has been released. Benchmarking POST requests is now supported, benchmark specific session data can be specified in the benchmark config file and ERB can be used as well.

added POST support:

    search:
      uri: /search
      method: post
      post_data: category=dessert

method defaults to get
post_data will be url encoded by railsbench

allow specification of session_data in benchmarks.yml:

    benchmark_name:
      uri: /
      session_data:
         user_id: 1
      post_data: category=dessert

load benchmarks.yml using ERB. Example: retrieve login ids using Rails)

    admin_interface:
      uri: /admin
      session_data:
        user_id: <%= User.find_by_name('admin').id %>

use benchmark names instead of urls in raw data files, old files can be converted using script convert_raw_data_files. change was required by post support
copy environments/production.rb to environments/benchmarking.rb, instead of providing a fixed one
don't create a broken benchmarks.yml file when running railsbench install
perf_plot and perf_plot_gc accept -geometry parameter, for exact dimensioning of the produced .png file
use -font_size=12 for setting font size when generating graphs
use 'perf_plot -colors=color1,color2,...' to get a different color set
use 'perf_plot -out=name' to write graph to file 'name'
new script perf_table: create a tabular overview of perf data
added -colorize option to perf_html, which results in green and red backgrounds in the factor columns
added -theme option to perf_plot

Happy benchmarking!

Rails 1.2 Performance

2007-04-01T12:28:00+02:00

Rails 1.2-stable is somewhat slower than Rails 1.1-stable, especially on action cached pages. But the slowdown I measured is nowhere near the numbers which have been reported by others. In fact, two recent patches to speed up session creation and pstore session retrieval, have resulted in improved performance for one of the benchmarked actions.

The following performance data table shows the speed difference for the fastest availabe configuration for the tested application.

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
empty	11.44034	10.91962	437.0	457.9	2.29	2.18	1.05
welcome	13.04648	14.29709	383.2	349.7	2.61	2.86	0.91
recipes	10.83617	13.51635	461.4	369.9	2.17	2.70	0.80
my_recipes	10.80137	13.48035	462.9	370.9	2.16	2.70	0.80
show	24.16722	27.37878	206.9	182.6	4.83	5.48	0.88
cat	26.56038	29.88683	188.3	167.3	5.31	5.98	0.89
cat_page5	27.24954	30.78081	183.5	162.4	5.45	6.16	0.89
letter	26.99067	30.17507	185.2	165.7	5.40	6.04	0.89
all requests	151.09216	170.43488	264.7	234.7	3.78	4.26	0.89
GC statistics	c1 total	c2 total	c1 #gc	c2 #gc	c1 gc%	c2 #gc%	c1/c2
	10.42790	12.90426	120.0	135.0	6.90	7.57	0.81

c1: 1.1-stable, c2: 1.2-stable, r/s: requests per second, ms/r: milliseconds per request

Read the full report here.

Railsbench gem updated

2007-01-14T15:45:00+01:00

Today I’ve released a new gem version (0.9.1) of railsbench. It’s mainly a bug fix release for the installation problems discovered for 0.9.0.

However, there’s also a new script called generate_benchmarks, which can be used to generate and maintain the benchmark configuration file used by railsbench.

railsbench generate_benchmarks updates the benchmark configuration file with

a benchmark for each controller/action pair (named <controller>_<action>)
a benchmark for each controller combining all actions of the controller into one benchmark (named <controller>_controller)
a benchmark comprising all controllers (named all_controllers)

This should get you started benchmarking in no time ;-)

After generating the file, you will need to edit the benchmarks and replace routing placeholders (:id, :article_id, etc.) by actual values.

If you add new controllers/actions later in the development process, you can invoke railsbench generate_benchmarks again to add new items to the configuration file. The older entries will not be removed by this process.

Currently, generate_benchmarks only works for the 1.2 release candidate and edge Rails (but I’ll gladly accept patches to make it work with 1.1.6).

Happy benchmarking!

Producing Performance Graphs

2006-12-26T19:16:00+01:00

The latest version of railsbench contains scripts to convert raw performance data files into pretty pictures. They depend on rmagick and gruff to be installed on your system.

railsbench perf_plot <files>

will generate a chart comparing benchmark data from all files passed as arguments.

Example:

Enjoy!

railsbench gem version available

2006-12-26T14:42:00+01:00

railsbench is now available as a gem from RubyForge. I recommend all users to switch to the gem version. This move should make it much easier to maintain railsbench installations for a larger number of machines.

Installation is now as simple as sudo gem install railsbench. Or, if you have installed the 0.8.4 gem version: sudo gem update railsbench.

The gem release will install only one ruby driver script, named railsbench in the default ruby path; individual commands like perf_run etc. are accessible through this driver. So instead of typing

perf_run 100 "-bm=all -mysql_session"

you would use

railsbench perf_run 100 "-bm=all -mysql_session"

The perf_ command prefix is now optional, thus

railsbench run 100 "-bm=all -mysql_session"

would give identical results.

If you’re accustomed to the old syntax and find typing railsbench awkward, you could add the gem’s script directory to your search path, by adding the following code to your shell profile:

eval `railsbench path`; export PATH

This will continue to work should you upgrade railsbench to a newer version.

Update: In order to call the scripts directly, you need to issue a sudo railsbench postinstall command after installing the gem. I haven’t found a way to create a gem that sets the executable bit on anything which lives outside the bin directory and isn’t a binary executable.

On Premature Optimization

2006-11-10T08:01:00+01:00

The alert reader of my blog might have noticed that I really care about performance. Some people even have accused me of being “obsessed” with it. Very often these statements are accompanied by Tony Hoare’s famous line: “Premature optimization is the root of all evil.”, which was later repeated by Donald Knuth.

I’ve always admired the work of Hoare and Knuth and I’m well aware of the famous quote on premature optimization. And they’re absolutely right: premature optimization is evil.

However, what lots of people seem to miss out, is that the most important word in his statement is “premature”.

If you don’t give any consideration to performance, you will be sure to avoid premature optimization. But you might have made a design decision which simply cannot be reverted without huge cost at a later stage of the project.

Let me give you an example: I once was asked to evaluate a project because it had performance problems. The software was intended to be used in a call center, which requires really fast response times. It turned out that the external company hired to implement this software, had made the decision to write the UI dialogs in a custom templating language, for which they had written an interpreter. The interpreter had been written in Visual Basic, which, at that time, was an interpreter as well.

So in effect, the dialogs were running c1*c2 times slower than necessary, where c1 and c2 are the interpreter overheads. It’s probably safe to assume the software ran about 25-50 times slower than necessary. And even top of the line hardware with 2 processors wasn’t able to make the package run fast enough.

In the end, the whole project was canceled, after pouring millions of EUR into it. And the problem was solved by buying another call center wholesale. I still believe today, that if some consideration had been given to performance questions during the architectural design phase, this disaster could have been avoided.

So, yes, premature optimization is evil, but no consideration for performance issues is just as bad.

Update: BTW, I was prompted to write this little piece after reading a blog entry
about a talk delivered at the Web 2.0 conference. It clearly shows that speed matters down to sub second resolution, even for web applications. I wouldn’t be surprised, if a well known UI axiom gets proven to hold in web context as well: response time should be below 100ms for graphical user interfaces. Making it faster is almost unnoticeable to the end user, but slower than 100ms does have negative effects.

SqlSessionStore gains support for SQLite

2006-11-09T12:37:00+01:00

Ted X Toth has submitted the code for a SQLite session class implementation for SqlSessionStore. I have checked it into the subversion repository.

Thank you very much Ted!

Make ruby-mysql create less garbage

2006-10-05T19:56:00+02:00

During the preparation of my RailsConf2006 talk I did some comparative benchmarks with Mysql and Postgresql. To my astonishment, I observed that Rails with Mysql created many more objects. After digging into the C part of ruby bindings of Mysql, I discovered that the Mysql bindings create a copy of the columns name string for each record retrieved from the database. This obviously leads to quadratic memory use and calls for correction.

I’ve written a small patch for the ruby-mysql bindings and submitted it to the author, but never got a response, although a tried several times.

You can get the patch here.

The patch actually goes a little farther than just correcting the n*m problem. I have added a Mysql::Result.all_hashes method, so that the entire array of hashes creation runs in C.

For pages that retrieve small datasets, you will probably see only a small increase. I measured around 5% increase for pages that load 25 objects. Which is neat. But the patch really shines when you retrieve large data sets. For example, for loading 1000 objects, I got this perf data:

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/alphabetic	6.86126	5.20610	14.6	19.2	68.61	52.06	1.32
GC statistics	c1 total	c2 total	c1 #gc	c2 #gc	c1 gc%	c2 #gc%	c1/c2
	2.14250	1.18728	24.0	12.0	31.23	22.81	2.00

30% faster!

But more important, a lot less gargabe gets created (in the example GC runs twice as often with the unpatched bindings).

You can take advantage of this by creating your own garbage ;-)

Oracle support for sql_session_store

2006-10-05T16:38:00+02:00

Due to a contribution from Tiago Macedo, sql_session_store has gained Oracle support.

If you’re on Oracle and want improved session handling performance, you should give it a try.

If you find any problems, please submit bug reports/patches here.

And if you now how to and have time to implement support for other database adapters, please send an email or better, create a ticket and attach the patch.

New plugin: query_builder

2006-09-23T06:33:00+02:00

I’ve added another plugin for you to check out: query_builder.

This plugin enables you to define finder methods that bypass the overhead of construct_finder_sql.

Inside an ActiveRecord model definition,

  define_finder query_name, query_type, options_hash

will create a SQL query method called query_name from a given options_hash. query_type can be :first or :all. The plugin supports all options except :include, but ignores with_scope options. Example:

class Recipe
  define_finder :find_all_of_user, :all,
     :conditions => 'user = :user AND priv < :priv'
  end

This defines a query method which can be called like so:

Recipe.find_all_of_user :user => 'martin', :priv => 1

This call is equivalent to

Recipe.find :all, :conditions =&gt;
            [ 'user = :user AND priv < :priv',
              {:user => 'martin', :priv => 1} ]

If options[:positional] is not nil or false, the created query method will use positional paramaters instead of a hash. In this case, arguments are created in the order of appearance on the parameters passed to define_finder. Therefore

define_finder :find_all_of_user, :all,
      :conditions => 'user = :user AND priv < :priv',
      :positional => true

will create a query method with parameters user and priv, which can be called like so:

Recipes.find_all_of_user('martin', 1)

I converted one of my pages to use this style and it gained 10% more performance.

If you like this plugin, drop me a line. If you find a bug, please submit a bug report to Trac

Enjoy.

Piggy back plugin updated

2006-09-16T10:42:00+02:00

Timo Hentschel, one of my coworkers at the company for which I’m doing contract work at the moment (AutoScout24), has added a nifty little extension to my piggy_back plugin.

It’s now possible to piggy back from has_many :through associations, provided the association will return at most one record from the underlying SQL query.

This type of association currently needs to be used as a replacement for the non existent has_one :through.

There have been rumours that has_one :through will find it’s way into rails core. If it does, I’ll probably remove the has_many :through support from piggy back plugin again, or at least raise a deprecation warning.

SqlSessionStore now available as a plugin

2006-09-15T09:56:00+02:00

Rails core team member Rick Olson, the author of many rails plugins, mephisto and techno-weenie, has turned SQLSessionStore into a plugin and sent his code for me to publish. Thanks a lot Rick!

I incorporated the latest changes from my sql session store implementation, most notably postgresql support, and added sql_session_store to my trac installation.

Using sql_session_store is now easier than ever before. Check it out into your vendor/plugins directory:

svn co http://railsexpress.de/svn/plugins/sql_session_store/trunk  \
 sql_session_store

and follow the instructions in the README file.

If you find any problems with the plugin, which would be entirely my own fault, not Rick’s, please use Trac to report them.

BTW, I’d love to get support in for additional database adapters, especially Oracle. I you find the time to write an adapter, please submit the code using Trac, or email me.

Rails Template Optimizer Beta Test

2006-08-15T23:01:00+02:00

It's there. Finally. Get it while it's still hot. Be among the first to experience unprecedented rendering speed for Rails Erb templates ;-)

We all know and love the little helpers that Rails offers to speed up template coding. They're convenient and help us to keep the template code readable. But convenience almost always comes with a hidden performance cost. It's an unavoidable fact, at least in the IT business.

I started the Rails Template Optimizer project in order to reduce the hidden cost in using Rails helper methods. And I think the current version of this software shows that this is possible, to a great extent, although it is still in a beta state.

You can download the code from subversion. Since I'm pretty sure that there are still some bugs and problems with the code, I have set up a Trac installation. The start page contains information on how to obtain the template optimizer and links to additional information.

So, how does it work? And what kind of performance improvement can you expect?

Very early during my involvement in Rails performance improvement I observed that some of the Rails helpers would run rather slowly, most notably helpers that use route generation in some form. I worked around the problem by coding my own specialized methods (as decribed in my InfoQ article to avoid invoking the routing code at all. This was acceptable for my own purposes, but not really usable for larger applications, with complicated route setups. In addition, some of the helpers, especially those that create AJAX code, produce code so complicated that it would be insane to write it manually.

But after thinking about the problem for a while it dawned on me that a good part of the work the helpers were doing could be done once and for all at template compile time:

First, there are helper calls that have either no argument or all of their arguments specfied in the template code. These can safely evaluated when the template gets compiled. (Well, most of the time, the actual story is a bit more complicated than that. Use the source to find out more.)

Second, there are helper calls where the output produced depends to a large extent on the structure of the input data, and only slightly on the exact value of that data.

Consider the following link_to call:

link_to "Edit User",
   {:controller => "user", :action => "edit", :id => @user},
   {:class => "user_link"}

Assuming just the default routes, we know that this will always produce

<a href="/user/edit/#{@user.to_param}" class="user_link">Edit User</a>

And this is exactly the code that will be produced by the template optimizer at template compile time. It achieves this by performing a partial (symbolic) evaluation of the template code. (For details I'd like to refer you to the source code, for now.)

How much improvement will you get by using the optimizer? That's a tough question. It all depends on the actual source and may vary between no improvement at all and spectacular improvements.

As usual, I've benchmarked it. Here are two data sets. First, my own app with every link using link_to instead of my helpers.

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/rezept/show/413	8.15451	5.23208	122.6	191.1	8.15	5.23	1.56
/rezept/cat/Hauptspeise	9.85367	4.68909	101.5	213.3	9.85	4.69	2.10
/rezept/cat/Hauptspeise?page=5	10.10721	4.84142	98.9	206.6	10.11	4.84	2.09
/rezept/letter/G	9.86153	4.67950	101.4	213.7	9.86	4.68	2.11

Second, the admin interface of an older Typo version:

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/admin	1.49967	0.85967	66.7	116.3	15.00	8.60	1.74
/admin/content	7.01567	1.84333	14.3	54.2	70.16	18.43	3.81
/admin/pages	2.99467	1.31767	33.4	75.9	29.95	13.18	2.27
/admin/categories	4.51033	3.12500	22.2	32.0	45.10	31.25	1.44
/admin/sidebar	4.88533	4.04133	20.5	24.7	48.85	40.41	1.21
/admin/themes	3.58300	2.29667	27.9	43.5	35.83	22.97	1.56
/admin/users	1.41167	0.77100	70.8	129.7	14.12	7.71	1.83

I would say these examples show some really nice improvement factors.

Enjoy.

Update:

You can svn co from http://railsexpress.de/svn/plugins/template_optimizer/trunk

And the template optimizer should now be compatible with rails 1.1.6

RailsConf2006 Aftermath

2006-06-28T16:25:00+02:00

As promised during my talk at RailsConf, I have uploaded a new version of my railsbench package to rubyforge. You can either download the tar ball or get the new version using svn. Apologies to all users which have a checked out and modified version of the old CVS archive. I decided to convert from CVS to subversion since I kept typing svn commands at my CVS archive, so it had to go.

The svn url for railsbench is svn://rubyforge.org//var/svn/railsbench/trunk/railsbench

The new version now supports running ruby-prof as an alternative to SVL's Ruby Performance Validator. This should make some Mac and Linux users happy. However, ruby-prof is only a weak substitute for a good profiler.

Profiling with ruby-prof requires ruby 1.8.4 and works like this:

perf_prof n "-bm=benchmark_name other_options" config_name

This will run the named benchmark n times using ruby-prof and store the resulting HTML formatted profiler graph into $RAILS_PERF_DATA

Happy profiling!

PS: My presentation given at RailsConf2006 is available for download.

Common Rails Performance Problems

2006-06-13T23:17:00+02:00

InfoQ has published an article on common Rails performance problems (written by me). You can head over to read all about it.

Simpler Piggy Backing

2006-05-29T12:06:00+02:00

A while ago I posted an article about the performance improvements obtainable by piggy backing. Although the speedups where substantial, I received some negative comments about exposing too much SQL to the programmer.

Exposure to the underlying SQL is not such a big problem for me, that it would deter me from applying the technique. But I found that I had a lot of repetition in my code, which I didn't like.

After ignoring the problem for a while, I decided it was about time to deliver a better solution than manually coding selects and joins.

I wrote a small extension to ActiveRecord, which makes piggy backing extremely simple.

Instead of coding joins and selects manually, you can now declare the piggy backed attributes from a has_one or belongs_to association in your model classes like so:

class Article
  belongs_to :user
  piggy_back :user_name_and_birthday, :user, :name, :birthday
end

This will generate all necessary select and join statements, which can then be added to a query rather elegantly:

article = Article.find :first,
            :piggy => :user_name_and_birthday,
            :conditions => "articles.title LIKE '%piggy%'"

would retrieve the first article which contains 'piggy' in its title attribute, along with the birthday and name attributes of the corresponding User object, storing them in the attributes hash of the AR instance.

But this is not all! After declaring the piggy backed attributes, you will also get reader methods which retrieve the attribute for you from the AR instance, and, should a type cast be required, convert the value to the correct type.

The piggy backed attributes are accessed like this:

article.user_name        ==> a string
article.user_birthday    ==> a date

Additionally, should you forget to piggy back a query,

article = Article.find :first,
            :conditions => "articles.title LIKE '%piggy%'"

methods user_name and birthday will silently use the user association to retrieve the values for you. This make is possible to use user_name and user_birthday regardless of whether your original query was piggy backed or not.

I've ensured that all of this works with pagination. In fact, using the extension is slightly faster than coding the select and joins manually (1%-3%).

You can download the code here. I haven't turned it into a plugin yet, so you need to add a require 'piggy_back' to your environment.rb.

Enjoy!

Update:

I failed to mention that piggy backing also works with finds on associations, so

article.comments.find(:all, :piggy => :user_name)

works as expected.

Upon popular demand it's now possible to use a more verbose syntax. You can write

class Article
  belongs_to :user
  piggy_back :user_name_and_birthday,
      :from => :user, :attributes => [:name, :birthday]
end

for the example above.

Piggy backing is now available as a plugin using the URL svn://railsexpress.de/plugins/piggy_back/trunk

Ruby en Rails 2006

2006-05-11T17:24:00+02:00

Next week, there’s a Rails conference in the Netherlands, Ruby en Rails 2006, were I will present my wisdom about Rails performance tuning. If you live nearby, and haven’t registered yet, now’s your last chance!

I’m sure it will prove to be a very interesting event. I’m especially looking forward to hear about a Ruby.net implementation project, presented by Wilco Bauwer.

And, of course I anticipate the after conference event, where we should have plenty of time to relax and dream up new ideas to save the world from joyless programming!

Update: I must say this was a very well organized event at a beautyful location, were I’ve met very nice people. It was a real pleasure to be there.

My slides are now available for download.

Performance related changes in Rails 1.1

2006-04-03T17:52:00+02:00

In addition to being packed with new features and also some forgotten ones, the recently released Rails 1.1 contains a number of performance related changes. For some cases, you will see significant improvements.

The components implementation received a complete refactoring, making it both faster and simpler to use.

inside a controller you can easily determine whether your current action is being run on behalf of an embedding controller by calling component_request?
session and flash information will now be passed on from the embedding controller to the component. As a consequence, it is no longer necessary to save session information to the session container before a component action is invoked by the embedding controller. For pages embedding n components, this saves n session queries and updates, reducing DB load significantly if you use ActiveRecordStore or MysqlSessionStore.

It is also possible to specify the controller as a class constant, bypassing the inflector code to compute the controller class at runtime. So

render_component :controller => "greeter",
                 :action => "hello_world"

becomes

render_component :controller => GreeterController,
                 :action => "hello_world"

Due to the component refactoring, it is no longer necessary to save newly created sessions twice. This has the biggest impact for people using ActiveRecodStore.

We have changed the default for ActiveRecord::Base.allow_concurrency from true to false, since almost all apps run in a single threaded request container. In addition, you can now safely set ActiveRecord::Base.allow_concurrency in environment.rb or any other config file.

Connection cache management got a serious overhaul. Previously, the connection cache was emptied after each request, in order to solve problems with dropped database connections. You can now use lazy connection verification by setting ActiveRecord::Base.verification_timeout to a number of seconds timeout interval. It defaults to 0, to be compatible with the previous release. However, I strongly recommend setting it to a higher value, for a simple reason:

Dropped database connections occur mostly on sites not seeing traffic for the database internal connection timeout period. High traffic sites will see them rarely. However, for high traffic sites, reducing the number of database accesses by setting verification_timeout to a value slightly below the database internal value will result in a significant reduction of DB requests. For example, if you use memcached for session management, the DB won't see any hit for action cached pages.

Additionally, the connection cache size was reduced from the number of ActiveRecord classes involved in a query, to the number of active database connections. So for most applications this means that instead of retrieving and verifying n connections, at most 1 retrieval and verification will occur, and only when the timeout interval has passed.

Update: the correct way of determining whether you're in a component request is calling component_request?. In a view you can call controller.component_request?

Rails 1.1 Release

2006-03-30T19:25:00+02:00

As you probably know already, Rails 1.1 was released a few days ago.

We have worked hard to ensure that 1.1 performs on the same level as 1.0. For some cases, it performs significantly better than 1.0 (explanation will follow in a future post).

The following performance data table shows the speed difference for the fastest available configuration for my application.

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/empty/index	7.86139	7.10545	636.0	703.7	1.57	1.42	1.11
/welcome/index	8.38743	8.32744	596.1	600.4	1.68	1.67	1.01
/rezept/index	8.87038	8.75717	563.7	571.0	1.77	1.75	1.01
/rezept/myknzlpzl	8.86172	8.76325	564.2	570.6	1.77	1.75	1.01
/rezept/show/713	22.22530	20.12046	225.0	248.5	4.45	4.02	1.10
/rezept/cat/Hauptspeise	25.29051	24.70123	197.7	202.4	5.06	4.94	1.02
/rezept/cat/Hauptspeise?page=5	25.92528	25.40904	192.9	196.8	5.19	5.08	1.02
/rezept/letter/G	25.06242	24.96315	199.5	200.3	5.01	4.99	1.00

You can find additional information and lots of performance data in the full report .

Presenting at RailsConf 2006

2006-03-28T09:51:00+02:00

I’m proud to announce that I will be giving a talk at RailsConf2006 on Rails Application Optimization.

If you have specific topics or questions you’d like to have addressed besides the ones mentioned in the talk proposal, or some general suggestion, I invite you to add an entry to the comment section of this post.

Looking forward to meet you in Chicago!

Using memcached for Ruby on Rails session storage

2006-01-24T12:42:00+01:00

A few days ago Eric Hodel announced the availability of a new pure Ruby memcached client implementation (memcache-client-1.0.3) with performance improvements over the older Ruby-Memcache-0.4 implementation.

I had measured the old version previously, for use as a Ruby on Rails fragment cache storage container, but found its performance to be abysmal and completely unusable for my purposes.

memcache-client-1.0.3 provides much better performance: much faster than either the old implementation, pstore or ActiveRecordStore, but also faster than my optimized SQLSessionStore using MysqlSession (see Roll your own SQL session store).
In order to determine the relative performance I ran my usual benchmarks using railsbench against memcache-client-1.0.3 and SQLSessionStore using MyslSession.

All components resided on a single machine. Before running the tests, both DB tables and memcached were populated with 10000 sessions.

The first test was run with Mysql query caching disabled:

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/empty/index	1.23318	0.96322	810.9	1038.2	1.23	0.96	1.28
/welcome/index	1.44314	1.15051	692.9	869.2	1.44	1.15	1.25
/rezept/index	1.52869	1.19616	654.2	836.0	1.53	1.20	1.28
/rezept/myknzlpzl	1.52159	1.19022	657.2	840.2	1.52	1.19	1.28
/rezept/show/713	4.14906	3.95939	241.0	252.6	4.15	3.96	1.05
/rezept/cat/Hauptspeise	9.48101	9.29202	105.5	107.6	9.48	9.29	1.02
/rezept/cat/Hauptspeise?page=5	9.84404	9.65393	101.6	103.6	9.84	9.65	1.02
/rezept/letter/G	5.42370	5.20227	184.4	192.2	5.42	5.20	1.04
c1: DB sessions, c2: memcached sessions, r/s: requests per second, ms/r: milliseconds per request

On the second run, Mysql query caching was enabled:

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/empty/index	1.26804	0.95403	788.6	1048.2	1.27	0.95	1.33
/welcome/index	1.47070	1.13807	679.9	878.7	1.47	1.14	1.29
/rezept/index	1.55976	1.18066	641.1	847.0	1.56	1.18	1.32
/rezept/myknzlpzl	1.55402	1.17650	643.5	850.0	1.55	1.18	1.32
/rezept/show/713	3.50876	3.20310	285.0	312.2	3.51	3.20	1.10
/rezept/cat/Hauptspeise	4.31720	4.00799	231.6	249.5	4.32	4.01	1.08
/rezept/cat/Hauptspeise?page=5	4.41150	4.11811	226.7	242.8	4.41	4.12	1.07
/rezept/letter/G	4.31190	4.02345	231.9	248.5	4.31	4.02	1.07
c1: DB sessions, c2: memcached sessions, r/s: requests per second, ms/r: milliseconds per request

What can we learn from this data?

Pages which force the creation of new sessions (empty, welcome) and action cached pages (rezept/index and rezept/knzlpzl) experience a speedup between 25% and 33%.

Pages involving DB queries other than session retrieval don’t see as much speedup.

If query caching is disabled and the query is really DB expensive (/rezept/cat issues a LIKE %Hauptspeise%), the speed improvement is negligible.

Enabling the Mysql query cache will result in slightly slower creation of new sessions, but can speed up complex queries tremendously (table below).

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/empty/index	1.23318	1.26804	810.9	788.6	1.23	1.27	0.97
/welcome/index	1.44314	1.47070	692.9	679.9	1.44	1.47	0.98
/rezept/index	1.52869	1.55976	654.2	641.1	1.53	1.56	0.98
/rezept/myknzlpzl	1.52159	1.55402	657.2	643.5	1.52	1.55	0.98
/rezept/show/713	4.14906	3.50876	241.0	285.0	4.15	3.51	1.18
/rezept/cat/Hauptspeise	9.48101	4.31720	105.5	231.6	9.48	4.32	2.20
/rezept/cat/Hauptspeise?page=5	9.84404	4.41150	101.6	226.7	9.84	4.41	2.23
/rezept/letter/G	5.42370	4.31190	184.4	231.9	5.42	4.31	1.26
c1: DB sessions without query cache, c2: DB sessions with query cache

Like all benchmarks, these results have to be taken with a grain of salt. Choice of either option involves more than just looking at the above numbers.

Database and memcached have quite different scaling properties: adding more memcached daemons is easy, whereas DB based session storage scales mostly by buying a bigger DB machine or running the session DB on a separate machine using a session database.

On the other hand, DB session storage makes it easy to get application usage statistics using SQL queries, e.g. displaying the number of active sessions or checking whether a particular user is currently logged in. I don’t know an easy way to do this using the memcachd API.

It seems to me that using memcached for session storage is a good choice if your DB server experiences a very high load and your scaling options are already exhausted. I wouldn’t recommend to use it per default.

Using memcachd for Ruby on Rails fragment cache storage is a completely different story, on which I hope to report in the future.

French translation of Railsexpress articles

2005-12-27T13:19:00+01:00

Tremeur Balbous has translated some of my blog articles into French. You can find them on www.railsfrance.org.

This is great news, for me, and the French Rails community.

Merci beaucoup, Tremeur.

Roll your own SQL session store

2005-12-19T10:30:00+01:00

The alert reader may have noticed that I'm using my own SQL session store, which provides much better performance than the default ActiveRecordStore shipping with Rails.

I have decided to publish the source code here. If you are using Mysql, simply unpack the files into your lib directory and make SQLSessionStore your session storage.

require 'sql_session_store'
require 'mysql_session'
ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.
    update(:database_manager => SQLSessionStore)
SQLSessionStore.session_class = MysqlSession

If you're using components, which you probably shouldn't, you'll need to activate eager session saving:

MysqlSession.eager_session_creation = true

The improved performance is due to a number of factors:

fixed number of known DB columns
retrieve only fields into the session that are actually used to create the session for a controller
offloading most of the work to the DB, including updating the created_at and updated_at fields
hard coded Mysql statements

As a consequence, if you want another DB, other DB column names, don't want created_at or updated_at fields, you'll have to change the code.

I have uploaded HTML representations of the call trees for ActiveRecordStore and SQLSessionStore. If you inspect them closely, you'll see how much time ActiveRecordStore spends in just creating the SQL statements and how much simpler the call tree is for SQLSessionStore.

If you adapt the provided code for other database systems, I'd be happy if you sent me the code, and I'll add it to the download after reviewing. The required changes are probably pretty small, but I currently don't have the time to test other DB systems.

Enjoy!

Rails 1.0 Release

2005-12-18T20:44:00+01:00

It's party time! Rails 1.0 hit the streets this week.

Somehow I didn't manage to release a performance report on the new release on the very same day, as planned ..., but here it is.

Rails 1.0 performs slightly worse than the previous 0.14.1 release for my application. However, 1.0 is still much faster than the 0.13 release series.

The following performance data table shows the speed difference for the fastest available configuration for the tested application.

page	c1 total	c2 total	c1 r/s	c2 r/s	c1 ms/r	c2 ms/r	c1/c2
/empty/index	1.26111	1.40676	793.0	710.9	1.26	1.41	0.90
/welcome/index	1.41814	1.57333	705.1	635.6	1.42	1.57	0.90
/rezept/index	1.50077	1.66533	666.3	600.5	1.50	1.67	0.90
/rezept/myknzlpzl	1.49374	1.65309	669.5	604.9	1.49	1.65	0.90
/rezept/show/713	3.59988	3.99798	277.8	250.1	3.60	4.00	0.90
/rezept/cat/Hauptspeise	4.54024	4.86669	220.3	205.5	4.54	4.87	0.93
/rezept/cat/Hauptspeise?page=5	4.64729	4.97066	215.2	201.2	4.65	4.97	0.93
/rezept/letter/G	4.59054	4.84387	217.8	206.4	4.59	4.84	0.95
c1: 0.14.1, c2: 1.0, r/s: requests per second, ms/r: milliseconds per request

The reasons for this significant slowdown have been identified and are being worked on (tickets 3155 and 3174).

You can find additional information and lots of performance data in the full report.

Trim your output

2005-12-06T01:13:00+01:00

The 0.14 Rails series includes a new option setting for template rendering, which allows for reducing the number of newlines generated by template processing. It's a string valued class attribute for ActionView, called erb_trim_mode. Its value is passed to the ERB compiler when a template gets compiled, with a default value of "-".

Some documentation for possible settings is part of the ERB class doc.

However, the doc is incomplete, as it doesn't mention Rails' default setting "-".

erb_trim_mode affects the way template processing handles newline characters in the template. Per default, (when erb_trim_mode is set to nil), processing leaves all newline characters occurring in the template in the generated html code. Which means that many extraneous newlines get inserted into your output.

For example,

<% if expr %>
some text
<% else %>
other text
<% end %>

will generate the following processing code:

if expr ; _erbout.concat("\n")
_erbout.concat("some text\n")
else ; _erbout.concat("\n")
_erbout.concat("other text\n")
end ; _erbout.concat("\n")

With the default setting of erb_trim_mode, you can suppress extraneous newline characters by adding a - before the closing %>:

<% if expr -%>
some text
<% else -%>
other text
<% end -%>

will generate the following template code:

if expr
_erbout.concat("some text\n")
else
_erbout.concat("other text\n")
end

This version looks much cleaner, suppresses 2 superfluous newline characters in the generated HTML code and saves 2 function calls when processed. Which is no big deal, unless the above call occurs inside a loop. If the loop body gets executed 100 times, you have already saved 200 characters and, more importantly, 200 function calls.

Manually adding all the minuses to closing %> markup is somewhat tedious. erb_trim_mode comes to rescue: if set to ">", newline characters will be suppressed for <%...%> markup closing a line; if set to "<>", newline characters will be suppressed for <%...%> occurring as a line of its own.

I have measured the relative performance of setting erb_trim_mode to ">". The following table shows that my app's pages receive a performance increase ranging from 1 to 4%.

configuration 1: 12-06.uncached.untrimmed
  requests=1000, options=-bm=uncached -lib=r141 -mysql_session
                         -fast_routes -fast_readers

configuration 2: 12-06.uncached.trimmed
  requests=1000, options=-bm=uncached -lib=r141 -mysql_session
                         -fast_routes -fast_readers -trim

page   c1 real   c2 real  c1 r/s  c2 r/s   c1 ms/r  c2 ms/r  c1/c2
 1:    3.52722   3.47518   283.5   287.8      3.53     3.48   1.01
 2:    4.30330   4.26213   232.4   234.6      4.30     4.26   1.01
 3:    4.49761   4.36027   222.3   229.3      4.50     4.36   1.03
 4:    4.45229   4.27840   224.6   233.7      4.45     4.28   1.04

urls:
 1: /rezept/show/713
 2: /rezept/cat/Hauptspeise
 3: /rezept/cat/Hauptspeise?page=5
 4: /rezept/letter/G

Unfortunately, ERB doesn't support combining the "-" and ">" setting. So you'll have to decide which way to go upfront. If you choose on ">", you may find that you have to add empty lines in some places where they weren't needed before.

Don't load stuff you don't need

2005-11-28T23:27:00+01:00

Using Rails 0.14 and onward, you can give your app a slight performance boost by not loading parts of the framework that you don't need. For example, if you don't expose parts of your app as a web service, then you don't need to load actionwebservice. Likewise, if you don't send mail from your app, you don't need to load actionmailer. If you're using the Rails initializer to start your app, you can do it like so:

Rails::Initializer.run do |config|
  # Skip frameworks you're not going to use
  config.frameworks -= [ :action_web_service, :action_mailer ]
  ...
end

If you're still using an old environment.rb, just comment out the corresponding require lines:

# Require Rails libraries.
require 'active_support'
require 'active_record'
require 'action_controller'
# require 'action_mailer'
# require 'action_web_service'

Additionally, you should also load only the database adapters for the database(s) which you are using. Add the following to your environment.rb, before Rails gets required:

RAILS_CONNECTION_ADAPTERS = %w(mysql)

By default, Rails will load all connection adapters it can find. Minimizing your code footprint will speed up Rails, because Ruby traverses all abstract syntax trees stored on the heap during garbage collection. If you can avoid loading significantly sized code, you will reduce the memory footprint of your app, thereby speeding up garbage collection. It might also help to increase object lookup at runtime due to smaller sized data structures containing meta information.

Speeding up the creation of new sessions

2005-11-22T04:13:00+01:00

There's a tiny peculiarity in the cgi/session.rb, which slows down the creation of new sessions quite a bit. The library digest/md5 gets required on creation of new sessions in method create_new_id. This can be rather expensive to execute. Move the require line to the beginning of the file, and your session creation will run faster. In environments where require is really slow, it can make a big difference (almost 3 times faster), as you can see from this data:

perf data file 1: 11-22.new.md5_unfixed
  requests=1000, options=-bm=new_sessions 
perf data file 2: 11-22.new.md5_fixed
  requests=1000, options=-bm=new_sessions

page              c1 real   c2 real    r/s    r/s   ms/r   ms/r  c1/c2
/empty/index      7.07800   2.39067  141.3  418.3   7.08   2.39   2.96
/welcome/index    7.07800   2.65100  141.3  377.2   7.08   2.65   2.67

On my Linux machine, the difference is less pronounced, but still significant:

page              c1 real   c2 real    r/s    r/s   ms/r   ms/r  c1/c2
/empty/index      1.47967   1.25513  675.8  796.7   1.48   1.26   1.18
/welcome/index    1.63819   1.41946  610.4  704.5   1.64   1.42   1.15

Maybe the require was placed inside function create_new_id to avoid loading the digest/md5 package in CGI environments when the request already passes a session id. But since Rails will usually run under FCGI/SCGI, moving it surely doesn't hurt! If you also have Ruby CGI apps running on the same machine, you can copy the file into your load path for Rails and modify this copy instead.

Components may not be evil, but they sure can be slow

2005-11-18T12:11:00+01:00

Let me start with a confession: I've been ignoring Rails components for a while, as I just couldn't figure out what they where good for. Meaning of course, that I didn't need them for my application ;-)

However, after DHH posted his opinions on components on the Rails weblog, I thought I'd better a have a closer look, what they could do for me and also inspect their implementation details. Additionally, I discovered that Typo uses components to display various parts of the front page of my blog. And it seemed pretty slow at doing that. So I smelled a performance rat.

Basically, components enable you to capture the presentation logic of a controller and embed it into another controller. This is implemented by duplicating the current request object, starting all over with request processing, capturing the rendering results and embedding them in the page that called in the first place.

This comprises creating a new controller object, in order enable request processing, and a new view object for template rendering. This already sounded like it wouldn't be particularly efficient, but the results I got, after creating a benchmark configuration for my blog and running railsbench to get some performance data, were shocking. Here's my benchmark data:

page request                            total  stddev%     r/s    ms/r
1:/                                  14.86220   0.1405    6.73  148.62
2:/articles/2005/11/08/accelera...   13.83662   0.1071    7.23  138.37
3:/admin                              1.47821   0.9429   67.65   14.78
4:/admin/content                      2.95742   0.2235   33.81   29.57
5:/admin/pages                        2.33728   0.0951   42.78   23.37
6:/admin/categories                   2.09189   0.2452   47.80   20.92
7:/admin/sidebar                     13.99597   0.9474    7.14  139.96
8:/admin/themes                       5.44847   0.3377   18.35   54.48
9:/admin/users                        1.48696   1.0194   67.25   14.87

Page / measures the main blog page, the second entry measures a single article, and the remaining entries present data on the blog's admin interface.

I just couldn't believe my eyes. Only 6.7 requests per second to generate my blogs entry page? Unbelievable (on an Athlon 64 3000+). Especially since my own app runs over 200 requests per second on the same box.

After studying call patterns using Ruby Performance Validator, which is a really good tool for this task btw., I found out that Typo was loading its configuration from the database several times per request. It also tried to figure out the current page's feed url just as often. That seemed awfully wasteful of resources.

The repetitions of the aforementioned actions were caused by 2 before filters installed on the application controller:

class ApplicationController < ActionController::Base
  before_filter :reload_settings
  before_filter :auto:discovery_defaults

  def reload_settings
    config.reload
  end

  def auto_discovery_defaults
    auto_discovery_feed(:type => 'feed')
  end

  def auto_discovery_feed(options)
    options = {:only_path => false, :action => 'feed',
               :controller => 'xml'}.merge options
    @auto_discovery_url_rss = url_for({:format => 'rss20'}.merge options)
    @auto_discovery_url_atom = url_for({:format => 'atom10'}.merge options)
  end
end

And sidebar content (archive, flickr, tada, etc.) gets delivered through components, which are all derived from ApplicationController, thus inheriting the filters. Bingo!

There are, of course, several ways to fix this problem, I chose the simplest one, taking advantage of the fact that the request object passed to components gets duplicated for component processing. Upon entering the controller responsible for the request, config will be loaded and a processing flag is stored as an instance variable in the request object. Similarly, the values retrieved during feed get stored there as well. When a component gets invoked it retrieves these values from the request object.

def reload_settings
  unless @request.instance_variable_get(:@config_reloaded)
    @request.instance_variable_set(:@config_reloaded, true)
    config.reload
  end
end

def auto_discovery_defaults
  @auto_discovery_url_rss =
      @request.instance_variable_get(:@auto_discovery_url_rss)
  @auto_discovery_url_atom =
       @request.instance_variable_get(:@auto_discovery_url_atom)
  unless @auto_discovery_url_rss && @auto_discovery_url_atom
    auto_discovery_feed(:type => 'feed')
    @request.instance_variable_set(:@auto_discovery_url_rss,
                                    @auto_discovery_url_rss)
    @request.instance_variable_set(:@auto_discovery_url_atom,
                                    @auto_discovery_url_atom)
  end
end

Running railsbench again and comparing the results shows that the most important pages are now rendered 4 to 5 times as fast:

page    c1 real   c2 real  c1 r/s  c2 r/s  c1 ms/r  c2 ms/r  c1/c2
1:     14.86220   3.75458     6.7    26.6   148.62    37.55   3.96
2:     13.83662   2.70294     7.2    37.0   138.37    27.03   5.12
3:      1.47821   1.29960    67.6    76.9    14.78    13.00   1.14
4:      2.95742   2.71447    33.8    36.8    29.57    27.14   1.09
5:      2.33728   2.13827    42.8    46.8    23.37    21.38   1.09
6:      2.09189   1.73313    47.8    57.7    20.92    17.33   1.21
7:     13.99597   2.73185     7.1    36.6   139.96    27.32   5.12
8:      5.44847   1.42805    18.4    70.0    54.48    14.28   3.82
9:      1.48696   1.15061    67.3    86.9    14.87    11.51   1.29

Note: c1 and c2 label data of the original/modified application. r/s are requests per second and ms/r are milliseconds per request.

Now that's some niiiiiice improvement!

I think that it's pretty easy to shoot oneself in the foot using components. I tend to avoid them. But sometimes they can be nice, in that case: watch out for da slow filters! Either don't derive the components from ApplicationController, make sure that the filters are fast, possibly using the technique presented in this article, or turn them off completely, e.g. if you don't really need to run them and or can get at the necessary data using some other technique.

The possibility to disable filters in derived controller classes was recently added to Rails. So we could have said

  skip_before_filter :reload_config

on the component classes, instead of storing a flag inside the request object. It should be even faster than storing flags on the request object, because the filter will be removed from the filter chain during class load time. Maybe I'll measure that at a later time.

Accelerate your dates

2005-11-09T00:49:00+01:00

When I started work on improving my app's speed, I noticed that on pages that display lots of date information, a large percentage of CPU time was spent on formatting date strings. I added a quick fix to my code, was satisifed and quickly forgot about it.

But recently I read a message on the ruby talk mailing list, where someone more or less rejected Rails after writing a small Ruby program for log file analysis that ran for about 27 minutes on a 100MB log file and a literally translated Java program finished in only 2.5 minutes blog entry.

It turned out that the inefficiency of the Ruby script was caused by the Ruby Date class, as someone rewrote the program using regexps to parse date info that needed only 3 minutes to complete the task.

I'm pretty sure that the Date class performance could be much improved, for example by rewriting it in C, but we'll leave this to the ruby core team. For page speedup we simply resort to avoiding construction of Date objects altogether. For Mysql, the Rails database layer will always return date information as strings. The strings are formatted as 'YYYY-mm-dd', i.e. they will have ISO date format. Such a string can of course be turned into 'dd.mm.YYYY' easily using Ruby primitives. So instead of writing

<%= r.created_at.strftime "dd.mm.YYYY" %>

<%= r.created_at %>

we'll change this into

<%= r.created_at_formatted %>

and add a function created_at_formatted to our model, which will access the attributes hash directly and transform the string into our desired format. One possible way to code this function is

def created_at_formatted
  d = @attributes['created_at']
  "#{d[8,2]}.#{d[5,2]}.#{d[0,4]}"
end

another is

def created_at_formatted
  @attributes['created_at'] =~ /^(\d\d\d\d)-(\d\d)-(\d\d)/
  "#{$3}.#{$2}.#{$1}"
end

This assumes that created_at is a non null column. If it isn't, you'll need to add some guards for null values.

The first version comes out slightly faster than the second, which I expected. But how much faster than the original code?

On pages with 25 dates displayed it is approximately 70% faster than the original version. On a page with 1 date, it's already 6% faster.

If you have pages with a large number of date displays, you might consider this technique to make them a tad snappier :-)

Update: As mentioned by henri, the technique does not apply to the oracle adapter, as it returns Time objects. As time permits I will look into the other adapters, to find the "bad boys" which return strings.

The case for piggy backed attributes

2005-11-06T20:35:00+01:00

If you're subscribed to the rails mailing list, you might have seen some people talking about piggy backing attributes onto SQL queries.

This is a feature of the Active Record ORM mapping that I like very much, because it can be used to speedup you queries enormously. And I hope it won't be eliminated, ever.

Suppose there's a 1:n relationship between models A and B, i.e., for each A record there's exactly one B record (think B owns A). In my case that would be recipes and users.

In ActiveRecord you'd have class declarations like this:

class User < ActiveRecord::Base
  has_many :recipes
end
class Recipe < ActiveRecord::Base
  belongs_to :user
end

Your usual code to retrieve some recipes with their associated authors will probably look like this:

def show_some
  @recipes = Recipe.find(:all, :limit => 50)
end

with the show_some template being rendered implicitely and looking similar to:

<% for r in @recipes %>
  <%= r.title %>, <%= r.user.name %>
  <br>
<% end %>

There's nothing wrong with this code, except that it's kind of slow:

there will be 51 queries to the database, because for each recipe the owning user (author) will be fetched in a separate query
all these queries are constructed dynamically upon calling r.user.name

We can do a little better than that: since approx. 0.13.1 Rails supports fetching associated objects by specifying our intention to access them via the :include syntax for find (called eager loading of associations). So we'd actually write our query like this:

def show_some
  @recipes = Recipe.find(:all, :limit => 50, :include => [:user])
end

But how much faster is this? The answer is: it depends. On how fast DB access is, for example. Whether your DB lives on another host, network speed, etc.

For a simple installation on one box, I have measured this (using railsbench, 1000 requests each, patched GC):

page request               total  stddev%     r/s    ms/r
/rezept/some_default    17.83765   0.2579    56.1   17.84
/rezept/some_include    13.25508   0.3499    75.4   13.26

Speedup is 34%, which is OK, but not spectacular.

We can do better than that using piggy backed attributes. First, we'll change the query to:

Recipe.find(:all, :limit => 50,
            :conditions => "r.user_id=u.id",
            :joins => 'r, users u',
            :select => 'r.*, u.name AS user_name')

We still have only one query to the DB, this time involving a join, which could be slow. But don't worry, databases were constructed to perform such queries as fast as possible.

In addition to the recipe columns we now get 'user_name' included in the retrieved records. To make access to this attribute fast, we'll add a user_name function to our model:

class Recipe < ActiveRecord::Base
  belongs_to :user
  def user_name
    @attributes['user_name']
  end
end

and change our template to:

<% for r in @recipes %>
  <%= r.title %>, <%= r.user_name %>
  <br>
<% end %>

Is this faster? Answer: you can bet!

page request               total  stddev%     r/s    ms/r
/rezept/some_default    17.83765   0.2579    56.1   17.84
/rezept/some_include    13.25508   0.3499    75.4   13.26
/rezept/some_piggy       4.19032   0.1059   238.6    4.19

It's whopping 4 times faster than the first version. And 3 times faster than the :include version.

I can almost hear you scream "But this code sucks! The original version was sooo much nicer. I hate adding this extra function to my model." My answer is: if your app is fast enough for you, don't do it. If it isn't, this is a nice opportunity for speedup.

Also, if your rails app lives in a shared hosting environment, it means being nice to everyone else to make it faster.

And if you weren't looking for speed, you wouldn't be reading this blog in the first place ;-)

session :off !

2005-11-05T09:02:00+01:00

Rails 0.14 contains the possibility to turn off sessions on a controller or even action basis.

class WelcomeController < ApplicationController
  session :off
  ...
end

This will turn off session creation and update on all actions of class WelcomeController.

Depending on the complexity of the logic inside your actions, you can get a big speedup. For example, on my welcome controller, index is action cached. Turning sessions off for this action, I see a huge performance improvement on my windows development box:

c1 real   c2 real  c1 r/s   c2 r/s  c1 ms/r  c2 ms/r  c1/c2
2.29700   0.27633   217.7   1809.4     4.59     0.55   8.31
2.45833   0.29200   203.4   1712.3     4.92     0.58   8.42

This was measured against the default ActiveRecordStore session implementation. But even with a faster session implementation, the difference is astonishing:

c1 real   c2 real  c1 r/s   c2 r/s  c1 ms/r  c2 ms/r  c1/c2
1.20333   0.26600   415.5   1879.7     2.41     0.53   4.52
1.35933   0.27633   367.8   1809.4     2.72     0.55   4.92

So it is certainly worthwhile to consider turning sessions off for controllers that don't need them.

Don't use Pstore as a session container

2005-11-04T04:41:00+01:00

You really shouldn’t be using Pstore for Rails. Apart from the fact that it doesn’t scale to a setup with more than 1 box, it is much slower than using the database for session storage, for example.

Using a Mysql session store can be up to 90% faster. Under, Linux, that is.

If you’re on a Windows box, the difference can be staggering: up to 560%!

Faster Pagination

2005-11-03T21:24:00+01:00

Rails version 0.14 and upward contain a massive performance improvement for pagination. Most of it is free and any app will benefit from it without even knowing.

However, there's also a little api addition to the pagination helper module, which can be used to speed up your paginated pages even more, if necessary.

We're talking about pagination_links_each. It takes the same parameters as pagination_links but instead of calling link_to to create the html code for each link, it yields the page number to a block. It is the block's responsibility to create the html code for the link.

So for example, you could write

pagination_links_each(pages, :window_size => 4) do |n|
    "<a href='?page=#{n}'>#{n}</a>"
end

instead of

pagination_links(pages, :window_size => 4)

The first version is faster than the second one, because it avoids creating lots of intermediate data structures to generate the html code. Additionally, the created html is shorter and less data gets transmitted over the wire. And the code will work for any page embedding it.

The speedup obtained will of course depend on your app. Here's some data for paginated pages of my application, showing speedups ranging from 11 to 26%. Quite an improvement for adding one line of code!

Blog opened!

2005-11-02T00:24:00+01:00

So finally I managed to get me a place in the blogosphere. Hey!

The main purpose of this blog will be to keep a log of (my) activities related to improving performance of the Rails web application framework.

A good deal of work already went into performance improvement since realease 0.12.1. To sum it up: I managed to improve the throughput of a recipe database application by a factor of 4 since I started this work, going from around 50 requests per second to over 220 requests per seconds.

In order to ease performance measurements of Rails applications, I have developed the package railsbench. It really does make performance regression tests easy, so give it try!

And stay tuned for further improvements :-)