small medium large xlarge

Keeping Rails on the Rails

System Testing Rails Itself

by Sam Ruby

Generic image illustrating the article
  How the system that keeps the Rails book on track also aids in the development of Rails itself.  

A minimal expectation of any book that purports to teach you about a subject is that the examples it provide actually work.

A few years ago, an excellent book by Dave Thomas and David Heinemeier Hansson was no longer meeting that expectation. Not because it wasn't well written (it was!), or even that it wasn't accurate at the time it was written, but rather because the topic it covered (Ruby on Rails) changed tracks underneath them.

I was added to the team for the third edition as an author with the intent of bringing the book up to date. As I was doing so, I found that the book's subject matter was a constantly moving target: Rails was continuing to change underneath me.

As an agile web developer myself, I did what came naturally: I started automating my testing.

You see, the book takes you though a real-world scenario in which you execute commands, edit files, and visit web pages. So I wrote a script that did each of those steps, captured the output, and presented it as a series of web pages. I then wrote a script using the same functional testing methods that are made available to Rails developers to compare the output produced against what is expected.

As you might imagine, from time to time there are failures. Failures generally mean that either the book needs to be updated, or Rails needs to be fixed, or, in some cases, both.

In the process, I've found that this automated testing has not only been helpful to me, but also to the Rails team. The primary focus of their testing is the extensive unit tests coupled with running the code in production. What that combination misses, however, is the running of system tests of Rails using the actual defaults that are provided. This means that I routinely find problems before the Rails team themselves are aware of them. I captured such an example on my weblog last year:

 rake db:migrate
 rake aborted!
 An error has occurred, this and all later migrations canceled:
 uninitialized constant Arel::Relation
 Tasks: TOP => db:migrate
 (See full trace by running task with --trace)

What this means is that anywhere from a couple of times a week to a couple of times a month I get alerted to a failure. Simply by rerunning tests on other configurations, I can determine if the failure is unique to a particular version of Ruby, and what editions of the book the failure affects.

As the scenario sections are numbered, and because I have the ability to rerun specific steps and I know when the tests last passed, I can generally run git bisect to identify the offending commit.

Once I have this information, I approach the Rails team with the following question: "Was this change intentional?" Sometimes the change is intentional. Often it is not, and my output and identification of the commit that introduced the problem is all they need to know to fix the problem. Sometimes they ask me for smaller test cases, which I then produce. Example:

In rare cases, such as this one, I may even offer a fix:


For intentional changes, I obviously update the book. Turning around an eBook with changes in a timely manner is a bit of a challenge, but well worth it. Generally we seem to be managing it within 48 hours. Turning around dead-tree versions in a timely manner takes a bit longer, but those versions benefit from the feedback provided by the tens of thousands of people who have had early access to this material in ebook form.

Still more details are available in my 2010 RailsConf presentation on System Testing Rails itself.

Output of runs can be found here.

More recently, there was a change intended for Rails 4.1 to use a new version of minitest. I left a comment containing a description of the failure as well as the output of a minimal test case that demonstrates the problem. In addition to showing the commands that were issued and the output that was produced, a description of the environment is included. This resulted in a fix to minitest.

The scripts themselves are available and can be found here.

In that git repository there is a subdirectory named vagrant that contains all of the necessary configuration to create a virtual machine and run the entire scenario.

You are welcome to poke around, and if you have any questions feel free to participate in the online forum. But most of all, if you are a Rails user, I encourage you to set up your own tests and try out Rails betas and release candidates as they become available. Nothing frustrates the Rails core developers more than to be told about a problem only after a release is made. Don't be that guy! :-)

Sam Ruby is a prominent software developer who has made significant contributions to many of the Apache Software Foundation’s open source software projects, and to the standardization of web feeds via his involvement with the Atom web feed standard and the web service.

Send the author your feedback or discuss the article in the magazine forum.