Shoulda Looked At It Sooner

Just a little bit ago I twittered:

I’ve been using shoulda with rspec for past week. Now trying it on fresh rails project and liking it ok thus far.

To which Brandon Keepers replied with:

@jnunemaker what do you like about it?

I started to send a tweet back and realized it would make an ok post here.

History

I remember asking Brandon at RailsConf last year why he liked RSpec so much (I was using test/spec at the time) and his answer was, “I don’t know, just because.”

Of course after that response, he laughed and tried to explain. One thing I’ve noticed is that it is sometimes hard to explain why you prefer a certain tool over another. That said, I am going to give it a shot.

Shoulda with RSpec

When Joe Ferris announced that shoulda macros could now be used with RSpec, I switched away from rspec-on-rails-matchers pretty quickly. I’ve been using shoulda’s macros with RSpec for about a little while and today when I started a new side project, because of my little bit of familiarity and interest due to using it the past while, I thought what the heck, and went all or nothing with shoulda.

Toe Dipping

What is funny, is that I was completely anti-shoulda until they announced RSpec compatibility. I remember thinking, oh great, here comes another stupid testing framework. I looked it over several times and couldn’t find enough coolness to interest me in switching. When they announced that I could dip my toe in while still using RSpec, I gave it a shot. Honestly, without the toe dipping, it would have been a long time, if ever, before I even gave shoulda a fair shot.

The dipping of the toe method reminds me of git-svn. My first git experience was working with an svn repository. The same thing happened. I figured I had nothing to lose by dipping my toe in and a few hours later, I was hooked.

At first, as usual with new things, I was frustrated, followed by excited, followed by frustrated. After an hour or two of adding tests to the project, with the shoulda source code right by my side, things started to kind of click.

My Two Favorite Shoulda Things

1. As I look at my tests from various projects using test/unit, test/spec, rspec and this new project with shoulda, the shoulda tests just seem more readable. I don’t know if it is the context/should verbiage that I like better than describe/it or what, but my test files seem easier to scan and aesthetically more pretty (if that makes sense).

2. I love the shoulda controller macros. They put the few matchers that I created and used with RSpec to shame. That is not RSpec’s fault, I just love shoulda’s. I’m usually most interested in code and syntax, so here is a sample from the project I’m playing with that tests a basic sessions controller.

class SessionsControllerTest < ActionController::TestCase
  context "on GET to :new" do
    setup { get :new }
    
    should_render_a_form
    should_respond_with :success
    should_render_template :new
  end
  
  context "on POST to :create with valid credentials" do
    setup do
      User.stub!(:authenticate, :return => users(:jnunemaker))
      post :create, :username => 'jnunemaker', :password => 'secret'
    end
    
    should_return_from_session :user_id, "users(:jnunemaker).id"
    should_redirect_to 'root_url'
    should_filter_params :username, :password
  end
  
  context "on POST to :create with invalid credentials" do
    setup do
      User.stub!(:authenticate, :return => nil)
      post :create, :username => 'jnunemaker', :password => 'fake'
    end
    
    should_respond_with :success
    should_render_template :new
    should_set_the_flash_to /Could not authenticate/
  end
  
  logged_in_as :jnunemaker do
    context "on DELETE to :destroy" do
      setup { delete :destroy }

      should 'log user out' do
        session[:user_id].should be(nil)
      end
      
      should_redirect_to 'login_url'
    end
  end
end

Note: I’m also using Jeremy McAnally’s stump for stubbing the User#authenticate method which makes an external web service call, his matchy library for the fancy session[:user_id].should assertion, and a macro I stole (logged_in_as) to easily setup authentication for controller tests. Nothing fancy, but I just like the way it flows.

Simple Model Test

If you are in the mood for more code, here are some of the tests from my user model. Yes, I’m making a twitter client. I’m dissatisfied with pretty much all the twitter clients out there (and I’ve used them all) so I decided to whip one together. I’ll probably open source it at some point.

class UserTest < ActiveSupport::TestCase
  should_have_many :user_statuses
  should_have_many :statuses, :through => :user_statuses
  
  context "#sync_with_twitter" do    
    should "not assign ignored attributes" do
      tweeter = new_tweeter(:id => '1234', :name => 'Shaq', :screen_name => 'THE_REAL_SHAQ', :created_at => '2006-08-13 22:56:06')
      User.sync_with_twitter(tweeter, 'secret')
      
      user = User.find_by_twitter_id('1234')
      user.created_at.should_not == user.twitter_created_at
    end
    
    should "create non-existant user" do
      assert_difference 'User.count' do
        tweeter = new_tweeter(:id => '1234', :name => 'Shaq', :screen_name => 'THE_REAL_SHAQ')
        User.sync_with_twitter(tweeter, 'secret')
      end
    end
    
    should "update existing user" do
      user = users(:jnunemaker)
      tweeter = new_tweeter(:id => user.twitter_id, :name => 'New Name')
      
      assert_no_difference 'User.count' do
        User.sync_with_twitter(tweeter, 'secret')
      end
      
      user.reload
      user.name.should == 'New Name'
    end
  end
  
  def new_tweeter(attrs)
    tweeter = Twitter::User.new
    attrs.each { |k,v| tweeter.send("#{k}=", v) }
    tweeter
  end
end

Again, I’m using Jeremy’s matchy (mentioned above) to get the .should == syntax. I still enjoy RSpec, but I’m pretty impressed with Shoulda. It feels very scannable/readable and the macros are really handy, both for testing models and controllers.

Oh, and for those who are wondering, all I did to set things up is install the gems and add the following to my config/environments/test.rb file.

config.gem 'thoughtbot-shoulda', :lib => 'shoulda', :source => 'http://gems.github.com'
config.gem 'jeremymcanally-stump', :lib => 'stump', :source => 'http://gems.github.com'
config.gem 'jeremymcanally-matchy', :lib => 'matchy', :source => 'http://gems.github.com'

Shoulda thoughts and reactions? Maybe the shoulda users out there could chime in with what they like best. I am also curious what is holding back others who haven’t tried shoulda out yet.

12 Comments

Morgan
Feb 21, 2009

Greetings,
I really enjoy Shoulda. Just…be careful with the macros.

Because the macros define functions when the class is loaded, and those function definitions are off deep in the Shoulda source, tracing exceptions in your code back to the right macro that spawned it is a major pain. Enough of a pain that one team I was on completely rejected Shoulda after a few devs went on shoulda-macro-making binges that ended up hiding the source of bugs, instead of revealing them.

I argued for keeping the baby, and making it a policy to avoid the bathwater, but c’est la vie.

Just keep it in mind; macros can obfuscate the source of problems.

Other than that, I really like the syntax; it just feels natural to me, so I use it whenever possible.

— Morgan
grosser
Feb 22, 2009

Great article, i am still in the “great another testing framework” camp, but these controller examples really look nice. Maybe ill give it a try on my next project, hope it supports mocha too…
iain
Feb 22, 2009

I also get excited everytime I see those macro’s, but I still am not using shoulda. I always get put off by string coding like:

should_redirect_to ‘root_url’

And to be honoust, most macro’s don’t do as much as you think, because it could easily be written like this in rspec:

specify { should redirect_to(root_url) }

I know, a bit more of special characters with the accolades and stuff, but not horrible at all. But I feel (I know that is subjective) safer when I don’t see some string getting eval’d.

Am I alone in the fact that I would rather wait for more marco’s to become available in RSpec?
John Nunemaker
Feb 22, 2009

@Morgan – I have noticed that the backtraces and failure messages are pretty much no help in shoulda. It does make debugging the tests failures a bit more difficult.

@grosser – I don’t know about mocha, should be fine. I’m enjoying the simplicity of stump a lot.

@iain – I don’t think you are alone. I’m probably just a bit more restless and using the same thing and not trying out new things when the come out. I’m not having problems with RSpec, I just enjoy new.
Dan Croak
Feb 22, 2009
Hey John,

Great writeup. Glad you’re finding Shoulda feels good.

A few common requests and complaints have floated around Shoulda for some time.
- “How do I use context with Shoulda macros?”
- “I don’t like Shoulda’s string evaluation.” (should_redirect_to ‘login_url’ in your example above)
- “Shoulda’s backtraces sometimes point to the wrong line number.”
- “Custom shoulda macros are dangerous.” (this is the gist of what Morgan said above)
I don’t know the answer to the first question, but it’s a valid question. Shoulda is basically two libraries in one: context & should blocks for Test::Unit (with a few other niceties like good test output, should_eventually, etc.) and Shoulda macros.

I don’t know whether it currently feasible to test a Rails app with context and Shoulda. Has anyone done it successfully?

It’d be pretty cool if you could use context to replace library one but still use Shoulda’s macros. For instance, you might want to do that if you wanted shared behaviors. This is something James Golick submitted as a patch to shoulda, which wasn’t well-received. It was accepted in context. I believe at this point, context may have a feature or two advantage on shoulda’s library one. There’s a decent argument that context’s internals are cleaner than Shoulda’s library one.

Shoulda’s library two has been the main focus of the Shoulda core team lately. We’ve been working to make it testing framework-independent, in particular so RSpec users can have access to things like belong_to (it { should belong_to(:government) }). In this form, Shoulda’s “macros” could be better called “matchers.” In fact, the new Shoulda matchers has the same interface as matchy, RSpec matchers, and other libraries. A Shoulda matcher always responds to matches?, description and optionally responds to failure_message and negative_failure_message.

The second important reason we’ve been focusing on refactoring Shoulda macros to use matchers under the hood is that they are much, much easier to unit test. For instance, look at the association matcher test which has comprehensive test coverage of the belongs_to, have_many, have_one, and have_and_belong_to matchers. A dirty little secret of Shoulda’s history is that, for the most part, the macros were tested by example. There’s a Rails app in the test directory and all of the macros are used, but never truly unit tested. The matcher refactoring changes that.

In regards to the second bullet, as part of our push to “Shoulda 3.0”, which started with this matcher refactoring, we’re also trying to replace string evaluation with blocks where possible.

In regards to the third bullet, there haven’t been any commits to fix this. There’s been some discussion about “backtrace injection” which may require a specific macro DSL (macro :have_db_column do … end) syntax.

This and the “testing the tests” paragraph above are both related to the fourth bullet. One thing people really like about Shoulda is how easy it is to create your own custom macros. They’re usually just class methods defined on TestUnit::TestCase. Very easy, very quick, very clear what the process is: extract them right out of a test you just wrote. If that causes a bad backtrace, however, then this solution isn’t without problems. Additionally, how do you know that there isn’t a problem in your macro code? (Call this the “Hampton Catlin” argument) A stronger matcher culture may be the way to go here, but more discussion and debate is needed in the community.

Thanks for the write up, John!

Dan
John Nunemaker
Feb 22, 2009
@Dan – Thanks for the comment. The eval doesn’t bother me that much but what I did for RSpec is used a block, just as you mentioned. For example, my rpsec macro for redirection was this:
```
# it_should_redirect_to { admin_sites_url }
def it_should_redirect_to(&amp;block)
  it "should redirect" do
    do_request
    redirection = instance_eval(&amp;block)
    response.should redirect_to(redirection)
  end
end
```
It still uses eval but at least gets away from strings. Honestly, I’m not picky enough to care whether something is a string or a block as long as it works. :) Every spec had to have a do_request method that would obviously perform the request in question.

The backtraces aren’t bad through rake. I use Textmate a lot to run my specs and that is where the backtraces are unusable. If you run them from Textmate and there is a failure, it won’t even show the “expected but was” stuff. It just shows some weird Textmate error. This could be specific to my install. Who knows.
Sanbit
Feb 22, 2009

So if I am already using rspec and factory_girl, is shoulda something I should look into adding into my existing project or will it be overkill with my existing setup? Any ideas would be greatly appreciated.
Alex Vollmer
Feb 23, 2009

OMG, if you write a twitter client that doesn’t suck on the desktop, I promise HTTParty-level contributions. Put that sucker on GitHub and let’s fix this! Interested in using MacRuby?
John Nunemaker
Feb 24, 2009

@Sanbit Shoulda wouldn’t be overkill. If you use it with RSpec, it just adds the matchers.

@Alex Haha. No desktop Twitter client. I’m making a web one.
Mark Coates
Mar 05, 2009

@John Thanks for the post. I always seem to learn something new when I look at others’ tests, and I appreciate the way you’ve brought several elements together in a creative way to produce elegant, easy-to-read code. Keep it coming.