Test Or Die

As you might have noticed, approximately 6 days ago, we experienced what is known as a new year. With new years come a feeling of starting over and the ridiculous idea that new habits form overnight. Resolutions are made, gyms are filled, and a month later they clear out for the disciplined to enjoy in peace once again.

Back in the old days, you know, like 2006, I was occasionally rogue and didn’t write tests. Tests? Isn’t that what the browser is for? Right? Right? Yeah, you know. I mean I only used the browser for tests when I wrote PHP. Wrong. You are sorely wrong my friend. In 2007, I pushed myself to learn testing and in 2008 I pushed myself to learn RSpec. You know what? I don’t regret a minute I spent learning how to write automated tests and now I actually find writing tests enjoyable. Crazy, I know.

I kind of assumed everyone was on the testing bandwagon. I mean lots of people seem to talk about TDD and BDD so everyone must be doing it right? Wrong. I’m in no way perfect, but I am still shocked and abhorred at the number of people out there in Ruby and Rails land that don’t test.

Test Awareness Month

Test first, test last, TDD, BDD, whatever. I don’t care how you do it, I just care that you do it. My mission for the month of January (and maybe longer) is to get all of you who don’t know where to start or don’t think you have time to learn or think you don’t need to learn, starting, learning and testing. Period. I’m going to start with the basics and post like a superhuman fiend until everyone is testing and the excuses are gone like donkey kong.

Your First Test

So where do we start? It all starts small and you build. You can’t download it like Neo in the Matrix unfortunately. To show you how easy it is to get started, lets run a few commands. Open up terminal and make sure you have Rails 2.2.2+ installed. Onward!

$ rails first_test
cd first_test
rake db:create
rake db:create RAILS_ENV=test
script/generate model Post title:string
rake db:migrate
rake db:test:prepare
rake

At this point, you have created an app, created both your development and test databases, added a model named post, migrated your database, ensured that your test database is equivalent to your development database and run your test suite for the first time. Wow, that was easy, right?

Let’s go one step further for this post. We added a title to the post, wouldn’t it be nice to ensure the presence of that title? Who has used validates_presence_of before? Go ahead, raise your hand. Uh huh, everyone. Why don’t we add a test quick to make sure that it is a requirement in our app. Open up your favorite editor and view the file test/unit/post_test.rb. It should look like this:

require 'test_helper'

class PostTest < ActiveSupport::TestCase
  # Replace this with your real tests.
  test "the truth" do
    assert true
  end
end

Let’s replace the truth test with a check that a post is not valid without a title. Our post_test.rb file should now look like this.

require 'test_helper'

class PostTest < ActiveSupport::TestCase
  test "should require a title" do
    post = Post.new
    post.valid?
    assert_not_nil post.errors.on(:title)
  end
end

Now that we have that in place, let’s run our test suite and see if it worked. Go back to your terminal and run rake like this:

$ rake
/opt/local/bin/ruby -Ilib:test "/opt/local/lib/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/rake_test_loader.rb" "test/unit/post_test.rb" 
Loaded suite /opt/local/lib/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/rake_test_loader
Started
F
Finished in 0.235756 seconds.

  1) Failure:
test_should_require_a_title(PostTest)
    [./test/unit/post_test.rb:7:in `test_should_require_a_title'
     /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/testing/setup_and_teardown.rb:94:in `__send__'
     /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/testing/setup_and_teardown.rb:94:in `run']:
<nil> expected to not be nil.

1 tests, 1 assertions, 1 failures, 0 errors

FAIL! You are a failure. Give up. You can’t even write one test. I guess it is too hard. I guess it is not worth learning! Oh, wait, we didn’t add it to the post model, did we? Open up app/models/post.rb and add the validation rule.

class Post < ActiveRecord::Base
  validates_presence_of :title
end

Now run rake again. You should get something like this:

$ rake
/opt/local/bin/ruby -Ilib:test "/opt/local/lib/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/rake_test_loader.rb" "test/unit/post_test.rb" 
Loaded suite /opt/local/lib/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/rake_test_loader
Started
.
Finished in 0.253822 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

Yay! Our only assertion passed. That is it! Literally. You now know how to test validates_presence_of. There are some things that are hard to test, but most of it is simple stuff like this. I can promise you, testing will save your butt and eventually you’ll be able to actually code faster and better when you test, than when you don’t.

How Can You Help?

So here is the deal, I’ll try to think of a lot of the basics of testing and I’ll also try to show how to do things with different testing frameworks, but I’ll probably need some help from you. I see two ways you can help out.

If you haven’t tested at all or have very little and you have something you want to test, but don’t know how, let me know? Suggest it. Email me. Send a pigeon or a fax. If I have time and the ability I will try to help out and post the results on RailsTips.
If you know how to test, be it in test/unit, test/spec, context, shoulda, RSpec or whatever, start thinking about what it took to get you started. If you have a blog, post articles. If you don’t, I’ll help you get them posted here. I seriously want to fill up all the Ruby and Rails blogs with testing articles from simple to complex (ie: from validates_presence_of to stubbing Net::HTTP).

Friends don’t let friends code without tests. Together lets make 2009 the year of practicing best practices in Ruby and Rails and lets start it by hopping on the test bandwagon.

35 Comments

Karmen Blake
Jan 06, 2009

Amen!
Justin
Jan 06, 2009

I’ve made it a project of mine to learn more about testing ASAP. I’m really glad to read this article, and look forward to more of them!
John Topley
Jan 06, 2009

Although I wholeheartedly applaud your sentiment here, doesn’t your example merely replicate the unit tests for ActiveRecord itself? The validates_* methods are already well catered for. Unit tests should test the business functionality that’s unique to your application’s domain.

My apologies if the purpose was to use a very simple example to illustrate how to get going.
Jonathan
Jan 06, 2009

I agree with John. Maybe testing the validation stuff was just an example of how simple testing can be. If not, what’s the point?

Otherwise, great post and thanks for the reminder.
Morgan
Jan 06, 2009

Greetings,

My biggest frustration (and thus what I’d love to hear you expound on) with testing is mocking, specifically 3rd party services. There seems to be no daylight between the two sides of (1) making the mock so complex that it needs debugging, and (2) mocking out so much of the infrastructure that you’re not really testing the code that does the work anymore.

I do a LOT of programming that involves this kind of thing, integrating with other services in various ways, and so I’ve run into this problem in many guises.

My current instance of this is an app I’m trying to build on the Lighthouse API, and my mock for it has grown to around 80 lines of code and 13 YAML files containing OpenStruct versions of the API’s real return values. (This time around I’m obviously pretty deep in the ‘make a mock that behaves like the real service, at least as much as is necessary for testing’ approach.)

I haven’t had this problem yet, but I worry about wasting time fixing & finding mock bugs instead of fixing & finding real bugs.

The opposite extreme, where you mock out all the entrypoints in a model that might end up talking to the API means that you might not actually TEST the code that’s beneath those points, and you can have bad logic and flat-out wrong code, and not know it.

Balancing between those is rough, and not something any simple example can show, unfortunately.

I’ve done both types on various projects, and while it’s more work, I find I prefer the ‘mock at the lowest level’ approach for its ability to allow actual testing of nearly all the code I write. I’m very much open to being told I’m being silly, and there’s a simpler way. :)

So, how would you write tests for your own Google Reader API library, or how would you test an application that was built on it?

— Morgan
Pete O'Grady
Jan 06, 2009

The example isn’t simply testing the validates_* methods, it is testing the business logic that a “Post should require a title”. Before the validation is added to the model the test fails, with it added it passes.
Nate Klaiber
Jan 06, 2009

Hi, my name is Nate, and I don’t test regularly. There I said it. Feels good to get it off my chest.

With that behind me – it’s time for me to find the right path and test in all projects in the near future.

I will be suggesting some different tests, interested to see how you mock/stub requests or information from 3rd party API’s (Net::HTTP, as you have said). While I understand the testing of the internals – I want to learn some of the more tricky or advanced testing – and these are the areas I run into bugs at times, so count me in as interested.

You have my attention, Mr. Nunemaker.
Jamie Macey
Jan 06, 2009

John, In this case the feature is very simple, but the purpose here is to have a test that will verify the presence of a validation check on title. We don’t terribly care what exactly the validation checks for (is it non-nil? non-blank? at least two words? capitalized?), so we’re only asserting that the title field has an error.

I do the same thing with associations – make sure I can traverse the association naively, and assume that it will behave properly based on ActiveRecord’s tests.
Ben Mabey
Jan 06, 2009

For those of you interested in testing third party services (or even internal web services) you should checkout fakeweb. Here is a great writeup about it:

http://technicalpickles.com/posts/stop-net-http-dead-in-its-tracks-with-fakeweb
Bill
Jan 06, 2009

I’m truly looking forward to Test Awareness Month here on RailsTips. I’ve been a bad programmer for a while and not doing as much testing as I should be doing. But this snippet from this post is where I sometimes end up

“be it in test/unit, test/spec, context, shoulda, RSpec or whatever”

Not that any one testing method is best, but they all seem to evolve quickly, with new approaches popping up all the time. How does shoulda relate to test/unit? Or is it RSpec? How is it different from cucumber/stories? etc. etc. It would be nice to get some better explanations and comparisons of the various testing frameworks for someone completely new to application testing.

There are so many options out there. I guess I should just pick one and learn as much as I can.
Morgan
Jan 06, 2009

Greetings,
I fubar’ed my own email address last time, which is why my gravatar didn’t show up. :(

@Ben: laugh! The YAML approach described there (with a few embellishments to make it easier) is very much like what I’ve been doing recently.

Since I’m using the Lighthouse API gem, I just don’t include it in test mode, requiring my mock instead, so anything that actually makes it past my mock will throw an exception, and I take that test failure as my guide to write another method/class/etc. onto my mock-API version. (Developing ‘test-middle’, instead of first or last…)

I’m glad to know I’m not the only one who’s gone down that path.

— Morgan
Morgan
Jan 06, 2009

Greetings,

@Bill: Start with test/unit. Everything else is sugar; some of it’s TASTY sugar, but it’s just C12H22O11.

Oh, and everybody has test/unit.

— Morgan
Chris
Jan 06, 2009

@John: Testing Validations. I think the point of that is testing that your application has the necessary validations in it, not that you are testing the way ActiveRecord does them, and I think it’s important to do so, the fact that x needs to be validated is part of the spec of your program, so write a spec/test for it.
Noel Rappin
Jan 06, 2009

Everybody reading this post is invited to take a look at Rails Prescriptions which will be publishing a free tutorial on Rails Testing followed by a PDF ebook. Also check out the blog for up to date info.
Alex Vollmer
Jan 06, 2009
As someone with a long history of XP and TDD, I would offer a handful of important tips:
1. When adding or changing a new feature, write a test that fails first, run it, then implement it
2. Never “refactor on a red” (change implementation while tests are broken)
3. Get into the habit of red (fail), green (pass), refactor (clean it up)
4. Use Autotest!
5. Read Kent Beck’s “Test Driven Development: By Example”
Cheers!
Lenary
Jan 06, 2009

I’m very guilty of this.

I want to take the time to learn, but whatever time i do have to code, i spend on doing the features, not the tests.

I should learn now. i got bitten tonight by a bug that would have shown up in testing i think. well, if not ruby testing then browser testing. It showed my why you need to do ruby testing, because you won’t hit all the stops with browser testing, but also to do a little browser testing of caching and the like so that you can check that it’s all working.
Jamie Hill
Jan 06, 2009
Great idea! I can’t believe I went so long without solid tests. I have found Jay Fields’s blog an invaluable resource for testing.

One think I really didn’t get when starting out was what to actually test. Here are a few rules I stick to when using plain ol’ Test::Unit which makes tests easier to maintain and on a whole easier to work with:
- One assertion per test.
- Create a Factory module with singleton methods for instantiating classes that need testing.
- No setup or teardown methods in unit tests.
- Use Mocha to stub external methods in unit tests and by any external methods I mean any public methods that are not the method I am testing.
- Don’t use fixtures in unit tests.
- Don’t hit the database in unit tests.
- Test every path the code can take in unit tests.
- Test overall functionality (full stack) in functional tests.
John Nunemaker
Jan 06, 2009

@John Topley – I understand what you are saying and yep I was starting simple. However, I still put tests like this in my app as I see tests as more of a contract then just verifying that something is working. If someone removes that validates_presence_of, I want tests to start failing.

@Morgan – I feel your sentiment. I don’t have a “set in stone” way of testing stuff like this and I’ve been feeling around lately but usually if the design is right, you can find a nice point in the code to stub/mock out. I typically have fixtures that are made up of example responses and then stub net/http or possibly a request method to just return that response. I also always have a directory of “live” examples that hit up the API for real. I run my automated tests while working on the API of the library and such and then make sure I run the live tests before releasing a new version. I’ll try to cover what has been working for me here but you can look at HTTParty’s tests for some examples.

@Pete O’Grady – Exactly! Much better put than me.

@Nate – Whew! Big relief getting that off your chest, eh? I’m glad you are willing to give it a shot. Hope I can actually teach it properly.

@Jamie – Yes. Spot on. Sometimes for associations I just test that the association is declared.

@Ben – I’ve used fakeweb before but I noticed issues with query params being in different order than the registered fakeweb url causing problems. Never got around to patching it though so I can’t complain too much. :)

@Bill – The first step is admitting. :) I always recommend that you start with test/unit. That is where we all started. Then, as you progress, if you find yourself wanting more, move on to another framework but you’ll never regret learning test/unit.

@Alex – Great tips. I’ll definitely cover autotest as I use that extensively as well.

@Lenary – That is just a mental block. See in the end you always end up spending more time, it is just the initial hump that you have to jump over. You spend way more time manually testing things than just taking an hour or two to learn how to automate things.

@Jamie – The no setup or teardown is interesting. I can see why but I’m not sure that I’m fully in agreement. You seem to be much more of a purist than me. I won’t be presenting that pure of options but that doesn’t mean they don’t deserve coverage. I hope that when I post stuff, you take it, convert it to how you would do it and post it on your site.

And thus ends my longest comment ever. You all are chatty today!
Adrian
Jan 06, 2009

Here here!

Second time I’ve heard about Fakeweb in the past few days, definitely going to check it out. When testing net/http I ended up just mocking out the methods I needed so it never made the call (using rspec).

Also, Test Driven Development: By Example can be previewed on Google Books.
crayz
Jan 06, 2009

I migrated to ruby/Rails back in early 2006 from PHP, and testing seemed to be viewed somewhat optionally in the community back then. I couldn’t make any sense of it, so just didn’t bother

After becoming a better programmer(thanks to ruby and the community) I realized a lot of the problems we were having could be caught or at least made less likely if we tested, learned RSPec, and started testing… a little. Over time I’ve been testing increasing amounts, and writing tests first, but I still don’t have anywhere near a proper level of coverage

This post has been a bit of a kick in the ass though, and I’m going to try to get the testing religion starting today :)

I’d be interested in hearing some talk about the “theory” of testing more as well. There’s a number of really interesting philosophical debates that go on in, e.g. the rspec mailing lists, about fixtures vs. mocks or the proper way to write tests such that you’re not just writing the same code twice, that don’t seem to leak out onto the community blogs very much

The other thing I think is really important and not talked about enough is, if you’re writing bad code(cramming everything into views and controllers and fat model methods, law of demeter, etc etc), it’s going to be nearly impossible to test it. Writing worthwhile tests tends to force you to have decently structured
Dave Woodward
Jan 06, 2009

As I’m writing some Javascript unit tests to cover ahem some of your code :) … I’ll respond with a YES!

We’re going to be tad busy this month… but does showing my wife how to write tests for her Rails project she’s working on count?

I’ve also had the urge to use github’s Pages thing to start a blog there, maybe I’ll write a quick post about the javascript_test plugin for Rails.
Jeff Wigal
Jan 06, 2009
Great post… quick question though… is the
```
test "should do such and such" do
  ...stuff goes here
end
```
syntax part of test/unit, or is this part of rspect or some other framework/plugin? all of my tests look like this:
```
def test_this_should_do_such_and_such
   ...
end
```
syntax.

BTW, I’m loving the one-two punch of Machinist and Faker . Good riddance to fixtures…
John Nunemaker
Jan 07, 2009

@crayz – I agree that theory should be discussed too. I can explain mine but I’m not as much as a purist as some are. I’m more about making sure the things I care about don’t break. I could care less if my unit test hits the database in all honesty.

@Dave – Snap! I do actually have a bunch of unit tests that I wrote for that originally lying around somewhere, though they are probably no good anymore. It does count if you show your wife but I’d love to see you start blogging. It is so gosh darn rewarding. :)

@Jeff – The test “should do such and such” is now built into Rails. I’m guessing it is in ActiveSupport::TestCase. RE:Fixtures – I haven’t really totally dropped fixtures. I do have some factory-ish things that I do though. I’ll definitely be researching factory girl and machinist for posts this month.
Paul
Jan 07, 2009

I’m looking forward to the rest of the series.
I’ve just written my first few tests as a result of this post, and it feels good.
These simple tests are easy to grasp, so I would like to see some info on how to test relationships between objects, i.e. order_line must belong to a order and so on.
smn
Jan 07, 2009

I’m currently working with a system (written by ‘somebody’) without tests and horribly written code.

The problem is that I dont really want to start writing test without refactoring the methods so they dont have side effects etc.
But I dont want to start refactoring without having proper tests of course.

It’s awful and a huge pain in the ass. that’s why no one up to this point has written any tests.

I started the year with the resolution to change this and as I know now..I am not alone :)

We’re using RSpec here which (for me personally) is a little strange at first but once you know how to use it it’s an awesome tool.

I am trying to enforce a policy here that every peace of code you touch must be covered in a test unless it already was before.

Let’s see how this works out! Wish me luck…
Chris Schmitt
Jan 07, 2009

John, I really like your attitude. For a slightly different take on the importance of test driven development, take a look at my post on applying TDD to non-programming related projects
Milan
Jan 07, 2009

Here is a post about how to start testing I wrote just a couple of days ago (actually it’s my first blog post ever :) ). It contains some motivation points for people afraid of making the really first steps.
And thank you John for very nice post and everyone else for useful comments.
Jason
Jan 07, 2009

Thanks John,

This type of push is great for those who need some encouragement to starting the testing lifestyle. I think one type of article you could write about is how to get an app that exists without tests, all tested up. Not sure if that requires going through each model and testing each method and the same for controllers, but it could be just a guideline.

Anyway, thanks for getting this started!
Nick Hoffman
Jan 07, 2009

@Morgan, for mocking and stubbing 3rd-party services, I’d suggest a multi-tiered approach:

1) Write “high-level” stories/features/tests that actually let your application run without interference (IE: without mocking or stubbing), and let it actually make the external API calls. I’d use Cucumber¹ for this.

2) Write “low-level” specs/tests along the lines of “the opposite extreme”. I’d use RSpec for this.

¹ Cucumber: http://wiki.github.com/aslakhellesoy/cucumber/

Cheers,
Nick
Jimmy Soho
Jan 07, 2009
Nice… so starting from very basic then, I’m curious how you would write tests for the following:
```
class Post &lt; ActiveRecord::Base
  belongs_to :account
  has_and_belongs_to_many :authors, :class_name =&gt; "User", :uniq =&gt; true, :validate =&gt; false

  attr_accessible :title, :body, :published_at

  validates_presence_of :account_id, :title, :body, :published_at
  validates_uniqueness_of :title, :case_sensitive =&gt; false, :scope =&gt; [:account_id]
end
```
How would you test your attributes (belongs_to, has_many, etc.) ?

How would you test mass assignment?

How would you test the validates_uniqueness_of thing?

If a month later someone adds an attribute e.g. :subtitle, and adds that to the attr_accessible, but “forgets” to add a test for that, is there a way to detect such a missing test?

Would be much appreciated if we could have a discussion around these basic questions.
John Nunemaker
Jan 07, 2009

@Jimmy – Those are all in the plan. Thanks for the suggestions though. Keep them coming people.
Don
Jan 08, 2009

Only a couple of other people have mentioned Kent Beck’s book “Test Driven Development”. In my opinion, there’s no better place to start. It’s not about a specific language or a framework, but how to think and proceed developing from a testing perspective. It’s not a big book.

Choosing between Rspec and Test::Unit, fixtures or FactoryGirl… that’s step 135 in the process. First is becoming used to that Red/Green/Refactor flow, getting to that point where the test is an integral step in your development and not an annoyance.

So bring on Test Awareness Month! I’m a freelancer, I want everyone to test in case I inherit your project some time in the future… :)
Evgeny
Jan 10, 2009

I love how restful_authentication adds it’s specs and stories to a Rails app, if only more plugins would do something like this. It gives the app all the required infrastructure like spec/stories folders, and a lot of example code that can be re-used.

For some reason, even though most vendor/plugins/… have testing code in them, it is rarely used as something to look at when looking for examples. When spec/* stories/* usually is a very good place to just “extend” the existing, instead of starting with a blank slate.

PS: your bottom on the blog is too big, when hitting “end” I don’t see the textarea where I type my comment. Just a lot of text that I don’t really care about. Better place for all that is in a sidebar or something.
Charlie
Jan 16, 2009

True story: I work in a Java shop (I hope not to for long – I’ve been learning other stuff including ruby/rails/merb on my own) and was actually yelled at by a coworker yesterday for writing automated tests. The practice at our shop seems to be, you have to test everything but you have to do it all manually. This can take hours to do the same level of testing that some JUnit tests could do in under a minute.

This person was upset because the tests were broken. Why were they broken? Because somebody changed some code that broke the tests, and didn’t realize this because they didn’t run the tests. Why didn’t they run the tests? Because nobody at this shop ever does tests except when they get to the ‘test’ part of the schedule, because it’s a huge amount of manual labor to do so.

I have to get out of there. If somebody is out there looking for a developer, and who has embraced some more or less ‘agile’ practices, including writing automated tests and using a good distributed VCS, I’d love to talk to you!
Collin
Jan 20, 2009

Nice post. For those of you that are using autotest on OS X, take a peek at growl-glue . It simplifies growl and autotest integration.