My Testing Theory

I figured I would start of this Test or Die month of January with the theories behind why and how I test. I am far from a “purist”, meaning what I care about most is that my butt is covered and that I can refactor until my little heart is content. I could care less about whether my unit tests hit the database and I really don’t want this emphasis on testing to become that.

Vision

In fact, my previous statement is really important and I’m going to say it again. I don’t want this emphasis on testing to turn into philosophical discussions on what is the true way. I want it to be down and dirty, in the mud, at the beginner level and then step slowly but surely up to intermediate and end where I am actually at (most likely not expert). I want it to be practical. I want people to learn the basics and not worry if they are doing it the “best” way. This month is all about starting.

Mission Statement

That said, what are my beliefs? Well, like I said above, I like to cover my butt. I want to know if something breaks when I change something. If I update to the latest released version of Rails or I dare to ride on the edge, I want to know if the update broke anything in my application. I have heard some speak of testing as a contract. It is an agreement with your code that it will behave under certain terms and if it doesn’t, someone call the lawyer (or turn red in autotest).

History

Just for the heck of it, I’ll give you a little look at where I am at tool-wise. I started with good old test/unit. Then, a long time ago, in a windy city (literally, the windy city, Chicago), I attended a a chirb.org meeting on BDD with RSpec. I thought it was cool and loved the syntax, but wasn’t ready for a switch. test/spec came along and I greeted it with open arms. I used it for all of 2007 and part of 2008.

I don’t even remember what my first project was that used RSpec, but it was after RailsConf so it must have been June or July. I’ve used pretty much only RSpec since then. I drank the kool-aide and it went down smooth. Then, I started running into errors that my tests weren’t catching, I think, in part, because I was isolating too much and not testing enough. I realized that I needed to write more tests in RSpec to get the coverage I wanted but I didn’t feel it was worth all the extra effort. Again, just my opinion, you are free to disagree, but you are wrong. Just kidding. :)

Then, in December, after reading about the current state of ruby testing, I felt swung back to good old test/unit, so I gave context and stump a try and really liked it. They provide just enough of what I need from RSpec by adding it onto vanilla test unit. I haven’t started a new Rails app since trying them out, but my next one will most likely have them both included in my test helper. Enough with the history lesson.

Coverage

Speaking of test coverage, what all do I think I’ll be covering in the next month? Well, truth is, I didn’t really think that far ahead, I just felt like testing was something I needed to push and hoped that the rest of the community would step up too. I’m actually amazed at the response thus far to be honest.

I would like to show the different options out there in a post or two and maybe elaborate a bit on my thoughts of them. I would like to run through all the basic contracts that one would want to have in their app. Simple tests like validations, associations, controllers, etc. From there, I will probably jump out of Rails and into plain old Ruby and show how I have begun to test my gems, specifically those that interact with other services and such.

That is really as far as I have thought ahead. I hope the discussions on each post are positive. I hope those in the know of testing help and encourage those that aren’t. I hope that this spurs those in the know that disagree with me to post their own articles and explain why I am wrong (and I will gladly link to them if you notify me). I just want to get all the opinions out there. I am fine if this takes longer than a month. Also, don’t expect every post for the next month or two to be about testing. I have some other stuff up my sleeve too.

10 Comments

khelll
Jan 07, 2009

Thanks a lot, we are waiting your upcoming posts
Matthew Lang
Jan 07, 2009

Looking forward to your future posts on testing and all related topics. Testing isn’t my strong point and it’s something that I want to start changing now in my development projects.
Bryce
Jan 07, 2009

I too am looking forward to this series. Testing is one of the things I really don’t know how to do yet.
Soleone
Jan 07, 2009

I really like Shoulda (I’d describe it as a mixture of test/unit and rspec), as it is really easy to use and brings some cool macros in.

You just need to require the Gem in your environment.rb and you’re ready to go. It allows mixed versions of test/unit and shoulda syntax, so it’s cool for transitioning.
Harry
Jan 07, 2009

You should take a shot at doing acceptance testing with Cucumber and Webrat as well, it’s all the rage, y’knows.
Peter Browne
Jan 07, 2009

As a newcomer to automatic testing I have noticed I spend as much, if not more, time coding the tests (which is the idea I believe).

But how do you manage to do this when you have barely enough time to code the app itself?

Anyways, I look forward to this month of posts, thanks!
Michael McClenaghan
Jan 07, 2009

Thanks for the post. I do Rails as a side gig for fun so I don’t get as much time to stay up to speed as I’d like. Because of that, I find that I spend a lot of time trying to figure out “the right way to test” rather than just trying something. I’ve recently started to classify that internal debate as just another type of procrastination.

Glad to see that guys doing Rails full-time also struggle with choosing the right way. Keep posting on your experiences!
Jason Morrison
Jan 07, 2009

Peter,

Make time for it. Delivering an untested application is compromising – it’s delivering uncertainty, fragility, and a ticking time bomb of unmaintainability. If your boss/client/devil-on-your-shoulder doesn’t think that time should be taken to write tests during initial development, help them understand that it’s really a time saver in the long run.

Next time you add a feature, refactor a module, or upgrade to a new version of a library or framework with the confidence of having a test suite, take a moment to consider how much time you would have spent debugging new code, manually re-testing every feature in your app, or deploying a hotfix to production because of a regression issue. Testing goes a long way toward mitigating these time losses.
John Nunemaker
Jan 07, 2009

Truth is I don’t think you will see and understand the time savings until you have a good test suite on an app and you have needed to make maintenance changes.

@Peter – At first you will notice it takes a while to write tests. It is just like coding though in that you start to recognize that certain ways of testing go well with certain pieces of code. Once you start to notice this, when you go to test, you will have an example to work with which makes writing the test much faster. Practice makes perfect!

Also, Jason’s comment is a great response to the time issue. The time savings that it might feel like you get when you first code something will be completely drained and then some down the rode when you maintain the app with no tests. I recently worked on one of my own apps that I cut corners on a few years ago and didn’t test much. The parts that were tested were easy to make changes to and the parts that weren’t burned me bad.

@Jason – Great comment.

@Michael – It definitely is procrastination. I experience that a lot, even outside of tests. Sometimes I sit there and think about the best way, when I could have just written it, tested it and then refactored.
Chris
Jan 09, 2009

I’m what I guess you can call an intermediate tester. I’m very comfortable with unit tests, but I need some guidance on higher level testing.

For instance I’m working on a gem. Even though I can write unit tests for the (cmdparse) command objects that make for the cli, it feels very little elegant. I guess this is where tools like cucumber comes in. In any case, it’d be great to see some examples on testing at this level.