Testing in Rails: Way More Trouble Than Flossing

Posted by amy on June 27, 2007

So the other day Max and I had a joint interview for a Rails job, and the hiring manager obviously felt that our knowledge of testing in rails left something to be desired. Too true, too true. But he also told us that he thought that learning to test in rails, and learning TDD in general, was basically straightforward and simple. As in, “you are idiots that you flubbed your answers to my questions about it.” I am sure we are idiots for some other reasons, but not completely grokking TDD should not be one of them. If TDD were easy to grok and easy to do, there wouldn’t be so much evangelizing around it. There wouldn’t be courses and lectures and demos and new articles every day about why you might want to bother doing it. Here’s Gregory Brown, just a couple weeks ago, still having to argue that rails testing is Not Just for the Paranoid.” :

The real issue most people have with testing is not that they think it’s a bad idea, but that it often means a whole lot more configuration, a whole lot more to learn, and lots of things that smell like extra work. Folks who have been in that crowd will be pleasantly surprised when working with Rails.

Thanks Gregory, but no, they really won’t be pleasantly surprised. Or at least, not immediately. We’ve been working with rails testing since March, and it still smells like more stuff to learn and more work to do. I’m sure it’ll start seeming easy in the future, but right now, not so much.

First off, there’s fixtures. Apparently everyone important agrees that rails fixtures suck. But mostly no one tells you this when you’re first learning to write tests. They reload before every damn method, which would be useful, except that they slow your testing down amazingly. I was feeling very proud of myself for using a Rails Recipe for erb in my fixture files to generate lots and lots of test data for very little effort, and wow, did it show in test slowness, not so much on my machine, but on Max’s, which is older and slower. Then there’s that whole weird transactional fixtures flag that Max wrote about, that no one tells you about but that can cause your db not to look anything like what you expect at the end of a test method. Lots of people refuse to use fixtures at all, or only use them to, hmm, load data directly into the test db, not for testing. Other people point out that fixtures are a pain to keep updated when you make changes to your db, so every time your model changes, all your tests may break because your fixtures no longer work properly. So then a model change involves: change your model. Make a db migration. Change your fixtures. Change your test code. Someone has written a nice little fixture migration task, but at this point, I think we’ll just change our testing strategy to not involve much in the way of fixtures.

Then there’s the TDD/BDD and third-party testing tools thing. Do we invest our efforts in TDD even though BDD is the new big thing, so should we just skip straight to RSpec? Do we learn Test::Unit really well before we switch over to ZenTest‘s Test::Rails? What about MonkeyTest? And rcov?

Oh, and how about assert_select? I turned around the other day and suddenly I was knee-deep in the DOM. All I wanted to do was check to make sure that when my user was viewing an item that belonged to him, he also saw two buttons, ‘edit’ and ‘delete’. And that when he wasn’t, he didn’t. It wasn’t obvious how to do it, and it turns out that if I want to make it easier to use assert_select, I probably need to be putting way more ids in my html tags. And then, I added a new feature, and a new button, and my tests broke.

Of course, tests are always breaking. I understand that it is a feature, not a bug, to be forced to understand all the implications of new code. Still, it’s frustrating.

Finally, and this is, perhaps, the root of the problem, there’s all the additional decision-making overhead imposed by the very act of testing. Not all this overhead is ‘good’ overhead — that is, not all of it ultimately contributes to the quality of your code. Most of it is just one niggly decision after another. How do I organize my test code? What do I test? How many tests are enough? What should be its own test method, and what should just be a couple of lines in some other test method? Should I use setup or not? Fixtures or not? Which frameworks? There are fifteen different assert methods that will do what I need them to do, in slightly different ways: which one should I use? Oh wait, is that one deprecated? How come this assert method doesn’t work? Oh, because it’s a Test::Rails method, and I’m not actually using Test::Rails, I just saw it somewhere. All those decisions are exhausting.

Bitch. Bitch. Bitch. I realize I may now have destroyed all my credibility as a test-driven-developer by complaining about all the things I don’t get and don’t like about TDD in rails. But I think it’s annoying to be struggling with something and read nothing on the internet about it other than “it’s easy and awesome”. I have drunk the TDD kool-aid, obviously, because if I hadn’t I wouldn’t bother with it since it obviously causes me a fair amount of pain. And maybe in six months I too will say “hey, yeah, testing in rails is easy and fun!” But right now, it’s neither easy nor fun. But right now, today, I am saying it loud and saying it proud, to you, the internets, to you, other struggling Rails developers, to you, the world: TDD is good for you, like flossing, but it is much, much harder.

The moral of this story is that every night as you floss your teeth, you should thank your lucky stars that something so good for you is also so easy to do. I know I will.

Popularity: 12% [?]

Theoretically Related Posts
  • Testing Rails: the learning curve
  • From _A Guide to Testing the Rails_, a fantastic explanation of why we should write test code
  • 12 Steps to Testing in Rails
  • Database Constraints, Stereotypes, Rails, Culture, Talmud, Gender, MINASWAN, Religion. But ABSOLUTELY NO BONDAGE PLAY
  • Trackbacks

    Use this link to trackback from your own site.

    Comments

    Leave a response

    1. Gregory Brown Thu, 28 Jun 2007 10:39:26 UTC

      I was actually speaking in reference to other languages, especially C++. Try writing tests in CppUnit or NUnit without poking your eyes out. :)

      But you’re right. Testing isn’t easy, at first. With respect to the decision making overhead, I find writing the tests first help with that. Then you’re not trying to form your specifications around blobs of code that may-or-may not be testable, and instead are writing what you expect right up front.

      It does take a whole lot of getting used to, but in practice, I’ve found that to dissolve the ‘how should I be testing this’ problem and move it into “how can I write testable code”. This usually means tiny methods that are easy to poke at when needed. It does result in better code, too.

      The problem is, if you’ve got a job with a tight deadline you’re trying to hit, maybe that’s not the time to start learning to make TDD/BDD a common practice. I’ve been in that boat before, and I find that even if you can’t have a comprehensive test suite, it pays off to eventually layer testing in.

      The most important places?

      • Any bug reports you get, to avoid regressions
      • Any business logic that can be expressed clearly in tests
      • When you refactor code, add tests for that chunk to make sure you don’t introduce new bugs.

      I would have liked to layer in more advice like that into my article, but article length constraints and a pre-specified scope left me little room to talk about the “why?” part.

      Anyway… best of luck. Hopefully things will get better over time, as if you’re working in Ruby/Rails you really won’t be comfortable without a strong appreciation for testing. Most jobs sort of expect it, too. :-/

    2. Sidu Sun, 01 Jul 2007 14:08:59 UTC

      I’ve generally found writing tests in Rails to be a more complex task than writing tests for plain Ruby code (or C# or Java) – mostly because Rails is all about the database. Usually, the focus of a test is one small piece of logic, but in Rails theres all this other stuff going on so it’s easy to get distracted and find you’re testing larger and larger chunks rather than a ‘unit’.

      One thing I can add which I wish someone had told me when I’d started with TDD – you can’t do TDD if you don’t know where you’re going. I used to spend a great deal of time trying to figure out how to test first when spiking or prototyping. If you’re exploring an idea or evolving your design in code, but aren’t sure what exactly you’re aiming for, you cannot be expected to test first. Once you’ve got your design fleshed out then you go ahead and write tests for your logic.

      One deficiency in Rails testing is the lack of integrated support for functional tests. It helps to start building a functional test suite quite early in your application life-cycle. We usually use either Selenium or Sahi which simulate user interactions with the browser to build these tests.
      Funtional tests are usually run in a secondary build (because it can take quite long to run as the number of tests increase) triggered by the first build (the regular rake -t build) passing.

      Setting up your build in a CruiseControl.rb instance (it takes under 30s) also helps streamline things a great deal.

    3. max Sun, 01 Jul 2007 14:20:59 UTC

      Sidu, thanks for the useful notes. My first readthrough was pretty funny, as I interpreted “CruiseControl.rb [...] takes under 30s” to mean that one must be under the age of 30 to install CruiseControl successfully.

    4. Testing in Rails: Way More Trouble Than Flossing Fri, 03 Aug 2007 16:26:40 UTC

      Kramer auto Pingback[...] gst via thirdbit.net Submitted: Aug 03 / 17:26 Testing in Rails: Way More Trouble Than Flossing So the other day Max and I had a joint interview for a Rails job, and the hiring manager obviously [...]

    5. ryan Mon, 06 Aug 2007 08:24:23 UTC

      Testing. Is. Not. Easy. And that’s the truth. I, too, have moments when I’m questioning TDD. In fact, lately, I’ve been doing some things differently.

      Some argue that it’s best to test while you work, and some argue it’s best to write the test before the method. I kind of disagree with both. For me, testing is an entirely different mindset than writing traditional Rails code. I mean, it’s all Ruby, but it’s different.

      I have to be in a “test mode” to write solid tests, even if that means doing it later on in the project. I’m not particularly good at switching back and forth, so I’ll develop a reasonable “chunk” of functionality, and then go back and write a series of tests for that chunk, later.

      By separating those two, testing almost becomes addicting in some ways. I love seeing 0 failures, 0 errors, but you’re absolutely right, it’s often not as easy as it seems. I’m still in Test::Unit, which is fine for me (right now). I hear there are better libraries and ways to test, but this is just fine so far.

      Good luck with it…

    6. [...] I trust it much more than I trust either my own code or DHH’s. Database constraints are easy to use and good for you, just like flossing. And that appears to be the majority [...]

    7. bytesized Fri, 27 Jun 2008 04:20:57 UTC

      Kramer auto Pingback[...] Aug 3rd, 2007 The moral of this story is that every night as you floss your teeth, you should thank your lucky stars that something so good for you is also so easy to do. I know I will. — Testing in Rails: Way More Trouble Than Flossing [...]

    8. Kramer auto Pingback[...] RSpec, Shoulda or Cucumber. You can also write your own custom RSpec matchers. It can become a nightmare to make the right decisions and find the right tool. Remarkable tries to unify the syntax and adds [...]

    Comments