An introduction to mutation testing

Test-driven development (TDD) is one of the most useful skills you'll learn as a software developer since it allows you to add new features, refactor old code, as well as catch bugs you may have missed when coding with ease. With tests, you can rest assured you'll know first whether your changes broke something somewhere.

Nevertheless, it is virtually impossible to consider every use case or to avoid making mistakes, that's why things wreak havoc from time to time. Is there a way to reduce this margin error even more? You can bet on that!

Enter mutation testing. As the name implies, it's a testing method where you add mutants (also known as variations) to your code to make it fail. Some examples would be: changing true for false, calling an incorrect method, deleting parts of your code, you name it. The success is then measured by the amount of tests that fail.

If a test fails, it means that your code is covering (or handling correctly) that case and there's nothing to worry about. If it passes, that means there's something you're either a) not covering b) not testing correctly, and you must fix that.

What are my options?

At Nebulab, we're very big fans of Ruby on Rails and JavaScript. We went looking for mutation testing libraries for these languages and found the following:

• Mutant by @mbj, which works for Ruby
• Stryker, which works for JavaScript, but has two work-in-progress implementations for Scala and .NET

How do I get started?

For the sake of brevity, we will only use mutant for this post. Its GitHub repo contains a README you should read in depth, as it contains a guide with nomenclature associated to mutation testing, which you should get familiar with before going any further.

Once that is done and dusted, you need to integrate mutant to your project. We're going to add it to Solidus, as it's always better to use a real-life example when explaining new concepts, which is as simple as adding the following line to Solidus' Gemfile:

gem 'mutant-rspec', '~> 0.8.24', require: false

Then, run bundle to install the newly-added dependency.

If you're running the above command for the first time, you shouldn't have any problems. Otherwise, you might need to run bundle update to get things up and running.

Worth noting that there's also an integration with minitest.

And that's it! You don't have to do anything else, mutant is now available for you to use on your project. But how do you use it?

Killing mutants

Unlike other testing libraries, you don't run mutant against a specific test file or scenario, but rather against example group descriptions, like this:

bundle exec mutant --include lib --require spree --use rspec "Spree::Api::CouponCodesController#load_order"

That particular action for that controller has two mutants that will make the specs pass; that means there's something we're not testing/covering and we need to kill the mutants (make them fail)

In this case, we're not testing how our endpoints must respond when:

• an order is not found
• passing an argument other than an order number

How can we fix this?

Under Spree::Api::CouponCodesController#load_order, we can replace params[:order_id] with the order_id method found under Spree::Api::BaseController, which returns all the possible attributes passed as an argument when searching for an order. While this is not directly related to the killing of a mutant, it allows us to fix possible bugs in advance.

Now, to properly kill the mutants, under api/app/controllers/spree/api/coupon_codes_controller.rb, we'd need to add the following tests:

describe '#load_order' do
  context 'when trying to load a non-existing order' do
    it 'returns an error' do
      post spree.api_order_coupon_codes_path('not_found'), params: { coupon_code: 'not_found' }
      expect(response.status).to eq(404)
      expect(json_response).to eq({
        "error" => I18n.t('spree.api.resource_not_found')
      })
    end
  end
  context 'when passing an argument other than an order number' do
    it 'returns an error' do
      post spree.api_order_coupon_codes_path(self), params: { coupon_code: 'not_found' }
      expect(response.status).to eq(404)
      expect(json_response).to eq({
        "error" => I18n.t('spree.api.resource_not_found')
      })
    end
  end
end

And that's it! You just killed two mutants that were dwelling within your codebase, expecting to crash your application when you least expect it 🎉

There are two caveats when implementing mutation testing that you need to take into consideration:

• Mutation testing is slow, as it requires a lot of computational power. Unless the project you work on can afford to apply it to the whole spec suite, it's better to use it only where absolutely required (i.e.: methods / modules / classes you want as bug-free as possible) or if the project is small enough.

• Equivalent mutants, which are mutants that modify your code in a way that makes it behave as you'd expect it. While this might help you write better, non-redudant code, it also makes a lot of (white) noise, which might be very annoying on bigger codebases.

I also noticed that the output is not consistent for failing specs, at least on this scenario. I'm not sure whether it is the expected behavior, a bug on the parser, or mutant requires modifications to properly work with Rails engines, but it was very annoying to receive a different output everytime I ran specs for the aforementioned controller.

Closing notes

I think mutation testing adds a lot of value to your projects and it's really worth the effort, as it allows you to reduce redundancy within your codebase, remove dead code, forces you to leave no stone unturned and, as mutant's README says: coverage becomes a meaningful metric.

I plan to propose a proof of concept of this testing method for Solidus in the near future. If you wish to keep track of this idea, you can do so here.

What do you think about this testing method? Have you implemented it before on one of your projects? How well did you fare? Let us know in the comments below!