Why I dislike mocking

Date
Tags programming
Target Audience Programmers.
Epistemic Status A rant.

Mocking is a tech­nique in writing tests where some com­ponent ex­ternal to the thing you are testing is re­placed with a fake im­ple­ment­a­tion with con­strained be­ha­viour.

I do not like mock­ing, and think it often does more harm than good un­less you are very careful about what your test is ac­tu­ally test­ing.

An ex­ample

Let’s say your pro­gram in­volves loading some data from disk, and you use a lib­rary to do this load­ing. Let’s say that there are a few ways in which a file can be in­valid, and these are each sig­nalled by the lib­rary raising a dif­ferent ex­cep­tion.

Your code might look like this:

def cal­cu­late_thing
  Ex­tern­al­Lib­rary::Reader.new("data_­file").frob­n­icate
rescue Ex­tern­al­Lib­rary::Mal­formed­Data, Ex­tern­al­Lib­rary::Un­sup­por­te­dEx­ten­sion
  nil
end

And your test might look like this:

def cal­cu­late_th­ing_handles_­file_er­rors
  er­rors = %w(Ex­tern­al­Lib­rary::Mal­formed­Data Ex­tern­al­Lib­rary::Un­sup­por­te­dEx­ten­sion)
  er­rors.each do |err|
    Ex­tern­al­Lib­rary::Reader.any­_in­stance.stubs(:frob­n­icate).raises(err.con­stantize)
    as­ser­t_nil cal­cu­late_thing
  end
end

This looks good: you’re catching ex­cep­tions in your pro­gram, and your test is throwing those and checking that they are handled. But what is this really test­ing?

The test is ob­vi­ously cor­rect, which isn’t ne­ces­sarily a bad thing as it guards against the code chan­ging, but does it really test that you handle er­rors from the ex­ternal lib­rary? I don’t think so. If a new ver­sion of Ex­tern­al­Lib­rary comes along and adds a third ex­cep­tion type, this test will not help you.

This test guards against the ex­cep­tion list in the code being changed, but does not check that all er­rors from the ex­ternal lib­rary are handled.

The problem with mocking

The main problem with mocking is that it is very easy to write a reas­on­able test, and then to de­rive more con­fid­ence from it than you should.

Whenever you ar­ti­fi­cially change the be­ha­viour of something, you need to be very clear about what your test is ac­tu­ally test­ing. It is much better to avoid the change if pos­sible, pos­sibly at the price of a more com­plex (but more real­istic) test.

There’s a lesser problem that it’s easy to write a mock which doesn’t ex­er­cise all the be­ha­viour your pro­gram ex­pects (ima­gine cal­cu­late_thing handled the two ex­cep­tions dif­fer­ently, but your mock only threw one of them, for ex­ample). This problem can be over­come with branch cov­er­age.