To mock, or not to mock?, by Emily Samp

Abstract

Mocking: it’s one of the most controversial topics in the testing world. Using mocks, we can more easily test parts of our applications that might otherwise go untested, but mocks can also be misused, resulting in tests that are brittle or downright self-referential.

So… how do we know when to use mocks in our tests? In this talk, we’ll identify three important questions that can help us make that decision. By the end of the talk, you will have a framework in mind to help you answer the question: to mock, or not to mock?

Details

Intended Audience

The intended audience for this talk are developers with some experience writing tests. They might know how to set up basic tests but feel less confident using more advanced techniques such as mocking. While I will provide code samples in RSpec, the concepts discussed in this talk should be applicable to any testing framework.

Outcomes

After attending this talk, audience members should be able to:

  1. Explain the difference between mockist and classicist testing techniques
  2. Articulate a framework for deciding whether or not to use mocks when testing
  3. Implement mocks in their Ruby tests using the RSpec testing library

Outline

Introduction (3 minutes)

  • Introduce two schools of thought in the testing world:
    • Classical testing: use real objects and verify the state of the system is correct
    • Mockist testing: use fake objects to verify the behavior of system is correct
  • Give an example of a test written in both styles and explain the benefits and drawbacks of both techniques
  • There are people who would say you should always write classical tests or always use mocks, but most developers use a combination of the two -- mostly classical with some mocks when necessary/useful.
  • However, few developers have a framework in mind to help decide when to use mocks. This can result in mocks that are improperly used, creating tests that are brittle, complicated, or tautological.
  • Thesis of the talk:
    • While classical testing will get the job done in most cases, there are scenarios where mocks are helpful, or even necessary.
    • This talk provides a framework to help people decide when they would benefit from using a mock in their test by highlighting the three main benefits of mocks:
      • They prevent the use of expensive resources
      • They create deterministic tests for non-deterministic code
      • They test behavior when state can’t or shouldn’t be exposed
    • Define key phrases: test double, stub, and mock

Note: for the examples listed in the rest of the talk, I will provide code samples using Ruby and the RSpec testing library.

1. Mocking prevents the use of expensive resources (6 minutes)

  • Example A: Mocking external API calls
    • Scenario: Your code makes a call to an external API.
    • Reasoning: You don’t want to make that API call because it will slow your tests down, or you might be rate limited.
    • How mocking helps: By skipping over the network call, you can keep your tests fast while still verifying that your code properly handles the API response.
    • Be aware: The shape of the API response may change, and you won’t notice because you’re not actually calling the API in your tests.
  • Example B: Mocking an object that performs time or resource-intensive work
    • Scenario: Your code performs work that takes a lot of time or uses intensive resources (e.g. takes up a lot of memory).
    • Reasoning: You don’t want your tests to take a long time or crash your CI system.
    • How mocking helps: You can keep your tests performant while still ensuring that other systems in your application properly interface with the resource-intensive object.
    • Be aware: Remember to update the mocks if you modify the resource-intensive object, otherwise your tests will pass but your application will fail on production.

2. Mocking can create deterministic tests for non-deterministic code (8 minutes)

  • Example C: Mocking an unpredictable error case
    • Scenario: You’re testing the handling of an unpredictable error (e.g. a NetworkError).
    • Reasoning: It’s hard to reproduce the error using classical techniques.
    • How mocking helps: By using mocks to raise the unpredictable error, you can ensure that your application is properly handling the error in production.
    • Be aware: Make sure you are mocking the correct error in the correct place, otherwise your test will be misleading and your application could break.
  • Example D: Mocking the dependency of a not-pure function
    • Scenario: You’re testing a function with a non-deterministic input, such as I/O or time. (This is the opposite of a pure function, where all inputs and outputs are deterministic.)
    • Reasoning: If the output of a method is non-deterministic, it’s hard to set up a test that will pass every time you run it.
    • How mocking helps: Mocking the non-deterministic input allows you to either verify the interactions of your code with that input (e.g. ensure that a message is logged), or make that input deterministic (e.g. using Timecop to freeze time).
    • Be aware: You can simplify your tests by making the non-deterministic object a direct input to your function (e.g. pass a created_at time as an argument to the method, rather than calling Time.now from inside the method).

3. Mocking can test behavior when state shouldn’t/can’t be exposed (6 minutes)

  • Example E: Mocking a cache
    • Scenario: A cache is an object that stores query results so they can be reused, rather than performing an identical query. Its internal state (aka the results it has stored) is not meant to be shared with other objects.
    • Reasoning: When a method uses a cache, its result will be the same whether or not the result was returned from the query or from the cache, making state verification useless.
    • How mocking helps: By mocking the cache and tracking its behavior, you can verify that it is working as intended.
    • Be aware: Lean as little as possible on the mocks by using Rspec’s and_return_original method. Use the mocks for behavior verification (e.g. check that certain methods on the cache were called), rather than re-writing the cache’s responses. This will make your tests less brittle.

Conclusion (2 minutes)

  • When deciding whether or not to use a mock, ask yourself the following questions:
    • Would this test use expensive resources?
    • Is the scenario I’m testing unpredictable or hard to reproduce?
    • Does the object I’m testing have state that can’t be exposed?
  • Final takeaway: there are no right answers when it comes to testing. The goal of this talk is to provide you with a framework for making the decision whether to use mocks, so you can make that decision for yourself and feel confident in the result.

Q&A (5 minutes)

Sources

These are the articles I will draw on to provide examples and justification throughout the talk:

Pitch

When I meet new people in the Ruby community, I tend to ask them what they think is the hardest part of the Ruby ecosystem. Surprisingly to me, a lot of people say “testing in RSpec/Minitest/etc.” When I dig further, I often find that these Rubyists understand how to use testing libraries just fine; what they actually struggle with is knowing when to use various testing tools or techniques, such as mocks. This talk should be considered because it provides a framework for using a testing technique that many people struggle with. Ultimately, it will help people become more confident in their test writing skills.

I am qualified to give this talk not only because I have years of experience writing and thinking about testing in Ruby, but also because I am a Ruby mentor to my teammates at work. I enjoy spending my working hours building and debugging tests with my coworkers and teaching them what I know about RSpec. I believe that my testing knowledge, coupled with my desire to help others become better testers, makes me the ideal person to give this talk.

Edit proposal

Submissions

RubyConf 2021 - Accepted [Edit]

Add submission