Integrated Tests are a scam 27 Jul 2015

“Integrated Tests are a scam” is a talk by by J.B. Rainsberger. Lately I discovered this talk and wanted to share it with my company. So I presented the talk to my company.

These are my research notes.

Introduction

Makes a case for using as few integration tests as possible. One should focus on writing good unit tests instead.

Integrated tests are a scam—a self-replicating virus that threatens to infect your code base, your project, and your team with endless pain and suffering.

I use the term integrated test to mean any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behavior.

Unit test vs integrated test

Unit Test tests one component of your system in isolation.

Integrated Test definition:

Any test whose result (pass or failure) depends on the correctness of the implementation of more than one piece of nontrivial behavior

Why do we write integrated tests

• integrated tests are written because we can’t write perfect unit tests
• all unit tests pass, but you find a defect anyway
• sometimes an obvious unit test is missing, but sometimes you can’t
• then you write an integrated test to test the broken code path

Why are integrated tests problematic

If we can find defects even when our tests pass 100%, and if I can only plug the hole with an integrated tests, then we’d better write integrated tests everywhere.

• you don’t have to reason hard about your architecture
• just test the desired behaviour with an integrated test
• black box testing, how the implementation is implemented becomes irrelevant
• allows sloppy design

Test explosion

• combinatorial explosion of integrated tests

Imagine you have 10 layers with 4 branch points in each layer This leads to 4^10 = 10^6 code paths in your application. We can never write that many tests.

Bottom point:

• can’t test intensely (combinatorial explosion)
• takes too much time/money
• allows sloppy design

How to do it with unit tests

• avoid combinatorial explosion by only testing one unit at a time
• thoughtfully testing the collaboration between components via interfaces.
• most of the time if 2 components talk one can be seen as the client and the other as the server

This types of tests should be written

• client should ask the right questions (check with mock server)
• client should work with expected answers (how to handle 0, 1, many, a lot, oops)
• server should expect to get asked the same questions as in 1.
• server should be tested to return all the expected answers

Boils down to

• Client ask the right questions
• Client should expect all answers
• Server should expect all defined questions (as asked in 1)
• Server should return all types of answers (as expected in 2)

How to avoid Pass Tests & finding defects?

The test pairs need to match each other. Can get out of sync with refactoring. Talks about it: Who tests the contract tests Perhaps possible to automate? Could be possible to automate

Takeaway

• Strongly typed languages with interfaces are required.
• To do this you have to think more about your tests
• Can result in a nice testable system

Reference

• Blog Post
• Talk from 2013
• Talk from 2009
• External notes on the talk

What is a contract test

Contract Tests explain how a class should extend a superclass or implement and interface, so that I don’t have to read a bunch of prose to figure out how to do that. Typically, a contract test case class is abstract, then I extend it and implement a creation method or two to return instances of my own implementation of the given interface. That gives me a standard battery of tests I can run to drive my implementation. It might not be perfect (I’ll have n failing tests to start) but I prefer it to documentation written in prose.

Reference JUnit Recipes by J.B. Rainsberger How do contract tests look like

Wait what? Are you suggesting to not write end to end

End2End: Test a system (web app) from one end (webbrowser) to the other end (database)

Acceptance Test: Make the customer feel good about a feature being present System Test: Make the programmer feel good that the components work together correctly

Acceptance tests tell when you’ve written enough stuff

Reference

What is the test pyramid

Concept described by Martin Fowler. Not to be confused with the ice cream cone.