Regression Tests vs Unit Tests, Integration Tests, etc.
I saw some confusion today about what the difference is between regression tests versus unit tests, integration tests and end-to-end tests. The person in question pointed out that regression tests are intended to ensure that modifications to the code base don’t break the existing features. So, they reasoned, wouldn’t all unit tests also be regression tests?
When you call something a “regression test”, you’re saying something about the intention behind creating that test, but nothing about the form that that test takes. The intention, as mentioned above, is to ensure that changes don’t “regress” the code base to some previous state (for example, the state where a given feature doesn’t work). But the form could be anything: for example, “regression testing” could refer to QA experts manually testing the software.
When you call something a “unit test” or an “integration test”, you’re saying a bit about the intention behind the test, but you’re mainly saying something about the form of the test. These tests are always automated, for example, and not manually performed by a human.
Unit tests are designed to test one specific unit (a method or a class), and this is both a fact about its intent and its form (unit tests are more likely to involve mocks, and run quickly).
Integration tests are designed to test whether the individual pieces are able to cooperate (integrate) together to solve the business need. This is also a fact about its intent and its form (integration tests are less likely to use mocks, more likely to use “fakes”, and are more likely to take longer to run).
If you imagine a Venn diagram, the circle for “regression test” and the circle for “unit test” overlap, but neither one is fully contained in the other. Similarly, “regression test” and “integration test” overlap but not fully. Probably in most people’s mental models, the circles for “unit test” and “integration test” do not overlap at all.
Many (perhaps most?) unit tests have the intention of preventing regression bugs, and thus can be described as regression tests. However, someone could write a unit test with a different intention. For example, someone might write unit tests to document the contract of a method.
This is especially powerful in languages like Python, where you can embed your unit tests in the documentation comments of methods, where they serve as “executable documentation”.
def factorial(n):
"""Return the factorial of n, an exact integer >= 0.
If the result is small enough to fit in an int, return an int.
Else return a long.
>>> [factorial(n) for n in range(6)]
[1, 1, 2, 6, 24, 120]
>>> factorial(30)
265252859812191058636308480000000L
"""
result = 1
factor = 2
while factor <= n:
result *= factor
factor += 1
return result
The intent for these kinds of tests isn’t so much to check that the implementation hasn’t broken (although they also sometimes do this as a secondary side effect); rather, the fact that these “comments” are executed by the compiler is intended to increase confidence that the documentation and the code are still in sync.
I didn’t cover end-to-end tests because in my experience, different teams mean vastly different things by that term, to the point where it’s almost useless for communicating across teams. If someone says “end-to-end tests”, my advice is to ask them to clarify exactly what they mean by that.