The Pyramid of Automated Tests

So you've decided that automated testing is a good thing for your team. After all, it's an excellent way to improve productivity and reduce risk. Who wouldn't want that, right?

But then you decide to research how to get started on building an automated testing framework for your existing application, and things get a little tricky. Integration tests? Unit tests? Acceptance tests? How do all these tests work together and what should you focus on first?

At Corgibytes, we see this issue frequently. We're often brought on to help a team introduce automated testing into an existing application.

We like to think of testing as a pyramid. The goal is to build an entire pyramid and keep it growing at scale. This means that over time, you'll end up having a lot more unit tests than acceptance tests, but I'm getting a little ahead of myself. First, let's dive into the three different types of automated tests and why they're important.

Types of Tests

There are three different types of tests to consider when you're building your testing suite:

• Unit Tests - validates the functionality of a small unit of code in complete isolation from all external systems.
• Integration Tests - tests the interaction between two or more units
• Acceptance Tests - ensures all the units involved in a particular feature or workflow work as expected

Unit Tests

What in the world is a unit?
Before we go any further, it's useful to define the word "unit". Pay attention. Units are the building block for the rest of the concepts.

A unit is an intentionally vague term that's meant to be a stand-in for whatever your unit of focus is. If you're in a functional language, your focus would be, guess what — a function! For object-oriented languages, this could be an object or a method. You get the idea. The idea here is that you want to be consistent, so make sure you document what a unit test means to you and your team.

Okay, so with that, now we can proceed.

So, what's a unit test and why is it useful?

A unit test is designed to validate the functionality of a small unit of code in complete isolation from all external systems.

Read that last part again. We emphasized it for a reason. It's cool. I'll get some tea while you let the concept sink in.

Alright, I'm back. You good?

Let's parse that sentence a little bit. Unit tests are the foundation of our pyramid because they validate behavior in very discrete chunks. You don't need access to a file system. You don't need access to a database. You don't need access to a network. You don't need access to any functionality provided by a third party API or framework. If you do connect to any of these things, you do not have a unit test. You have an acceptance or integration test. Very important.

Why keep things so separated?
Well, you want there to be only one reason a test can fail. This way, if a test fails, you know exactly what you need to do to make it pass. Keeping unit tests small and independent is vital for maintaining the speed and focus that make developers more productive. If your unit tests are discrete and not tied to other aspects of your code, your test suite will be zippy, and your poor developers will not be able to rely on the "Sorry, my test suite is running" excuse any longer. Instead, they'll be focused. Keeping the problem domain in their brain and minimizing context switching, which is expensive.

Integration and acceptance tests rely on having a group of unit tests already in place, so if you have zero tests, you'll want to focus on getting some unit tests in place first.

A note just for Rails developers
Sorry to pop your balloon, but chances are your app has very few unit tests. That's because it's VERY common for tests that are called unit tests by rails to include database interactions. But as we've learned, a unit test needs to be discrete and not tied to another system. This means that you have lots of shiny integration and acceptance tests, but that you're probably frustrated by how long it's taking your suite to run. We love Rails, but this is one area where it falls short in that the easiest way of doing something isn't the best. I'll plan on going over specific strategies for attacking this in a later post, but if you have questions in the meantime, just let me know.

Integration Tests

Whew! If this were a talk, this would be the point where everyone gets up and stretches. Go ahead. I'll wait.

Okay. And we're back.

As you'll recall from earlier, integration tests validate the connections between two units and the way data flows from one unit to another. These tests are incredibly useful but are slower to run than unit tests because there's more code that needs to run.

When clients complain that their test suite takes too long to run, one of the first things we look at is the ratio between integration and unit tests. When your integration tests lack a solid foundation of unit tests, your test suite becomes incredibly slow. Why? Let's take a familiar example from Rails to illustrate the point.

In Rails, a significant amount of application logic lives directly in controllers and the only way to reliably test controller actions is by simulating an HTTP call. This means that if you only rely on integration tests (which most Rails apps do), then you're repeatedly executing the same logic over and over and over again. Kind of goes against that whole DRY principle thing.

A better way to approach this scenario is to have unit tests that cover all of the different behaviors that are accessed through the controller. Then you'll only need one, maybe two, integration tests that ensure that data is flowing correctly from the HTTP request and the pieces that are already covered by unit tests.

Acceptance Tests

Acceptance tests make sure all of the components in your system are behaving when they all work together. If it sounds like a comprehensive and all-encompassing type of thing, you're spot on. Acceptance tests are useful because they're a great way of translating the criteria that a human would use to determine if a feature is acceptable or not. Product Owners LOVE these, and for good reason. Acceptance tests help ensure the app that they envisioned is working as expected.

There's a downfall, though. Acceptance tests are incredibly slow to run, precisely because they're so comprehensive. So if your test suite is bogged down, super slow, and you're questioning its value, take a look at the ratio between the types of tests you have.

Balance Your Pyramid

The goal here is not to build a test suite entirely out of unit tests. That would be useless. Instead, the idea is to make sure you have enough foundational tests to support the suite and make it run as efficiently as possible.

What's a good ratio? In general, you should aim for two to three times more tests for each descending level. So for each acceptance test you want to write, you'll have around three integration tests and nine unit tests.

If that's ever not true, then your pyramid is out a balance. Imagine what a pyramid would look like if it was upside down. I've worked with test suites that were structured that way: there were more acceptance tests than anything else.

Another way to measure balance is with test coverage. If you collect the test coverage of each level in your pyramid, you should find that your unit tests cover a larger percentage of your app than your integration tests, and your integration tests cover a greater percentage of tests than your unit tests.

You can also measure balance with execution time. Your entire unit test suite should run faster than all of your integration tests, which should run faster than your acceptance tests.

By using the pyramid of tests to design your test suite, you can keep your tests running quickly, mitigate risk, and keep all the members of your team happy and productive.