Is BDD and automated regression testing the same?

Introduction

When we run Automation in Testing training, BDD almost always comes up at some point with the terms BDD, automation, Gherkin, regression or Cucumber used interchangeably. This pattern has been a constant frustration for me to the point that it has become a running joke with Richard that the terms Gherkin and Cucumber set me off into a rant (for balance, Richard’s is JavaScript).

But I rant because I am still frustrated that as an industry we struggle to distinguish the differences between the purpose and activities defined within BDD versus the purpose and activities of automated regression testing. So in this post, I want to get it all off my chest and share how I define both and explain how they fit together.

Defining the confusion

Joking aside, it does highlight that in our industry we still have misconceptions around the purpose and value of BDD and its relation to automation. My observation is that the misconception comes from several points related to the automated tooling side of BDD:

1. We’re constantly being told that BDD is about ‘communication and collaboration’ but then the term is used casually in conjunction with test automation tools like Cucumber, RSpec, JBehave, etc.
2. The examples shown to us for these tools follow the same patterns as other ‘non-BDD’ test automation tools
3. We’re told the automated examples from these tools are typically run in a pipeline and are used to regression check our products

Or to put these points another way:

If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

It’s easy to assume BDD is the same as automated regression checking because the automated tools we use behave in almost the same way. So how do we clear up the confusion? We do that by focusing on the purpose, not the implementation of these two approaches.

Defining purpose

BDD and Outside-in-development

Firstly let’s clear up the concept of BDD and its relation to automation. Behaviour Driven Development isn’t just about automation, it’s a process that is made up of activities designed to increase collaboration and discussion within teams. All focused on creating a shared understanding of the system and what to deliver within a team. It’s an anti-pattern to use BDD as a synonym to automation because BDD is so much more.

But automation does make up a part of the BDD process in the form of ‘Outside-in-development’. Outside-in-development is an activity in which you take a captured Gherkin example, automate it so that it fails (because the code it’s checking hasn’t been implemented yet) and then develop just enough production code to make that automated example pass.

The purpose of Outside-in-development is to guide development towards delivering the right thing. It’s this process that is often mistaken for automated regression checking, but before we explore that issue, let’s first define the purpose of automated regression checking.

Automated regression checking

Let’s take a step back, and remove the automation part of this equation and look at the purpose of regression testing. When regression is typically discussed, it’s seen as a safety net or a way to prevent bugs but actually, regression testing is a way to challenge our assumptions about the system. As we test, we learn about our products which helps us build an understanding of what the system is and does. As new code is introduced or old code is removed from the system what we think the system does compares to watch it actually does begins to drift. So we carry out regression testing to discover the changes that have occurred between our understanding and the current system to decide if those changes are desirable or not. I like to think of regression testing as change detection.

Automated regression checking is the act of deferring that change detection to a machine. When creating an automated check, we are codifying our understanding of a system into code that a machine will run to see if our understanding of the system is still correct, or if something has changed. If an automated check completes without error, we can assume that our understanding is correct. If it doesn’t, then something has changed and we need to explore said change.

The purpose of automated regression checking is automated change detection.

Different roles for different problems

So we have two different activities at play here. Outside-in-development is about discovery and design. We capture the expectations and then use those expectations as boundaries to develop a solution to solve a problem. Whereas, automated regression checking is about executing codified knowledge to determine if our knowledge of the system has changed.

So am I saying: ‘stop using tool A’, no I’m not. But as an industry, we are focused too much on the tools and not the purpose behind them. A tool like Cucumber can be used for both Outside-in-development and regression checking, but depending on the problem you are trying to solve, the tool you use might not necessarily be the most effective.

Gherkin based tools are designed to encourage a team to capture behaviour about the system as a whole, they are focusing on the risks of delivering the right thing. So the tool is biasing you towards building full-stack, UI driven checks. These types of checks are not inherently bad, but when trying to apply this approach to automated regression checking that focuses on lots of different risks, you run the risk of creating a mountain of full-stack checks that:

• Aren’t targeted or informative because they only communicate behaviour from the UI layer
• Are slow to run and diagnose due to the additional layers of abstracts
• Require more maintenance due to the additional abstraction layers added
• Can be unreliable as they deal with many system complexities at the same time

In the end, I am encouraging you to think about what your problem is before choosing a tool. If you want to ensure you deliver the right thing the first time, then Gherkin based automation can help. Want to detect changes in your system rapidly, then look elsewhere.