Property-Based Testing with PropEr, Erlang, and Elixir

Find Bugs Before Your Users Do

by Fred Hebert

Property-based testing helps you create better, more solid tests with little code. By using the PropEr framework in both Erlang and Elixir, this book teaches you how to automatically generate test cases, test stateful programs, and change how you design your software for more principled and reliable approaches. You will be able to better explore the problem space, validate the assumptions you make when coming up with program behavior, and expose unexpected weaknesses in your design. PropEr will even show you how to reproduce the bugs it found. With this book, you will be writing efficient property-based tests in no time.

Most tests only demonstrate that the code behaves how the developer expected it to behave, and therefore carry the same blind spots as their authors when special conditions or edge cases show up. Learn how to see things differently with property tests written in PropEr.

Start with the basics of property tests, such as writing stateless properties, and using the default generators to generate test cases automatically. More importantly, learn how to think in properties. Improve your properties, write custom data generators, and discover what your code can or cannot do. Learn when to use property tests and when to stick with example tests with real-world sample projects. Explore various testing approaches to find the one that’s best for your code. Shrink failing test cases to their simpler expression to highlight exactly what breaks in your code, and generate highly relevant data through targeted properties. Uncover the trickiest bugs you can think of with nearly no code at all with two special types of properties based on state transitions and finite state machines.

Write Erlang and Elixir properties that generate the most effective tests you’ll see, whether they are unit tests or complex integration and system tests.

Top Five Property-Based Testing Tips

1. Sit back and think. The hardest part of writing property-based tests is coming up with properties, and that’s usually because we’re not quite sure of what the program should do in the first place. It’s fine to leave your IDE or text editor, and just think things over again a few times, draw a bit on paper, and take a step back. If you draw a small diagram of what a solution looks like, or write up an explanation to someone who knows nothing about your program, you may just find properties that way.

2. Start broad, and build up. You don’t need to be specific at first. Start with general data types and properties. It can be as simple as “this type of input always results in this type of output”. Then add some complexity: maybe you need more precise input for some scenario, and that will guide you to new fancier data generators and properties. This approach essentially helps get over blank page syndrome with your tests. Attach the exercise to something discrete and you might soon get going.

3. Assemble multiple well-defined properties rather than writing a more complex one. This is especially true if you take one of the previous iterative approaches. It might be tempting to add more and more checks and rules to a single property test, but by doing so you will make debugging harder, and exploration trickier. By using various well-targeted properties, you increase the chances of keeping everything much more maintainable. Add one or two general properties to unite them all together and you’ll have a solid test suite in no time.

4. Double-check the quality of properties, either through metrics, or by adding random bugs in your code and making sure your properties find them. Properties can often feel very trustworthy; metrics let you know if the generator distribution makes sense, and adding random bugs is just a good general way to let you know whether there are areas of your code that are not validated very well. If you can see a good distribution and your properties appear to catch most bugs you try to willingly add to code, chances are better they’ll find all the unexpected ones as well.

5. Think like an operator debugging an incident. How would you detect a bug in something? How would you search for it in any output? What data would you need to confirm there’s a problem? This is an iterative process. The patterns you’d look for in data to confirm a bug let you figure out your properties. For example, you might find that a backpressure mechanism does not work well if a request takes too long without being cancelled: this can be turned into a property where you check for the quality of response delays. The data you need to figure this out, on the other hand, may point to changes you must make to your system to make it observable. In the same example, you may need to expose metrics or logs regarding response times that the operator would use; then, make your tests use that same data. The testability of the system becomes connected to its operability!

Q & A - Interview questions:

Q: Why write a book on Property-Based Testing?

A: I think it’s a rather amazing technology. Property-based testing is the kind of stuff that looks like wizardry if you have never seen it before: developers writing about 50 lines of tests and then the test framework can find a few dozen bugs that no one would have ever seen coming. That’s how I perceived it at first many years ago too.

Then you decide to play around with it and figure out “wow, this is much harder than I thought!” The frameworks are often documented in a way where people who know the frameworks find the documentation useful, but people who are just starting have a real hard time figuring things out. It took me a real long time to get comfortable just learning on my own, and asking around for help. It’s a bit like going from object-oriented programming to thinking in a functional manner, but for tests. You really have to change how you approach problems and their solutions.

After a bit of practice, I started to feel really confident about property-based testing, and I wanted to use more of it, both in open source projects and at work. The problem, though, is that if my own ramp up took considerable amounts of time. It would not be a good idea to impose this on colleagues and coworkers who are not familiar with it: the bus factor would be uncomfortable.

So I wrote this book in order to get the word out. I believe in property-based testing so much I want more people to know about it, and I found it hard enough to learn that I want them to have a good time learning it. Hopefully, people will enjoy learning property-based testing and will become productive really fast with it.

Q: Why Both Erlang and Elixir?

A: The Erlang and Elixir communities possibly suffer from a kind of narcissism of minor differences; a kind of hostility exists based on small differences between the two languages and how programmers do things, despite Elixir and Erlang being so much closer to each other than any other languages or platforms out there.

This book represents a conscious effort to bridge the gap between the two communities and see both groups join strengths rather than compete with each other; it is one small part, attempting to use one property-based testing tool, with one resource, to improve the code and tests of one community. There’s a significant challenge to writing a book that caters equally to two languages at once, but I think readers will be pleasantly surprised by how readable it is — proving how close both languages are at the same time.

Q: What can Property-Based testing do that other tests frameworks can’t?

A: The biggest difference is that other test frameworks mostly ask you to list a bunch of example cases of how your software works, and then they make sure that they always remain the same. If you change an interface or some arguments, you need to change them in all examples for each test. Property-based testing, on the other hand, requires you to specify what your software should do in more general terms, and from these descriptions, it generates example test cases for you.

The consequence of this is that Property-Based testing is really good at finding “failures of imagination,” cases where it did exactly what you asked of it, but in a manner you did not see coming, which results in a bug. One example I’ve had was one where I encoded some data, wrote it to disk, and then parsed it back. There’s dozens of ways to write example tests for that, but the short property I wrote for it managed to generate sessions that corrupted the content. Specific Unicode byte sequences had distinct encoding and decoding rules that the various libraries I used would handle properly individually, but not together.

I can’t imagine easily finding such bugs—because I wouldn’t even think of them as possible—without using property-based testing. This leads to a specific behavior you’ll see in people who know property-based testing: they may not use it everywhere, but they use it in every critical component they have that absolutely has to work.

Q. What kind of problems can I use property-based testing for?

A: Property-Based testing can be used from anything as simple as unit tests up to very broad system tests. For unit tests, there are stateless properties that mostly validate functions through their inputs and outputs. For system and integration tests, you can instead use stateful properties, which let you define ways to generate sequences of calls and interactions with the system, the same way a human tester doing exploratory testing would.

Stateless properties are easiest to use when you can think of rules or principles your code should always respect, and there is some amount of complexity to the implementation. It might be a bit of a humblebrag, but stateless properties are least interesting when the code you want to test is trivial.

Stateful properties work well in general. The two harder things with them is handling the initial setup and teardown of more complex stateful systems, and dealing with hard-to-predict behavior of code, such as handling timeouts or non-deterministic results. They’re possible to deal with, but rather trickier. Anything else can be fair play.

Q: Should I expect property-based testing to replace other tests?

A: Yes and no. The big difference is that when you write example-based tests, you have to write examples that exercise as much of the program as possible: boundary conditions, tricky sequences of operations, and so on. At some point you go “okay, these are convoluted enough, I’m done,” but it’s still possible you have only validated a small part of the possible program space. When you write property-based tests, you instead try to encode the general rules that your program should obey, and let the framework generate examples for you.

On the other hand, some problems are not straightforward when it comes to having very precise properties, and the pragmatic approach is often to write properties to cover a lot of cases, and rely on a few specific examples as a kind of anchor to make sure the properties are well-grounded.

In a nutshell, real world projects tend to use properties for particularly tricky components; they never fully replace standard tests. Standard tests mixed in with properties let you be lazy and still reap massive benefits: you test with a lot more depth, and yet you have to write and maintain far fewer example tests.

• Errata, typos, suggestions

• Source Code (zip file)

Releases:

• P1.0 2019/01/18
• B5.0 2019/01/04
• B4.0 2018/11/01
• B3.0 2018/09/10

Introduction

• The Basics
  • Foundations of Property-Based Testing excerpt
    • Promises of Property-Based Testing
    • Properties First
    • Property-Based Testing in Your Project
    • Running a Property
    • Wrapping Up
  • Writing Properties
    • Structure of Properties
    • Default Generators
    • Putting It All Together
    • Wrapping Up
  • Thinking in Properties excerpt
    • Modeling
    • Generalizing Example Tests
    • Invariants
    • Symmetric Properties
    • Putting It All Together
    • Wrapping Up
  • Custom Generators
    • The Limitations of Default Generators
    • Gathering Statistics
    • Basic Custom Generators
    • Fancy Custom Generators
    • Wrapping Up
• Stateless Properties in Practice
  • Responsible Testing
    • The Specification
    • Thinking About Program Structure
    • CSV Parsing
    • Filtering Records
    • Employee Module
    • Templating
    • Plumbing It All Together
    • Wrapping Up
  • Properties-Driven Development
    • The Specification
    • Writing the First Test
    • Testing Specials
    • Implementing Specials
    • Negative Testing
    • Wrapping Up
  • Shrinking
    • Re-centering with ?SHRINK
    • Dividing with ?LETSHRINK
    • Wrapping Up
  • Targeted Properties
    • Understanding Targeted Properties
    • Targeted Properties in Practice
    • Thinking Outside the Box
    • Wrapping Up
• Stateful Properties
  • Stateful Properties
    • Laying Out Stateful Properties
    • How Stateful Properties Run
    • Writing Properties
    • Testing a Basic Concurrent Cache
    • Testing Parallel Executions
    • Wrapping Up
  • Case Study: Bookstore
    • Introducing the Application
    • Writing Generators
    • Broad Stateful Testing
    • Precise Stateful Modeling
    • Refining the Tests
    • Debugging Stateful Properties
    • Parallel Tests
    • Wrapping Up
  • State Machine Properties
    • Laying Out State Machine Properties
    • How State Machine Properties Run
    • Writing Properties
    • Testing a Circuit Breaker
    • Modeling the Circuit Breaker
    • Adjusting the Model
    • Wrapping Up