small medium large xlarge

Property-Based Testing with PropEr, Erlang, and Elixir: Find Bugs Before Your Users Do

by

Cover image for Property-Based Testing with PropEr, Erlang, and Elixir

Property-Based Testing with PropEr, Erlang, and Elixir

Find Bugs Before Your Users Do

by

Property-based testing helps you create better, more solid tests with little code. By using the PropEr framework in both Erlang and Elixir, this book teaches you how to automatically generate test cases, test stateful programs, and change how you design your software for more principled and reliable approaches. You will be able to better explore the problem space, validate the assumptions you make when coming up with program behavior, and expose unexpected weaknesses in your design. PropEr will even show you how to reproduce the bugs it found. With this book, you will be writing efficient property-based tests in no time.

Customer Reviews

Property-based testing through PropEr and QuickCheck is one of the most powerful, yet underutilized, testing approaches of the Erlang Ecosystem. Fred, through his book, has reduced the barrier to entry, making property-based testing accessible to everyone. This book is unarguably a must-have if you are serious about Erlang and Elixir. A long time coming!

- Francesco Cesarini

Director, Erlang Solutions Ltd.

In his inimitable and engaging way, Fred lays out a strong case for why property testing is a revolutionary idea in software testing and, throughout the book, masterfully teaches readers how to become proficient in property testing. As we move toward property testing at Bleacher Report, Property-Based Testing with PropEr, Erlang, and Elixir is our indispensable guide. For anyone considering property testing, it should be yours, too.

- Ben Marx

Software Architect, Bleacher Report

Fred Hebert has crafted a text with breadth and depth in his inimitable approachable style. Property-Based Testing with PropEr, Erlang, and Elixir thoroughly demystifies property testing with detailed examples and practical tips in a way that is useful to beginners and experts alike.

- Sean Cribbs

Staff Software Engineer, Postmates, Inc.

Fred breaks these topics down in understandable ways and provides a lot of insight into how to think about property-based testing. There’s a real emphasis on ensuring that your tests are working to find the hidden bugs and not just generating random data.

- Chris Keathley

Senior Engineer, Bleacher Report

This book is an excellent resource if you want to introduce property-based testing to your Erlang or Elixir project. It will guide you from the basics to the challenges of applying property-based testing in practice, giving invaluable advice on how to get your tests correct and efficient.

If you want to get started with property testing—you need look no further!

- Dr. Annette Bieniusa

Senior Researcher and Lecturer,, University of Kaiserslautern

See All Reviews

Choose Your Format(s)

  • $47.95 In Stock

    Save $14.95 on the combo pack.

  • $37.95 In Stock
  • $24.95 In Stock
  • Ebooks are DRM free.

  • Ebook delivery options.

About this Title

Skill-meter-4-8
Pages: 374
Published: 2019-01-10
Release: P1.0 (2019-01-18)
ISBN: 978-1-68050-621-1

Most tests only demonstrate that the code behaves how the developer expected it to behave, and therefore carry the same blind spots as their authors when special conditions or edge cases show up. Learn how to see things differently with property tests written in PropEr.

Start with the basics of property tests, such as writing stateless properties, and using the default generators to generate test cases automatically. More importantly, learn how to think in properties. Improve your properties, write custom data generators, and discover what your code can or cannot do. Learn when to use property tests and when to stick with example tests with real-world sample projects. Explore various testing approaches to find the one that’s best for your code. Shrink failing test cases to their simpler expression to highlight exactly what breaks in your code, and generate highly relevant data through targeted properties. Uncover the trickiest bugs you can think of with nearly no code at all with two special types of properties based on state transitions and finite state machines.

Write Erlang and Elixir properties that generate the most effective tests you’ll see, whether they are unit tests or complex integration and system tests.


Top Five Property-Based Testing Tips

1. Sit back and think. The hardest part of writing property-based tests is coming up with properties, and that’s usually because we’re not quite sure of what the program should do in the first place. It’s fine to leave your IDE or text editor, and just think things over again a few times, draw a bit on paper, and take a step back. If you draw a small diagram of what a solution looks like, or write up an explanation to someone who knows nothing about your program, you may just find properties that way.

2. Start broad, and build up. You don’t need to be specific at first. Start with general data types and properties. It can be as simple as “this type of input always results in this type of output”. Then add some complexity: maybe you need more precise input for some scenario, and that will guide you to new fancier data generators and properties. This approach essentially helps get over blank page syndrome with your tests. Attach the exercise to something discrete and you might soon get going.

3. Assemble multiple well-defined properties rather than writing a more complex one. This is especially true if you take one of the previous iterative approaches. It might be tempting to add more and more checks and rules to a single property test, but by doing so you will make debugging harder, and exploration trickier. By using various well-targeted properties, you increase the chances of keeping everything much more maintainable. Add one or two general properties to unite them all together and you’ll have a solid test suite in no time.

4. Double-check the quality of properties, either through metrics, or by adding random bugs in your code and making sure your properties find them. Properties can often feel very trustworthy; metrics let you know if the generator distribution makes sense, and adding random bugs is just a good general way to let you know whether there are areas of your code that are not validated very well. If you can see a good distribution and your properties appear to catch most bugs you try to willingly add to code, chances are better they’ll find all the unexpected ones as well.

5. Think like an operator debugging an incident. How would you detect a bug in something? How would you search for it in any output? What data would you need to confirm there’s a problem? This is an iterative process. The patterns you’d look for in data to confirm a bug let you figure out your properties. For example, you might find that a backpressure mechanism does not work well if a request takes too long without being cancelled: this can be turned into a property where you check for the quality of response delays. The data you need to figure this out, on the other hand, may point to changes you must make to your system to make it observable. In the same example, you may need to expose metrics or logs regarding response times that the operator would use; then, make your tests use that same data. The testability of the system becomes connected to its operability!

Q & A – Interview questions:

Q: Why write a book on Property-Based Testing?

A: I think it’s a rather amazing technology. Property-based testing is the kind of stuff that looks like wizardry if you have never seen it before: developers writing about 50 lines of tests and then the test framework can find a few dozen bugs that no one would have ever seen coming. That’s how I perceived it at first many years ago too.

Then you decide to play around with it and figure out “wow, this is much harder than I thought!” The frameworks are often documented in a way where people who know the frameworks find the documentation useful, but people who are just starting have a real hard time figuring things out. It took me a real long time to get comfortable just learning on my own, and asking around for help. It’s a bit like going from object-oriented programming to thinking in a functional manner, but for tests. You really have to change how you approach problems and their solutions.

After a bit of practice, I started to feel really confident about property-based testing, and I wanted to use more of it, both in open source projects and at work. The problem, though, is that if my own ramp up took considerable amounts of time. It would not be a good idea to impose this on colleagues and coworkers who are not familiar with it: the bus factor would be uncomfortable.

So I wrote this book in order to get the word out. I believe in property-based testing so much I want more people to know about it, and I found it hard enough to learn that I want them to have a good time learning it. Hopefully, people will enjoy learning property-based testing and will become productive really fast with it.

Q: Why Both Erlang and Elixir?

A: The Erlang and Elixir communities possibly suffer from a kind of narcissism of minor differences; a kind of hostility exists based on small differences between the two languages and how programmers do things, despite Elixir and Erlang being so much closer to each other than any other languages or platforms out there.

This book represents a conscious effort to bridge the gap between the two communities and see both groups join strengths rather than compete with each other; it is one small part, attempting to use one property-based testing tool, with one resource, to improve the code and tests of one community. There’s a significant challenge to writing a book that caters equally to two languages at once, but I think readers will be pleasantly surprised by how readable it is—proving how close both languages are at the same time.

Q: What can Property-Based testing do that other tests frameworks can’t?

A: The biggest difference is that other test frameworks mostly ask you to list a bunch of example cases of how your software works, and then they make sure that they always remain the same. If you change an interface or some arguments, you need to change them in all examples for each test. Property-based testing, on the other hand, requires you to specify what your software should do in more general terms, and from these descriptions, it generates example test cases for you.

The consequence of this is that Property-Based testing is really good at finding “failures of imagination,” cases where it did exactly what you asked of it, but in a manner you did not see coming, which results in a bug. One example I’ve had was one where I encoded some data, wrote it to disk, and then parsed it back. There’s dozens of ways to write example tests for that, but the short property I wrote for it managed to generate sessions that corrupted the content. Specific Unicode byte sequences had distinct encoding and decoding rules that the various libraries I used would handle properly individually, but not together.

I can’t imagine easily finding such bugs—because I wouldn’t even think of them as possible—without using property-based testing. This leads to a specific behavior you’ll see in people who know property-based testing: they may not use it everywhere, but they use it in every critical component they have that absolutely has to work.

Q. What kind of problems can I use property-based testing for?

A: Property-Based testing can be used from anything as simple as unit tests up to very broad system tests. For unit tests, there are stateless properties that mostly validate functions through their inputs and outputs. For system and integration tests, you can instead use stateful properties, which let you define ways to generate sequences of calls and interactions with the system, the same way a human tester doing exploratory testing would.

Stateless properties are easiest to use when you can think of rules or principles your code should always respect, and there is some amount of complexity to the implementation. It might be a bit of a humblebrag, but stateless properties are least interesting when the code you want to test is trivial.

Stateful properties work well in general. The two harder things with them is handling the initial setup and teardown of more complex stateful systems, and dealing with hard-to-predict behavior of code, such as handling timeouts or non-deterministic results. They’re possible to deal with, but rather trickier. Anything else can be fair play.

Q: Should I expect property-based testing to replace other tests?

A: Yes and no. The big difference is that when you write example-based tests, you have to write examples that exercise as much of the program as possible: boundary conditions, tricky sequences of operations, and so on. At some point you go “okay, these are convoluted enough, I’m done,” but it’s still possible you have only validated a small part of the possible program space. When you write property-based tests, you instead try to encode the general rules that your program should obey, and let the framework generate examples for you.

On the other hand, some problems are not straightforward when it comes to having very precise properties, and the pragmatic approach is often to write properties to cover a lot of cases, and rely on a few specific examples as a kind of anchor to make sure the properties are well-grounded.

In a nutshell, real world projects tend to use properties for particularly tricky components; they never fully replace standard tests. Standard tests mixed in with properties let you be lazy and still reap massive benefits: you test with a lot more depth, and yet you have to write and maintain far fewer example tests.

What You Need

You will need basic knowledge of Erlang, or optionally Elixir (all Erlang code is translated to Elixir). All Erlang examples have been tested and will run with Erlang/OTP 20 and Rebar3 3.6.1. All Elixir examples have been tested with Elixir 1.5.0 or newer. Any operating system with a network connection will be fine to follow along.

Contents & Extracts

  • Introduction
  • The Basics
    • Foundations of Property-Based Testing excerpt
      • Promises of Property-Based Testing
      • Properties First
      • Property-Based Testing in Your Project
      • Running a Property
      • Wrapping Up
    • Writing Properties
      • Structure of Properties
      • Default Generators
      • Putting It All Together
      • Wrapping Up
    • Thinking in Properties excerpt
      • Modeling
      • Generalizing Example Tests
      • Invariants
      • Symmetric Properties
      • Putting It All Together
      • Wrapping Up
    • Custom Generators
      • The Limitations of Default Generators
      • Gathering Statistics
      • Basic Custom Generators
      • Fancy Custom Generators
      • Wrapping Up
  • Stateless Properties in Practice
    • Responsible Testing
      • The Specification
      • Thinking About Program Structure
      • CSV Parsing
      • Filtering Records
      • Employee Module
      • Templating
      • Plumbing It All Together
      • Wrapping Up
    • Properties-Driven Development
      • The Specification
      • Writing the First Test
      • Testing Specials
      • Implementing Specials
      • Negative Testing
      • Wrapping Up
    • Shrinking
      • Re-centering with ?SHRINK
      • Dividing with ?LETSHRINK
      • Wrapping Up
    • Targeted Properties
      • Understanding Targeted Properties
      • Targeted Properties in Practice
      • Thinking Outside the Box
      • Wrapping Up
  • Stateful Properties
    • Stateful Properties
      • Laying Out Stateful Properties
      • How Stateful Properties Run
      • Writing Properties
      • Testing a Basic Concurrent Cache
      • Testing Parallel Executions
      • Wrapping Up
    • Case Study: Bookstore
      • Introducing the Application
      • Writing Generators
      • Broad Stateful Testing
      • Precise Stateful Modeling
      • Refining the Tests
      • Debugging Stateful Properties
      • Parallel Tests
      • Wrapping Up
    • State Machine Properties
      • Laying Out State Machine Properties
      • How State Machine Properties Run
      • Writing Properties
      • Testing a Circuit Breaker
      • Modeling the Circuit Breaker
      • Adjusting the Model
      • Wrapping Up

Author

Fred Hebert is a systems architect at a video surveillance, access control management, and license plate recognition company. He previously worked as a tech lead on a cloud platform’s networking services. He has also worked in real-time bidding systems, as a professional Erlang trainer and course maintainer, and is one of the maintainers of the principal Erlang build tool (rebar3).

Upcoming Author Events

  • 2019-02-28: Fred Hebert
    Operable Erlang and Elixir to make our systems operator-friendly, we have to understand how our mental models work, what constitutes good automation, and we need to be aware of all the tools the Erlang VM makes available to us to deal with surprises (CodeBEAM San Francisco)