Practical A/B Testing

Creating Experimentation-Driven Products

by Leemay Nassery

Whether you’re a catalyst for organizational change or have the support you need to create an engineering culture that embraces A/B testing, this book will help you do it right. The step-by-step instructions will demystify the entire process, from constructing an A/B test to breaking down the decision factors to build an engineering platform. When you’re ready to run the A/B test of your dreams, you’ll have the perfect blueprint.

With smart, tactful approaches to orchestrating A/B testing on a product, you’ll quickly discover how to reap all the benefits that A/B testing has to offer—benefits that span your users, your product, and your team. Take the reins today, and be the change you want to see in your engineering and product organizations.

Develop a hypothesis statement that’s backed with metrics that demonstrate if your prediction for the experiment is correct. Build more inclusive products by leveraging audience segmentation strategies and ad-hoc post analysis to better understand the impact of changes on specific user groups. Determine which path is best for your team when deciding whether to go with a third-party A/B test framework or to build the A/B testing platform in-house. And finally, learn how to cultivate an experimentation-friendly culture within your team.

Leverage the A/B testing methodology to demonstrate the impact of changes on a product to your users, your key business metrics, and the way your team works together. After all, if you aren’t measuring the impact of the changes you make, how will you know if you’re truly making improvements?

• Errata, typos, suggestions

Releases:

• P1.0 2023/05/24
• B4.0 2023/04/05
• B3.0 2023/03/22
• B2.0 2023/03/09

• Preface
  • How to Get Started
  • What to Expect
  • OK, So What’s Next?
• Why You Should A/B Test excerpt
  • What Is A/B Testing?
  • Who Should Run A/B Tests
  • Beginning Your Analyst Role
  • Evaluating Your Ideas
  • Assessing Multiple Designs at Once
  • Gaining User Insights
  • Conducting Additional Analysis for Deeper User Insights
  • Building More Inclusive Products
  • Taking Risks, Being Bold
  • Dispelling the Failed Experiment Myth
  • Empowering Your Team
  • Wrapping Up
• Learn the Fundamentals of an A/B Test
  • Creating a Clear Hypothesis
  • Measuring the Impact of an Experiment
  • Establishing Baselines
  • Leveraging Proxy Metrics
  • Defining Test and Control Variants
  • Increasing Confidence in Your Test Results
  • Summarizing the For You A/B Test
  • Wrapping Up
• Select the Right Type of Experiment
  • Illustrating Experimentation in the Medical Industry
  • Defining the Superiority A/B Test
  • Introducing the Non-inferiority A/B Test
  • Opting for an Equivalence A/B Test
  • Validating Longer-Term Impact with Holdbacks excerpt
  • Defining Degradation Holdbacks
  • Utilizing Long-Term Cumulative Holdbacks
  • Understanding the Cost of Long-Term Holdbacks
  • Tying It All Together
  • Wrapping Up
• Interface with Data and Visualizing Results excerpt
  • Working with Different Types of Data
  • Wrangling Your Data
  • Making Data Easily Accessible
  • Creating Insights About Your Data
  • Visualizing A/B Test Results
  • Wrapping Up
• To Build or to Buy an A/B Testing Platform
  • Navigating the Classic Engineering Struggle
  • Illustrating the Components of an A/B Testing Platform
  • Building Versus Buying
  • Extending the Scope of an A/B Testing Platform
  • Wrapping Up
• Cultivate a Test-Friendly Culture
  • Breaking Down the Resistance
  • Shifting Your Team’s Perspective
  • Rolling Out A/B Testing Within Your Organization
  • Growing Demand for A/B Testing
  • Wrapping Up