Next-Level A/B Testing

Repeatable, Rapid, and Flexible Product Experimentation

by Leemay Nassery

The better tools you have in your experimentation toolkit, the better off teams will be shipping and evaluating new features on a product. Learn how to create robust A/B testing strategies that evolve with your product and engineering needs. See how to run experiments quickly, efficiently, and at less cost with the overarching goal of improving your product experience and your company’s bottom line.

The long-term success of any product hinges on a company’s ability to experiment quickly and effectively. The more a company evolves and grows, the more demand there is on the experimentation platform. To continue to meet testing demands and empower teams to leverage A/B testing in their product development life cycle, it’s vital to incorporate techniques to improve testing velocity, cost, and quality.

Learn how to create an A/B testing environment for the long term that lets you quickly construct, run, and analyze tests and enables the business to explore and exploit new features in a cost-effective and controlled way. Know when to use techniques—stratified random sampling, interleaving, and metric sensitivity analysis—that let you work faster, more accurately, and more cost-effectively. With practical strategies and hands-on engineering tasks oriented around improving the rate and quality of testing on a product, you can apply what you’ve learned to optimize your experimentation practices.

A/B testing is vital to product development. It’s time to create the tools and environment that let you run these tests easily, affordably, and reliably.

• Errata, typos, suggestions

Releases:

• B5.0 2025/04/28
• B4.0 2025/02/11
• B3.0 2024/10/18
• B2.0 2024/10/04

Note: Contents and extracts of beta books will change as the book is developed.

• Introduction
  • Who should read this book
  • How this book is organized
  • Taking your experimentation to the next level
• Why Experimentation Rate, Quality, and Cost Matters
  • Advancing Your Experimentation Practices
  • Increasing Experimentation Rate
  • Improving Experimentation Quality
  • Decreasing Experimentation Cost
  • Chapter Roundup: Running an Experimentation Workshop
  • Wrapping Up
• Running Experiments More Effectively
  • Reasoning with Limited Testing Availability
  • Varying Testing Strategies excerpt
  • Shifting Experimentation Mindset
  • Illustrating Interaction Effects
  • Defining General Guidelines to Increase Testing Space
  • Chapter Roundup: What Type of Testing Strategy Best Suits Your Use Cases?
  • Wrapping Up
• Designing Better Experiments
  • Improving Experiment Design
  • Opting for Sensitive Metrics
  • Aligning on Experiment Goal
  • Reducing the Number of Variants
  • Increasing Power to Detect Small Changes with CUPED
  • Sharing Experimentation Best Practices
  • Chapter Roundup: Identifying Experiment Design Improvements
  • Wrapping Up
• Improving Machine Learning Evaluation Practices
  • Identifying Challenges with Machine Learning
  • Measuring Effect with Offline Methods
  • Increasing Reward with Multi-Armed Bandits
  • Comparing Multiple Rankers with Interleaving
  • Chapter Roundup: When to Implement New Strategies for Machine Learning Evaluations
  • Wrapping Up
• Verifying and Monitoring Experiments
  • Tracking Metrics to Measure Experimentation Strategy Effectiveness
  • Verifying Experiments Before Launch
  • Conducting Health Checks with A/A Tests
  • Recognizing the Spillover Effect
  • Structuring Experimentation Process
  • Chapter Roundup: Checklist for Creating an Experimentation Quality Roadmap
  • Wrapping Up
• Ensuring Trustworthy Insights
  • Why Insights Quality Matters
  • Comparing Effect with Meta-Analysis
  • Considering Metric Sensitivity in Relation to Quality Insights
  • Increasing Precision with Stratified Random Sampling
  • Measuring Outcomes with Covariate Adjustments
  • Navigating False Positive Risk
  • Doubling Down On Statistical Power
  • Chapter Roundup: Verifying You’re Measuring True Effect
  • Wrapping Up
• Practicing Adaptive Testing Strategies
  • What is Adaptive Testing?
  • Making Decisions Early with Sequential Testing
  • Making Multi-Armed Bandits Effective for You
  • Opting for Thompson Sampling Algorithm
  • Personalizing the Decision with Contextual Bandits
  • Generalizing Components to Support Adaptive Testing
  • Chapter Roundup: Engineering Team Requirements To Support Adaptive Testing
  • Wrapping Up
• Addressing the Cost of Long-term Holdbacks
  • Defining Long-Term Holdbacks
  • Illustrating Benefits of Holdbacks
  • Identifying Common Pitfalls of Holdbacks
  • Leveraging Post Period Analysis
  • Measuring Impact with Continuous Monitoring
  • Wrapping Up
• Making Experimentation Trade-offs
  • Managing Experimentation Tradeoffs
  • Considering Your Platforms Robustness
  • Comparing Experimentation Cost Versus Quality
  • Compromising Experimentation Quality for Rate
  • Building Your Experimentation Platform Roadmap
  • Wrapping Up