About This Title

Pages: 200
Published: March 2025
ISBN: 9798888651308
In Beta

Skill Level Meter

Next-Level A/B Testing

Repeatable, Rapid, and Flexible Product Experimentation

by Leemay Nassery

The better tools you have in your experimentation toolkit, the better off teams will be shipping and evaluating new features on a product. Learn how to create robust A/B testing strategies that evolve with your product and engineering needs. See how to run experiments quickly, efficiently, and at less cost with the overarching goal of improving your product experience and your company’s bottom line.

eBook Formats:

  • PDF for desktop/tablets

  • epub for Apple Books, e-readers

  • mobi for Kindle readers

Get all eBook formats here for $32.95 (USD)

Add to Cart we accept visa, mastercard, amex, discover, paypal

This book is in Beta, final version expected Mar 2025

Beta Books: What do I get?


The long-term success of any product hinges on a company’s ability to experiment quickly and effectively. The more a company evolves and grows, the more demand there is on the experimentation platform. To continue to meet testing demands and empower teams to leverage A/B testing in their product development life cycle, it’s vital to incorporate techniques to improve testing velocity, cost, and quality.

Learn how to create an A/B testing environment for the long term that lets you quickly construct, run, and analyze tests and enables the business to explore and exploit new features in a cost-effective and controlled way. Know when to use techniques—stratified random sampling, interleaving, and metric sensitivity analysis—that let you work faster, more accurately, and more cost-effectively. With practical strategies and hands-on engineering tasks oriented around improving the rate and quality of testing on a product, you can apply what you’ve learned to optimize your experimentation practices.

A/B testing is vital to product development. It’s time to create the tools and environment that let you run these tests easily, affordably, and reliably.

What You Need

N/A

Resources

Releases:

  • B2.0 2024/10/04
  • B1.0 2024/09/24

Contents & Extracts

Note: Contents and extracts of beta books will change as the book is developed.

  • Introduction
    • Who should read this book
    • How this book is organized
    • Taking your experimentation to the next level
  • Why Experimentation Rate, Quality, and Cost Matters
    • Advancing Your Experimentation Practices
    • Increasing Experimentation Rate
    • Improving Experimentation Quality
    • Decreasing Experimentation Cost
    • Chapter Roundup: Running an Experimentation Workshop
    • Wrapping Up
  • Running Experiments More Effectively
    • Reasoning with Limited Testing Availability
    • Varying Testing Strategies excerpt
    • Shifting Experimentation Mindset
    • Illustrating Interaction Effects
    • Defining General Guidelines to Increase Testing Space
    • Chapter Roundup: What Type of Testing Strategy Best Suits Your Use Cases?
    • Wrapping Up
  • Designing Better Experiments
    • Improving Experiment Design
    • Opting for Sensitive Metrics
    • Aligning on Experiment Goal
    • Reducing the Number of Variants
    • Increasing Power to Detect Small Changes with CUPED
    • Sharing Experimentation Best Practices
    • Chapter Roundup: Identifying Experiment Design Improvements
    • Wrapping Up
  • Improving Machine Learning Evaluation Practices
    • Identifying Challenges with Machine Learning
    • Measuring Effect with Offline Methods
    • Increasing Reward with Multi-Armed Bandits
    • Comparing Multiple Rankers with Interleaving
    • Chapter Roundup: When to Implement New Strategies for Machine Learning Evaluations
    • Wrapping Up
  • Verifying and Monitoring Experiments
  • Ensuring Trustworthy Insights
    • Why Insights Quality Matters
    • Comparing Effect with Meta-Analysis
    • Considering Metric Sensitivity in Relation to Quality Insights
    • Increasing Precision with Stratified Random Sampling
    • Measuring Outcomes with Covariate Adjustments
    • Navigating False Positive Risk
    • Doubling Down On Statistical Power
    • Chapter Roundup: Verifying You’re Measuring True Effect
    • Wrapping Up
  • Practicing Adaptive Testing Strategies
    • What is Adaptive Testing?
    • Making Decisions Early with Sequential Testing
    • Making Multi-Armed Bandits Effective for You
    • Opting for Thompson Sampling Algorithm
    • Personalizing the Decision with Contextual Bandits
    • Generalizing Components to Support Adaptive Testing
    • Chapter Roundup: Engineering Team Requirements To Support Adaptive Testing
    • Wrapping Up
  • Addressing the Cost of Long-term Holdbacks
    • Defining Long-Term Holdbacks
    • Illustrating Benefits of Holdbacks
    • Identifying Common Pitfalls of Holdbacks
    • Leveraging Post Period Analysis
    • Measuring Impact with Continuous Monitoring
    • Wrapping Up
  • Making Experimentation Trade-offs
    • Managing Experimentation Tradeoffs
    • Considering Your Platforms Robustness
    • Comparing Experimentation Cost Versus Quality
    • Compromising Experimentation Quality for Rate
    • Building Your Experimentation Platform Roadmap
    • Wrapping Up

Author

Leemay Nassery is an engineering leader specializing in experimentation and personalization. With a notable track record that includes evolving Spotify’s A/B testing strategy for the Homepage, launching Comcast’s For You page, and establishing data warehousing teams at Etsy, she firmly believes that the key to innovation at any company is the ability to experiment effectively.

eBook Formats:

  • PDF for desktop/tablets

  • epub for Apple Books, e-readers

  • mobi for Kindle readers

Get all eBook formats here for $32.95 (USD)

Add to Cart we accept visa, mastercard, amex, discover, paypal

This book is in Beta, final version expected Mar 2025

Beta Books: What do I get?

Related Titles:

Skill Level Meter

About This Title

Pages: 200
Published: March 2025
ISBN: 9798888651308
Edition: 1
In Beta