Switchback experimentation

Published: Oct 28, 2020
Oct 2020

Controlled experiments using A/B testing is a great way to inform decisions around product development. But it doesn't work well when we can't establish independence between the two groups involved in the A/B test — i.e., adding someone to the "A" group impacts the "B" group and vice versa. One technique to address this problem space is Switchback experimentation. The core concept here is we switch back and forth between the "A" and "B" modes of the experiment in a certain region at alternating time periods instead of both running during the same time period. We then compare the customer experience and other key metrics between the two time buckets. We've tried this to good effect in some of our projects — it's a good tool to have in our experiments toolbelt.