Product Happy Hour
Product Happy Hour
Episode 16: John Meakin! 🧪 How to Win at Experimentation as a Product Manager

Episode 16: John Meakin! 🧪 How to Win at Experimentation as a Product Manager

Learn the best stuff on how to win at experimentation from one of the best in the game.

🍻 Join us for happy hour in your sweatpants! In this episode, an interview with John Meakin, Data Scientist for the Facebook Experimentation Platform, formerly Head of Experimentation at Vrbo! He has over 10 years of experience with experimentation at companies that are some of the best in tech. We discuss his advice on how to win at experimentation as a product manager.

📣 Subscribe at for the latest episode updates! 🙏 If you haven't become a paid subscriber, now's the time to support the show! It’s either $5/month or $30/year, that’s one Starbucks latte a month or one Doordash order a year!

🎥 Check out the video on YouTube:

👨‍🔬 John’s Bio

John Meakin is Staff Data Scientist at Meta. John has worked with top tech companies Expedia and Meta specifically on the art of A/B testing. 

He began his career as a researcher at the American Institutes for Research, managing experiment design on research projects there. From AIR, he joined Vrbo, part of Expedia Group, as a statistician focused on experimentation, which is where we met and embarked on many experimentation adventures! John eventually became Head of Experimentation at Vrbo prior to joining Meta in San Francisco. At Meta, John continues to be focused on all things experimentation in his role on Meta’s experimentation platform.

✍🏾 Show Notes

00:00 Intro

01:30 Welcome John!

01:46 John's Bio

04:10 What's That Drink?!

05:14 Hard Truths about A/B Testing

09:39 What to Watch Out For When Experimenting

17:34 Exposure, SRM Level-Setting, How to Do It

24:15 Choosing an Evaluation Metric

26:14 Binarized vs. Non-Binarized Metrics

29:25 How Product Managers Prepare for A/B Testing

34:07 Being a Skeptic 38:24 Issues with A/B Testing Evaluation

46:35 Process for Product Managers When Engaging in A/B Testing

51:29 Clarity on Experimentation Policy Required

53:50 Parting Thoughts

58:44 Outro

Follow us for more, comment and let us know what else you'd like to learn or see:

You can also contact us here, leave us ideas, questions or feedback:

Product Happy Hour
Product Happy Hour
🧐 New to product management? Considering becoming a PM? Miss happy hour in our new remote/hybrid work world? Join us for Product Happy Hour 🍻🎙, where you can do happy hour in your sweatpants! 🎉 Drink with your favorite product people, Ajay Waghray and Era Johal, while learning more about building great products. Support this podcast at