🍻 Join us for happy hour in your sweatpants! In this episode, an interview with John Meakin, Data Scientist for the Facebook Experimentation Platform, formerly Head of Experimentation at Vrbo! He has over 10 years of experience with experimentation at companies that are some of the best in tech. We discuss his advice on how to win at experimentation as a product manager.
📣 Subscribe at https://www.producthappyhour.com/ for the latest episode updates! 🙏 If you haven't become a paid subscriber, now's the time to support the show! It’s either $5/month or $30/year, that’s one Starbucks latte a month or one Doordash order a year!
🎥 Check out the video on YouTube:
👨🔬 John’s Bio
John Meakin is Staff Data Scientist at Meta. John has worked with top tech companies Expedia and Meta specifically on the art of A/B testing.
He began his career as a researcher at the American Institutes for Research, managing experiment design on research projects there. From AIR, he joined Vrbo, part of Expedia Group, as a statistician focused on experimentation, which is where we met and embarked on many experimentation adventures! John eventually became Head of Experimentation at Vrbo prior to joining Meta in San Francisco. At Meta, John continues to be focused on all things experimentation in his role on Meta’s experimentation platform.
✍🏾 Show Notes
00:00 Intro
01:30 Welcome John!
01:46 John's Bio
04:10 What's That Drink?!
05:14 Hard Truths about A/B Testing
09:39 What to Watch Out For When Experimenting
17:34 Exposure, SRM Level-Setting, How to Do It
24:15 Choosing an Evaluation Metric
26:14 Binarized vs. Non-Binarized Metrics
29:25 How Product Managers Prepare for A/B Testing
34:07 Being a Skeptic 38:24 Issues with A/B Testing Evaluation
46:35 Process for Product Managers When Engaging in A/B Testing
51:29 Clarity on Experimentation Policy Required
53:50 Parting Thoughts
58:44 Outro
Follow us for more, comment and let us know what else you'd like to learn or see:
You can also contact us here, leave us ideas, questions or feedback: producthappyhr@gmail.com
Episode 16: John Meakin! 🧪 How to Win at Experimentation as a Product Manager