A/B Testing Metrics: What You Need to Know About Success, Driver, and Guardrail Metrics!

In this video, we will take a deep dive into the fundamentals of metrics in A/B Testing. By the end of the video, you will have a solid understanding of the common types of metrics and tips for applying them in real-world A/B testing applications. Some ideas are inspired by the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. I'm sure this video will be extremely helpful for both interview preparation and day-to-day work as a DS.
Blog post by Rob and me:
towardsdatascience.com/the-ul...
Part 1 of the tutorial:
• A/B Testing Fundamenta...
Part 2 of the tutorial:
• A/B Testing Made Easy:...
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001
====================
Contents of this video:
====================:
0:00 Intro
1:13 Goal Metric
3:13 Driver Metric
4:42 Guardrail Metric
6:14 Application of Metrics
8:17 Metrics for experiments

Пікірлер: 32

  • @emma_ding
    @emma_ding3 жыл бұрын

    Typo correction: Thanks Squiderify, at 10:38 "metics" should be "metrics".

  • @jackyzhang6848

    @jackyzhang6848

    3 жыл бұрын

    Hello Emma, thanks for your sharing, may I ask for a question, for weighted combination of, say 3 metric, how do you decide the weight among 3 metrics? For example, the engagement of FB newsfeed, the number of likes, comments and shares, there are 3 main metrics to measuring the FB newsfeed engagement, compare to like, share and comment are more meaningful to measure the interaction on FB, but how would you decide the weight among those 3 metrics in a scientific way? Really thanks for your answer.

  • @SM-gf7ux
    @SM-gf7ux10 ай бұрын

    Emma - watching your videos helped me land a data scientist position at a FAANG company. So grateful for your knowledge and your ability to share it with a wide audience :)

  • @tungthanhle6797
    @tungthanhle67973 жыл бұрын

    Thanks, Emma for your contributions to our DS community! I have been enjoying your content so much so I even clicked on the like button before I watch any of your new videos :D. Thank you very much!

  • @emma_ding

    @emma_ding

    3 жыл бұрын

    Glad my videos are helpful!

  • @TrinaMoitra
    @TrinaMoitra4 ай бұрын

    We actually use a 3rd category of guardrail metrics. Program metrics. To ensure overall quality of the experimentation program. One of the guardrails we have is ratio of iterative to exploratory tests.

  • @where_is_georgio
    @where_is_georgio3 жыл бұрын

    You explain things so well, very helpful for those of us breaking into the data science world. Thank you Emma

  • @omkarkhedkar1885
    @omkarkhedkar18853 жыл бұрын

    Really intuitive videos, Thanks! Keep up with the good content!!

  • @yashnikhare2210
    @yashnikhare2210 Жыл бұрын

    Amazing content and often asked in technical interview. Keep it up Emma 🙌

  • @tianyifang6601
    @tianyifang66012 жыл бұрын

    so useful and helpful!!! I learned a lot from your video! thanks so much

  • @saeidsamizade6257
    @saeidsamizade62573 жыл бұрын

    This channel is gonna be a great technical resource for professionals. Thanks 👍

  • @aymanz83
    @aymanz832 жыл бұрын

    Great source of information, and clear explanation for tough topics

  • @qingchen7835
    @qingchen78352 жыл бұрын

    给Emma点赞!!最近面试Product Analytics 多亏看了你的视频!!讲的又好又清晰!! 因为自己没有Product经验,所以面试还挺费劲儿哈哈~ 希望自己可以早日上岸!再次谢谢Emma的视频

  • @duchen1844
    @duchen18443 жыл бұрын

    这节说的真好,微软AnE说的也不过如此。口才好,写书的Rony若是听了也要点赞。

  • @robinhood3203
    @robinhood3203 Жыл бұрын

    good explanation, please keep up the good work!

  • @DataKnight
    @DataKnight3 жыл бұрын

    Thank you for sharing

  • @saishastech8023
    @saishastech80233 жыл бұрын

    Good explanation of metrics 😊 thanks for sharing 👌 keep going 👍

  • @DataKnight
    @DataKnight3 жыл бұрын

    Thanks for sharing

  • @doreenshen3690
    @doreenshen36902 жыл бұрын

    Hi Emma, thanks for the super clear and structured content! I am still a bit confused about what an "attributable" metric means. Could you add more clarity here? Thanks!

  • @tinawang1291
    @tinawang1291 Жыл бұрын

    6:13 for the sample ratio mismatch check, I think we should only apply Chi-square test, which is to compare a proportion with an expected value. Can you help clarify how to use a T-test for checking SRM? Thanks

  • @technicallyeasy
    @technicallyeasy2 жыл бұрын

    i love your channel! you should create amazon referral links for the book you mentioned in your video. that way you can get referral revenue when your followers buy that book based on your recommendation. i want to read that book now

  • @rubaldahat4813
    @rubaldahat48133 жыл бұрын

    Can u shows its implementation so that it easy to remember.

  • @sparklingamber792
    @sparklingamber7923 жыл бұрын

    Hi Emma, for the sample ratio mismatch, shouldn't we use z-test rather than t-test? I read that the variance p(1-p) and mean p are not independent for proportion comparison, so it will follow a z-distribution.

  • @ranpan2861
    @ranpan28612 жыл бұрын

    Hi Emma, could you dive deeper into how to answer the question “If one metric goes up, and the other goes down, what should we do?”

  • @vaibhavsingh39
    @vaibhavsingh397 ай бұрын

    Best videos

  • @yuzhouzhang9847
    @yuzhouzhang98473 жыл бұрын

    哈哈哈,你的内容真好,请问可以在Github或者medium上同步吗,给点文件啥的

  • @asyvli1226
    @asyvli12262 жыл бұрын

    hi Emma, thanks for your sharing, could you please share more detail about “If one metric goes up, and the other goes down, what should we do?” thanks

  • @emma_ding

    @emma_ding

    2 жыл бұрын

    Fantastic observation! When metrics go a bit wild we can experience strange outcomes for all kinds of reasons worth talking about. This has given me a great idea for more content about how to avoid analysis paralysis! Thanks for sharing your comment with me!

  • @Squiderify
    @Squiderify3 жыл бұрын

    At 10:38 there is a typo: "metics" instead of "metrics". Amazing content though!

  • @emma_ding

    @emma_ding

    3 жыл бұрын

    Thanks for spotting the typo!

  • @awang3139
    @awang31393 жыл бұрын

    数科秘籍Emma Ding,interview rat 大救星

Келесі