Regularization Part 3: Elastic Net Regression

Elastic-Net Regression is combines Lasso Regression with Ridge Regression to give you the best of both worlds. It works well when there are lots of useless variables that need to be removed from the equation and it works well when there are lots of useful variables that need to be retained. And it does better than either one when it comes to handling correlated variables. Dang!!!!
NOTE: This StatQuest follows up on the the StatQuest on Ridge Regression...
• Regularization Part 1:...
...and the StatQuest on Lasso Regression....
• Regularization Part 2:...
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
Also, here are some references that helped me put this video together:
The original manuscript on Elastic-Net Regression: web.stanford.edu/~hastie/Pape...
A webpage at North Carolina State University that shows different situations for Ridge, Lasso and Elastic-Net Regression: www4.stat.ncsu.edu/~post/josh...
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
KZread Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
#statquest #regularization

Пікірлер: 330

  • @statquest
    @statquest4 жыл бұрын

    NOTE: In this video, for some reason I used the word "variable" instead of "parameter" in the equations for elastic-net. We are trying to shrink the parameters, not the variables. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

  • @badoiuecristian

    @badoiuecristian

    4 жыл бұрын

    Clarification: Params {slope, intercept}. Variable {weight, hight} - for anyone that got confused

  • @statquest

    @statquest

    4 жыл бұрын

    @@badoiuecristian Exactly.

  • @s25412

    @s25412

    3 жыл бұрын

    3:56 "...when there are correlations between parameters..." this should be between variables instead. Similarly at 4:48 "...job dealing with correlated parameters..."

  • @statquest

    @statquest

    3 жыл бұрын

    @@s25412 Oops! You are correct.

  • @falaksingla6242

    @falaksingla6242

    2 жыл бұрын

    Hi Josh, Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so. Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.

  • @zhizhongzhu9524
    @zhizhongzhu95244 жыл бұрын

    Lasso: Yee-ha! Ridge: Brrr... Elastic Net: ...

  • @statquest

    @statquest

    4 жыл бұрын

    Great Question!!! :) "snap"?

  • @swaggcam412

    @swaggcam412

    4 жыл бұрын

    Lol my favorite part of the video by far

  • @rezahay
    @rezahay5 жыл бұрын

    Wonderful, brilliant, awesome. What a relief! Finally, I understand some important concepts of the statistics. Thank you very much Josh.

  • @becayebalde3820
    @becayebalde3820 Жыл бұрын

    I have now finished the 3 parts, ouf! Thank you a thousand times for the awesome content you provide 👏🏾

  • @statquest

    @statquest

    Жыл бұрын

    BAM! There's one more part here: kzread.info/dash/bejne/iqFmpcGghKTSaMY.html

  • @jbridger9
    @jbridger94 жыл бұрын

    When I first watched one of your videos I was struck by how entertaining it was. But the more videos I watch, the more I notice how well I'm understanding the explanations in your videos.Thanks a lot for your amazing efforts!

  • @statquest

    @statquest

    4 жыл бұрын

    Thank you! :)

  • @tymothylim6550
    @tymothylim65503 жыл бұрын

    Thank you, Josh, for this excellent video on Elastic-Net Regression! It was a great finish to this 3-part series on Regularization!

  • @statquest

    @statquest

    3 жыл бұрын

    Hooray!

  • @amitgoel2810
    @amitgoel28105 жыл бұрын

    Thank you very much Josh for explaining regularization so clearly ! The visuals that you use in your videos makes the learning easy.

  • @statquest

    @statquest

    5 жыл бұрын

    Hooray! Thank you. :)

  • @yannisskoupras4138
    @yannisskoupras41385 жыл бұрын

    Thank you for the amazing videos! Your ability to explain the concepts simply is incomparable..!

  • @statquest

    @statquest

    5 жыл бұрын

    Thank you so much! :)

  • @Paul-tl1du
    @Paul-tl1du5 жыл бұрын

    You have an uncanny way of explaining this material well. Thank you so much for creating these videos!

  • @statquest

    @statquest

    5 жыл бұрын

    Thank you! :)

  • @georgfaustmann852
    @georgfaustmann8525 жыл бұрын

    Thanks for the entertaining and informative channel. Keep up the good work!

  • @rachithrr
    @rachithrr4 жыл бұрын

    I can't say which is better, your albums or this amazing series.

  • @statquest

    @statquest

    4 жыл бұрын

    Thank you very much! :)

  • @jinlee1874
    @jinlee18743 жыл бұрын

    The best explanations I could find online for Stats!!! Thank you Josh!

  • @statquest

    @statquest

    3 жыл бұрын

    Thanks!

  • @pratiknabriya5506
    @pratiknabriya55064 жыл бұрын

    Hey Josh, thanks for the crisp explanation. Today after long procrastination, I managed to watch all three of the videos - L1, L2 and Elastic Net.

  • @statquest

    @statquest

    4 жыл бұрын

    Wow! Good work! :)

  • @rsandeepnttf
    @rsandeepnttf5 жыл бұрын

    Crisp and clear ! thanks for the video

  • @zdeacon
    @zdeacon Жыл бұрын

    Thank you so much Josh!! I was struggling with Lasso, Ridge, and ElasticNet Regression for my graduate class. Your 3 videos cleared up all the confusion. Thank you SO much for all that you do to make these topics accessible for all!

  • @statquest

    @statquest

    Жыл бұрын

    Happy to help!

  • @hanadiam8910
    @hanadiam89102 жыл бұрын

    Another great video!!! Keep it up!! Always big fan

  • @statquest

    @statquest

    2 жыл бұрын

    Triple bam! :)

  • @chitracprabhu2922
    @chitracprabhu29223 жыл бұрын

    You explain the concepts so well ......Thanks a lot for these videos

  • @statquest

    @statquest

    3 жыл бұрын

    Thank you! :)

  • @Abrar_Ahmed05
    @Abrar_Ahmed055 жыл бұрын

    whoa

  • @prodigyprogrammer3187
    @prodigyprogrammer31872 жыл бұрын

    Man that intro is the best it forces me to listen to the rest of the lecture. Thanks :)

  • @statquest

    @statquest

    2 жыл бұрын

    bam! :)

  • @dannychan0510
    @dannychan05103 жыл бұрын

    At first, I came here for the stats revision. Lately, I've been finding myself visiting to remind myself of the tunes instead!

  • @statquest

    @statquest

    3 жыл бұрын

    That's awesome! :)

  • @moeinhasani8718
    @moeinhasani87185 жыл бұрын

    really useful series, keep doing the great tutorials!

  • @statquest

    @statquest

    5 жыл бұрын

    Will do! I'm glad you like the videos. :)

  • @xiangnan-oz9hs
    @xiangnan-oz9hs5 жыл бұрын

    Thanks very much, StatQuest. each lecture is fantastic and interesting. Looking forward to your clearly explanation of Bayesian statistics, MCMC, MH, Gibbs sampling, etc.

  • @statquest

    @statquest

    5 жыл бұрын

    Glad you like the video! All of those topics are on the To-Do list, and hopefully I can get to them sooner than later. :)

  • @angelbueno7810
    @angelbueno78105 жыл бұрын

    Fantastic video Josh !! Thanks a lot, keep up the good work ! :)

  • @statquest

    @statquest

    5 жыл бұрын

    You're welcome! :)

  • @malinkata1984
    @malinkata1984 Жыл бұрын

    Ihaaa!😀 All tutorials are brilliant! A huge thank you.

  • @statquest

    @statquest

    Жыл бұрын

    Thanks! :)

  • @VishalBalaji
    @VishalBalaji4 жыл бұрын

    Wow, Ur channel is a boon to beginners like me in the world of Data Science.....Thanks a lot

  • @redpantherofmadrid
    @redpantherofmadrid7 ай бұрын

    you explain really well, better than the course I am following! thanks 🙏

  • @statquest

    @statquest

    7 ай бұрын

    Thank you!

  • @phamphuminhphuc5859
    @phamphuminhphuc5859 Жыл бұрын

    Thanks Josh its 2022 and your videos saved me well!

  • @statquest

    @statquest

    Жыл бұрын

    Thanks!

  • @_subrata
    @_subrata2 жыл бұрын

    OMG! I'm so happy I found your channel.

  • @statquest

    @statquest

    2 жыл бұрын

    Hooray! :)

  • @cosimocuriale8871
    @cosimocuriale88713 жыл бұрын

    Hi Josh, your lessons are so nice that I decided to support you. I bought your digita album "Made from TV". You rock!

  • @statquest

    @statquest

    3 жыл бұрын

    WOW!!!! Awesome! Thank you!

  • @sophiepan6182
    @sophiepan61822 жыл бұрын

    You are the best! I indeed learned a lot from you! Thanks!

  • @statquest

    @statquest

    2 жыл бұрын

    Bam!

  • @yourboyfazal
    @yourboyfazal2 жыл бұрын

    man i love your way of teaching

  • @statquest

    @statquest

    2 жыл бұрын

    Thank you!

  • @divyyy4358
    @divyyy43584 жыл бұрын

    I love your channel man, its the major reason I'll be majoring in Data science in college!

  • @statquest

    @statquest

    4 жыл бұрын

    Thank you very much and good luck with your degree! :)

  • @sachof
    @sachof5 жыл бұрын

    You are awesome... I gonna buy a t-shirt with "I love StatQuest" written on it !

  • @statquest

    @statquest

    5 жыл бұрын

    Hooray!!! One day I'll have those shirts for sale.... One day.

  • @naveedmazhar5186
    @naveedmazhar51863 жыл бұрын

    Sir! I really liked your style thank you for such entertainment and informative lecture🙏

  • @statquest

    @statquest

    3 жыл бұрын

    Thanks and welcome!

  • @javiermenac
    @javiermenac5 жыл бұрын

    I'm from Chile... i 've loved your videos of regularization, specially each intro!!!

  • @statquest

    @statquest

    5 жыл бұрын

    Hooray!!!! Thank you so much! :)

  • @javiermenac

    @javiermenac

    5 жыл бұрын

    @@statquest do you have any videos about SVM , Neural Net models?

  • @statquest

    @statquest

    5 жыл бұрын

    @@javiermenac Not yet, but I'm working on both. SVM will probably come first, followed by Neural Net.

  • @kkviks
    @kkviks4 жыл бұрын

    I love your channel!

  • @statquest

    @statquest

    4 жыл бұрын

    Thank you so much!

  • @cookiemonster6900
    @cookiemonster69003 жыл бұрын

    Your explanations are on point and easy to understand. (Can be used as quick reference) 🙆🏻👍🏻💯

  • @statquest

    @statquest

    3 жыл бұрын

    Thanks!

  • @resonatingvoice1
    @resonatingvoice15 жыл бұрын

    I mean how easy can it get ... these views are the perfect example of how complex algorithms can be explained in simple and then later people can dive into the actual math behind it to get the full picture... Awesome ... thanks

  • @statquest

    @statquest

    5 жыл бұрын

    That's exactly the idea. I'm so glad you like the videos. :)

  • @resonatingvoice1

    @resonatingvoice1

    5 жыл бұрын

    @@statquest you're welcome and thank you for creating awesome videos..... i really enjoyed the pca ones... as first time i understood the svd in a simple way :-)

  • @user-kb1uw7rd3m
    @user-kb1uw7rd3m5 ай бұрын

    I sometimes come just for the intros! Amazing work!!!

  • @statquest

    @statquest

    5 ай бұрын

    BAM! :)

  • @viralthegenius
    @viralthegenius4 жыл бұрын

    Only reason I subscribed you is because of your singing before every videos! No doubt you explain very well

  • @statquest

    @statquest

    4 жыл бұрын

    Thanks! :)

  • @scutily
    @scutily5 жыл бұрын

    Really nice videos! It's very well explained and helpful! Can you also do videos on adaptive elastic net and multi-step elastic net ? Thank you so much!

  • @yulinliu850
    @yulinliu8505 жыл бұрын

    Great! Thank-you Josh!

  • @statquest

    @statquest

    5 жыл бұрын

    You're welcome! :)

  • @TheParkitny
    @TheParkitny Жыл бұрын

    This clears things up a lot. 4 years on still the best explanation online. Yeeha

  • @statquest

    @statquest

    Жыл бұрын

    BAM!

  • @markaitkin
    @markaitkin5 жыл бұрын

    Legendary as always! 😁🤘🤙👌👍

  • @statquest

    @statquest

    5 жыл бұрын

    Hooray! :)

  • @datascienceandmachinelearn3807
    @datascienceandmachinelearn38075 жыл бұрын

    Hı Thank you so much. I have learned alot and look forward to new videos. Good luck

  • @mariapaulasancineto8871
    @mariapaulasancineto88715 жыл бұрын

    Love your clearly explained videos. And your songs are so sweet like Phoebe Buffay’s 😉

  • @statquest

    @statquest

    5 жыл бұрын

    Ha! Thanks. I sing the smelly song every day as a warm up. ;)

  • @hung717
    @hung7173 жыл бұрын

    Thank you, your video is very useful

  • @statquest

    @statquest

    3 жыл бұрын

    Glad it was helpful!

  • @jonibeki53
    @jonibeki535 жыл бұрын

    So amazing, thank you!

  • @statquest

    @statquest

    5 жыл бұрын

    You're welcome! :)

  • @haneensuradi
    @haneensuradi5 жыл бұрын

    Thanks for this amazing series! It is making my life way easier while I am taking Machine Learning course in university. Can you please 'clearly explain' what do you mean by correlated variables? And what Elastic Net regression does to them?

  • @statquest

    @statquest

    5 жыл бұрын

    An example of correlated variables is if I wanted to use "weight" and "height" measurements to predict something. Since small people tend to weight less than tall people, weight and height are correlated. Elastic-Net Regression would shrink the parameters associated with those variables together.

  • @nguyenthituyetnhung1780
    @nguyenthituyetnhung178010 ай бұрын

    excellent explanation for complexity of model 👍

  • @statquest

    @statquest

    10 ай бұрын

    Thanks!

  • @niklasfelix7126
    @niklasfelix71262 жыл бұрын

    Thank you so much! Once again, your videos are of invaluable help to my PhD dissertation! And the "Brrr" made me laugh out loud :D

  • @statquest

    @statquest

    2 жыл бұрын

    Hooray!

  • @yousufali_28
    @yousufali_285 жыл бұрын

    thanks for making elastic net this easy

  • @statquest

    @statquest

    5 жыл бұрын

    You're welcome!! :)

  • @wenwensi9597
    @wenwensi9597 Жыл бұрын

    I like it a lot when he said the super fancy thing is actually xxx.

  • @statquest

    @statquest

    Жыл бұрын

    :)

  • @ayushbhardwaj4168
    @ayushbhardwaj41685 жыл бұрын

    Awesome man .. Thanks a lot

  • @statquest

    @statquest

    5 жыл бұрын

    Thanks!!! :)

  • @shruti5472
    @shruti54723 жыл бұрын

    In the intro song, I thought you would say "simpler.. than you might expect **it to be**" cause that rhymes. Anyways, love your videos. Thanks for doing such great work.

  • @statquest

    @statquest

    3 жыл бұрын

    Noted

  • @user-xf1gm4en4e
    @user-xf1gm4en4e5 жыл бұрын

    I love this channel BAM~~~

  • @statquest

    @statquest

    5 жыл бұрын

    Hooray! :)

  • @heteromodal
    @heteromodal3 жыл бұрын

    Hey Josh! Thank you for this, watched your 4 regularization videos today and am happy! And a suggestion for a related, followup video - collinearity & multi-collinearity :)

  • @statquest

    @statquest

    3 жыл бұрын

    Thanks! I'll keep those topics in mind.

  • @heteromodal

    @heteromodal

    3 жыл бұрын

    @@statquest Thank YOU for all you do!

  • @RealSlimShady7
    @RealSlimShady74 жыл бұрын

    I am in love with your videos Josh! BAM! I just wanted to ask when we have so many features and multicollinear variables (real case datasets), is applying Elastic Net Regression always better than Ridge and Lasso? I mean, we cannot actually check that as there are so many variables ( Your Deep Learning Example) so can we say that Elastic Net is best of both worlds? We can apply it in most of the scenarios where making a hypothesis about the features not very simple?

  • @statquest

    @statquest

    4 жыл бұрын

    I talk about this in my video that shows how to do Elastic Net regression in R. The answer is, "Yes, elastic-net gives you the best possible situation". See: kzread.info/dash/bejne/laihsNNwdsrIpqw.html

  • @RealSlimShady7

    @RealSlimShady7

    4 жыл бұрын

    @@statquest Thank you so much!!! You are a savior!! BAMMMMM!!!

  • @rishipatel7998
    @rishipatel7998 Жыл бұрын

    I know answering these many comments is very boring I guess you using NLP to filter the comments and answer the important ones and auto reply the others Above video was wonderful! Thank You again Sir 😁

  • @statquest

    @statquest

    Жыл бұрын

    Thanks!

  • @johannesgengenbach6701
    @johannesgengenbach67014 жыл бұрын

    Thanks for all the Great Videos with decent musical intros! ;) I have a question concerning this one: You mention "lambdaX * variableX" but shouldn't it rather be "lambdaX * parameterX" (except the y intercept).

  • @statquest

    @statquest

    4 жыл бұрын

    You are exactly right.

  • @arunasingh8617
    @arunasingh8617 Жыл бұрын

    nice explanation

  • @statquest

    @statquest

    Жыл бұрын

    Thanks!

  • @babasupremacy5225
    @babasupremacy52253 жыл бұрын

    It doesn't gets more easier than this

  • @statquest

    @statquest

    3 жыл бұрын

    Bam! :)

  • @harsharavi8989
    @harsharavi89894 жыл бұрын

    Hey Josh, I love your videos!! Lasso and elastic net regression also do work well when there are lesser data points compared to the variable, rite?

  • @statquest

    @statquest

    4 жыл бұрын

    Yes.

  • @elinkim7212
    @elinkim72125 жыл бұрын

    I think you are missing parenthesis in penalty terms 3:31. But thank you so much for the videos!

  • @jiayoongchong2606
    @jiayoongchong26064 жыл бұрын

    POISSON REGRESSION, PARTIAL LEAST SQUARES AND PRINCIPAL COMPONENT REGRESSION PLEASEEE DR JOSHHH # WE LOVE YOU

  • @statquest

    @statquest

    4 жыл бұрын

    :)

  • @rahulsadanandan5076
    @rahulsadanandan50763 жыл бұрын

    Statquest staaaaat quest whaaat are we learning today....

  • @statquest

    @statquest

    3 жыл бұрын

    Looks like Elastic Net! :)

  • @sslxxbc4092
    @sslxxbc40922 жыл бұрын

    Great video, thank you!! I'm just a bit unsure about the scaling of features. Say, if we scale a feature, what would change for lasso and ridge?

  • @statquest

    @statquest

    2 жыл бұрын

    Scaling the data will change the original least squares parameter estimates, but it will not change the process that Elastic-Net uses to reduce the influence of features that are less useful.

  • @xiaomengzhang1646
    @xiaomengzhang16462 жыл бұрын

    you save my life

  • @statquest

    @statquest

    2 жыл бұрын

    :)

  • @Yzhang250
    @Yzhang2505 жыл бұрын

    Hi Josh, Just revisited this video and very clearly explained. But what are the disadvantages of elastic net? Is this model more computational expensive?

  • @statquest

    @statquest

    5 жыл бұрын

    As far as I know, it's pretty efficient.

  • @glaswasser
    @glaswasser2 жыл бұрын

    Thanks, I ended up googling that airspeed of a swallow thing and watching a Monty Python scene instead of learning how to do elastic net lol

  • @statquest

    @statquest

    2 жыл бұрын

    :)

  • @sowmy525
    @sowmy5255 жыл бұрын

    Thank you StatQuest...awesome series :) Can you do videos on time series methods as well ...clearly explained :P

  • @statquest

    @statquest

    5 жыл бұрын

    You're welcome! I'll add time-series to the to-do list. The more people that ask for it, the more I'll move it up the list. :)

  • @purneshdasari5667

    @purneshdasari5667

    5 жыл бұрын

    @@statquest Thanks for the video lectures josh sir , I am also waiting for timeseries forecasting classes

  • @saikumartadi8494

    @saikumartadi8494

    5 жыл бұрын

    @@statquest it would be great if u do it

  • @statquest

    @statquest

    5 жыл бұрын

    @@saikumartadi8494 Your vote has been noted and I bumped time series up on the list. :)

  • @saikumartadi8494

    @saikumartadi8494

    5 жыл бұрын

    @@statquest awaiting for the video :)

  • @rezab314
    @rezab3143 жыл бұрын

    The intro song with 2.0 speed is nice alternative to the original version :D

  • @statquest

    @statquest

    3 жыл бұрын

    bam!

  • @user-lt3wd5bd8g
    @user-lt3wd5bd8g2 жыл бұрын

    Thanks a lot for your amazing videos. I just wondering, when I use Elastic Net, the coefficient of useless variable seems will not go to zero because of the part of the Ridge Regression in the equation. So, why not just use Losso Regression first to eliminate the useless variable and then use the Ridge Regression to regularize?

  • @statquest

    @statquest

    2 жыл бұрын

    Interesting. You could try that. However, in theory, elastic net is supposed to do that for you. So there may be some aspect specific to your data that is giving you strange results.

  • @basavarajtm5018
    @basavarajtm50184 жыл бұрын

    Hi Josh, Thank you for another awesome video. I have one qn, how to decide which parameters to group for lasso and ridge penalty for Elastic net regression?? are they selected randomly? thanks in advance

  • @statquest

    @statquest

    4 жыл бұрын

    Elastic-net takes care of all of that for you. See it in action here: kzread.info/dash/bejne/laihsNNwdsrIpqw.html

  • @61_shivangbhardwaj46
    @61_shivangbhardwaj463 жыл бұрын

    Thnx sir

  • @statquest

    @statquest

    3 жыл бұрын

    No problem! :)

  • @perrygogas
    @perrygogas3 жыл бұрын

    Hey Josh another excellent video! I thank you very much! Quick question: Do you need brackets after the lambdas? ie λ1 Χ [variable1] + λ1 Χ [variable2] + ... or is it λ1 Χ { [variable1] + [variable2] + ... } ? and similarly for λ2 in ridge regression?

  • @statquest

    @statquest

    3 жыл бұрын

    The two ways you wrote out are equal to each other. So you can use one or the other, they are equivalent.

  • @perrygogas

    @perrygogas

    3 жыл бұрын

    @@statquest You 're right pff that was a mistake. I meant that you show this in 2:47: λ1 Χ [variable1] + ... + [variableX] And again with the squarred for Ridge. There must be a λ1 before the last bracket and λ2 for Ridge

  • @statquest

    @statquest

    3 жыл бұрын

    @@perrygogas Sorry! My notation is super sloppy here. There should be brackets around all of the variables. So it should be lambda1 * [ |v1| + |v2| + ...] + lambda2 * [ v1^2 + v2^2 + ...]

  • @perrygogas

    @perrygogas

    3 жыл бұрын

    @@statquest Great!!! thank you!!! you are the best!!!

  • @jy2502
    @jy25022 жыл бұрын

    Thank you for your wonderful videos. It really helped me to understand ridge/lasso/elastic net. I still have one question though, it seems like elastic net regression can delete some variables even though both lambda 1 and 2 are not 0 (I found it from other papers). but I am not sure how that is possible if lambda 2 is not 0..... do you have any idea for this? Thanks again!

  • @statquest

    @statquest

    2 жыл бұрын

    As long as the lasso penalty is in use, then you can eliminate variables.

  • @user-rn5wd9hl8x
    @user-rn5wd9hl8x3 жыл бұрын

    Thanks so much for this video!! I have a quick question if I may. Didn't u mention in the last video that the lamda for ridge can be close to 0 but never equals 0? I was confused in the part where you say lamda 2 in the elastic-net regression can be 0 which makes it a lasso regression. Thank you in advance for your explanation. :)

  • @statquest

    @statquest

    3 жыл бұрын

    I think you confused lambda for the parameter estimates. With Ridge Regression, the parameter estimates can be close to 0, but not equal to 0. However, lambda can be any value >= 0, and the value is determined using cross-validation.

  • @gonchedanesh6731
    @gonchedanesh67315 жыл бұрын

    Thank you so much for this ! I just don't see when ridge can be better than lasso ?

  • @statquest

    @statquest

    5 жыл бұрын

    This is a good question. There are a few things that Ridge is better at than Lasso. Ridge tends to do better when there were more variables than data. And if there are high correlations in the data, then Ridge can do better than Lasso when there is more data than the number of variables. For more details, check out this website: www4.stat.ncsu.edu/~post/josh/LASSO_Ridge_Elastic_Net_-_Examples.html ...and check out the original Elastic-Net publication: web.stanford.edu/~hastie/Papers/B67.2%20%282005%29%20301-320%20Zou%20&%20Hastie.pdf

  • @maskew
    @maskew5 жыл бұрын

    Hi Josh, great video! There's just one thing that I'm confused about. I understand that Elastic-Net is meant to provide the best of both worlds out of Lasso and Ridge regression but I'm struggling to get my head around what this means. You said that "Elastic-Net regression - groups and shrinks the parameters associated with the correlated variables and leaves them in the equation or removes them all at once". What's the advantage to keeping all of the correlated variables in the equation? I thought that this was a bad thing to do since they are likely providing the same information to the model more than once. Also, does Lasso always keep a single variable of a correlated variable group, even if the group doesn't actually help at all to make predictions?

  • @statquest

    @statquest

    5 жыл бұрын

    This is a great question, and there may be more than one good answer. However, here's my take on it. In a pure "machine learning" setting, retaining correlated variables my not be very useful, but in a research setting, it is very useful. If you have thousands of variables, it my be very useful to see which groups of variables are correlated because that could give you insight into your data that you didn't have before. Does that make sense? And if a variable or a group of correlated variables are not useful, then the corresponding coefficients will shrink, all of them.

  • @maskew

    @maskew

    5 жыл бұрын

    @@statquest Thanks for the quick answer. I can understand why this would be useful in a research setting but surely the purpose of regularization is to find the best set of parameter values to model the function? By holding onto these variables, I can only see them having a negative effect on the optimality of the model

  • @statquest

    @statquest

    5 жыл бұрын

    ​@@maskew So true! So I Iooked into this, and the answer is that correlated variables don't get in the way of predictions. They get in the way of trying to make sense of the effect of each variable on the prediction, but not in making the prediction itself. Said another way, if we used elastic-net regression and left a group of correlated variables in the model, we could conclude that they helped make good predictions, but we would not be able to make any conclusions about relationship between any one variable and the predicted response based on the coefficients. For more details, see point 5 on this page: newonlinecourses.science.psu.edu/stat501/node/346/

  • @maskew

    @maskew

    5 жыл бұрын

    @@statquest Right okay that makes sense then! Thanks a lot for getting back to me

  • @tanphan3970
    @tanphan39702 жыл бұрын

    Hello @Josh Starmer, Thank you for your videos, so easy to understand. But we are talking about Elastic_net (also Ridge/Lasso) technical in Regression model. So how about others model? They can apply to solve overfitting situation as Regression!

  • @statquest

    @statquest

    2 жыл бұрын

    Yes. Ridge, Lasso and Elastic net style penalties can be added to all kinds of models.

  • @tanphan3970

    @tanphan3970

    2 жыл бұрын

    @@statquest all kinds of models with same formulars as Regression?

  • @statquest

    @statquest

    2 жыл бұрын

    @@tanphan3970 No, pretty much any formula will work. For example, regularization can be applied to Neural Networks, which are very different.

  • @reactorscience
    @reactorscience4 жыл бұрын

    Sir your videos are amazing. I have a question though. In case of deep neural networks why make the model complex and then add regularization or dropout, isn't a better idea not to create the problem at the first place that is not make the model complex? And if the the model is overfitting, shouldn't trying to reduce the complexity of the model be the solution?

  • @statquest

    @statquest

    4 жыл бұрын

    Neural Networks are a little bit like black boxes and it's hard to know what they are doing - so it's hard to know if the model is "simple" or "complex". So, with NNs, regularization can just deal with the overfitting problem without you having to worry too much about the model.

  • @merrimac1
    @merrimac15 жыл бұрын

    Thank you for the tutorial. One thing I don't get is why Elastic Net can remove some variables. It has the component of Ridge regression, so a variable won't be removed all together. How come?

  • @yc6768
    @yc67682 жыл бұрын

    Thanks for the video Josh. Your explaination makes sense, but i can't wrap my head to think of a reason why would this work still. If we know some variables that are less important (e.g., Age in your previous example), don't we still have those variables that in the loss function? Is it just that their impact will be sitting in between none and when using L2?

  • @statquest

    @statquest

    2 жыл бұрын

    I'm not sure I understand your question. However, for less important variables, we can reduce their associated coefficients without drastically reducing the fit of the model to the data, and this will result in a significant reduction in the "penalty" that we add to the loss function.

  • @ruihanli7241
    @ruihanli72412 жыл бұрын

    Thanks for the clearly explanation, so since elastic regression is the best, should we just use elasitc regression every time instead of using lasso or ridge?

  • @statquest

    @statquest

    2 жыл бұрын

    Yes, because you can use Elastic Net to be pure Lasso or pure Ridge, and everything in between, so you can have it all.

  • @ashwinmanickam
    @ashwinmanickam2 жыл бұрын

    Hey Josh great video as usual ! I have a question for you, grateful if you can answer Let’s say I do a market mix modelling and I have closer to 250 variables and closer to 180 line items, which of these would be most suitable. Info about data A lot of these variables are super correlated, but I cannot afford to drop anyone off then since I need to present contribution of every channel to the business and they are naturally Co related since spending from business usually happens in clusters and are similar for similar channels like Facebook and Instagram. Any pointers on these will be very useful thanks!

  • @statquest

    @statquest

    2 жыл бұрын

    Try just using Ridge Regression and see how that works.

  • @boshbigcat9619
    @boshbigcat96195 жыл бұрын

    its sounds nice guitar!!! loooooooool

  • @AI_Financier
    @AI_Financier2 жыл бұрын

    Dear Josh, thank you so much for making life easier for us, if you believe in heaven you will be one the firsts in (lol). A question, how does the Lasso eliminate one of the correlated variables? lets say i have two identical variables v1 and v2 (with 100% correlation), how does the lasso work on them? thank you in advance

  • @statquest

    @statquest

    2 жыл бұрын

    Presumably, if you increase lambda enough, one parameter will go to 0.

  • @carloszarzar
    @carloszarzar5 жыл бұрын

    Hi everyone. Could I consider the lambda as a hyperparameter in Ridge Regression and Lasso Regression?

  • @statquest

    @statquest

    5 жыл бұрын

    Yes

  • @jeancarlobejaran3295
    @jeancarlobejaran32954 жыл бұрын

    'Yee-ha!'

  • @statquest

    @statquest

    4 жыл бұрын

    :)

  • @JSS11
    @JSS112 жыл бұрын

    Why do we not use parentheses after each lambda? I got confused as we did in the two earlier videos on regularization. Thanks for helping out!

  • @statquest

    @statquest

    2 жыл бұрын

    Oops. Looks like I forgot to add the parentheses. Sorry about the confusion that caused. :(

  • @JSS11

    @JSS11

    2 жыл бұрын

    ​@@statquest No worries! I thank you for your keeping my motivation level up there and getting back to me so quick.

  • @nashaeshire6534
    @nashaeshire65343 жыл бұрын

    Thanks a lot for your great videos. I don't understand why use Lasso reg or Ridge reg when we can use Elastic-Net reg? What is the draw back of Elastic Neg regression?

  • @statquest

    @statquest

    3 жыл бұрын

    None that I know of. However, not every ML method implements the full elastic net.

  • @nashaeshire6534

    @nashaeshire6534

    3 жыл бұрын

    @@statquest Thanks! I don't get why you don't get more thumbs up... Great show, thanks again

  • @antoniovivaldi2270
    @antoniovivaldi22705 жыл бұрын

    Your presentations are good - short, clear and well explained. However, the "lambda" parameters have to be arbitrarily chosen so the Lasso Regression and Ridge Regression methods lose objectivity - the result depends on the observer. I wonder where those methods are used. In my opinion, the classic Least Means Squares (LMS) or LMS with statistical weights (in different variations) are still the best methods/techniques for reduction of experimental data and modeling.

  • @statquest

    @statquest

    5 жыл бұрын

    Elastic Net is used all the time in Machine Learning and lambda is determined using cross validation.

  • @antoniovivaldi2270

    @antoniovivaldi2270

    5 жыл бұрын

    @@statquest Thank you! Would you kindly recommend a link to that "cross validation"?

  • @statquest

    @statquest

    5 жыл бұрын

    @@antoniovivaldi2270 Here's a link to the StatQuest on cross validation: kzread.info/dash/bejne/mIet1tyAp9qohto.html

  • @JMRG2992
    @JMRG29925 жыл бұрын

    I want to know something. Minimum sample size for ridge and lasso is?. I have checked tons of papers, where some journals use at least 4, and others use 30, and others requires to estimate (like Greenes) for about 250 observatoins. Would this change with ridge and lasso regressions?

  • @paulstevenconyngham7880
    @paulstevenconyngham78805 жыл бұрын

    hey man, might you be able to do a wee vid on z-score?

  • @statquest

    @statquest

    5 жыл бұрын

    Yep! I can do that fairly soon.

  • @paulstevenconyngham7880

    @paulstevenconyngham7880

    5 жыл бұрын

    @@statquest awesome Josh!

  • @mohitkushwaha8974
    @mohitkushwaha8974 Жыл бұрын

    hi , u mentioned that best lambda1 and lambda 2 we can find by cross validation. for cross vaildation do we need to write a seperate code or it is done by elastic net by its own???

  • @statquest

    @statquest

    Жыл бұрын

    It's done in the code. See: kzread.info/dash/bejne/laihsNNwdsrIpqw.html

  • @innamemetova5923
    @innamemetova59235 жыл бұрын

    Sorry, mb its a bit silly, but.. Don't we need brackets after lambda1 for all absolute parameters and brackets after lambda2? 3:46 in the video

  • @statquest

    @statquest

    5 жыл бұрын

    Yes! You are correct. That was a slight omission. I hope it's not too confusion.

  • @longkhuong8382
    @longkhuong83825 жыл бұрын

    BAM!!! Yee-ha!!! haha. Thank you

  • @statquest

    @statquest

    5 жыл бұрын

    Hooray!!! :)

  • @mrcharm767
    @mrcharm767 Жыл бұрын

    brilliant !! but have a doubt in mind that how are we sure that elastic net regression would not cause high variance since its summing both ridge and lasso and due to this it will guide the model to change through a higher range?

  • @statquest

    @statquest

    Жыл бұрын

    I'm not sure I understand your question, but, by using validation, we can test to see if elastic-net is increasing variance, and if so, not use it.

  • @mrcharm767

    @mrcharm767

    Жыл бұрын

    @@statquest I meant was since the line tries to adjust to lowest error from the target as possible with the gradient descent and all .. but we use ridge and lasso regression that would slightly variance the line from the data (predicted points line to the actual data points line ) and the accuracy would be slightly increased or decreased depending on the data .. so if we use elastic net regression which is combination of both ridge and lasso it would cause higher variance and it's confirm that accuracy would be bit reduced right ?? This was the question

  • @mrcharm767

    @mrcharm767

    Жыл бұрын

    here by variance i mean the distance between predict data points line and the actual data line

  • @statquest

    @statquest

    Жыл бұрын

    @@mrcharm767 To be honest, I still don't understand your question. But I think part of the problem is that the term "variance" has two meanings - the statistical one ( kzread.info/dash/bejne/ha6OmKmpk8nVgbw.html ) - and the machine learning one ( kzread.info/dash/bejne/d6l2pNxskqyTkaQ.html ). The whole point of regularization is to reduce variation in the sense of used in machine learning (and thus, increase long term accuracy) and we do that by desensitizing the model to the variables in the model. To see this in action, and to verify that it works correctly, see: kzread.info/dash/bejne/laihsNNwdsrIpqw.html

  • @mrcharm767

    @mrcharm767

    Жыл бұрын

    @@statquest yes u got right where i was actually i made a mistake interchanging bias and variance in the explanation

  • @MohamedIbrahim-qk3tk
    @MohamedIbrahim-qk3tk4 жыл бұрын

    Lasso does the job of shrinking the coefficients AND removing the useless parameters right?

  • @statquest

    @statquest

    4 жыл бұрын

    In this video I show the roles that both Ridge and Lasso play in Elastic Net: kzread.info/dash/bejne/laihsNNwdsrIpqw.html