Kolmogorov-Arnold Networks: MLP vs KAN, Math, B-Splines, Universal Approximation Theorem

Ғылым және технология

In this video, I will be explaining Kolmogorov-Arnold Networks, a new type of network that was presented in the paper "KAN: Kolmogorov-Arnold Networks" by Liu et al.
I will start the video by reviewing Multilayer Perceptrons, to show how the typical Linear layer works in a neural network. I will then introduce the concept of data fitting, which is necessary to understand Bézier Curves and then B-Splines.
Before introducing Kolmogorov-Arnold Networks, I will also explain what is the Universal Approximation Theorem for Neural Networks and its equivalent for Kolmogorov-Arnold Networks called Kolmogorov-Arnold Representation Theorem.
In the final part of the video, I will explain the structure of this new type of network, by deriving its structure step by step from the formula of the Kolmogorov-Arnold Representation Theorem, while comparing it with Multilayer Perceptrons at the same time.
We will also explore some properties of this type of network, for example the easy interpretability and the possibility to perform continual learning.
Paper: arxiv.org/abs/2404.19756
Slides PDF: github.com/hkproj/kan-notes
Chapters
00:00:00 - Introduction
00:01:10 - Multilayer Perceptron
00:11:08 - Introduction to data fitting
00:15:36 - Bézier Curves
00:28:12 - B-Splines
00:40:42 - Universal Approximation Theorem
00:45:10 - Kolmogorov-Arnold Representation Theorem
00:46:17 - Kolmogorov-Arnold Networks
00:51:55 - MLP vs KAN
00:55:20 - Learnable functions
00:58:06 - Parameters count
01:00:44 - Grid extension
01:03:37 - Interpretability
01:10:42 - Continual learning

Пікірлер: 100

  • @josephamess1713
    @josephamess171329 күн бұрын

    The fact this video is free is incredible

  • @umarjamilai

    @umarjamilai

    29 күн бұрын

    You're welcome 🤗

  • @edsonjr6972
    @edsonjr697229 күн бұрын

    Your videos are literally the only ones with 1hr+ I would ever watch on KZread. Keep going mate, extremely high quality content 👏🏽👏🏽

  • @AdmMusicc
    @AdmMusicc12 күн бұрын

    You're on a mission to make the best and friendliest content to consume deep learning algorithms and I am all in for it.

  • @nokts3823
    @nokts382328 күн бұрын

    Thanks a lot for making this accessible for people outside the field, for which reading and understanding these papers is quite tough. Thanks to you I'm able to stay slightly more up to date with the crazy quick developments in ML!

  • @brandonheaton6197
    @brandonheaton61974 күн бұрын

    Best explanations of splines i have seen. Legit 100%

  • @franciscote-lortie8680
    @franciscote-lortie868024 күн бұрын

    Incredibly clear explanations, the flow of the video is also really smooth. It’s almost like you’re telling a story. Please keep making content!!

  • @xl0xl0xl0
    @xl0xl0xl019 күн бұрын

    Wow this was a super clear an on-point explanation. Thank you, Umar.

  • @bensimonjoules4402
    @bensimonjoules44027 күн бұрын

    Amazing content, thanks! I'm very excited about the continual learning properties of these networks.

  • @mohamedalansary2542
    @mohamedalansary254229 күн бұрын

    Clearly explained and very valuable content as always Umar. Thank you!

  • @MrNathanShow
    @MrNathanShow29 күн бұрын

    The intro of a basic linked up linear layers was so well done and really makes this introduction friendly!

  • @goldentime11
    @goldentime1120 күн бұрын

    Thanks Umar for such a wonderful tutorial! I've been eyeing this paper for a while!

  • @manumaminta6131
    @manumaminta613128 күн бұрын

    Your videos help me (a grad student) really understand difficult, often abstract concepts. Thank you so much... I'll always support your stuff!

  • @andreanegreanu8750
    @andreanegreanu875010 күн бұрын

    Very clear, well explained, top notch!

  • @AlpcanAras
    @AlpcanAras27 күн бұрын

    This is life changing, in my opinion. Thank you for the efforts on the videos!

  • @ozgunsungar9370
    @ozgunsungar937016 күн бұрын

    awesome, easy to follow even person dont know anything :)

  • @stacks_7060
    @stacks_706028 күн бұрын

    One of the best math videos I’ve watched on KZread

  • @odysy5179
    @odysy517918 күн бұрын

    Fantastic explanation!

  • @anmolmittal9
    @anmolmittal921 күн бұрын

    This is really great! Power to you!!🚀

  • @MuhammadrizoMarufjonov-os5fv
    @MuhammadrizoMarufjonov-os5fv29 күн бұрын

    Thanks for including prerequisites

  • @anirudh514
    @anirudh51429 күн бұрын

    Thanks for the crystal clear explaination!!

  • @user-pu4oc9ek9u
    @user-pu4oc9ek9u29 күн бұрын

    Hello Umar, this video is my best birthday gift I have ever received, thanks a lot :)

  • @artaasadi9497
    @artaasadi949719 күн бұрын

    that is very useful, informative and interesting! Thanks a lot!

  • @arupsankarroy8722
    @arupsankarroy872226 күн бұрын

    Sir, you are great..💙💙

  • @JONK4635
    @JONK463528 күн бұрын

    Extremely clear explanation and content here! Very helpful. I am happy that you came from PoliMI as well :) keep it up!

  • @ansonlau7040
    @ansonlau704022 күн бұрын

    Thankyou Jamil, what a cool video

  • @kmalhotra3096
    @kmalhotra309622 күн бұрын

    Hats off, what an awesome video!!!

  • @MuhammadMuzzamil-ki4he
    @MuhammadMuzzamil-ki4he27 күн бұрын

    Thank you for such great and detailed explanation.

  • @bankayxy00
    @bankayxy0024 күн бұрын

    Thank you so so much for this amazing content.

  • @user-il1hu5xp2x
    @user-il1hu5xp2x29 күн бұрын

    What funny, is that i predicted your next video will be on KAN, after i see you in github. I WILL WATCH THIS VIDEO, AS I FEEL THIS WILL BE THE FUTURE OF NEUR NETWORK, THANK YOU FOR YOUR WORK AND CONTENT ❤

  • @ScottzPlaylists
    @ScottzPlaylists28 күн бұрын

    High quality explanations.. Thanks.

  • @lethnis9307
    @lethnis930728 күн бұрын

    Your explanations are the best, thank you so much😘🤗

  • @coolkaran1234
    @coolkaran123429 күн бұрын

    You are savior, without you mortals like me would be lost in the darkness!!!

  • @enricovompa1876
    @enricovompa187629 күн бұрын

    Thank you for making this video!

  • @howardmeng256
    @howardmeng25624 күн бұрын

    Amazing video! Thanks a lot !

  • @johanvandermerwe7687
    @johanvandermerwe768729 күн бұрын

    I saw this paper on papers with code, and thought to myself I wonder if Umar Jamil will cover this. Thanks for your effort and videos!

  • @jeunjetta
    @jeunjetta27 күн бұрын

    I think KAN will be the catalist of a significant tipping point in science. I want to apply this to power system grids and replace existing dynamic models with ones made from PMU data using KAN

  • @luigigiordanoorsini5980
    @luigigiordanoorsini598022 күн бұрын

    Ho appena letto la piccola bio del tuo canale, spero di non essere offensivo dicendo che adesso capisco perché il tuo ottimo inglese mi sembrasse comunque molto familiare. Ad ogni modo ti ringrazio enormemente per il tuo contributo hai spiegato tutta la teoria in un modo, a mio avviso, estremamente chiaro e soprattutto coinvolgente. Ti prego continua così, di nuovo un enorme grazie e complimenti per il tuo contributo alla scienza

  • @umarjamilai

    @umarjamilai

    22 күн бұрын

    Grazie a te per aver visitato il mio canale! Spero di pubblicare più spesso, anche se per fare contenuti di qualità ci vogliono settimane di studio e preparazione. In ogni caso, spero di rivederti presto! Buon weekend

  • @luigigiordanoorsini5980

    @luigigiordanoorsini5980

    22 күн бұрын

    @@umarjamilai Avevi già guadagnato un iscritto adesso hai guadagnato un fan. Ahahahahah

  • @zaevi6855
    @zaevi685529 күн бұрын

    crazy that it took me an hr video to understand that its the (control points) being trained on the spline graph vs weights with MLPs and CNNs, thank you!

  • @prathamshah2058
    @prathamshah205827 күн бұрын

    Thank-you so much for explaining the paper, it is so easy to understand now, btw can you also make a hands on video with the kan package developed by mit which is based off pytorch.

  • @wolfie6175
    @wolfie61759 күн бұрын

    Good video, quality content.

  • @GUANGYUANPIAO
    @GUANGYUANPIAO16 күн бұрын

    awesome explanation

  • @sergiorego6321
    @sergiorego632129 күн бұрын

    Phenomenal! Thank you :)

  • @hajaani6417
    @hajaani641729 күн бұрын

    You’re fantastic, mate.

  • @samadeepsengupta
    @samadeepsengupta29 күн бұрын

    Great Content !!

  • @RiteshBhalerao-wn9eo
    @RiteshBhalerao-wn9eo14 күн бұрын

    Amazingg explanation !

  • @user-wy1xm4gl1c
    @user-wy1xm4gl1c18 күн бұрын

    This is awesome!

  • @vaadewoyin
    @vaadewoyin26 күн бұрын

    Cant wait to watch this, saved! Will comment again when i actually watch it..😅

  • @dhackmt
    @dhackmt21 күн бұрын

    i loved it sir .

  • @p4ros960
    @p4ros96019 күн бұрын

    bruh so good. Keep it up!

  • @danielegiunchi9741
    @danielegiunchi974127 күн бұрын

    brilliant video!

  • @satviknaren9681
    @satviknaren968127 күн бұрын

    Please do post more ! please do more videos !

  • @pabloe1802
    @pabloe180224 күн бұрын

    An implementation video will be awesome

  • @JuliusSmith
    @JuliusSmith21 күн бұрын

    Excellent video, thanks! At the end, I _really_ wanted to see an illustration of the relatively "non-local" adaptation of MLP weights. Can that be found somewhere?

  • @subhamkundu5043
    @subhamkundu504328 күн бұрын

    Hey @Umar, great content as always. Looking forward to a KAN implementation video from scratch. Also I think in 31:01 there is a minor language mistake. I think it will be for using a quadratic Bspline curve rather than quadratic Bezier curve

  • @seelowst
    @seelowst27 күн бұрын

    Having a such good teacher is so adorable, i wish i could be your students.

  • @umarjamilai

    @umarjamilai

    27 күн бұрын

    哪里哪里啊,谢谢你的赞成!

  • @seelowst

    @seelowst

    27 күн бұрын

    @@umarjamilai 太棒了,您还会中文👍

  • @umarjamilai

    @umarjamilai

    27 күн бұрын

    @@seelowst 我就是刚刚从中国来的,在中国主了4年了,现在回欧洲了。

  • @seelowst

    @seelowst

    27 күн бұрын

    @@umarjamilai 我从没离开过我的城市,我希望像您一样👍

  • @fatemeshams9758
    @fatemeshams975813 күн бұрын

    awesome👍

  • @plutophy1242
    @plutophy124220 күн бұрын

    this video is so amazing!!!!!!!

  • @user-jb3ht1wq5l
    @user-jb3ht1wq5l23 күн бұрын

    THANK YOU

  • @akramsalim9706
    @akramsalim970629 күн бұрын

    awesome bro.

  • @ezl100
    @ezl10024 күн бұрын

    thanks Umar. Very nice explanation. Just 2 questions : 1 - Does it mean we can specify different knots per edge? 2 - I am not understanding how the backpropagation will work. Let's say we calculate the gradient from h1. It will update phi 1,1 and phi 1,2 but how the learning process will impact the knots to the desired value?

  • @bzzzzz1736
    @bzzzzz173627 күн бұрын

    thank you

  • @faiqkhan7545
    @faiqkhan754528 күн бұрын

    Umar bhai you the great

  • @Kishan31468
    @Kishan3146829 күн бұрын

    Thanks man. Next xLSTM please.

  • @ai__76
    @ai__7629 күн бұрын

    amazing

  • @emiyake
    @emiyake28 күн бұрын

    Thanks!

  • @daleanfer7449
    @daleanfer744929 күн бұрын

    刚好期盼这个!

  • @umarjamilai

    @umarjamilai

    29 күн бұрын

    期待你的评价😇

  • @daleanfer7449

    @daleanfer7449

    29 күн бұрын

    ❤很好的内容,有考虑做inverse rl的内容吗❤

  • @baba42kachari
    @baba42kachari24 күн бұрын

    Thanks

  • @shubhamrandive7684
    @shubhamrandive768427 күн бұрын

    Great explanation. What app do you use to create slides ?

  • @umarjamilai

    @umarjamilai

    27 күн бұрын

    PowerPoint + a lot a lot a lot a lot a lot of patience.

  • @DiegoSilva-dv9uf
    @DiegoSilva-dv9uf28 күн бұрын

    Valeu!

  • @fouziaanjums6475
    @fouziaanjums647529 күн бұрын

    Hi, can you please make a video on multimodal LLMs, fine tuning it for custom dataset...

  • @routerfordium
    @routerfordium26 күн бұрын

    Thank you for the great video! Can you (or anyone) help understand why you need to introduce the basis functions b(x) in the residual activation functions?

  • @user-hd7xp1qg3j
    @user-hd7xp1qg3j29 күн бұрын

    Could you please next explain multi modal llms, techniques like Llava, llava plus, llava next?

  • @Patrick-wn6uj

    @Patrick-wn6uj

    29 күн бұрын

    I waiting for that day too

  • @MrAloha
    @MrAloha29 күн бұрын

    Wow! 🙏

  • @Engrbilal143
    @Engrbilal14329 күн бұрын

    Time to implement it

  • @willpattie581
    @willpattie58128 күн бұрын

    One thing I didn’t catch: how are the functions tuned? If each function consists of points in space and we move around the points to move the B spline, how do we decide to move the points? Doesn’t seem like backprop would work in the same way.

  • @umarjamilai

    @umarjamilai

    28 күн бұрын

    The same way we move weights for MLPs: we calculate the gradient of the loss function w.r.t the parameters of these learnable functions and change them in the opposite direction of the gradient. This is how you reduce the loss. We are still doing backpropagation, so nothing changed on that front compared to MLPs.

  • @user-sy6xn7nq7s
    @user-sy6xn7nq7s18 күн бұрын

    There are continuous but indiferable points in the spline, right? What are you going to do?

  • @rohitjindal124
    @rohitjindal12429 күн бұрын

    Sir I have been a huge fan of your videos and have watched all of them . I am currently in my second year BTech and really passionate about learning ml sir if possible can work under you I don’t want any certificate or anything just want to see observe and learn

  • @jeremykothe2847
    @jeremykothe284728 күн бұрын

    fwiw I took a MLP solution for MNIST, substituted KAN for the MLP layers and no matter what I did (adding dimensions etc) it couldn't solve it. My intuition is that KANs only work well for approximating linear-ish functions, not irregular, highly discontinuous ones like image classification would need. But perhaps I just screwed it up :D

  • @ScottzPlaylists
    @ScottzPlaylists28 күн бұрын

    Please explain DSPy

  • @ChukwuemekaAmblessedchinenye
    @ChukwuemekaAmblessedchinenye7 күн бұрын

    can you make tutorial video on model like Perplexity that use website live search

  • @pratishdewangan132
    @pratishdewangan13223 күн бұрын

    In search of gold i found a diamond

  • @suman14san
    @suman14san29 күн бұрын

    Please add a payment option

  • @umarjamilai

    @umarjamilai

    29 күн бұрын

    Your love and support is enough! Have a great weekend!

  • @Patrick-wn6uj

    @Patrick-wn6uj

    29 күн бұрын

    @@umarjamilaiJust woow

  • @einsteinsapples2909
    @einsteinsapples290929 күн бұрын

    Your explenations are great. I think though, you should take breaks to blow your nose maybe, because you were sniffing a lot. It will make you videos more enjoyable.

  • @ln_exp1
    @ln_exp127 күн бұрын

    Interesting

  • @kiffeeify
    @kiffeeify20 күн бұрын

    Thanks!

  • @alfredmanto5487
    @alfredmanto548727 күн бұрын

    Thanks

Келесі