Dive Into Deep Learning, Lecture 2: PyTorch Automatic Differentiation (torch.autograd and backward)

In this video, we discuss PyTorch’s automatic differentiation engine that powers neural networks and deep learning training (for stochastic gradient descent). In this section, you will get a conceptual understanding of how autograd works to find the gradient of multivariable functions. We start by discussing derivatives, partial derivatives, and the definition of gradients. We then discuss how to compute gradients using requires_grad=True and the backward() method. Thus, we cover classes and functions implementing automatic differentiation of arbitrary scalar-valued and non-scalar-valued functions. We also discuss the Jacobian matrix in PyTorch. Differentiation is a crucial step in nearly all machine learning and deep learning optimization algorithms. While the calculations for taking these derivatives are straightforward, working out the updates by hand can be a painful and tedious task.
#Autograd #PyTorch #DeepLearning

Пікірлер: 41

  • @nitinsrivastav3541
    @nitinsrivastav3541Ай бұрын

    thanks man i was finding it extremely difficult to understand the maths behind backward and detach (although i have doen it in my high school) because no one was explaining them in this depth.........love you 😍😍

  • @michaelscience2481
    @michaelscience24812 жыл бұрын

    This is what I was searching for months. I m so thankful Dr. Science for breaking down this concept for us

  • @DrDataScience

    @DrDataScience

    2 жыл бұрын

    I'm glad you find it helpful!

  • @reddysekhar3765
    @reddysekhar3765 Жыл бұрын

    Truly thankful to you. To the point without confusion. Thank you once again

  • @atanudasgupta
    @atanudasgupta Жыл бұрын

    Excellent videos and the textbook, deeply admire your contributions

  • @parisahajibabaee2893
    @parisahajibabaee28932 жыл бұрын

    Amazing as always!! Very helpful and valuable video!

  • @sktdebnath
    @sktdebnath Жыл бұрын

    This is what a proper tutorial should be. Thanks a lot. Subscribed

  • @DrDataScience

    @DrDataScience

    Жыл бұрын

    Thanks!

  • @rohithpeesapati8840
    @rohithpeesapati88402 жыл бұрын

    thank you very much for uploading this video! very helpful!

  • @user-vm9hl3gl5h
    @user-vm9hl3gl5h Жыл бұрын

    14:50 x.grad contains the valuesd of partial{y} / partial{x} 17:50 x.grad.zero_() 25:00 gradient for multiple inputs -> multiple outputs. Since the Jacobian is a matrix, we need to input a 1-d tensor to get a valid vector-output. => But our loss function has been a scalar, so this is why I am not accustomed to this form. 34:10 explaining .detach(). => treat those as constants, not a variable that we differentiate w.r.t.

  • @edd-bp8mj
    @edd-bp8mj2 жыл бұрын

    why do you need to multiply v vector to the jacobian matrix? and what is v vector exactly?

  • @azzyfreeman
    @azzyfreeman Жыл бұрын

    Thanks for the video, it really cleared up Pytorch autograd, now I will be making notes on this gold nugget

  • @DrDataScience

    @DrDataScience

    Жыл бұрын

    Thanks!

  • @Gibson-xn8xk
    @Gibson-xn8xk2 жыл бұрын

    First of all, I want to express my gratitude to you for the work you have done. There is one thing i want you to ask: why do we write the partial derivatives of the scalar function in the form of column? Whereas, following the logic of Jacobian matrix, it should be a row. Thanks in advance!

  • @AJ-et3vf
    @AJ-et3vf Жыл бұрын

    Awesome video! Thank you!

  • @DrDataScience

    @DrDataScience

    Жыл бұрын

    Thanks!

  • @sadpotato5111
    @sadpotato5111 Жыл бұрын

    This video deserves more view.

  • @DrDataScience

    @DrDataScience

    Жыл бұрын

    Thanks a lot!

  • @md.zahidulislam3548
    @md.zahidulislam3548 Жыл бұрын

    Thanks a lot.

  • @nicolaemaria4380
    @nicolaemaria43802 жыл бұрын

    great explanation! I hope it will get the likes it deserves

  • @DrDataScience

    @DrDataScience

    2 жыл бұрын

    Thanks a lot!

  • @asfandiyar5829
    @asfandiyar5829 Жыл бұрын

    Thank you sooo much! This finally clicked for me.

  • @DrDataScience

    @DrDataScience

    Жыл бұрын

    I am glad you found it helpful!

  • @user-gr2jy6xq4h
    @user-gr2jy6xq4h2 жыл бұрын

    Nice explanation. Found it from youtube recomendations

  • @DrDataScience

    @DrDataScience

    2 жыл бұрын

    I am glad you found it helpful!

  • @greender644
    @greender644 Жыл бұрын

    29:49 - As far as I understand in 'a.grad' should turn out [12., 18.]

  • @azharhussian4326
    @azharhussian43268 ай бұрын

    Thank you

  • @DrDataScience

    @DrDataScience

    8 ай бұрын

    Thank you!

  • @ramincybran
    @ramincybran2 ай бұрын

    thnks myfriend - in last section - whats is the z.sum ? what is the the (SUM) function for ? why yyou put the sum ?

  • @user-wz3np2hf3g
    @user-wz3np2hf3g6 ай бұрын

    way better than my prof...

  • @DrDataScience

    @DrDataScience

    6 ай бұрын

    Thank you!

  • @danieleboch3224

    @danieleboch3224

    27 күн бұрын

    Loser.

  • @callpie2398
    @callpie2398 Жыл бұрын

    Please Do cover a playlist on Graph Neural Networks (at least discuss all the basics and methods of GNNs). The internet world lacks quality contents on this topic

  • @DrDataScience

    @DrDataScience

    Жыл бұрын

    Great suggestion! I'll work on it.

  • @arpit743
    @arpit7432 жыл бұрын

    hi !! thanks for the wonderfull video :) can you please explain why vector v is 1?? and what is derivative w.r.t to self??

  • @DrDataScience

    @DrDataScience

    2 жыл бұрын

    The derivative of any variable. let's say x, wrt itself, i.e., x, is equal to 1. That is why we use a vector of v of all ones. If for some reason, you want to exclude one variable from the gradient vector, you can then replace the corresponding entry of v with 0.

  • @arpit743

    @arpit743

    2 жыл бұрын

    @@DrDataScience that does make sense thanks! But why do we multiply with v or ones what does the dot product of jacobian and vector v represent? Why do we multiply derivative of self with jacobian?

  • @mahdiamrollahi8456
    @mahdiamrollahi84562 жыл бұрын

    The way you define the y is different from what i am thinking. In first example, I thought we should define y as x**2 or in second one y as x. But if i define the y like this, I will get an error which say that it needs just one number not sequence of numbers.

  • @mahdiamrollahi8456

    @mahdiamrollahi8456

    2 жыл бұрын

    Oh I got it. But i did not get what v vector is?

  • @vtrandal
    @vtrandal Жыл бұрын

    No!

  • @vtrandal
    @vtrandal Жыл бұрын

    Fixed your broken web page that forces us to bend to your flaws.