Пікірлер

• 
  @s3_build23 сағат бұрын

  ok based

• 
  @jacobrintamaki23 сағат бұрын

  Moore’s Law, also known as “let’s just draw a line and call it a day”.

• 
  @mechadenseКүн бұрын

  Good video. Minor nitpick 2:57 not so clear why active components give more precision than passive ones. And the context of "precision" here. My guess would be that you refer to what you mention later on analog circuits needing to be bigger for energy storage. (And opportunity of error correction in digital cicuits - in usage)

• 
  @jacobrintamakiКүн бұрын

  Thank you for the nitpick. I agree that, in retrospect, that point was not the right one to make.

• 
  @Miles-li9jyКүн бұрын

  Hell yeah

• 
  @jacobrintamaki23 сағат бұрын

  I bet it feels really good to be a transistor in an integrated circuit.

• 
  @zuberzubreme-bm6lrКүн бұрын

  damn i got a degree in applied math/physics and this basically covers the whole degree, and some stuff you'd learn on your own doing undergrad research and shit. do they really test you on everything? and are the questions made to be trivial (ive seen this before so i know how to do it) or is it spend 30 min trying to solve it just to end up where you started?

• 
  @jacobrintamakiКүн бұрын

  I will be going over exam questions on the weekend! That should hopefully answer a lot of your questions. :)

• 
  @abjohnson1172 күн бұрын

  Great video! As an Applied Math PhD student I think you explained all the concepts very succinctly at a level that a lot of people with some math background can understand. I think it would also be interesting to go through some problems from exams to apply some of these topics you've just explained very well. Thanks!

• 
  @jacobrintamaki2 күн бұрын

  Thank you for the comment! There are past exams online; I think this weekend I will make a video walking through them.

• 
  @abidhossain80742 күн бұрын

  Gonna Join this Fall 25 PhD program in ICME Stanford. KZread recommendations at times is actually helpful !

• 
  @nysewerrat65772 күн бұрын

  Am I the only one here who hears dimenxional instead of dimensional? For real though, nice video!

• 
  @YoutubeHandlesSuckBalls3 күн бұрын

  Damn, I hoped you were going to talk about Autobots.

• 
  @emir27504 күн бұрын

  Thank you for your clear explanations! I am a second year maths undergraduate student and thinking about these make me feel two things: (1) I feel like I might not be able to qualify for all of these in the rest 2 years I have in UG, (2) I feel motivated and want to work for these things starting now. We will see I guess xD

• 
  @saraandsammyb.95994 күн бұрын

  awesome vid!! Can you do the one for physics?

• 
  @jamesrothwell17385 күн бұрын

  Watched this whole mf and I got expelled from my senior year of high school lol I thought AP Calc BC was hard but I guess I just dipped my toe in the world of numbers…. But now I’m a refrigeration tech so I have no reason to dive in. So thank you for showing me some of what I would’ve ended up learning

• 
  @jacobrintamaki5 күн бұрын

  Two things: 1. I genuinely appreciate your comment. Thank you. 2. “But now I’m a refrigeration tech so I have no reason to dive in.” Don’t limit yourself like that. You’re clearly driven and curious. Keep going.

• 
  @jamesrothwell17385 күн бұрын

  Watched this whole mf and I got expels from my senior year of high school lol I thought AP Calc BC was hard but I guess I just dipped my toe in the world of numbers…. But now I’m a refrigeration tech so I have no reason to dive in. So thank you for showing me some of what I would’ve ended up learning

• 
  @yashashwinkarthikeyan17016 күн бұрын

  so cool

• 
  @imeprezime12856 күн бұрын

  I've never heard of such

• 
  @rumikang7566 күн бұрын

  amazing

• 
  @SianaGearz8 күн бұрын

  REAL TRANSFORMERS HAVE WINDINGS (i am an electrical engineer)

• 
  @jacobrintamaki8 күн бұрын

  real I’ll make more EE videos soon

• 
  @pauljnellissery709610 күн бұрын

  Hey Jacob, these videos are really cool and informative. Please do more on differential equations. Thanks

• 
  @jacobrintamaki8 күн бұрын

  Will do

• 
  @Andre-fm9zx10 күн бұрын

  loved the explanation of what a green function is!

• 
  @milos_radovanovic10 күн бұрын

  you skipped quarks and gluons

• 
  @psychii67810 күн бұрын

  honestly this doesnt seem too difficult to pass, ive used most of the topics covered in CFD in undergrad

• 
  @psychii67810 күн бұрын

  the analytical section would probably be harder for me i guess

• 
  @jacobrintamaki10 күн бұрын

  @@psychii678 that's fair. I would say that this qual is in the mathematics department rather than MechE/Aero/Astro/CS, and is done on paper, so I wouldn't expect it to be incredibly involved as compared to industry sims.

• 
  @adrianzyskowski198910 күн бұрын

  crazy value

• 
  @Mcflush110 күн бұрын

  Me as a ML PhD watching and listening along for fun. Great job

• 
  @MelodySparkleroni49010 күн бұрын

  me as a n international relations student watching and i feel the same way, very fun to follow along (although idk whats going on)

• 
  @jacobrintamaki10 күн бұрын

  what specifically did you not understand? I was kind of bad in this video for background (since usually you need a physics or a math degree to work up to this), but I still don't like having my videos be unintelligible.

• 
  @MelodySparkleroni49010 күн бұрын

  @@jacobrintamaki No no, l think you did a great job talking through concepts and explaining everything, l just don't have the prerequisite knowledge to keep up. If anything that goes to show you're a good orator

• 
  @sandygrungerson11777 күн бұрын

  ML/AI/statistics/"data science"/etc is *not* science, it's a technical dead-end to keep the 120IQs occupied while the 140+'s do physics/maths. am i saying this to be offensive? no, you should be aware of the narrative traps created to corral or "tranche" you, just like all standardized testing.

• 
  @baby-maegu2810 күн бұрын

  14:50

• 
  @jacobrintamaki10 күн бұрын

  Time-Stamps: 0:00 Introduction 1:13 Finite Methods (Difference, Element, Volume) 1:24 Finite Difference Methods 1:41 Laplace and Poisson 2:45 Heat Equation and Wave Equation 3:42 Elliptic Equation In Divergent Form 4:50 Finite Element Methods 6:11 Finite Volume Methods 7:09 Scalar Conservation Laws 7:56 Entropy Solution 9:14 Conservative Schemes 10:11 Godunov Scheme 10:59 Gradient Flow/Descent Methods 11:26 Gradient Flow 12:37 GD, SD, MBGD 14:24 Connection With Optimization 15:00 Hamiltonian Flow 15:35 Basic Symplectic Operators 16:42 Monte Carlo Methods 17:17 Metropolis Algorithm 19:00 Variance Reduction 19:35 Importance Sampling 21:02 The Euler-Maruyama Method 22:10 The Milstein Method 22:29 The Feynman-KAC Method 24:09 Wavelets 25:07 Multiresolution Scheme 25:41 Conjugate Mirror Filters 26:19 Orthogonal Wavelets 27:37 References

• 
  @jacobrintamaki10 күн бұрын

  Time-Stamps: 0:00 Introduction 1:16 Elliptic PDEs/Laplace Equation 2:03 Poisson Equation 2:37 Green’s Function (Elliptic) 4:39 Parabolic PDEs 5:10 Heat Equation 6:14 Green’s Function (Parabolic) 7:33 Random Walks/Brownian Motion 8:25 Connection To Parabolic PDEs 9:26 Hyperbolic PDEs 10:12 Wave Equation 11:00 1-D Schrodinger Equation 11:42 Hamilton-Jacobi Equations 13:25 Conservation Laws 13:51 Shocks 14:38 Weak And Entropy Solutions 15:29 Lax = Oleinik Formula 15:57 Stochastic Modeling 16:19 Brownian Motion (Wiener Process) 17:25 Stochastic Integral 18:21 Ito’s Formula 19:03 Stochastic Differential Equations 19:39 Forward Kolmogorov Equation 20:22 Backward Kolmogorov Equation 20:52 References

• 
  @baby-maegu2810 күн бұрын

  thanks thanks you make my day.

• 
  @baby-maegu2810 күн бұрын

  I apreciate it. AAAAA make me down here.

• 
  @mrstopanimate12 күн бұрын

  Respect for being willing to revise !

• 
  @logan456512 күн бұрын

  This is awesome. Keep it up

• 
  @burnytech12 күн бұрын

  Lovely!

• 
  @ramanShariati13 күн бұрын

  LEGENDARY 🏆

• 
  @CitraCerita13 күн бұрын

  fun to learn all of these processes in a simple but concise explanation, nice vid!

• 
  @jacobrintamaki13 күн бұрын

  Time-Stamps: 0:00 Cooked 0:43 Atoms 0:55 Semiconductors 3:56 Transistors 8:16 Logic Gates 9:20 Flip-Flops 11:26 Registers 12:07 ALUs/Tensor Cores 14:22 Streaming Multiprocessors (SMs) 15:27 GPU Architecture 17:09 Instruction Set Architecture (ISA) 18:25 CUDA 19:51 PyTorch 21:31 Transformers 26:17 Fun Thought

• 
  @JJW8364115 күн бұрын

  That's a pretty interesting tour. I know that you said that you would add music to the vid if you had an editor but I think that the video is better without music. In case you're interested in one, kdenlive is good because its free and open source. (Open source means that the programming code is publicly available so someone can call out a virus or vulnerability. Others could request new things to be added). Or you could just use davinci resolve but I haven't used it yet. Or you could just buy one. Apart from the tour, why are the snacks stored in the same place as the used rockets? At my school, theres sometimes a guy who comes in and inspects the place to make sure that its "legal". I like to call them the IRS of pre-eng due to the teacher sometimes saying he could lose his job on the spot.

• 
  @jacobrintamaki15 күн бұрын

  Hey, thanks for the comment! Just so that everyone is fully aware, we are 100% cleared with EH&S (Environmental Health And Safety) and routinely do quarterly and sub-quarterly safety inspections for any and all procedures. The rockets shown in this video are just structural, they don't have active motors or engines in them.

• 
  @Barc0d315 күн бұрын

  damn that's interesting

• 
  @flazerflint16 күн бұрын

  looks like young tony starks engineering basement

• 
  @codenocode16 күн бұрын

  Great timing for me personall (I was just dipping my toes into A.I.)

• 
  @michellepark971316 күн бұрын

  This is SO EPIC

• 
  @jacobrintamaki15 күн бұрын

  my favorite current co-president apart from the other one

• 
  @nicholasdominici16 күн бұрын

  This video is my comp sci degree

• 
  @En1Gm4A17 күн бұрын

  Highest Signal to noise ever observed

• 
  @aniksamiurrahman63653 күн бұрын

  I'll say noise to singal. Not the other way around.

• 
  @tsugmakumo206417 күн бұрын

  i was talking with gpt-4o about exactly this abstraction layers from the atom until a compiler. So this video will be a great refresher.

• 
  @PRFKCT17 күн бұрын

  wait Nvidia invented GPUs? wtf

• 
  @d_polymorpha17 күн бұрын

  GPUs have only existed for about 25 years!🙂

• 
  @enticey16 күн бұрын

  they weren't the first ones, no

• 
  @NICKCIN17 күн бұрын

  sick

• 
  @user-ig8pp3st4v17 күн бұрын

  wow had no idea this is right behind Huang

• 
  @Nurof3n_17 күн бұрын

  you just got 339 subscribers 👍 great video

• 
  @boymiyagi17 күн бұрын

  Thanks

• 
  @sophiawisdom342917 күн бұрын

  Some thoughts as I watched the video: Tensor cores don't do FMAs, they do MMAs (matrix-multiply add). FMA is a different thing they can also do that typically refers to a *single* fused multiply-add. Kudos for mentioning they do the add though, most people skip over this. At 12:58 you have a slide with Register/Sram/L1$/SRAM/L2$/DRAM. All of these are made of SRAM. Under ISA you mention the ISA for a tensor core, which I don't think makes sense. The tensor core is within the SM and is called just like any other part of the chip like the MUFU. All of the stuff you put on the slide at 14:24 is also not part of the ISA as most people would understand it. Outputs also can't be written to memory (though as of Hopper they can be read from shared memory!). You're correct that CUDA is compiled to PTX and then SASS, but SASS probably doesn't stand for Source And Assembly (it probably stands for Shader Assembly but NVIDIA never specifies) and CUBIN is a format for storing compiled SASS. What you're saying is equivalent to "C gets optimized to LLVM IR then to armv9-a aarch64 then to ELF" on CPU. Ignoring Inductor, Torch does not compile pytorch into CUDA -- this is an important distinction that is meaningful for both Torch's strengths and weaknesses. It calls pre-existing CUDA kernels that correspond to the calls you make. For transformers, I find it somewhat confusing you're teaching encoder-decoder instead of decoder-only, but whatever. The dot product of things that are close would not be close to 1 -- the *softmax* of the dot product of things that are close would be close to 1. MHA is also not based on comparing the embeddings, but on comparing "queries" for each token to "keys" for each other token. The network *learns* specific things to look for. The addition is *not* about adding little numbers to it but about adding *the previous value* to it. The intuition is that attention etc. compute some small *update* to the previous value as opposed to totally transforming it. I think your explanation of MLP also leaves something to be desired -- there are already nonlinearities in the network you described (layer norm and softmax). It also doesn't do an FMA, but a matrix multiply. your explanation of the linear embedding at the end is confusing. Typically the unembedding layer *increases* the number of values per token because the number of tokens is larger than d_model. you say all the matrix and addition happen in the tensor cores, inside of the SM, whereas the intermediate stuff happens in the registers. All of the stuff "happens in the registers" in the sense that the data starts and ends there, but more correctly it happens in the ALUs or the tensor cores. When you say that DRAM hasn't kept up as much, DRAM is made of the same stuff as the SMs -- it's all transistors. You mention you would have to redesign your ISA -- the ISA is redesigned every year, see e.g. docs.nvidia.com/cuda/cuda-binary-utilities/index.html .

• 
  @d_polymorpha17 күн бұрын

  Hello do you know of any resources to dive deeper into this higher level intro video? Specially towards cuda/pytorch/ actual transformer?

• 
  @maximumwal16 күн бұрын

  Very good post, but Jacob's right about DRAM. DRAM also uses capacitors to store the bits, and then transistors for reading, writing, and refreshing the bit. In addition, the manufacturing process is quite different. Moore's law for DRAM has been consistently slower than Logic scaling, which is why nvidia pays 5x as much for HBM than the main die, and still, the compute : bandwidth ratio keeps getting more and more skewed every generation towards compute. Even SRAM, which is purely made of transistors, can't keep up because leakage gets worse and worse, and if you're refreshing it all the time, it's unusable. Logic is scaling faster both due to 1. physics and 2. better/larger tensor cores.

• 
  @sophiawisdom342916 күн бұрын

  @@maximumwal ah true, though i thought DRAM uses a transistor and a capacitor (?). I feel like you should expect they pay more for HBM than the die because the main die is 80B transistors but the HBM is 80GB*8 bits/byte=640B transistors+640B capacitors. HBM is also much more expensive than regular DRAM I believe, like $30 vs $5 per GB.

• 
  @maximumwal16 күн бұрын

  @@sophiawisdom3429 Yes, there's 1 transistor per capacitor, whose channel and gate connect to the bit and word lines. Branch education has a great video on this. as for HBM being roughly the same transistors/$: True, but they used to be much cheaper, because logic has tens of layers of wires/vias on top of the transistors at the bottom, vs just 2 simple layers of wires on dram. With b100 and beyond, HBM will be more expensive than logic on a transistor basis. There are many reasons for this, including the fact that smaller capacitors have to be refreshed more often, and the hard limits of memory frequency + bits per pulse (a100 -> h100 doubled bits per pulse, but lowered frequency, probably since it's harder to parse the signal at low power, but possibly because of greater resistance with thinner bitlines), which were previously leaned on to improve GB/s/pin, whereas on the die you can just build a larger systolic array/tensor core, and get more flops/(transistors * clock cycles), and increase clock frequency more easily, you just have to manage power. Right now we're stacking HBM with even more layers (8 -> 12 -> 16), and using more stacks (5 -> 8). Nvidia will eat the cost, and lower their margins. The normalizations + activations are soon going to use more gpu seconds than the matmuls. Everyone knows this, so tricks on the algorithms, scheduling, and hardware sides are being aggressively pursued to provide life support to Huang's law.

• 
  @CheeYuYang17 күн бұрын

  Amazing