CORRECTIONS: • At 5:22, instead of "more random" I probably should have said "less predictable," and instead of "less random, more deterministic," I probably should have said "more predictable," since it's only deterministic if one class has a value of 1 and all others 0, and other than that it represents randomness just with different distributions. Also, this is related to the context of when one is sampling from this distribution, for example when choosing the next character or word in a sentence when doing text generation. • At 16:51 I said, "It's basically the idea that as our networks are training, they're trying to push these logit values towards those Gaussian distributions," but this should only be taken loosely since the gradients don't point towards any single point. It would be more accurate to describe the direction of the gradient as being a weighted combination of both towards the subspace occupied by that Gaussian and towards the intersection of all the other subspaces, which can be seen by the slope of the gradient shown at 10:04.

@apophenic_

Жыл бұрын

Thank you!

After 30s you know that this lecture is worth its weight in gold.

@elliotwaite

2 жыл бұрын

Thanks!

thank you for taking the effort to produce this, it is super helpful!

@elliotwaite

3 жыл бұрын

Thanks, catthrowvandisc! Glad you found it helpful.

Awesome visuals! Thank you for a great work!

@elliotwaite

3 жыл бұрын

Thanks, Yuriy! Glad you liked it.

Awesome illustrations! Thanks

Well thought out and extremely informative. Thank you for making this.

@elliotwaite

3 жыл бұрын

Thanks!

This content is amazing, and the visualizations are so intuitive and easy-to-understand. Awesome video!

@elliotwaite

Жыл бұрын

Thanks, Harry! I appreciate it.

What you've done is just amazing!!

@elliotwaite

3 жыл бұрын

Thanks!

wow you make desmos sing! I love desmos as well. such as an amazing thing, incredible visualization tool. Great video, please keep making more!

@elliotwaite

3 жыл бұрын

Agreed, Desmos is great! Fun to use and helpful in gaining insights. Thanks, bArda26! I'll keep the videos coming.

Amazing video Elliot!!! Great insghts, keep it up.

@elliotwaite

3 жыл бұрын

Thanks, Wenceslao! Comments like this make my day.

very helpful, content was amazing😍

Amazing video, by far one of the best and most vidual explanations I have yet to see. Thanks for making such a great video. Can't wait to watch more of your videos.

@elliotwaite

2 жыл бұрын

Thanks!

So useful! Thanks a lot

I have no words for you !!! Awesome

Ultimate explanation. Thank you a million.

@elliotwaite

2 жыл бұрын

Thanks!

Best, most thorough explanation possible!

@elliotwaite

10 ай бұрын

Thank you!

I love you man, this video is what I needed. Much love and best of luck mate.

@elliotwaite

10 ай бұрын

Thanks!

I started a neural nets from scratch tutorial and your videos are amazing for support/understanding.

@elliotwaite

3 жыл бұрын

Thanks!

This is great! You have put so much effort into this to make it easy to understand

@elliotwaite

4 ай бұрын

Thanks!

Thanks so much for creating this thorough amazing explainer exposing the hidden details of softmax visually. Amazing!

@elliotwaite

7 ай бұрын

:) I'm glad you liked it.

Man. This is the best video I've seen on this topic.

@elliotwaite

3 жыл бұрын

Thanks!

Very helpful! The visualisation was very good!! Thank you...

@elliotwaite

Жыл бұрын

Thanks! Glad you liked it.

The best video I have ever seen about softmax

@elliotwaite

3 жыл бұрын

Thanks!

I particularly love the way you explain it in graphical calculator, which most youtubers won't dare to use. The whole ML operation is a math, love you videos!! please update us by uploading more videos like this!

@elliotwaite

10 ай бұрын

Thanks, I appreciate the comment. I hope to make more videos eventually.

Very well explained 👍

@elliotwaite

3 жыл бұрын

Thanks!

I was searching for intuitive explanation for Sigmoid and Softmax because they both have something in common for that something there are rare materials.And to my surprise Elliot you explained most of nuances in the concept if not all. I was hoping to find intuition on some other aspects in activation functions but sadly there you didn't made any videos for the last many years. I understand due to lack of time and probably nothing comes out of this hobby of yours you left making videos on technical content. But the way you have perceived these ideas have produced gold.

@elliotwaite

9 ай бұрын

Thanks. I'm glad you liked the explanation. I may make more videos in the future. The main reason I stopped is because I want to build an AI company, and I felt like making these videos was slowing down my progress towards that goal. But I think once I get my business going a little more, it will actually be beneficial to get back into making videos. But we'll see. Thanks again for the kind comment.

I just did a presentation with Desmos and I thought mine was good, but WOW props to you. Ur Desmos skills are insanely good man

@elliotwaite

2 жыл бұрын

Thanks! Yeah, I'm a big Desmos fan. It's a great learning tool.

Excellent !

Thank you so much.

very good explanation

Addition (shift) to all classes is changing the scores. For example we have 3 classes with values: 5, 3 and 2. So we can say: 5/10 = 3/10 + 2/10. After addition 1 to value of all calsses: 6/13 != 4/13 + 3/13. So addition must change scores. It is also easy to see if we add 1000 for example. So it will be 1005, 1003 and 1002. And score of each roughly become to 0.3333....

@anton_98

2 жыл бұрын

It was comment to this part of video: kzread.info/dash/bejne/q6iWu7SCfpO0ZMo.html.

@elliotwaite

2 жыл бұрын

The reason an addition to all inputs doesn't change their outputs is because instead of using the raw input values in the fractions, you exponentiate the inputs first before using them in the fraction: e^2 / (e^2 + e^3 + e^5) = e^1002 / (e^1002 + e^1003 + e^1005) Thanks for the question.

@anton_98

2 жыл бұрын

@@elliotwaite Thank you for explanation.

very helpful!

Great video. The point where you take the log of all these functions to make the loss seems like it could use a bit more motivation: The log makes sense for exponential-like sigmoids and if you're requiring a probabilistic interpretation, but perhaps there is some other function more appropriate for algebraic sigmoids? (Although I can't think of any that would produce the "nice" properties that you get from cross-entropy loss & logistic sigmoid, apart from maybe a piecewise scheme like "Generalized smooth hinge loss".)

@elliotwaite

3 жыл бұрын

Good point. I didn't go into the reasoning of why I chose the negative log-likelihood loss other than mentioning that it is the standard used for softmax outputs that are interpreted as probabilities. I wasn't sure of a concise way to explain it, so I thought I would just take it as a given without going into the reasoning. But maybe I could have at least pointed viewers to another resource that they could read for if they wanted to better understand why it is typically used any how other loss functions could also potentially be used. Thanks for the feedback.

THANK YOU

Thanks!

great stuff

awesome video, thank you

@elliotwaite

9 ай бұрын

:) thanks for the comment. I'm glad you liked it.

Superb, Excellent Elliot!!! BTW, which software you used?

@elliotwaite

3 жыл бұрын

Thanks, Snehotosh! For the 2D graph, I used Desmos, and for the 3D graphs, I used GeoGebra. They both have web-based versions and GeoGebra also has a native version. I posted links in the description to all the interactive graphs shown in the video if you want to get some ideas for how to use them.

Please keep doing it!

@elliotwaite

3 жыл бұрын

Thanks, Cankun! I plan to make more videos soon.

this is probably the best math video I have ever watched

@elliotwaite

Жыл бұрын

Thanks!

great video, many thanks! One question: around 5:00, you mentioned that getting probabilities together make them more random, and separating them more deterministic, I can see that graphically but I can't reconcile with the definition of variance which asserts the more spread of the points the more variance, maybe I'm mixing things up here??

@elliotwaite

2 жыл бұрын

Thanks, glad you liked the video. About your question, it is a bit counterintuitive at first, but I think the key to understanding how having more similar probabilities can lead to a more random outcome is remembering that the probabilities we are talking about are the probabilities for different outcomes. And when I say "random," I'm using it a bit vaguely and not in the same way as variance, which I explain below. So for example, if we were rolling a six sided die, we could think of plotting the bar heights of the probabilities of getting each of the six sides. If it were a weighted die that almost always rolled a 5, then the bar for 5 would be large and the others would be small, and we might say the die was not very random. In this case, the distribution of the value we'd get from rolling the die would also have a low variance with most of the distribution mass being for 5. On the other hand, for a fair die, the probabilities for the sides would almost all be equal with their bar heights all being about the same, and we might say the die was more random. In this case the distribution of the value we'd get from rolling the die would also have a higher variance with the probability mass distributed more evenly across the outcomes. To see the difference in my usage of the word "random" and the precise term "variance," let's consider a weighted die that rolls a 1 half the time and a 6 the other half. We might say that this die was less random than a fair die, but the distribution of the outcome values would actually have a higher variance than a fair die. And in fact it would be the way to weight the die to have the highest variance possible. So when I say "random" I just mean that we are less certain about which class will be the outcome, but if those classes correspond to specific values in a field (a 1d, 2d, 3d, or higher dimensional space), then the variance of the distribution of those values could be higher or lower depending on which values the classes correspond to. I hope that helps.

@alroy01g

2 жыл бұрын

@@elliotwaite Ah, yes you're right, those are different outcomes, appreciate the detailed explanation, thanks for your time 👍

Thanks Elliot, great video! 👍 Could you please clarify why the output space is evenly distributed among classes? Shouldn't there be more space for the dominant class? Or you consider that all the classes are equivalently dominant? Thanks 🙌

@elliotwaite

Жыл бұрын

So the visuals are actually showing all possible cases in one graph, both what the output would be when one class is dominant and what the output would be when the classes are equivalent. The spaces where one color is above the others is where that class is dominant and the place in the middle where the curves cross and are all equal height is what the output would be if the classes were equivalent. And the spaces are all the same size because the softmax function doesn’t give any preference to any of the inputs before knowing their values. Let me know if it is still unclear.

@my_master55

Жыл бұрын

@@elliotwaite oh, okay, thank you 👍 So what we see is just "the middle of the decision" and therefore it only looks like the classes are equal. Which means, this does not imply that the classes are like 50/50 everywhere in case of 2 classes, but at the intersection it indeed looks like they are 50/50. Hope I got it right 😊

@elliotwaite

Жыл бұрын

@@my_master55 I’m not sure what you mean by "the middle of the decision." But yes, at the intersection they are 50/50 in the 2 class example and 1/3, 1/3, 1/3 in the 3 class example.

Even without watching this video, I liked it! because I know this is really going to change my thought toward softmax function and I learn a lot through this video.

I didnt get why exactly the smaller curve itself is changing shape when you decrease / increase green value at 2:25. I guess going through this desmos will bring more clarity. Can you open source it? Or its already shared?

@elliotwaite

2 жыл бұрын

The link to the publicly accessible Desmos graph is in the description. The bottom graph is a copy of the top graph, but scaled down vertically so that if you add up the heights of all the dots, that total height has a height of 1. So when you increase the height of the green dot, that increases the total height of all the dots, so the bottom graph has to be scaled down even more to make that total height stay at a height of 1, which is why the bottom graph squashes down more as the height of the green value increases.

Hi, can I see your neural network training (at 12.12) in some environment such as Jupiter notebook?

@elliotwaite

3 жыл бұрын

I don't have a notebook of it, but you can get the code here: github.com/elliotwaite/softmax-logit-paths

why you have so few subscribers :

@elliotwaite

10 ай бұрын

I know 😥. I think I need to make more videos.

The comment (5:06) about temperature parameter is not accurate, when temp parameter

@elliotwaite

Жыл бұрын

I'm not sure I understand your comment. Can you clarify what you mean by "theta values"? In the video I was trying to say that increasing the temperature flattens the distribution (making it more uniform), and that lowering the termpature pushes the arg max up the others down (making it more like a spike for only the arg max probability). Are you saying that you think it is the other way around, or was there something else that you thought was inaccurate?

"To imagine 23-dimensional space, I first consider n-dimensional space, and then I set n = 23"

@elliotwaite

11 ай бұрын

Haha. For real though, that probably is the best way to do it.

Lol "I am not confortable with geometry"... If you aren't, imagine marginal people... Haha

@elliotwaite

3 жыл бұрын

😁 Thanks, I try. The higher dimensional stuff is tough, but I've slowly been getting better at finding ways to reason about it. But I'd imagine there are people, perhaps mathematicians who work in higher dimensions regularly, that have much more effective mental tools than I do.

@DiogoSanti

3 жыл бұрын

@@elliotwaite Maybe but higher dimensions are hypothetical as we are trapped in 3 dimensions ... the rest is just imagination hehe

Great video. Thank you