To learn more about Lightning: lightning.ai/ Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

In simple words, word embeddings is the by-product of training a neural network to predict the next word. By focusing on that single objective, the weights themselves (embeddings) can be used to understand the relationships between the words. This is actually quite fantastic! As always, great video @statquest!

@statquest

10 күн бұрын

bam! :)

Probably the most important concept in NLP. Thank you explaining it so simply and rigorously. Your videos are a thing of beauty!

@statquest

Жыл бұрын

Wow, thank you!

Josh; this is the absolutely clearest and most concise explanation of embeddings on KZread!

@statquest

2 ай бұрын

Thank you very much!

@davins90

Ай бұрын

totally agree

Can't believe this is free to watch, your quality content really helps people develop a good intuition about how things work!

@statquest

5 ай бұрын

Thanks!

this channel is the best resource of ML in the entire internet

@statquest

8 күн бұрын

Thank you!

This channel is literally the best thing happened to me on youtube! Way too excited for your upcoming video on transformers, attention and LLMs. You're the best Josh ❤

@statquest

Жыл бұрын

Wow, thanks!

@MiloLabradoodle

Жыл бұрын

Yes, please do a video on transformers. Great channel.

@statquest

Жыл бұрын

@@MiloLabradoodle I'm working on the transformers video right now.

@liuzeyu3125

Жыл бұрын

@@statquest Can't wait to see it!

Statquest is by far the best machine learning Chanel on KZread to learn the basic concepts. Nice job

@statquest

3 ай бұрын

Thank you!

So good!!! This is literally the best deep learning tutorial series I find… after a very long search on the web!

@statquest

8 ай бұрын

Thank you! :)

Thank you Josh, this is something I've been meaning to wrap my head around for a while and you explained it so clearly!

@statquest

11 ай бұрын

Glad it was helpful!

Wow!! This is the best definition I have ever heard or seen, of word embedding. Right at 09:35. Thanks for the clear and awesome video. You guy rock!!

@statquest

16 күн бұрын

Thanks! :)

I was struggling to understand NLP and DL concepts, thinking of dropping my classes, and BAM!!! I found you, and now I'm writing a paper on neural program repair using DL techniques.

@statquest

8 ай бұрын

BAM! :)

When I watched this,I have only one question which is why all the others failed to explain this if they are fully understood the concept?

@statquest

8 ай бұрын

bam!

@rudrOwO

5 ай бұрын

@@statquest Double Bam!

@meow-mi333

4 ай бұрын

Bam the bam!

Damn, when I first learned about this 4 years ago, it took me two days to wrap my head around to understand these weights and embeddings to implement in codes. Just now, I need to refreshe myself the concepts since I have not worked with it in a while and your videos illustrated what I learned (whole 2 days in the past) in just 16 minutes !! I wished this video existed earlier !!

@statquest

8 ай бұрын

Thanks!

Keep up the amazing work (especially the songs) Josh, you're making live easy for thousands of people !

@statquest

11 ай бұрын

Wow! Thank you so much for supporting StatQuest! TRIPLE BAM!!!! :)

Absolutely the best explanation that I've found so far! Thanks!

@statquest

Жыл бұрын

Thank you! :)

On of the best videos I've seen till now regarding Embeddings.

@statquest

6 ай бұрын

Thank you!

The phrase "similar words will have similar numbers" in the song will stick with me for a long time, thank you!

@statquest

Жыл бұрын

bam!

StatQuest is great! I learn a lot from your channel. Thank you very much!

@statquest

Жыл бұрын

Glad you enjoy it!

Thank you so much for this playlist! Got to learn a lot of things in a very clear manner. TRIPLE BAM!!!

@statquest

6 ай бұрын

Thank you! :)

This is the best explanation of word embedding I have come across.

@statquest

Жыл бұрын

Thank you very much! :)

haha I love your opening and your teaching style! when we think something is extremely difficult to learn, everything should begin with singing a song, that make a day more beautiful to begin with ( heheh actually I am not just teasing lol, I really like that ) thanks for sharing your thoughts with us

@statquest

Жыл бұрын

Thanks!

That was the first time I actually understood embeddings - thanks!

@statquest

8 ай бұрын

bam! :)

Hey Josh. A great new series that I, and many others, would be excited to see is bayesian statistics. Would love to watch you explain the intricacies of that branch of stats. Thanks as always for the great content and keep up with the neural-network related videos. They are especially helpful.

@statquest

11 ай бұрын

That's definitely on the to-do list.

@mamdouhdabjan9292

11 ай бұрын

@@statquest looking forward to it.

Great presentation, You saved my day after watching several videos, thank you!

@statquest

21 күн бұрын

Glad it helped!

BAM!! StatQuest never lie, it is indeed super clear!

@statquest

Жыл бұрын

Thank you! :)

This is one of the best sources of information.... I always find videos a great source of visual stimulation... thank you.... infinite baaaam

@statquest

Жыл бұрын

BAM! :)

your work is extremely amazing and so helpful for new learns who want to go into detail of working of Deep Learning models , instead of just knowing what they do!! Keep it up!

@statquest

Ай бұрын

Thanks!

Hopefully everyone following this channel has Josh's book. It is quite excellent!

@statquest

Ай бұрын

Thanks for that!

This video explains the source of the multiple dimensions in a word embedding, in the most simple way. Awesome. :)

@statquest

5 ай бұрын

Thanks!

the best video I saw about this topic so far. Great Content! Congrats!!

@statquest

6 ай бұрын

Wow, thanks!

It's so nice to google and realize that there is a StatQuest about your question, when you are certain of that there hadn't been one some time before

@statquest

Жыл бұрын

BAM! :)

highly valuable video and book tutorial, thanks for putting this kind of special tuts out here .

@statquest

11 ай бұрын

Glad you liked it!

I love all of your songs. You should record a CD!!! 🤣 Thank you very much again and again for the elucidating videos.

@statquest

5 ай бұрын

Thanks!

Thank you statquest!!! Finally I started to understand LSTM

@statquest

Жыл бұрын

Hooray! BAM!

Way better than my University slides. Thanks

@statquest

Жыл бұрын

Thanks!

Hey Josh, i'm a brazilian student and i love to see your videos, it's such a good and fun to watch explanation of every one of the concepts, i just wanted to say thank you, cause in the last few months you made me smile beautiful in the middle of studying, so, thank you!!! (sorry for the bad english hahaha)

@statquest

7 ай бұрын

Muito obrigado!!! :)

Thank you sir. Your explanation is great and your work is much appreciated.

@statquest

9 ай бұрын

Thanks!

I admire your work a lot. Salute from Brazil.

@statquest

6 ай бұрын

Muito obrigado! :)

Hey Josh! Loved seeing your talk at BU! Appreciate your videos :)

@statquest

Жыл бұрын

Thanks so much! :)

This is by far the best video on embeddings. A while university corse is broken down in 15minutes

@statquest

29 күн бұрын

Thanks!

Thank you so much Mr.Josh Starmer, you are the only one that makes ML concepts easy to understand Can you , please , explain Glove ?

@statquest

Жыл бұрын

I'll keep that in mind.

BAM! Thanks for your video, I finally realize what the negative sampling means ~

@statquest

Жыл бұрын

Happy to help!

You are a beautiful human! Thank you so much for this video! I was finally able to understand this concept! Thanks so much again!!!!!!!!!!!!! :)

@statquest

8 ай бұрын

Glad it was helpful!

Thank you Statquest!!!!

@statquest

4 ай бұрын

Any time!

Bro , i have my master degree in ML but trust me you explain it better than my teachers ❤❤❤ Big thanks

@statquest

8 ай бұрын

Thank you very much! :)

Great video! One suggestion is that you could expand on the Negative Sampling discussion by explaining how it chooses purposely unrelated (non-context) words to increase the model's accuracy in predicting related (context) words of the target word.

@statquest

11 ай бұрын

It actually doesn't purposely select unrelated words. It just selects random words and hopes that the vocabulary is large enough that the probability that the words are unrelated will be relatively high.

I love the way you teach!

@statquest

11 ай бұрын

Thanks!

Great video for explaining word2vec!

@statquest

Жыл бұрын

Thanks!

Thanks for enlightening us Master.

@statquest

7 ай бұрын

Any time!

Awesome as always. Thank you!!

@statquest

Жыл бұрын

Thank you! :)

This is an amazing video. Thank you!

@statquest

4 ай бұрын

Thanks!

Thank you so much for these videos. It really helps with the visuals because I am dyslexic… Quadruple BAM!!!! lol 😊

@statquest

7 ай бұрын

Happy to help!

Great video as always!

@statquest

Жыл бұрын

Thanks again!

Absolutely mind blowing and amazing presentation! For the Word2Vec's strategy for increasing context, does it employ the 2 strategies in "addition" to the 1-Output-For-1-Input basic method we talked about in the whole video or are they replacements? Basically, are we still training the model on predicting "is" for "Gymkata" in the same neural network along with predicting "is" for a combination of "Gymkata" and "great"?

@statquest

5 ай бұрын

Word2Vec uses one of the two strategies presented at the end of the video.

You ARE the Batman and Superman of machine learning!

@statquest

2 ай бұрын

:)

Wow, Awesome. Thank you so much!

@statquest

8 ай бұрын

You're very welcome!

Love this channel.

@statquest

11 ай бұрын

Glad to hear it!

Just incredible!

@statquest

11 ай бұрын

Thank you!

great stuff as usual ..BAM * 600 million

@statquest

Жыл бұрын

Thank you so much! :)

That was quite informative

@statquest

Жыл бұрын

BAM! Thank you so much for supporting StatQuest!!! :)

thanks for your tutorial!!!

@statquest

Ай бұрын

You're welcome!

Keep going statquest!!

@statquest

Жыл бұрын

That's the plan!

Awesome explanation..

@statquest

3 ай бұрын

Thanks!

Hi, I love your videos! They're really well explained. Could you please make a video on partial least squares (PLS)

@statquest

Жыл бұрын

I'll keep that in mind.

Extremamente didático! Parabéns.

@statquest

Ай бұрын

Muito obrigado! :)

Great Explanation. Please make a video on how do we connect the output of an Embedding Layer to an LSTM/GRU for doing classification for say Sentiment Analysis

@statquest

10 ай бұрын

I show how to connect it to an LSTM for language translation here: kzread.info/dash/bejne/fmx8rdmeiqy1nco.html

@tupaiadhikari

10 ай бұрын

@@statquest Thank You Professor Josh !

For those of you who find it hard to understand this video, my recommendation is to watch it at a slower pace and make notes of the same. It will really make things much more clear.

@statquest

5 ай бұрын

0.5 speed bam!!! :)

you da bessssst, saved me alota time and confusion :..)

@statquest

3 ай бұрын

Thanks!

It would also be nice to have a video about the difference between LM (linear regression models) and GLM (Generalized Linear Models). I know they're different but don't quite understand thAT when interpreting them or programming them in R. THAAANKS!

@statquest

Жыл бұрын

Linear models are just models based on linear regression and I describe them here in this playlist: kzread.info/head/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU Generalized Linear Models is more "generalized" and includes Logistic Regression kzread.info/head/PLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe and a few other methods that I don't talk about like Poisson Regression.

@mariafernandaruizmorales2322

Жыл бұрын

@@statquest Thanks Josh!! I'll watch them all 🤗

Machine learning explained like Sesame Street is exactly what I need right now.

@statquest

11 ай бұрын

bam!

so goooood! Thank alot!

@statquest

Жыл бұрын

Glad you like it!

the best channel ever.

@statquest

9 ай бұрын

Double bam! :)

My favourite topic its magic. Bam!!

@statquest

Жыл бұрын

:)

Thank you Josh for this great video. I have a quick question about the Negative Sampling: If we only want to predict A, why do we need to keep the weights for "abandon" instead of just ignoring all the weights except for "A"?

@statquest

5 ай бұрын

If we only focused on the weights for "A" and nothing else, then training would cause all of the weights to make every output = 1. In contrast, by adding some outputs that we want to be 0, training is forced to make sure that not every single output gets a 1.

Great vid. So your going to do a vid on transformer architectures? That would be incredible if so. Btw bought your book. Finished it in like 2 weeks. Great work on it!

@statquest

Жыл бұрын

Thank you! My video on Encoder-Decoders will come out soon, then Attention, then Transformers.

@thomasstern6814

Жыл бұрын

@@statquest When the universe needs you most, you provide

Thank you very much for your excellent tutorials! Josh. Here I have a question, at around 13:30 of this video tutorial, you mentioned to multiply by 2. I am not sure why 2? I mean if there are more than 2 outputs, will we multiply the number of output nodes, instead of 2? Thank you for your clarification in advance.

@statquest

11 ай бұрын

If we have 3,000,000 words and phrases as inputs, and each input is connected to 100 activation functions, then we have 300,000,000 weights going from the inputs to the activation function. Then from those 100 activation function, we have 3,000,000 outputs (one per word or phrase), each with a weight. So we have 300,000,000 weights on the input side, and 300,000,000 weights on the output side, or a total of 600,000,000 weights. However, since we always have the same number of weights on the input and output sides, we only need to calculate the number of weights on one side and then just multiply that number by 2.

@surojit9625

9 ай бұрын

@@statquest Thanks for explaining! I also had the same question.

@jwilliams8210

5 ай бұрын

Ohhhhhhhhh! I missed that the first time around! BTW: (Stat)Squatch and Norm are right: StatQuest is awesome!!

Thanks!

@statquest

5 ай бұрын

Hooray!!! Thank you so much for supporting StatQuest!!! TRIPLE BAM! :)

Please make a video about the metrics for prediction performance: RMSE, MAE and R SQUARED. 🙏🏼🙏🏼🙏🏼 YOURE THE BEST!

@statquest

Жыл бұрын

The first video I ever made is on R-squared: kzread.info/dash/bejne/ZHWFrc-wYZfTeLA.html NOTE: Back then I didn't know about machine learning, so I only talk about R-squared in the context of fitting a straight line to data. In that context, R-squared can't be negative. However, with other machine learning algorithms, it is possible.

watched this video multiple times but unable to understand a thing. I'm sure I am dumb and the Josh is great!

@statquest

5 ай бұрын

Maybe you should start with the basics for neural networks: kzread.info/dash/bejne/daWDyMttYa_MdNo.html

Nice!

@statquest

Жыл бұрын

Thanks!

Thanks ❤

@statquest

11 ай бұрын

:)

Hello. Thank you very much. Great, great video. I have a question. In the negative sampling procedure we never use A = 1 as input at any step in the training process. I am wondering about the time the embeddings for A are trained. I can see how the weights for A at the right of the activation functions are trained, but not for the weights at the left. I can see that because we use a lot of training steps, in some moment A will be a word we don't want to predict at the input; therefore the embeddings for A will change, however, the prediction won't be A for those steps.

@statquest

Жыл бұрын

Why would we never use "A = 1" in training?

funny and very nicely explained.

@statquest

8 ай бұрын

Thanks! 😃

Thank you so much for the video! I have one question, at 15:09, why we only need to optimize 300 steps? For one word with 100 * 2 weights? not sure how to understand the '2' as well.

@statquest

8 ай бұрын

At 15:09 there are 100 weights going from the word "aardvark" to the 100 activation functions in the hidden layer. There are then 100 weights going from the activation functions to the sum for the word "A" and 100 weights going from the activation functions to the sum for the word "abandon". Thus, 100 + 100 + 100 = 300.

@neemo8089

8 ай бұрын

Thank you!@@statquest

Great video! Was just wondering why the output of the softmax activation at 10:10 are just 1 and 0s. Wouldn't that only be the case if we applied ArgMax here not SoftMax?

@statquest

Жыл бұрын

In this example the data set is very small and, for example, the word "is" is always followed by "great", every single time. In contrast, if we had a much larger dataset, then the word "is" would be followed by a bunch of words (like "great", or "awesome" or "horrible", etc) and not followed by a bunch of other words (like "ate", or "stand", etc). In that case, the soft max would tells which words had the highest probability of following is and we wouldn't just get 1.0 for a single word that could follow the word 'is'.

@MadeyeMoody492

Жыл бұрын

@@statquest Ohh ok, that clears it up. Thanks!!

Hi Josh, thank you for your excellent work! Just discovered your videos and consuming like a pack of crisps. I was wondering about the desired output when using the skip-gram model. When we have a word as input, the desired output is to have all the words found within the window size on any sentence of the corpus activate to 1 at the same time on the output layer, right? It is not said explicitly but I guess it is the only way it can be.

@statquest

7 ай бұрын

The outputs from a softmax function are all between 0 and 1 and add up to 1. In other words, softmax function does not allow more than one output to have a value of 1. See 12:16 for an example of outputs for the skipgram method.

@guillaumebarreau

7 ай бұрын

@@statquest, thanks for your prompt reply! You are right, I didn't look carefully enough. I guess I got confused because after watching the video, I read other sources which seem to consider every skip-gram pair as a separate training example, which confused me.

Amazing lecture, congrats. The audio was also made from an NPL (Natural Language Processing), right?

@statquest

Ай бұрын

The translated overdubs were.

That's awesome! But how would the multilingual word2vec be trained? Would the training dataset simply include corpus of two (or more) languages? or would additional NN infrastructure be required?

@statquest

Жыл бұрын

Are you asking about something that can translate one language to another? If so, then, yes, additional infrastructure is needed and I'll describe it in my next video in this series (it's called "sequence2sequence").

@user-rj6wc7bm8x

Жыл бұрын

@@statquest not exactly, it's more like having similar words from multiple languages to be mapped within the same vector spaces. so for example King and "King" in French, German and Spanish - would appear to be the same.

@statquest

Жыл бұрын

@@user-rj6wc7bm8x Hmmm.. I'm not sure how that would work because the the english word "king" and the Spanish translation, "rey", would be in different contexts (For example, the english "king" would be in a phrase "all hail the king", and the spanish version would be in a sentence that had completely different words (even if they meant the same thing).

This guy really loves Troll 2!

@statquest

4 ай бұрын

bam!

Awesome video! This time, I feel I miss one step through. Namely, how do you train this network? I mean, I get that we want the network as such that similar words have similar embeddings. But what is the 'Actual' we use in our loss function to measure the difference from and use backpropagation with?

@statquest

2 ай бұрын

Yes

@balintnk

2 ай бұрын

@@statquest haha I feel like I didn't ask the question well :D How would the network know, without human input, that Troll 2 and Gymkata is very similar and so it should optimize itself so that ultimately they have similar embeddings? (What "Actual" value do we use in the loss function to calculate the residual?)

@statquest

2 ай бұрын

@@balintnk We just use the context that the words are used in. Normal backpropagation plus the cross entropy loss function where we use neighboring words to predict "troll 2" and "gymkata" is all you need to use to get similar embedding values for those. That's what I used to create this video.

Fantastic video! How do you apply in powerpoint the style of "pencil-written" boxes?

@statquest

Жыл бұрын

I use Keynote, and it's one of the default line types.

@robott12

Жыл бұрын

@@statquest Thanks!

Can you do GloVe? i really enjoyed Word2Vec it will be great to see how GloVe works...how factorization based method works. Thank you for this amazing content!

@statquest

3 ай бұрын

I'll keep that in mind.

i understood word embedding in first 10 seconds

@statquest

6 ай бұрын

bam!

I'm not sure if I understood correctly. Have those millions of word embeddings been preprocessed and are public? Or ar they dependent on context? I need to do a project on word clustering of movie genres and I'm not sure if this is my way to go. Any help is appreciated!

@statquest

Жыл бұрын

I'm not sure this is the way to go either - these specific embeddings are usually used for processing natural language. However, you can download some publicly available embeddings here: fasttext.cc/docs/en/crawl-vectors.html

Hello Josh, thanks for your video.May I know if we could use 3 neuron network to predict the next words?

@statquest

6 ай бұрын

Sure

Hi josh firstly thank you for all your videos. I had one doubt , in skip gram what will be the loss function on which the network is been optimized, in CBOW i can see that cross entropy is enough

@statquest

5 ай бұрын

I believe it's cross entropy in both.

Thank you so much for this video. Could you do something like this for audio embedding as well? or how could we merge (do fusion) audio and text embedding? I really appreciate it.

@statquest

3 ай бұрын

Unfortunately, I'm not familiar with audio embedding.

I appreciate the knowledge you've just shared. It explains many things to me about neural networks. I have a question though, If you are randomly assigning a Value to a word, why not try something easier? For example, In Hebrew, each of the letters of the Alef - Bet is assigned a value. these values are added together to form a sum of a word. It is the context of the word, in a sentence that forms the block. Sabe? Take a look at Gamatra, Hewbew has been doing this for thousands of years. Just a thought.

@statquest

5 ай бұрын

Would that method result in words used in similar contexts to have similar numbers? Does it apply to other languages? Other symbols? And can we end up with multiple numbers per symbol to reflect how it can be used or modified in different contexts?

@kimsobota1324

5 ай бұрын

I wish I could answer that question better than to tell you context is EVERYTHING in Hebrew, a language that has but doesn't use vowels, since all who use the language understand the consonant-based word structures. Not only that, but in the late 1890s Rabbis from Ukraine and Azerbaijan developed a mathematical code that was used to predict word structures from the Torah that were accurate to a value of 0.001%. Others have tried to apply it to other books like Alice in Wonderland and could not duplicate the result. You can find more information on the subject through a book called, The Bible Code, which gives much more information as well as the formuli the Jewish Mathameticians created. While it is a poor citation, I have included this Wikipedia link: en.wikipedia.org/wiki/Bible_code#:~:text=The%20Bible%20code%20(Hebrew%3A%20%D7%94%D7%A6%D7%95%D7%A4%D7%9F,has%20predicted%20significant%20historical%20events. The book is available on Amazon if you find it peaks your interest. Please let me know if this helps. @@statquest

@kimsobota1324

5 ай бұрын

@starquest, I had not heard from you about the Wiki?

Hi Josh, great video. I have one question, how are the 2-20 words selected for being dropped while doing negative sampling

@statquest

Жыл бұрын

This is answered at 13:44. We can pick a random set because the assumption is that when the vocabulary is large, the chances of selecting a similar word are small. And I believe you select a different subset each iteration, so even if you do pick a similar word, the long term effects will not be huge.

@Rex389

Жыл бұрын

@@statquest Got it. Thanks