I've changed the output layer a bit... this: self.head_ln = nn.LayerNorm(emb_dim) self.head = nn.Sequential(nn.Linear(int((1 + self.height/self.patch_size * self.width/self.patch_size) * emb_dim), out_dim)) Then in forward: x = x.view(x.shape[0], int((1 + self.height/self.patch_size * self.width/self.patch_size) * x.shape[-1])) out = self.head(x) The downside is that you'll likely get a lot more overfitting, but without it the network was not really training at all.

@DeepFindr

Ай бұрын

Hi, thanks for your recommendation. I would probably not use this model for real world data as there are many important details that are missing (for the sake of providing a simple overview). I will pin your comment for others that also want to use this implementation. Thank you!

Keep making content like this, I am sure you will get a very good recognition in the future. Thanks for such amazing content.

This is very underrated channel. You deserve way more viewers!!

Awesome man!! You code and explain with such simplicity.

This channel is amazing. Please continue making videos!

You're awesome man!!! I clicked your video so fast, you're one of the my favorite AI youtubers. I work in the field and I think you have a wonderful ability of explaining complex concepts in your videos

@DeepFindr

11 ай бұрын

thanks for the kind words :)

Thank you! Very clear and informative.

Awesome! Thanks for excellent explanation!

Really great explanation. Nice visuals

@DeepFindr

Жыл бұрын

Much appreciated!

Very helpful video, thanks!

Great tutorial

감사합니다!!

There was a error on your published code but not in the video. attn_output, attn_output_weights = self.att(x, x, x) It should be attn_output, attn_output_weights = self.att(q, k, v) Anyway, thanks for sharing the video and code base. It helped me a lot while learning ViT

thank you

Cool video! What do you think about the implementation of ViT on signal processing (spectrogram analysis) , applied to audio for example. Which advantages could it have over the classic convolutional networks?

@sorvex9

Ай бұрын

Take a look at gpt4-omni to find out, lol

Awesome video! But I wonder if you reverse the order of LayerNorm and Multi-Head Attention? I think the LayerNorm should be applied after Multi-Head Attention but your implementation apply the LayerNorm before it.

@DeepFindr

11 ай бұрын

Hi! Thanks! There is a paper that investigated pre- VS post-layernorm in transformers (see here arxiv.org/pdf/2002.04745). The "Pre" variant seems to perform better as opposed to the traditional suggestion in the transformer paper. This is also what most public implementations do :)

Nice video! However, I think it's incorrect that you would get separate vectors for the three channels? This is not how they do it in the paper; there they say that the number of patches is N = HW/P^2, where H, W and P is the height and width of the original image and (P, P) is the resolution of each patch, so the number of color channels doesn't affect the number of patches you get.

Hope you could explain the swim transformer object detection in new video please

thank u ,

can someone help me with the training codes in the Google Colab link in the description?

Hey,great video. I have a question though. Isn't the entire point of 'pre' norm is that the normalization is applied before attention computation is performed? But from the code ,norm = PreNorm(128, Attention(dim=128, n_heads=4, dropout=0.)) , it seems like you are performing attention first and then normalizing aka post-norm. Please correct me if I'm wrong :)

@DeepFindr

9 ай бұрын

Hi! In the forward pass of the PreNorm layer is this line: self.fn(self.norm(x), **kwargs) So normalization is applied first and then the function (such as attention in this example) The line you are referencing is just the initialization, not the actual call Hope that helps :)

@chinnum9716

8 ай бұрын

@@DeepFindr stupid of me to not see that first. Thank you for the reply

Isn't the embedding layer redundant? I mean we have then the projection matrices meaning that embedding + projection is a composition of two linear layers.

Hello, first of all great tutorial video. I've tried running provided code for training, but after ~400 epochs loss is still the same (~3.61) and model always predicts the same class. Do you have an idea what might be a problem with it?

@DeepFindr

7 ай бұрын

Hi, have you tried a Lower learning rate? Also is the train loss decreasing or also stuck?

@user-px7zm2hy3c

7 ай бұрын

​@@DeepFindr Actually I've already found one bug in notebook. In forward method of Attention module, input is directly passed to MultiheadAttention bypassing linear layers. Changing learning rate doesn't affect training at all. Also, when training I've noticed that model's output converges to all zeros. I've checked gradients in network and it turns out that gradient flow stops at PatchEmbedding layer. All layers after it have non-zero gradients. Still don't know why this happens

@DeepFindr

7 ай бұрын

Thanks for finding this bug. But I actually think it's not super relevant for this issue - I experimented with the attention previously and tried both ways (with linear layers and without), that's how this bug was created in the first place. When I started the training back then the loss was definitely decreasing, but I didn't expect it to get stuck at some plateau. Typically when models predict always the same class there can be a couple of reasons. I already checked this: - Input data is normalized - Too few / too many parameters (I would recommend to count the model parameters to get a feeling for this) - Learning rate - SGD Optimizer (seems to work a bit better) - Batch size, I put it to 128 - Embedding size, make it a bit smaller After 100 epochs the Loss is also converging to 3.61, but the model is predicting different classes. Maybe the Dataset is not big enough. What about trying another dataset? Alternatively, try data augmentation. As stated in the video, transformers need to see a lot of examples.

I think there is a confusion between cls token and positional embedding? At 6:09?

can you please make a video on how to perform inference on VIT like googles open source vision transformer?

Is this better for the MNIST challenge compared to a simple conv network like LeNet

in 05:08. how we calculated that? when I calculated the patch shape I got a different result. Could someone explain that?

@chandank5266

Ай бұрын

yes exactly, I also have same doubt, for me its 192 instead of 324

@user-wm8xr4bz3b

25 күн бұрын

original image size was 144 (h) x 144 (w), after patch embedding, it was transformed to 324 patches, and each patch has 128 dimension size. 324 was derived from (144/8) x (144/8) = 18 x 18 = 324 patches. 8 is the patch size of each patch.

Is the Colab using cuda? IF so how can I tell if it is useing cuda

Where to get slides? Used in video

hi there I have used the code for binary class classification, but encountering the problem on accuracy , showing 100% accuracy only on label 1 and some times on label 2. So it would be helpful for me if u provide me any solution

@DeepFindr

Ай бұрын

Hi, please see pinned comment. Maybe this helps :)

Why are the positional embeddings learnable? It doesn't make sense to me

@trendingtech4youth989

6 ай бұрын

Bcoz, positional embedding represent the adress or position of image information of image patches

@cosminpetrescu860

6 ай бұрын

@@trendingtech4youth989 in "Attention is all you need", afaik, positioal embeddings are not learnable

@Omsip123

2 ай бұрын

​@trendingtech4youth989 so..they are given like the patches, why shall they be learned

@sohangundoju8940

Ай бұрын

Imagine you have a sentence "I made a pizza and put it in an oven, it was tasty". Here we know that it refers to pizza, but for a model, it could mean pizza is delicious or oven is delicious. Positional encodings are learnt based on each other, the position of it here is with respect to pizza and oven. Therefore it is a learnt parameter. Here the video mentions that it is doesn't have any significant advantage over numbering, but when you are trying to teach the model to identify features, it will identify feature and position of one patch wrt other patches, ergo making it a learnable parameter

@isaakcarteraugustus1819

Ай бұрын

It’s not really needed, in the original transformer paper have been tests with learnable pe and basic-not-learnable-sinusoidal pe and that did not make a big difference, gpt2 also uses learnable pe, nowadays it’s more RoPE doing the positional encoding

The video is great but the training in the code didn't work for the entire 1000 epochs. Despite the code looks logical, there is endless of things that can go wrong so I think it was better to do a tutorial with working ViT notebook.

@DeepFindr

Ай бұрын

Hi! I think this is because the Dataset is too small. Transformers are data hungry. It should work with a bigger dataset

@DeepFindr

Ай бұрын

Also have a look at the pinned comment, maybe that helps :)

And now sora uses the same algorithm. this video aged so well

@simpleplant606

3 ай бұрын

Sora is using DiT (Diffusion Transformer)