Implementation video: kzread.info/dash/bejne/m42mk9qTnpWcgps.html

This was beautifully explained. As someone who relies exclusively on self learning from online materials, the mathematical barriers of differig notations and overly complex wording is the most time consuming challenge. Thank you for such a distilled explanation, with only the notation and wording that makes sense for an intuitive and initial dive understanding. Subscribed!

@chrisalexiuk

10 ай бұрын

Thanks so much for your kind words.

@jacobjakubek1076

2 ай бұрын

I'm not kidding feed it to ChatGPT, helps me a ton

This is the first video I watched from your channel, and I loved it! now I have 12 tabs open, to watch rest of your videos. Simply Amazing! (and WOW, I could not imagine LoRA can be so good. it will save me a ton of resources in developing the product I am working on.

The high-level intuition you gave at the end of the video was great. As a mathematician I'm aware of the theory behind low-rank decomposition and the classic applications, but the way it is applied in the context of LLMs is interesting.

Thank you for thus nice, deep yet easy to follow explanation of LoRA. Nice job

Thanks for the great intro to LoRA. I liked your graphics and your take-aways, also you energetic presentation :)

I particularly appreciate the depth of research and preparation that clearly goes into this video. It's evident that you're passionate about the topics you cover, and your enthusiasm is contagious. Your dedication to providing accurate information while maintaining an accessible and entertaining format is commendable.

Best explanation of the whole KZread so far. Great job!

Love your enthusiasm and great explanation!

Thank you so much Chris! This was an awesome video and has stopped me from going down the fine-tuning rabbit hole! Just dipping my toe into AI so it’s really great to find an informative channel like yours!

@chrisalexiuk

Жыл бұрын

Fine-tuning (full parameter) can still be a great idea! But if you're just dipping in, LoRA is great!

The AutoGPT team is learning about LoRA and I recommended this because it is such a clear explanation. Thanks for the awesome resource!

@chrisalexiuk

Жыл бұрын

Thanks so much!

I came across this video a month back, at that time, I didn't understand your excitement, though I understood the technique. Presently, I have better understanding of LLMs, and how they are trained and fine tuned for down stream tasks, and now I share your excitement.

Love your clear and passionate explanations!!!

@chrisalexiuk

Жыл бұрын

Thanks!

Fantastic video and a really concise and well-considered explanation. Thanks!!!

Really simple explanation without skimming through the intuition and other important details. Subscribed for more content like this :)!

@chrisalexiuk

10 ай бұрын

Thanks so much!

One of the BEST explanations!!

Man that video is fireee! Thank you for your work!

Woow the best explanation I've found so far.Coming from mathematics background it's really amazed me. Thank you

Love you Chris! It really helps me a lot!

Great explanation! Keep up the great work!!

Great explanation and first video I've seen from your channel. Somehow, I had the impression that you have 624K subscribers. Was shocked to see there's no actual "K" in there, when browsing your whole video history and seeing your channel is only ~2 months old. You'd deserve that "K" from my pov. Looking forward to the implementation video! (new subscriber)

@chrisalexiuk

Жыл бұрын

Thank you so much!

Really a wonderful explanation. Best I've seen so far

@chrisalexiuk

Жыл бұрын

Thanks so much, Shawn!

bears repeating that you should continue doing these videos, wish I had your communication skillz

Thanks a lot for this amazing explanation. I am fine tuning Mixtral 8x7B and using QLoRA have been able to perform test runs on Colab Pro using A100 machine.

Thank you very much. I didn't understand the paper so this helped me a lot. I'd be curious to hear about some of the limitations people are experiencing. I know it's not all completely characterized yet because this is new. I've noticed that Lora can redirect the type of output that I get but it's not very good at adding information to the model. This makes sense

Dude, you're great at explaining stuff. 👍

@chrisalexiuk

Жыл бұрын

Thanks!!

Amazing explanation, Thank you!

Amazing explanation! Thankyou😊

Beautifully explained my friend thabk you very much!

Thank you for the good explanation of LoRA! Super!

Thanks a lot for this video! This is the first time I see a good explanation on this LoRA thing! 14:45 One minor note, is that it would indicate that the model has a low intrinsic info only if you could get rid of the original weights and just stick to the lora. That is, during the lora finetune training, if you could get away with while decaying the original (non-lora) weights down to zero. So I think that what has a low intrinsic info is "what you have to change from the base model" for your finetuning needs - but not the base model itself.

You explained this very well. thanks!

Thank you for the explanation, It helps me a lot.

Absolutely brillant video!

the best LoRA explanation I ever get. Better than ChatGPT.

I got here thinking this would be explaining how my waifu appears in the pretty image but instead I got a precise mathematical description of how this system works. Very good!

@chrisalexiuk

Жыл бұрын

>

love the enthusiasm!

thanks man, love the genuine!

great work dude please do some more videos god bless you

Chris, actually, what you are describing when you relate to this kind of application on the attention matrix, is specifically at inference, because the attention matrix is constantly being fine tuned (the only part of the GPT model to be constantly fine tuned at inference). This will enable to actually construct an AI that will enjoy the ability to think long term, since this will make use of so much less memory, so no need to reset the attention matrix so often. This, however, could be dangerous🙂.

Amazing video! Thanks

Incredible video, just subscribed to your channel!

You are really good at explaining it. Keep it up

@chrisalexiuk

Жыл бұрын

Thank you very much!

Thanks. This was very useful

Thank you Chris .

this saved me a ton of time. thank you!

@chrisalexiuk

Жыл бұрын

Great to hear!

amazing, thank you so much

Wow, what a great video!

Dude. This video is amazing

@chrisalexiuk

6 ай бұрын

Thank you! That means a lot!

great explanation!!

@chrisalexiuk

Жыл бұрын

Thanks, Praveen.

this was explained so well. why can't the authors write their paper like the way you explained it.

@chrisalexiuk

Жыл бұрын

It's certainly easier to distill information after they did the important work of creating it!

@mohammedhelal5778

Жыл бұрын

The explanation was good, but the paper also had a decent explanation.

@ko-Daegu

Жыл бұрын

just cuz you know something doesn't necessary mean you can explain it clearly to others

@nicolas_boumal

Жыл бұрын

Also, papers, talks, lectures, videos etc. are different forms of communication, consumed differently and also aiming for different (complementary) goals. Papers have the difficult job of conveying both Intuition about new ideas while also detailing all of the technicalities about it and connecting them to a vast literature, rigorously substantiating pros and cons, all under a page limit constraint. Talks/videos have the difficult job of truly making the intuition and potential impact of an idea "pop", with the advantage that they can drop many of the other concerns above. We need both.

@anshumansabath4362

Жыл бұрын

It takes 'work' to come up with good explanations - which may or may not be the priority at the time of writing (the first draft) of the paper.

Great explanation! I have gained an intuitive insight from your video. Thank you. Looking forward to your new videos! It seems that there is a missing x in the formula on the slide LoRA Explained (Cont.), which h should equal to w0x + wAwBx, if I understand your explanation correctly.

@chrisalexiuk

Жыл бұрын

You 100% correct! Thanks for catching that!

Oh my God, thank you very much for this.

Couple of basic questions: Is the 'h' or embedding layer an implementation detail related to LoRA? I thought the embedding layer is only used for the initial conversion of tokens into vectors during input feeding in the beginning Does LoRA only apply to the finetuning or training process? To prepare for inference, I'd imagine we'd combine the pretrained weights with the delta matrix and store that resulting matrix right? I wasn't really sure what you meant by the whole swapping out of weights for different problem types, because I assumed we just store the resulting weight matrix after training is complete.

bless up nice work thank you

Thank you so much for this video. The explanation is super clear. I do have a question though. When we reduce the WA and WB, does that wind up causing a lower fidelity in terms of the quality of inference results? Of course the difference is negligible, otherwise Lora wouldn't be useful, but I wonder if there is a loss, to what degree that might have a cumulative effect across many operations over time? In other words, yes, it's super that we are saving time and money, but is there a sense of at what cost overall? Thanks again for a great video!

@chrisalexiuk

Жыл бұрын

For the most part, the fidelity of LoRA hinges on the fact that the downstream tasks are intrinsically low dimensional - and so can be represented by low-rank weight matrices. The authors claim that LoRA can *fully* represent the downstream task without the full weight matrix. You're absolutely correct to identify that cumulative error could be a determining factor in whether or not you use LoRA to fine-tune, but in most cases (from available literature), the savings on both the training and potential savings on the inference side outweigh the minimal, if any, error accumulation.

Thank you so much! I'm working on a project to develop a chatbot for student advisory services and i am contemplating between two approaches: Fine-Tuning and Retrieval-Augmented Generation (RAG). Here are the key tasks the chatbot needs to address: Answering general queries about university courses and requirements. Providing personalized advice on study plans and schedules. Assisting with administrative processes (enrollment, documentation, etc.). Offering support for common academic and personal challenges faced by students. Given these responsibilities, which approach would be more suitable? Fine-Tuning could offer more precise, tailored responses, but RAG might better handle diverse, real-time queries with its information retrieval capabilities. Any insights or experiences with similar projects would be greatly appreciated!

@AlexG4MES 8 months ago This was beautifully explained. As someone who relies exclusively on self learning from online materials, the mathematical barriers of differig notations and overly complex wording is the most time consuming challenge. Thank you for such a distilled explanation, with only the notation and wording that makes sense for an intuitive and initial dive understanding. Subscribed!

Thank you

this is a masterpiece

Hello ... new to the field of llm training .. thanks for putting up these two videos .. However , I am a bit confused by the comment ' ...you can do it during inference time' ? As per my understanding the weight updates are done during fine tuning ... and they are later used during inference ... if the task is changed we just revert back to the pertained weight by getting rid off the weight update for current task .. and fine tune the model based on the new task to get the new weight update ... the new update are then again used later during inference .. so the weight updates are during fine tuning only ... which I think why the authors mentioned that batch processing is not obtained by LoRA (base) ... though possible and difficult ... may be there is some future version where its implemented ? I am not sure .... but please correct me if I am conceptually wrong ...

thank You

LoRA is a game changer

Hi a question, can we use lora to just reduce the size of a model and run inference, or we have to train it always?

how is Lora fine-tuning track changes from creating two decomposition matrix? How the ΔW is determined?

The one issue I have had is that this causes memory footprint to grow. But it sounds like you should be able to merge it into the base model at the end to keep the same footprint. Maybe that is something for me to try. I wonder if this low rank decomposition can be used for model distillation. Instead of just quantizing weights.

cool video tahnk you!

Nice!

So it's basically a linear algebra trick. Kinda sorta like pricipal component analysis. Very interesting. I can see this is essentially lossless compression post-training, but I have to wonder about subtle effects on the quality of the model during training as it's left with fewer parameters to discover features.

@chrisalexiuk

Жыл бұрын

LoRA does need to *learn* the decomposed matrices - but otherwise, what you said about the subtle effects is correct. The assumption is that the downstream tasks are low rank, which mostly holds true.

@alexisravanel5275

Жыл бұрын

Are they low ranked or is it the weight update ? When added to original weights maybe it's not low ranked anymore?or do I make a mistake? Ilk read the paper to be sure 😅. Good vid!

Great!

Please make a playlist on llm and different architecture and there implementation..🙂

Could you explain why this saves memory? Don't you need the pre-trained weights in backprop to calculate the difference matrixes and during the forward pass?

@chrisalexiuk

6 ай бұрын

We only need to pass through the frozen weights, which means we don't need them in the optimizer. That is where the significant memory load reduction comes from.

Отличный туториал)

@chrisalexiuk

6 ай бұрын

Thanks!

I wonder why this is such a recent and new concept. Matrix decomposition applied to machine learning models. Like, matrix decomposition has been known to work really well for so many different applications and tasks. How is it possible that we only thought about applying it to machine learning recently? Also, the authors talk about fine-tuning with low rank adaptation, but why are we not assuming low rank for the whole training process as well (meaning NOT from pretrained). If it's worth it to decompose a model before starting to train it, then the implications are huge for machine learning as a whole.

@chrisalexiuk

Жыл бұрын

It's interesting, for sure - but I believe the main issue is that people assumed you needed the full rank (and for the generalized tasks you do). It's only once we fine-tune that the idea that downstream tasks are low-dimensional creeps in. I'm not certain about full training; but I'd imagine someone is trying that - and if not...it's a perfect thing for you to try!

@clray123

Жыл бұрын

I suspect the people who do ML and the people who do maths are not exactly the same group.

Hi Chris, just trying to understand. Assume that I have two downstream tasks, and I have two adapters trained for these tasks. During inferencing and I want to use both of them simultaneously. Will LoRA allow me to do this with one base model along with the two different adapters ? Basically, can I use multiple adapters simultaneously ?

@chrisalexiuk

Жыл бұрын

You can basically do exactly that.

@gagangayari5981

Жыл бұрын

@@chrisalexiuk I want two different outputs for the two adapters. Not trying to fuse them.

First of all, thanks for the video. I know its just a minor thing but still for the mathematical correctness, the equation in the LoRA explained slide should be updated as: h = W_o x + W_A W_B x Annoying reviewer1 out ! :)

@chrisalexiuk

11 ай бұрын

Yes, you're absolutely correct! Another person also suggested this - though I'm not sure how to edit the video!

Hmmm.. Isn't it also extremely faster inference, due to that little thing called "the associative law of matrix multiplication"?

love your explanation it helped me a lot

But dW matrix changes with each propagation. When do we determine how many dimension is needed to represent dW?

@chrisalexiuk

Жыл бұрын

While the contents of dW change - the shape does not. We don't really determine how many dimensions we need ahead of time, or empircally (yet) and intead select a "reasonable" r (rank) based on overall generation after fine-tuning.

What's the difference between less parameters, and low intrinsic weights? because weights are parameters of Neural Net isn't it?

Follow up video on LyCORIS locon?

@15:28, there is nothing great about adding the extra LoRA parameters to the weights that makes it easier to swap the behaviour of the model at inference time because the difference between adding the matrices and loading the entirely new weight matrix from different finetuned models to the model architecture is negligible.

@chrisalexiuk

Ай бұрын

I actually think that winds up being largely incorrect, looking at platforms like Gradient - and initiatives like multi-LoRA (LoraX, etc), seem to be a testament to that.

Why does deltaW need to be represented by both WA x WB? Why couldn't it be represented using just smaller matrix?

@chrisalexiuk

6 ай бұрын

In order to preserve the original shape of the weights and to avoid needing to change the model architecture!

what does the author mean by "parameter budget"? which in this case is 18M.

@chrisalexiuk

Жыл бұрын

Artificially limiting the number of trainable parameters. In the specific example in the paper they were choosing between adapting Wq by itself, Wq & Wv, or all of them. So we know that choosing lower values for rank (r) means less parameters - and the idea is that if we have Wq by itself, we can choose a higher r and have the same number of parameters than if we have Wq & Wv together. Let me know if that explains things well!

It doesn't train the entire Lora model, so I came up with an idea to divide the model and train each part on Lora in each epoch. Wouldn't this approach, which requires less RAM like Lora and involves full fine-tuning, yield the same results?

@chrisalexiuk

11 ай бұрын

It does train all of the attention weights that have been adapted by the LoRA model! We target a specific module, but that module is represented many times throughout the model's architecture.

What’s with the latency, I didn’t get it. Can’t we swap with traditional fine tuning, we store delta_W but it’s just way bigger than A & B.

@chrisalexiuk

Жыл бұрын

While we could swap them technically, yes - the massive reduction in size of the LLaMA weights means we're only loading ~10-20MB of information, versus potential GBs of data. That size difference could be the difference between ~1ms additional latency versus 50-100ms or more of additional latency. Additionally, the changes in traditional fine-tuning are not stored as delta_W, and so it would be an extra process to be able to do so in the first place.

@creativeuser9086

Жыл бұрын

@@chrisalexiuk oh, so in the case of LoRA the operation W+delta_W will require 10M operations while in the traditional fine-tuning, it would require 1B+ operations (depending on the size of the delta_W_traditional), right? Also, why can't we preload all the different versions of the model (W_original, W_finetuned_1, W_finetuned_2, etc.). Is it strictly because it will require lots of storage?

sub’d and even hit the spam bell

how is Lora different from PEFT? or are they related?

@chrisalexiuk

Жыл бұрын

LoRA is a subset of PEFT. So it's a method of Parameter Efficient Fine-tuning!

If the key and query matrices are of dimensions m×n but have rank r=max(rank_k, rank_q) in the pretrained net then why don't we transform these matrices into m×r matrices?

@chrisalexiuk

Жыл бұрын

Essentially that's what LoRA is doing! We decompose the MxN matrix to a Mxr and an rxN matrix pair. I feel as though I likely misunderstood your point, though, so would you mind expanding a bit on it?

@clray123

Жыл бұрын

@@chrisalexiuk I think the question is if LoRA is so great and the tweaked source matrices have so little information in them as to allow it to work, why don't we dispose of the big pretrained matrices altogether and replace them with small ones. In other words, why keep a huge model if you could use a much smaller one.

@alexisravanel5275

Жыл бұрын

Its delta w that is low ranked, not W. Model got info that we should keep. It's the delta that can be condensed. (If I got it right)

@chrisalexiuk

Жыл бұрын

@@clray123 It's actually the case that only the *downtstream* tasks have intrinsically low dimensionality (and the assumption the LoRA paper makes is that they by extension have intrinsically low rank) So when it comes to the "full" model - we need that large rank! Hopefully that answers your question.

@clray123

Жыл бұрын

@@chrisalexiuk Good points, thank you for answering.

Holy shit this is huge

what is "prefix-based approaches"?

@chrisalexiuk

Жыл бұрын

aclanthology.org/2021.acl-long.353/ arxiv.org/pdf/2110.07602.pdf Here are some papers that go over the idea!

Is it feasible to exchange the pre-trained model as well, as you explained for LoRa in 16:40? By the way, amazing content! Subscribed!

@chrisalexiuk

Жыл бұрын

It is unfortunately not going to be as easy to hots-wap them! Also, the LoRA weights will be architecture specific, and so won't work on with a different base model.

@omarnomad

Жыл бұрын

@@chrisalexiuk I'd be awesome to work on some meta-architecture for such hot-swap use cases. Thanks for answering!

good

Is this the basis of why we think model distillation works?

@chrisalexiuk

10 ай бұрын

AFAIK, no. I haven't dove deep into the original distillation papers, but I believe that process relies on some other methods - though they definitely seem related. I'll have to re-up on the distillation papers though!

Subscribed ;-)

So I don’t get why, if the weight matrix ranks is much less than the original matrix. Why doesn’t the larger mega model just use the low rank matrix instead of their monster weight matrix?

@chrisalexiuk

Жыл бұрын

The base model needs that monster weight matrix to be good at many different things! It's the downstream tasks that have intrinsically low rank, according to the cited paper in the LoRA work.

What about smaller AIs networked together

@chrisalexiuk

Жыл бұрын

Something like that is an excellent idea - especially if you're using different LoRA adapters for the same base model!

Yes yes yes - D, K, matrix, 100 etc etc etc - but what does actually LoRa do?

@chrisalexiuk

Жыл бұрын

LoRA learns the decomposed matrices of the target weight matrix training on your downstream task.

@BR-hi6yt

Жыл бұрын

@@chrisalexiuk Sure - decomposed matrices .... lol

@chrisalexiuk

Жыл бұрын

I'm sorry! I'm missing something here lol What precisely do you mean by what does it actually do? I must've misunderstood your original comment, sorry!

doing a f'cking god's work

as clear as mud