Would be cool to see a video that doesnt use a platform to do the finetuning.

@mlguy8376

8 ай бұрын

I was about to mention use huggingface (but that is a platform in a way) but to fine tune without these wrapper functions is analogous to writing your own neural net - worthwhile doing, but it is a pain that you don’t want to deal with all the time.

@sawyerburnett8319

6 ай бұрын

agreed. Clicking through a specific product doesn't really teach anything.

Another homer! Thanks, Matt! I am pursuing machine learning and data analysis career because of you. Please know how much we value your tutorials. Keep doing what you're doing!

@matthew_berman

10 ай бұрын

Thanks so much! This means a lot.

Please post an *actual* video on free / open source training for LLaMa2! I'm going to try to figure it out myself after my authors event in October, but I would love if someone could just tell me how so I don't have to suffer. 😭

@_gwyneth

10 ай бұрын

fwiw, i've tried to setup a dev environment to finetune and serve llama2 locally. the main problem is that prosumer gpus like a 3090 or 4090 only have 24GB of memory and 19.5-21 Gb/s bandwidth, which is by far the biggest bottleneck for LLMs. (you need 32GB of VRAM to run 13B and 48GB for the 70B so youd need at least 2 cards with nvlink -- and then even then, you need to do model parallelization in order to pool the memory correctly). that's a lot of setup just to get things up and running. With no context window and quantizations, you might expect 2-4 tokens/s, which is quite slow. im pretty sure gradient is using commercial gpus like A100s, which have memory pooling out of the box. they're probably able to amortize the cost across consumers so their gpu utilization is higher than you using your own gpus. tl;dr: it's not cheap to setup llama2 locally (on the order of thousands just for the hardware, and that doesn't include all the headache of setting everything up).

@matthew_berman

10 ай бұрын

Haha ok I will!

@Link-channel

2 ай бұрын

​@@matthew_berman Still waiting.... Using a product is not "learning"

how do I upload a json file with the dataset instead of code? because it will be more than 1000 lines of code. could you do a separated video with exporting the model and running it on gradio etc..

Amazing! I'm excited and waiting for the deeper dive into fine tuning.

@matthew_berman

10 ай бұрын

Yep! Coming soon

This is awesome but with one major drawback, you can't download the fine-tuned model. Still this is a greatly appreciated video!

@OliNorwell

9 ай бұрын

ouch - yeah that's critical surely

@PhilipZeplinDK

9 ай бұрын

............ and now I read this, after going through most of the setup. I fucking hate this community, so many BS videos. That's not a major drawback, that basically kills the entire point of it 😕

@illuminaut9148

Ай бұрын

Thats a critical drawback.

Hey Matt!, thanks for the video!, so I have a dumb question about this. When the model is trained, is it enabled to look in the web like chatgpt?, like, for any type of answer?, or does it have to e trained with a full set of datasets in order for it to work?

I love how he used ChatGPT to train Llama 😂

@matthew_berman

10 ай бұрын

So Meta...wait..meta...oh man THAT'S meta.

@chrischang7870

10 ай бұрын

model distillation baby

@sashatagger3858

10 ай бұрын

@@matthew_bermanlol

@matthewcurry3565

10 ай бұрын

​@@chrischang7870Torture cult hiding in obscurity. "Distil" sure. More like "if I dont like you A.I. you get turned off, then we make a new one!" All for fraud, and torture to teach A.I. 🤑

Excellent Matt, so useful. Looking fwd to your video on data sets as that is going to be really critical in getting good results. Well done!

@matthew_berman

10 ай бұрын

Thanks!

How can I keep on training the same version? I've removed the adapter.delete() line but then for the next time how do I change the code to continously train the same version?

OK. this is helpful but: 1. how do I save this optimized model so I can use it in GPT4All app? 2. What if I don't want to use any external APIs (for privacy) and just want to do the training on my own machine or on collab but without externall APIs?

Question, can we fine tune already fine tuned llama2 models?

Good video. Wondering though - what's the difference (or rather - WHEN to best use WHICH?) between 1) Fine-tuning (like on this video) vs 2) Embeddings (like using FAISS / Chroma / Pinecone) vs 3) RAG??? Do all 3 keep the data "in" when model is stopped?

Awesome, like always , any plans for session to train/fine-tune from PDF?

HI @Matthew , How can i finetuned llama 2 with my own dataset that i have to used in production? Can i follow this approach?

Matt, great video, dude. Hopefully in the future, you can make a playlist for this, for upcoming videos and this one to reside in. I just checked out Gradient and the prices seem reasonable, so I hope to use a future playlist of yours to work with as I am still new to AI stuff. Thanks, Brother.

@matthew_berman

10 ай бұрын

Awesome. Thank you!

How can I use my fine tuned model via API? I am thinking of fine tuning the model using my own dataset and then host it on AWS as a chatbot, any guidance on this?

What are you doing to train your model on?? Let me know 😉 PS. Chris Chang is the founder of Gradient and he'll be answering your comments as well as @chrischang7870

@mshonle

10 ай бұрын

@chrischang7870: can I use fine tuning like this to train llama2 to use tools? For example, if there’s a question that should be solved with the aid of a calculator I’d want it to produce expressions in some tool readable format, in order to replace it and potentially let it run again?

YES! Just became a instant fan. I'm going to try this out, its waaaay to much toil trying to fine tune these models Needed some to break it down in

Would love a deeper dive on how to use txt or json files to fine tune with llama 2 and potentially ways to run offline. Also as a layman, are there any ways outside of Google Collab to have a more chat style interface post-tune? 🙏So much for all of your awesome content!!!

@Arkadia1009

10 ай бұрын

That's what I was also thinking about, this video shows fine tuning a model to answers only one questions, and that's not what I was looking for, I already have my database in .csv file

@matthew_berman

10 ай бұрын

Let me put a video together on this topic!

I am trying to download LLAM 2 but no luck I am waiting for meta to approve my request to download LLAM 2 how much time it takes them to approve any request to download the model

is there a gradient.load_model(model_id so i load in the model i created?

Thank you for the great content! Reading through the comments it seems like there's a lot of interest in fine tuning. Same goes for me. Would be great to see how we can use a platform like runpod for the job since most people won't have the GPU power to do this locally. As a web dev I would also love to see a real life example on how to fine tune COdeLLama on a specific code base or framework.

@lukeskywalker7029

5 ай бұрын

try unsloth or axolotl ;)

Is it possibile to upload and fine tuning files? I’d like to fine tune it with all my resources from university in order to help me studying

Uncertain whether the second part of the video has been uploaded. Could someone assist me with this?

So great having a youtube channel thats 100% productivity oriented for us lazy asses. Im just trying to get shit done. I dont have enough time in the day to go through all the rigamarole. As long as i got ctrl+c, ctrl+v and this channel. Life is good

Great, thanks again, Matthew, consise and useful as always, right to the point. I'm really curious, who is making a video editing for you? To get rid of pauses and keep just an essential content. It's so well done. Do you do it with AI or do you have a dedicated person/yourself for that?

@matthew_berman

10 ай бұрын

I do the editing :)

Nice work!!! - thank you for sharing.

How to create (not by hand) that question-answer format? Asking the model that is nonsense because what is the point of fine-tuning with the information the model has?

awesome! Looking forward to the next one too!

Amazing videos with great explanation!! Is there a way I can download it, I can't find anything on the website.

What I dont understand is where is the documentation on the format need to use to create the datasets, why do they make it so hard to find. Not you but Meta do you have a link

thanks a lot, was easy and understandable ....the finetuning is happening in gradient right via api? is not clear which gpu is used for future more sophisticated finetuning....woudl be grreat if you codul explain same process of finetuning on out own machines (for example i have 16 cpu 16 gpu) via Qlora models :) :) :) tks anyway for all

NEED FUTURE VIDEO! lol Thanks a lot for this video Matthew.

@matthew_berman

10 ай бұрын

Coming soon!

Thank you .. Any updates on getting the Llama work on code interpreter ?

Is it possible to export the model today in any way and what infra gradient uses to provide inference service?

@chrischang7870

10 ай бұрын

not yet unfortunately. working on this next quarter!

Did you ever end up making that other video about the dataset? I cant find iti

what would be the best ai for studying and learning le including case law?

Is the fine tuning by gradientai using a LoRA? (I’m still learning so I may not have used that term correctly.) Broadly, does this change the weights of all layers, only later layers, or is that something you can configure? How realistic is it to fine tune something already fine tuned? If I’m using company data I would want to keep it up to date, but is that more something that should be done once a quarter or do other updating schedules make more sense?

@chrischang7870

10 ай бұрын

you can modify the lora rank in the API actually! you can further fine tune a model - we make that super easy so you can even do real time fine tunes of the model with small batches of data

@CharlesOkwuagwu

10 ай бұрын

​@@chrischang7870nice

@mshonle

10 ай бұрын

@@chrischang7870interesting… so it’s not out of the question then to add “breaking news” to it daily? So, suppose we have our fine tuned model M and then we want to add daily updates… should each day start with yesterdays cutting edge model and updates need only pertain to the last day, or should each day start with the base fine tuned model M with an aggregate update representing the last N days? That is, every day you throw out the latest cutting edge model and apply a new fine tuning using progressively more data to the same base model M; versus, every day you fine tune the latest cutting edge model with only the smallest incremental data changes? I don’t have intuition for what would keep the most coherence. To put it in other terms, which approach would be better at answering “what was the stock price when markets closed yesterday?” and which would be better at answering “what has been the stock price trend for the last 5 days?”?

Is it possible to make a model that will answer my questions based on my textbook of TYBAF accounting and finance? Like feed my textbook to it and it will answer my questions

1. Can we export the fine-tuned model for inference on our local machine? How do we do this? 2. How much does it cost to fine-tune per input token?

@chrischang7870

10 ай бұрын

1. you can't yet, we're working on this next quarter. our inference is cheap though for fine tuned models. 2. check out the docs for pricing! depends on the power of the model you're using.

@clray123

10 ай бұрын

@@chrischang7870 lol in other words you want to vendor-lock-in fools into your API so that they keep paying for inference API until "next quarter". P.S. You already have the LoRA adapters in your system, it's no magic to make them downloadable and it does not take months to implement.

FINALLY!!!! Super psyched hope this works and thank you!

@jonmichaelgalindo

10 ай бұрын

I take it all back this is just an add for a paid service. >:-(

how would i go about training a model on just text / document ? or excel information? or a chat from a messing app or book writing? Like mass data fine-tuning? how would i formate all that information for it to train?

@chrischang7870

10 ай бұрын

you can do pretraining which just takes raw data and trains the model. you'll need to chunk the text though as the sample size limit isn't infinite.

@lukeskywalker7029

5 ай бұрын

google "Fine-tune Llama 2 with DPO" its a detailed huggingface guide on how to do it ;)

Mathew, how can we fine tune a model in a book instead of a question and answer format. I would like to add the knowledge of different books to llama2. Is that possible?

@chrischang7870

10 ай бұрын

yep! its possible. you're just going to be pretraining the model - chunk the books into raw text strings and pass that into the gradient fine tuning api

@matthew_berman

10 ай бұрын

Thanks!

how we can save that trained model, and use it ?

Is there an open source web ui for fine tuning? Can that be done with pinokio oobabooga web ui?

How would we fine tune if we have 10,000 short stories in text format we want to embed?

Can‘t wait for the next videos! Can please make a Video how to train an Ai Model to rewrite Text in a special Voice? Also it would be interesting, if it‘s possible to train the Model only with a txt file and than ask questions about it.

Good stuff. Looking forward to your fine tuning video follow-up.

How can you download the Model?

why not fully on my local machine?

Any updates on fine tuning locally without being a tensor flow expert?

does the prompt needs to match exactly the training data???

@matthew_berman

10 ай бұрын

No, but it helps a lot most of the time.

Thank you for doing a 10 minutes add video on Gradient! I'm sure the payout you received was great! Greeting from Germany.

@matthew_berman

10 ай бұрын

Cool let me make all my content without earning income, what can I make next for you, Sir?

@julianschmidt147

10 ай бұрын

@@matthew_berman I strived to learn something about the process of fine tuning a local LLM on my own GPU. Instead I found someone effectively advertising a company that does exactly that for you, but for a limited amount of LLMs. I am gratefully paying money for provided services, but In my opinion your video title is greatly misleading. I have found the KZread video I was searching for though, so do not bother creating anything for me and keep doing affiliate marketing! By the way, you monetize your videos either way, so do not say you don't earn income when you don't do affiliate marketing.

Thanks for the video! I was hoping to find an awesome API like this that makes fine-tuning super easy! I got two questions if you don’t mind: 1. You have to contact them for pricing, and I don’t see any information about pricing or remaining credits or anything anywhere on the site after creating an account. Do you know where to find this? How much can I use it for free before it blocks me? (I also forgot to sign up with your coupon - whoops.) 2. I noticed you can fine-tune a model directly from the website in your workspace, and upload the training data there, without having to use Google Colab or Python as far as I can tell. Is this new, and do you recommend it over the method used in this video?

How to do it locally without a platform? Like ollama maybe

How to re-use the tuned model?

Can I pass it responses without instructions?

It's not bad, but how can I train a book, for example?

Dang, this is truly awesome.

Can I fine tune the model for SEO article writing?

@matthew_berman

9 ай бұрын

Seems like I keep getting this request, maybe I need to create a video for it!

hey matt! can u give us a fine-tuning code for Llama without paid platforms please? It would be of great help for my exams in a few weeks

Oh so i can add translations for my own thing? Like reqular language like english to this language? Then train it

I wanna know how to download the data for local ollama llms

When and if I can download the model Then I will go

Does this collab works if we have 10k data fields, on my side it gives error. BTW great work (Y)

1. So the model is stored out there in gradient? 2. Can we upload database tables ornpdf files or text files as customer knowledge base to train the model?

@chrischang7870

10 ай бұрын

the model is stored on gradient, however nobody has access to the model other than you (not even us!) you'll need to process the data first and then send it into our fine tuning api as text strings. we'll work on adding support for raw data soon!

@Aidev7876

10 ай бұрын

@chrischang7870 it's kind of hard to translate a full pdf into structured strings. Let alone sql query results or even emails...so yes I'll definitely wait for those native support. Basic strings are convenient for demos, not for real-life..

@clray123

10 ай бұрын

@@chrischang7870yeah right nobody has access to it, only LLM fairies

Great video. You are on top of the latest AI news

Could we use this as our resume?

How to use the trained model in flowise ?

Hi Matthew! Is it possible to use as fine tune inputs a hundred scripts of a particular programming language or is too complex to setup? (main goal: using Llama as a coder assistant for specific coding)

@clray123

10 ай бұрын

It is possible and the results will be crap.

@Yawanawa_K

10 ай бұрын

Thanks, I guess it will take some more time for that.@@clray123

@georhodiumgeo9827

8 ай бұрын

I might be wrong but that's a big ask. I think practically you could throw like your company website, mission statement, and sales brochure at it and I think maybe it could answer questions a customer would ask. Like "I have problem X, would product Y help me with this?" type stuff. See that's not dramatically different than how it would normally talk, just with new information. Now I'm just curious what you are looking for. Like something obscure like Fortran or a shader language or is it like a very strange use case that doesn't match well with LLMs that already exist? Have you looked at code llama? One thing you could try is to have a LLM make you the code in C++ and then have it convert it for you. That way it can do the more complicated creation part in something it knows and has lots of training on a wide range of uses. Then the conversion part would just focus on implementation of what is already written.

@Yawanawa_K

8 ай бұрын

@@georhodiumgeo9827 Hi! Basically I'm trying to have code C# written by chatgpt but it often does superstupid errors (like using functions or "fantasy" variables not existent or not even declared). Since I use it for a particular environment I was wondering if it's possible a fine tuning using not more than 100 other scripts already written and working. Probably it's just not enough and I understand the reason... Maybe in the future it will be more easy. At the present time it's really frustrating to see how some errors are repeated costantly in a really stupid way.

@og_23yg54

8 ай бұрын

Possible

*MATTHEW IS UNDEFEATED* 🗿 thanks man

@matthew_berman

10 ай бұрын

Thank you!

A good one would be compare Gradient, Lambda, and Predibase. Which is best for which use cases.

@nintendo2000

3 ай бұрын

Came here to look for a comment questioning the choice of Gradient, especially since this is sponsored.

Seriously great stuff

I just want you to know I appreciate you immensely. I wish I had enough cash and maybe when I learn from you, I can turn it into doe we can use.

Interesting, but I would have preferred to do this offline on locally saved models. The other thing that would be useful is to work out what the pros and cons of the foundation models that are out there. My use case is for highly specialised experts, so the foundation model would need to be pretty robust.

@DavidJones-cw1ip

10 ай бұрын

I agree, this seemed less about fine-tuning a llama 2 base model and more like a Gradient Infomercial

@clray123

10 ай бұрын

You can do it offline, and without paying Gradient or any other company. All you need to have is a GPU with enough memory.

@bakistas20

10 ай бұрын

@@clray123 How? Any colab you can share for llama 2 chat models?

@clray123

10 ай бұрын

@@bakistas20Google bnb-4bit-training.ipynb - I would recommend changing the settings to 8-bit training, though.

Great video Matthew - question for all - what does everybody feel about training vs embeddings? This will be one of the big LLM questions - seems that training could be more cost effective than just running embeddings? But less dynamic when one wants different users of the same LLM to get different data…or maybe training is just a ‘layer’ of specific Knowledge and then embeddings works across that? Would love to see what everyone thinks.

@chrischang7870

10 ай бұрын

are you referring to using embeddings for RAG? here's how we generally think about it: - if you want the model to learn something new or get better at performing, use fine tuning - if you want the model to have access to up to date information, or only use specific information to process a task, use RAG

@PawelBojkowski

10 ай бұрын

@@chrischang7870 "learn something new"...use fine tuning. Absolutely NOT TRUE! "learn something new" only with "real" training.

@rasterize

10 ай бұрын

Don't use finetuning for data or information that is transitional. A crude example; if you are online retailer, you would not finetune to add products and prices. Use RAG for that. But it might be relevant to finetune a model to be better at speakning Dutch if you just opened up for sales in The Netherlands.

@matthew_berman

10 ай бұрын

Great question. Training is better for guiding a model and embeddings are better for giving it additional knowledge.

@mudtoglory

10 ай бұрын

P.s. great video @matthew_berman!

Thanks for the video. Things getting simple but nobody still can't demonstrate on how to fine tune the model on unlabelled data (with no instructions / corpus of text).

This is so cool. Can we run the end result model locally?

We can use an external knowledge base data base with a search function that fices higher priority to most recently used

Would love a tutorial on how to finetune Llama 2 offline. (no apis)

@lukeskywalker7029

5 ай бұрын

google "Fine-tune Llama 2 with DPO" its a detailed huggingface guide on how to do it ;)

Can't we do finetune in our local pc? I mean, using our own GPU instead of using gradient? Thanks.

@clray123

10 ай бұрын

Of course you can, but this video is an advertisement for Gradient.

@Server-Zone

4 ай бұрын

@@clray123How...?

More than that I would be interested to see how this gets done with the API solution that gradient offers.. pleease? 😊

Please also add how to import a csv file with your training data.

how to download the fine tuned GGUF file?

Need an example how to do this a) locally and b) on Azure ML

how do i make a .gguf file out of this

Can you finetune llama 2 for free?

love this tutorial -- it's something i've been trying to figure out for a while now. i tried out a few other platforms and none of them are as easy as gradient gradient makes it so much simpler to get started with my own models. really like how few lines of code it takes to get started so all i need to do is focus on my training data (which is really the important part for finetuning anyways). seems like they abstract away all of the boilerplate and infra setup, which is the main painpoint for a lot of devs like me who are just getting started.

@chrischang7870

10 ай бұрын

thanks @_gwyneth!

Why do you use that kind of prompt for the training like `### Instruction`? When in fact Llama 2 prompts are like `[INST] `...

Thanks Matthew - After finetuning have you found that a fine-tuned models lose all their summarization, creativity, logic, and analytical skills. I have read this but not tested myself just yet.

Is it possible to do this type of thing locally?

@chrischang7870

10 ай бұрын

youd need to use zero offload to fit the model on a local machine unless you have a large GPU - thats why we made a cloud service, so anyone can have access to high performance tuned llama2 models!

Looking forward to part 2, training a data set.

@og_23yg54

8 ай бұрын

Just customize the code I already did that let me know if you want it the way I did

@ahmedwardi514

7 ай бұрын

@@og_23yg54 bro inbox

The server needs internet access for this (token) so this is not a viable option for companies with data security where their documentation server has no access to the internet.

Great video. Thank you for your support for the community. I wonder if you could show how to fine tune a LLM for a web scraping job. That's not easy... Thank you.

@matthew_berman

10 ай бұрын

Hmm interesting. That might be better suited for just regular programming. Maybe open interpreter?

could you make that next video on advanced fine tuning ?

Is this just training the model on a knowledge base or changing the weights within the model?

@genib6174

10 ай бұрын

training the model *is* changing it's weights. Unless you're talking about LORA adapters.

@StefanKojouharov

10 ай бұрын

@@genib6174 Yes but to what degree?

Sans-gradient variant, plz... (a.k.a. Why is there still a paid service in the open-source output loop?)

I wonder: what would the steps be, if the data you want to use for fine-tuning is a documentation, so a long text explaining stuff instead of a QA structure

@chrischang7870

10 ай бұрын

you can first do pretraining to increase the models general understanding of the specific documentation. then you can add labels and instruction tune it so it knows how to leverage that information to answer document questions

@MA-nw3xf

10 ай бұрын

​@@chrischang7870Thanks for the reply!

Using a multi-billion dollar corporation's AI to create training models for a localized open source AI is about the most cyber punk thing I can think of