Thank you for illustrating both the concepts and then doing the math on pre-training, fine-tuning and ICL!

@code4AI

Жыл бұрын

Thank you. But: It was not mathematics, rather an estimation. But when I'll summarize JAX-FLAX-T5X there will be some mathematics ...

Great video! Congratulations!

@code4AI

9 ай бұрын

Glad you liked it!

Can ICL be for something like ai streamers that learn from their stream chat? Even though there's no correct answer for how they should respond, there is the stream chat's response which could be used for feedback?

hi do you know any good papers for icl that describe this topic in more detail?

Maybe I misunderstood but you seemed to imply that ICL persists between sessions and across users and that's not the case. It only exists in the conversation

@code_your_own_AI - thank you for this video, I think the analogy is super helpful, though one key thing is that when you learn on the job you retain that knowledge while ICL does not affect the model

@code4AI

Жыл бұрын

Prompt engineering by definition does not alter the weights in the transformer, but all eg ChatGPT conversations are recorded in your user history and given your decision to allow OpenAI to use your conversational data, could be used to further be part of the next training run.

If the ICL affects the model responses for any user that will use it, how are the weights & biases parameters of the DNN not modified?

Great video.

@code4AI

Жыл бұрын

Thank you!

Thank you

Can you continue training an already trained model ? I.e. you start from a gpt 3.5 and continue training it with your custom raw data ? Or is it that once a model has been trained, it's no longer retainable anymore ?

@ZZ-yw4xf

2 ай бұрын

Im also looking for an answer to this question but after watching this clip I guess the answer is it depends on if the model is available for continued pretraining, and the costs. GPT models are only released for FT, so we cant do more pretraining on them. Some posts on the internet say you can prepare you FT data as [prev word, next word] pairs, but that won't work. I tried it and all the model learns is to output a token given your input. On the other hand, some open source models are fully accessible and I guess we can. Have you found other answers? I am also interested to learn.

Is zero-shot and few shot learning part of ICL, depending on how many examples we give?

@code4AI

11 ай бұрын

In-context learning (ICL) refers to the LLM's ability to understand and use the context provided in the prompt to generate appropriate responses. The context can include zero or more examples (hence we talk about: zero-shot or few-shot learning). The LLM uses this context to infer the task it needs to perform. And yes, the number of examples can influence the model's performance, but more important: the quality of the examples. Even a single well-chosen example can sometimes be more effective than several poorly chosen ones. Therefore "prompt engineering" - or the art of crafting effective prompts-is also crucial in this aspect. Hope it answers your question.

Is it necessary that in the fine-tuning process, all weights/parameters of transformer blocks are modified? I think some layers can be frozen. Correct me if I am wrong.

@code4AI

Жыл бұрын

I just have 3 videos on how to freeze transformer architecture elements for increased speed, so how would I know?

@shubham_chime

Жыл бұрын

@@code4AI Of course you know :). I wanted to confirm my understanding.

8:15 Could you please provide examples of public LLMs which learn from ICL (in the case where future users can also benefit from your input data)? So far I've only seen ICL help with the *current* prompt, but not seen cases where it helps with *future* prompts of other users.

@code4AI

Жыл бұрын

I was talking that the system learns from the user interactions, but regarding the OpenAI system admin, not the external single user, although all your user conversations with ChatGPT are recorded in your history and could (theoretically) be used to further train the system.

@ArtOfTheProblem

9 ай бұрын

Yes you can use ICL to affect other users prompts, for example recall the hacks where people made chatgpt spit out it's hidden "commands" where were appended to the front of any user prompt "i.e. "you are an AI model...you won't do dangerous things...."

@CptAJbanned

4 ай бұрын

That part is poorly explained. ICL doesn't persist between sessions. You could add it as a fixed prompt to your system so that every session gets the data, but you have to burn part of your context budget with the examples each time you use it.

the last piece of ICL for task QA, I thought those are fine tuning, no? I thought ICL is mostly about context in the prompt.

Austria!?! Interesting. Guess you teach!? Where? Do you personally know Hochreiter Sepp? Are you a fan of Károly Zsolnai-Fehér too? Thanks a ton for your effort!! And, ahh, yes, Colabs are great!

Might be a bit of a stupid question,………… but : in the depicted way, what is the difference between ICL and RLHF? Or is ICL just a subset of RLHF ?

@code4AI

Жыл бұрын

Reinforcement learning (RL) is a type of machine learning that involves an agent interacting with an environment and learning to make decisions that maximize a reward signal. The RL algorithm learns by trial and error, with the agent receiving feedback in the form of rewards or penalties based on its actions. The goal of the agent is to learn a policy that maps states to actions, such that the expected cumulative reward over time is maximized. The RL algorithm interacts with the GPT model by selecting actions based on the current state of the model and the desired objective. The actions may correspond to specific words or phrases to be generated, or they may involve modifying the internal state of the model in some way. The RL algorithm then receives feedback in the form of a reward signal based on how well the GPT model is performing with respect to the objective. One challenge in using RL for GPT training is defining a suitable reward function that accurately reflects the desired objective. The reward function may be based on external evaluation metrics, such as human judgment of the quality of the generated text, or it may be based on internal metrics that reflect the coherence, fluency, or other aspects of the generated text. After the GPT model generates a sequence of output tokens, the RL algorithm evaluates the quality of the generated text based on the desired objective. Based on the evaluation, the RL algorithm provides a reward signal to the GPT model that indicates how well it is performing with respect to the objective. The reward signal is used by the GPT model to update its internal parameters through a process known as backpropagation, which adjusts the model's weights to improve its performance on the task.

What about Lora? Which is also technically fine tuning but at a fraction of the cost.

But does previous context persist into different sessions under ICL?

@code4AI

Жыл бұрын

If you can save your LLM and reload your LLM, yes. This is why OpenAI new Foundry will sell you dedicated machines for 3 months to 1 year.

I didn’t realize that you get charged for converting your training data into embeddings. Can we use open source?

@code4AI

Жыл бұрын

Whatever takes compute resources, somebody has to pay. Of course you can use open source resources, but you will have to run them on a (cloud cluster) compute infrastructure.

Is it possible to do fine tuning simply using open source options?

@code4AI

Жыл бұрын

Yes.

how to inject company data into LLM memory without using vector database?

@code4AI

Жыл бұрын

If you buy your own machine from OpenAI, no problem, otherwise your are limited to NN activations, not memory.

What about fine-tuning with your own embeddings?

@code4AI

Жыл бұрын

Your embeddings are a vector representation of the words chosen to be in the set of the vocabulary, that itself has been created in the pretraining phase of your model. Do you see the point?

@GenzhPuff

Жыл бұрын

@@code4AI I guess the point is that you cannot fine-tune the embedding right?

So that was fine tuning before PEFT/LoRA

Pre training takes 416 days?

@code4AI

Жыл бұрын

Do you know about the size of the internet?

@dataai514

Жыл бұрын

Hehe

What, from Austria???? 🙈 How geil is das bitte? You sound a bit like an English-speaking Viennese 😁