Retrieval-Augmented Generation (RAG)
Ғылым және технология
This video explains the Retrieval-Augmented Generation (RAG) model! This approach combines Dense Passage Retrieval with a Seq2Seq BART generator. This is tested out on knowledge intensive tasks like open-domain QA, jeopardy question generation, and FEVER fact verification. This looks like a really interesting paradigm for building language models that produce factually accurate generations!
Thanks for watching! Please Subscribe!
Paper Links:
Original Paper: arxiv.org/pdf/2005.11401.pdf
FB Blog Post (Animation used in Intro): / retrieval-augmented-ge...
HuggingFace RAG description: huggingface.co/transformers/m...
Billion-scale similarity search with GPUs: arxiv.org/pdf/1702.08734.pdf
Language Models as Knowledge Bases? arxiv.org/abs/1909.01066
REALM: Retrieval-Augmented Language Models: arxiv.org/pdf/2002.08909.pdf
Dense Passage Retrieval: arxiv.org/pdf/2004.04906.pdf
FEVER: arxiv.org/pdf/1803.05355.pdf
Natural Questions: storage.googleapis.com/pub-to...
TriviaQA: arxiv.org/pdf/1705.03551.pdf
MS MARCO: arxiv.org/pdf/1611.09268.pdf
Thanks for watching!
Time Stamps
0:00 Introduction
2:05 Limitations of Language Models
4:10 Algorithm Walkthrough
5:48 Dense Passage Retrieval
7:44 RAG-Token vs. RAG-Sequence
10:47 Off-the-Shelf Models
11:54 Experiment Datasets
15:03 Results vs. T5
16:16 BART vs. RAG - Jeopardy Questions
17:20 Impact of Retrieved Documents zi
18:53 Ablation Study
20:25 Retrieval Collapse
21:10 Knowledge Graphs as Non-Parametric Memory
21:45 Can we learn better representations for the Document Index?
22:12 How will Efficient Transformers impact this?
Пікірлер: 29
Welcome back.. It was a long time since you posted on KZread.. We were waiting for you. :) Thank you for sharing knowledge
@connorshorten6311
3 жыл бұрын
Thank you so much! Really glad you liked the video as well!
Glad to see you again. I was getting worried. Your videos are great, thanks so much for the content.
@connorshorten6311
3 жыл бұрын
Thank you so much!
Very nice video! Really excited to try these techniques out
Very comprehensive overview, thanks!
New background, who dis?? Great to see you back making videos!
@connorshorten6311
3 жыл бұрын
Lol, Thank you so much!
Welcome back,thanks for the great work you are doing .
@connorshorten6311
3 жыл бұрын
Thank you so much!
Great video!
We’ve missed you so much 🤍
@connorshorten6311
3 жыл бұрын
Thank you so much!
Any ideas on some methodologies to perhaps evaluate the performance of the retrieval mechanism within the RAG model? thanks
nice!
Really great paper. To some extent all of NLP can be treated as a QA task.
@connorshorten6311
3 жыл бұрын
Thanks! I think the "Text-in, Text-out" unifying framework for all tasks really set the stage for this, interesting stuff!
is it possible for you to add the link to the ppt presentation used for this video in the description?
Ooooooooooo 🙌😎
Nice one
@connorshorten6311
3 жыл бұрын
Thank you!
Amazing video. What about finetuning this for different tasks? Authors say we do not need to fine-tuning the document encoder.. but other things.sny comments on that?
@connorshorten6311
3 жыл бұрын
Fine-tuning the document encoder would be very tedious because you need to continually re-build the index and centroids that speedup searching for nearest neighbors. Fine-tuning the query and BART seq2seq generation is much easier and any NLP task can be setup with this, as in the Text-input Text-output formulation. I cover that in more detail in the T5 video if interested.
@shaz7163
3 жыл бұрын
@@connorshorten6311 Yeah I went through those videos. So basically their doc-encoder is trained with 21 million wiki-dumps and it kind of enough for the network to encoder any type of document into a vector right? My other question is what if I want to look at a different set of documents ? How should I index it?
@bivasbisht1244
8 ай бұрын
@@shaz7163 did you get the answer to that ? cuz i have the same question :(
Can you explain why this RAG model seems popular? It seems like all they've done is connect a pre-trained retrieval model and connected it to a pretrained seq2seq model, and trained them together. They also just did a simple concatenation of the retrieved passages with the initial input before inputting it to the seq2seq model. This all seems like really basic stuff. So am I just missing something here? Because you could just also get rid of the retrieval model if you already knew which documents you wanted the seq2seq model to use and could just directly concat those with the original input.
Can someone expand on the snippet at 4:45 explaining how the query works with the encoded samples? In the video, the speaker states, "And then when we ask a query, like we have this new x sequence with a mask at the end of it, we're going to treat that like a query, encode that query, and then use this maximum inner product search...". My understanding is that we encode the masked x (where x is the input) with the same query encoder as what encodes the context information, then use MIPS to find essentially the most similar context to x, which is then processed by the generator to append to x. Any help clarifying would be much appreciated.
i want to know if RAG is a model or a framework or just an approach ? question might be dumb to ask , but i really want to know
@user-uy7gp6pu5c
5 ай бұрын
approach