PEGASUS Explained!
Ғылым және технология
This video explains the PEGASUS model and new pre-training objective from researchers at Google AI. PEGASUS shows that there's no free lunch for downstream NLP tasks with respect to the pretraining task. Using this new self-supervised learning task, gap sentences generation, they dramatically surpass the baseline performance, especially in the low resource setting.
Thanks for watching! Please Subscribe!
Links:
PEGASUS (arxiv): arxiv.org/abs/1912.08777
PEGASUS (blog post): ai.googleblog.com/2020/06/peg...
Follow Anna Rogers on Twitter!: / 1270058442523369473
What is ROUGE: www.ccs.neu.edu/home/vip/teach...
The Illustrated Transformer: jalammar.github.io/illustrated...
GPT-3: arxiv.org/pdf/2005.14165.pdf
Пікірлер: 18
1:45 A Better Aligned Pre-training Objective 2:53 PEGASUS Architecture and Gap Sentences Generation pre-training 4:45 Inspiration from T5 6:22 HugeNews pre-training dataset 7:15 Downstream Abstractive Summarization Datasets 9:15 Ablations of Gap Sentences Generation 14:23 Results 16:17 Low-Resource Summarization 17:10 Human Evaluation and Examples of Generated Summaries 19:01 PEGASUS vs. BART 20:00 WikiSum Dataset 21:25 Comparison with Ideas in GPT-3
Finally! A technical deep learning channel which doesn't have its last video 8 months ago. Keep up the good work!
@connorshorten6311
4 жыл бұрын
Haha, thank you so much! I also recommend checking out Yannic Kilcher's videos!
I need exactly this video for a presentation in my master, thanks a lot
Great video!
It was very helpful, keep up the good work 😊
Awesome Explanation
Perfect timing, I had just heard about the model and was interested in trying it out for myself. Similar to pegasus, your videos are pre-training for me, in a sense, before I actually go and read the paper and try out the technique. (someone said this pre-training part before, not my original meme)
@connorshorten6311
4 жыл бұрын
Haha! I use yannic's videos in the same way!
Interesting stuff. Keep up the great work.
@connorshorten6311
4 жыл бұрын
Thank you!
I need to present this paper in a class and find a video with an explanation by you is just... awesome
@connorshorten6311
3 жыл бұрын
Awesome, really glad to hear it!
Yay!
What's your opinion on adapting the pretrained PEGASUS model to a different domain (e.g. German)?
Is it still SOTA? Also, it looks like an extension to T5 objective like from span to sentence apart from prefix context.
I'm confused about if PEGASUS deal with noise in text or not, otherwise Great WORK!!
What was the compute used for pre training this?