Dive into the cutting-edge generative AI at the Pathway Bay Area Meetup, featuring two groundbreaking talks from notable experts in the field - Łukasz Kaiser (Researcher at OpenAI, previously at Google Brain) and Jan Chorowski (CTO at Pathway, previously at Google Brain).
💬 First Talk: "Deep Learning Past and Future: What Comes After GPT?" by Łukasz Kaiser
Łukasz Kaiser, a renowned researcher at OpenAI, co-author of TensorFlow, and co-inventor of the Transformer model through his seminal paper "Attention is All You Need," explores the evolution and future of deep learning technologies. He emphasizes that more data and compute lead to better results but highlights the impending data scarcity. Łukasz Kaiser delves into teaching models to think more efficiently, discussing how training with fewer, high-quality data points can enhance performance. He explains the importance of powerful retrieval mechanisms, integrating personal and organizational knowledge graphs, and efficient context provisioning for efficient reinforcement learning. By refining how models process and retrieve information, Łukasz Kaiser explains the future of LLMs with more powerful and intelligent AI systems.
💬 Second Talk: "Taming Unstructured Data: Which Indexing Strategy Wins?" by Jan Chorowski
Jan Chorowski, CTO of Pathway and a prominent figure in AI and NLP, extends the discussion from Łukasz Kaiser's talk by focusing on the essential role of context and retrieval in AI systems. He explores the "yin and yang" relationship between Large Language Models (LLMs) and retrieval systems, emphasizing that effective LLM performance and reinforcement learning require robust retrieval mechanisms, as highlighted by Łukasz. Efficient retrieval, in turn, relies on the processing power of LLMs. By integrating preprocessing with smart retrieval techniques, infinite context lengths and cost-effective AI solutions can be achieved. This comprehensive strategy for managing unstructured data and optimizing AI applications is crucial for enhancing AI efficiency and intelligence. Chorowski invites the audience to explore these concepts further, offering resources and examples at Pathway's developer site.
📅 Event Date: April 30, 5:30 PM - 8:00 PM PDT
📍 Location: MindsDB SF AI Collective, 3154 17th St, San Francisco, CA 94110
Explore more about Pathway:
Website: pathway.com/
GitHub: github.com/pathwaycom/pathway/
Documentation: pathway.com/developers/
#deeplearning #ai #datascience #openai #PathwayMeetup #LukaszKaiser #JanChorowski #retriever #reinforcementlearning #generativeai
-
Chapters
00:00 Introduction to Łukasz Kaiser, Co-author of Transformers Architecture
00:42 History of deep learning
02:17 Deep Learning for Parsing and Seq2Seq Models
02:56 Parsing Paper with Oriol Vinyals and Ilya Sutskever
04:02 Invention of Transformer Model and Belief in Scaling Up
05:00 Scaling laws for Neural Language Models
05:55 RLHF and Generalization of LLMs
07:14 The Future of Deep Learning
08:00 Missed Observation on Parsing from Attention is All You Need
09:08 Open AI Paper on Improving Chains of Thought (CoT)
09:28 All you need is more layers and quality, limited data.
09:49 Retrieval for efficient reinforcement learning (RL)
10:24 Live Q&A with Łukasz Kaiser
14:44 Jan Chorowski: Context in Model Applications
15:40 Enhancing LLM Context Windows with Better Context
17:05 Introduction to Retrieval Augmented Generation (RAG)
18:42 Differences Between Retrievers and LLMs
19:30 Retrievers are engineered. LLMs are complex: Retriever vs LLMs.
20:50 Maintaining LLMs in Production: Retriever vs LLMs
22:00 Read and Write Code Path: Retriever vs. LLM
22:52 Retrievers Beyond Vector Search
24:03 Using Graph Indices with Retrievers and RAG
25:00 Yin-Yang Relationship Between Retrievers and LLMs
26:01 LLMs for Better Retrieval Augmented Generation (RAG)
26:43 Achieving an Infinite LLM Context Window
27:48 State-of-the-Art RAG for Up-to-date Information
29:14 CPU-GPU Negotiation for Cheaper, Efficient RAG
35:16 Reduced complexity of Retriever with Pathway