This video can be a terrific example of overfitting 😁😆
@JMSproduction-qk2rx19 күн бұрын
Pleass my youtube compleat now
@dj...channel254920 күн бұрын
So amazing 😊
@hanhphuclaemtn9227 күн бұрын
Yydhkxhxhx
@hanhphuclaemtn9227 күн бұрын
Đìh
@shrirangmoghe378428 күн бұрын
Please upload slides and perhaps sync it to the audio if you can. Can’t believe we are in age of AI and humans are already losing it
@shrirangmoghe378428 күн бұрын
What are we even looking at here ? Are we at an edge of some black hole? Totally nauseating
@davidbadmus-yh9boАй бұрын
like this
@swenicАй бұрын
Not liking the current layout. It is not possible to make up a lot of detail (you can see a large room w people in it but nothing of value is discernible) in the small image, and placing it side by side w the presented material essentially makes that harder to see as well, resulting in a need for more screen estate or get used to watching things in half the size that is customary. How about instructing presenters to leave a small box area in their presentations and paste the room in that?
@ericgibson2079Ай бұрын
Please support The End and Prevention of Homelessness USA 2030!
@JOHNSMITH-ve3rqАй бұрын
If y’all could make sure you actually have clean audio in future that’d be great. The banging is super super distracting and brings down the quality of the whole thing.
@harvarddatascienceinitiati332027 күн бұрын
Noted! Thank you for your feedback.
@santaespinal15402 ай бұрын
🎯 Key Takeaways for quick navigation: 00:12 *🎓 Introduction of Sasha Rush by David Parks* - Introduction of Sasha Rush, a professor at Cornell University and a contributor to Hugging Face, now part of Google, known for his work in AI and large language models. - Sasha's background in computer science, his contributions to AI research, and his commitment to open-source initiatives. 04:28 *🗣️ Overview of Large Language Models (LLMs)* - Large language models (LLMs) are discussed in terms of their significance, complexity, and implications. - LLMs are described as extremely useful, expensive, important, and sometimes perceived as intimidating, but also capable of providing remarkable outputs with consistency and creativity. - The talk aims to provide insights into LLMs and address questions regarding their functioning, reasoning, and impact on various domains. 07:38 *📊 Dividing LLM Understanding into Five Formulas* - Sasha introduces the concept of understanding LLMs through five formulas: perplexity, attention, gem, chinchilla, and rasp. - Each formula represents a different aspect of LLMs, such as generation, memory, efficiency, scaling, and reasoning, providing a comprehensive view of their functionalities. - These formulas aim to simplify complex concepts and enable a deeper understanding of LLMs for both technical and non-technical audiences. 11:33 *🔍 Understanding Perplexity in Language Models* - Perplexity in language models is explained, focusing on the probabilistic model of documents and the probability distribution of word sequences. - The Markov assumption and the representation of Theta as categorical distributions are discussed as historical aspects of language models. - Modern advancements in LLMs are highlighted, indicating departures from past assumptions and embracing more complex neural network architectures. - Language models' historical evolution is discussed, from early Markov models to modern neural network-based LLMs. - Changes in assumptions and approaches, such as fixed-length history and categorical representations, are contrasted with contemporary methods emphasizing neural network architectures. - The session concludes with reflections on the enduring relevance and evolution of language models in the field of natural language processing. 21:04 *🧮 Language Model Development Pre-2010* - Explanation of early methods to develop language models. - Discussion of Shannon's paper from 1948 outlining language model development. - Overview of the challenges and limitations of early language models. 22:32 *📊 Quantifying Language Model Performance* - Introduction to evaluating language model performance. - Explanation of the challenges in quantifying language model effectiveness due to the lack of a definitive correct answer. - Illustration of the evaluation process using an example sentence and the concept of word prediction difficulty. 29:17 *📏 Metric Relating to Compression and Shannon's Work* - Introduction to a metric related to language compression inspired by Shannon's work. - Explanation of encoding words with binary strings based on their probabilities. - Discussion on how the metric, perplexity, measures the efficiency of language compression. 38:21 *📈 Evaluation and Practical Applications of Perplexity* - Discussion on the practical evaluation of perplexity using a test set. - Exploration of the relationship between perplexity and model quality. - Overview of historical perplexity scores achieved by various language models and their implications. 44:09 *📊 Correlation between Perplexity and Task Accuracy* - Perplexity correlates almost perfectly with translation accuracy. - General language perplexity correlates extremely well with task accuracy on different tasks. 45:59 *📈 Impact of Perplexity Reduction on Model Performance* - Lower perplexity corresponds to better model performance. - Graphs illustrate the relationship between perplexity reduction and resource investment. 53:17 *🧠 Introduction to Memory and Attention Models* - Explains the concept of memory and attention in neural network models. - Introduces the limitations of Markov models and the need for attention mechanisms. 01:05:03 *🧠 Understanding Query, Key, and Value Operations in LLMs* - Query, key, and value operations in LLMs enable prediction of the next word based on contextual information. - The process involves querying a lookup table using the query, retrieving the key, and using it to predict the next word. 01:07:03 *🔄 Transitioning from Argmax to Softmax* - Argmax selection poses challenges in neural networks due to its discontinuous nature. - Softmax provides a smooth alternative for selecting the best choice, enabling meaningful training. 01:15:08 *📊 Evolution of Attention Mechanisms in Language Modeling* - The attention mechanism, popularized by the "Attention is All You Need" paper, revolutionized language modeling. - Transformers utilize attention mechanisms to process input sequences efficiently and capture long-term dependencies. 01:18:05 *🖥️ Efficient Computation with Attention Mechanisms* - Attention mechanisms efficiently compute long-term context by utilizing matrix operations. - Matrix multiplication and softmax operations allow for effective computation of attention scores. 01:27:10 *🖥️ Transition from CPUs to GPUs for efficient computation* - The shift from CPUs to GPUs revolutionized the efficiency of running applications like language models. - GPUs allowed for faster computation of complex operations like softmax, significantly altering the research landscape. 01:29:19 *🧠 Understanding GPU architecture and parallel computing* - GPUs operate as parallel computers with multiple threads running simultaneously. - Threads within blocks share block memory, enabling fast data exchange and computation. 01:34:24 *🧮 Efficient matrix multiplication on GPUs* - Matrix multiplication on GPUs involves loading data into block memory, performing computations within blocks, and minimizing reads from global memory. - Leveraging shared memory and parallel processing allows for efficient computation of matrix multiplication. 01:47:04 *💡 Maximizing GPU performance for ML applications* - ML applications benefit from efficient GPU performance, measured in ML operations per second (MLops). - Optimizing data formats, such as using smaller floating-point values, enhances GPU efficiency. 01:53:20 *🚀 GPU Optimization: Why Speed Matters* - GPU programming optimization is crucial for efficient computation. - Efficient GPU programming enables faster computation, which is essential for large-scale models like LLMs. 01:53:49 *🔍 Scaling in LLMs: Model Size vs. Training Data* - The performance of LLMs depends on the interplay between model size and training data. - Increasing model size and training data improves the model's ability to generalize and understand complex patterns. 01:54:42 *📊 Compute Optimization Formula: The Chinchilla Formula* - The Chinchilla Formula extrapolates the expected perplexity of LLMs based on model size and training data. - It suggests a proportional relationship between model size and training data for optimal performance. 02:11:35 *📊 Cost Comparison and Scaling Laws* - Understanding the costs associated with large language models (LLMs). - Comparing and incorporating costs related to data acquisition and model scaling. 02:13:09 *💰 Cost Considerations in Model Development* - Addressing cost considerations in LLM development, especially focusing on compute and data. - Exploring scenarios where reducing specific costs benefits certain stakeholders. 02:14:46 *🔄 Data Reusability and Synthesis* - Discussing the reusability of training data and its diminishing returns. - Exploring the potential of synthetic text generation and its current applications. 02:17:29 *📰 Importance of Training Data Quality* - Highlighting the significance of high-quality training data, such as from sources like the New York Times. - Discussing challenges in quantifying and assessing the quality of training data. 02:18:19 *🛑 Addressing Token Exhaustion Concerns* - Discussing concerns related to token exhaustion and its potential impact on model training. - Exploring strategies to address token scarcity, including alternative data sources. 02:32:52 *🧠 Exploring Formal Logic in Language Models* - Introducing a formal logic approach to understanding language model behavior. - Discussing the use of logical operations to manipulate and analyze model behavior. 02:34:57 *🔄 RASP Language and Multi-Layered Models* - Explaining the RASP language and its deterministic, logical approach to modeling. - Highlighting the benefits of using multiple layers in language model architectures. 02:36:30 *🔍 Potential and Limitations of Formal Logic in LLMs* - Discussing the implications of formal logic approaches for understanding model capabilities. - Exploring the feasibility of implementing complex tasks using logical formulations in language models. Made with HARPA AI
@noninvasive_rectal_probe89902 ай бұрын
Lmao this talk is trash
@420_gunna2 ай бұрын
What do you think is bad about it? Haven't listened yet, but Sasha always puts out great content in the past.
@imaspacecreature2 ай бұрын
Wanted to hear more!
@ShivaprakashYaragal2 ай бұрын
This is awesome. I loves these tools and Taxi data
@user-wr4yl7tx3w2 ай бұрын
If Harvard wasn’t so woke. Only woke DEI look down upon Extension students.
@AlgoNudger2 ай бұрын
Thanks.
@AlgoNudger2 ай бұрын
Thanks.
@r2internet2 ай бұрын
Thanks for the informative talk. Can you please share the slides?
@harvarddatascienceinitiati33202 ай бұрын
Hi there, we have reached out to request permission to share the slides. We will add a link to the description if approved. Thank you!
@harvarddatascienceinitiati332027 күн бұрын
See the description!
@mikaackermann40722 ай бұрын
Why 360? How about a normal Video?
@kalmyk2 ай бұрын
you can just imagine formulas
@noomadeАй бұрын
what do you mean?
@jaredtweed78262 ай бұрын
Can you upload this with the slide not in VR
@harvarddatascienceinitiati33202 ай бұрын
Working on it :)
@RocketmanUT2 ай бұрын
Going to need to reupload this, the slides are distorted.
@harvarddatascienceinitiati33202 ай бұрын
Working on it :)
@jaredtweed78262 ай бұрын
Thank you!
@pankajsinghrawat10562 ай бұрын
make normal videos please
@harvarddatascienceinitiati33202 ай бұрын
Working on it :)
@travelcatchannel86572 ай бұрын
Thanks very much for this presentation. It helps a lot. Could you kindly tell me the tool you use for demo?
@user-wr4yl7tx3w3 ай бұрын
why does every harvard presentation have such a long preamble. just skip the first 10 minutes if you don't want to sit through it.
@KyleXLS3 ай бұрын
I'd love to have access to the data to play around with in ArcGIS Pro.
@TheNighter5 ай бұрын
Harvard should not exist.
@kinmacpherson93025 ай бұрын
'Promosm' 🤪
@mgophern6 ай бұрын
could you share slides?
@toddbrous_untwist Жыл бұрын
This was such an awesome video! Thank you HDSI for posting this.
@lorenzo2775 Жыл бұрын
I like your style!!!! #1 YT views provider -> Promo>SM!
@optimism90 Жыл бұрын
Thanks for sharing the tutorial video. Could you share the R code and slides if possible, thanks!
@marilinfykes7558 Жыл бұрын
p̷r̷o̷m̷o̷s̷m̷ 😑
@chinwevivianaliyu Жыл бұрын
Would it be possible to have a recorded version for those who registered?
@muhammadsyukri7462 жыл бұрын
Thanks so much indeed for this video. Helps me a lot
@haow852 жыл бұрын
Are their any open datasets for fair AI ?
@somewheresomeone39592 жыл бұрын
Great work and thanks for sharing! Is it just me or the video is a lil bit asynchronous of the voice and the pictures?
@somewheresomeone39592 жыл бұрын
Nvm I shut the page and reentered, everything works charmingly rn.
Пікірлер
How can I be part of this program?
This video can be a terrific example of overfitting 😁😆
Pleass my youtube compleat now
So amazing 😊
Yydhkxhxhx
Đìh
Please upload slides and perhaps sync it to the audio if you can. Can’t believe we are in age of AI and humans are already losing it
What are we even looking at here ? Are we at an edge of some black hole? Totally nauseating
like this
Not liking the current layout. It is not possible to make up a lot of detail (you can see a large room w people in it but nothing of value is discernible) in the small image, and placing it side by side w the presented material essentially makes that harder to see as well, resulting in a need for more screen estate or get used to watching things in half the size that is customary. How about instructing presenters to leave a small box area in their presentations and paste the room in that?
Please support The End and Prevention of Homelessness USA 2030!
If y’all could make sure you actually have clean audio in future that’d be great. The banging is super super distracting and brings down the quality of the whole thing.
Noted! Thank you for your feedback.
🎯 Key Takeaways for quick navigation: 00:12 *🎓 Introduction of Sasha Rush by David Parks* - Introduction of Sasha Rush, a professor at Cornell University and a contributor to Hugging Face, now part of Google, known for his work in AI and large language models. - Sasha's background in computer science, his contributions to AI research, and his commitment to open-source initiatives. 04:28 *🗣️ Overview of Large Language Models (LLMs)* - Large language models (LLMs) are discussed in terms of their significance, complexity, and implications. - LLMs are described as extremely useful, expensive, important, and sometimes perceived as intimidating, but also capable of providing remarkable outputs with consistency and creativity. - The talk aims to provide insights into LLMs and address questions regarding their functioning, reasoning, and impact on various domains. 07:38 *📊 Dividing LLM Understanding into Five Formulas* - Sasha introduces the concept of understanding LLMs through five formulas: perplexity, attention, gem, chinchilla, and rasp. - Each formula represents a different aspect of LLMs, such as generation, memory, efficiency, scaling, and reasoning, providing a comprehensive view of their functionalities. - These formulas aim to simplify complex concepts and enable a deeper understanding of LLMs for both technical and non-technical audiences. 11:33 *🔍 Understanding Perplexity in Language Models* - Perplexity in language models is explained, focusing on the probabilistic model of documents and the probability distribution of word sequences. - The Markov assumption and the representation of Theta as categorical distributions are discussed as historical aspects of language models. - Modern advancements in LLMs are highlighted, indicating departures from past assumptions and embracing more complex neural network architectures. - Language models' historical evolution is discussed, from early Markov models to modern neural network-based LLMs. - Changes in assumptions and approaches, such as fixed-length history and categorical representations, are contrasted with contemporary methods emphasizing neural network architectures. - The session concludes with reflections on the enduring relevance and evolution of language models in the field of natural language processing. 21:04 *🧮 Language Model Development Pre-2010* - Explanation of early methods to develop language models. - Discussion of Shannon's paper from 1948 outlining language model development. - Overview of the challenges and limitations of early language models. 22:32 *📊 Quantifying Language Model Performance* - Introduction to evaluating language model performance. - Explanation of the challenges in quantifying language model effectiveness due to the lack of a definitive correct answer. - Illustration of the evaluation process using an example sentence and the concept of word prediction difficulty. 29:17 *📏 Metric Relating to Compression and Shannon's Work* - Introduction to a metric related to language compression inspired by Shannon's work. - Explanation of encoding words with binary strings based on their probabilities. - Discussion on how the metric, perplexity, measures the efficiency of language compression. 38:21 *📈 Evaluation and Practical Applications of Perplexity* - Discussion on the practical evaluation of perplexity using a test set. - Exploration of the relationship between perplexity and model quality. - Overview of historical perplexity scores achieved by various language models and their implications. 44:09 *📊 Correlation between Perplexity and Task Accuracy* - Perplexity correlates almost perfectly with translation accuracy. - General language perplexity correlates extremely well with task accuracy on different tasks. 45:59 *📈 Impact of Perplexity Reduction on Model Performance* - Lower perplexity corresponds to better model performance. - Graphs illustrate the relationship between perplexity reduction and resource investment. 53:17 *🧠 Introduction to Memory and Attention Models* - Explains the concept of memory and attention in neural network models. - Introduces the limitations of Markov models and the need for attention mechanisms. 01:05:03 *🧠 Understanding Query, Key, and Value Operations in LLMs* - Query, key, and value operations in LLMs enable prediction of the next word based on contextual information. - The process involves querying a lookup table using the query, retrieving the key, and using it to predict the next word. 01:07:03 *🔄 Transitioning from Argmax to Softmax* - Argmax selection poses challenges in neural networks due to its discontinuous nature. - Softmax provides a smooth alternative for selecting the best choice, enabling meaningful training. 01:15:08 *📊 Evolution of Attention Mechanisms in Language Modeling* - The attention mechanism, popularized by the "Attention is All You Need" paper, revolutionized language modeling. - Transformers utilize attention mechanisms to process input sequences efficiently and capture long-term dependencies. 01:18:05 *🖥️ Efficient Computation with Attention Mechanisms* - Attention mechanisms efficiently compute long-term context by utilizing matrix operations. - Matrix multiplication and softmax operations allow for effective computation of attention scores. 01:27:10 *🖥️ Transition from CPUs to GPUs for efficient computation* - The shift from CPUs to GPUs revolutionized the efficiency of running applications like language models. - GPUs allowed for faster computation of complex operations like softmax, significantly altering the research landscape. 01:29:19 *🧠 Understanding GPU architecture and parallel computing* - GPUs operate as parallel computers with multiple threads running simultaneously. - Threads within blocks share block memory, enabling fast data exchange and computation. 01:34:24 *🧮 Efficient matrix multiplication on GPUs* - Matrix multiplication on GPUs involves loading data into block memory, performing computations within blocks, and minimizing reads from global memory. - Leveraging shared memory and parallel processing allows for efficient computation of matrix multiplication. 01:47:04 *💡 Maximizing GPU performance for ML applications* - ML applications benefit from efficient GPU performance, measured in ML operations per second (MLops). - Optimizing data formats, such as using smaller floating-point values, enhances GPU efficiency. 01:53:20 *🚀 GPU Optimization: Why Speed Matters* - GPU programming optimization is crucial for efficient computation. - Efficient GPU programming enables faster computation, which is essential for large-scale models like LLMs. 01:53:49 *🔍 Scaling in LLMs: Model Size vs. Training Data* - The performance of LLMs depends on the interplay between model size and training data. - Increasing model size and training data improves the model's ability to generalize and understand complex patterns. 01:54:42 *📊 Compute Optimization Formula: The Chinchilla Formula* - The Chinchilla Formula extrapolates the expected perplexity of LLMs based on model size and training data. - It suggests a proportional relationship between model size and training data for optimal performance. 02:11:35 *📊 Cost Comparison and Scaling Laws* - Understanding the costs associated with large language models (LLMs). - Comparing and incorporating costs related to data acquisition and model scaling. 02:13:09 *💰 Cost Considerations in Model Development* - Addressing cost considerations in LLM development, especially focusing on compute and data. - Exploring scenarios where reducing specific costs benefits certain stakeholders. 02:14:46 *🔄 Data Reusability and Synthesis* - Discussing the reusability of training data and its diminishing returns. - Exploring the potential of synthetic text generation and its current applications. 02:17:29 *📰 Importance of Training Data Quality* - Highlighting the significance of high-quality training data, such as from sources like the New York Times. - Discussing challenges in quantifying and assessing the quality of training data. 02:18:19 *🛑 Addressing Token Exhaustion Concerns* - Discussing concerns related to token exhaustion and its potential impact on model training. - Exploring strategies to address token scarcity, including alternative data sources. 02:32:52 *🧠 Exploring Formal Logic in Language Models* - Introducing a formal logic approach to understanding language model behavior. - Discussing the use of logical operations to manipulate and analyze model behavior. 02:34:57 *🔄 RASP Language and Multi-Layered Models* - Explaining the RASP language and its deterministic, logical approach to modeling. - Highlighting the benefits of using multiple layers in language model architectures. 02:36:30 *🔍 Potential and Limitations of Formal Logic in LLMs* - Discussing the implications of formal logic approaches for understanding model capabilities. - Exploring the feasibility of implementing complex tasks using logical formulations in language models. Made with HARPA AI
Lmao this talk is trash
What do you think is bad about it? Haven't listened yet, but Sasha always puts out great content in the past.
Wanted to hear more!
This is awesome. I loves these tools and Taxi data
If Harvard wasn’t so woke. Only woke DEI look down upon Extension students.
Thanks.
Thanks.
Thanks for the informative talk. Can you please share the slides?
Hi there, we have reached out to request permission to share the slides. We will add a link to the description if approved. Thank you!
See the description!
Why 360? How about a normal Video?
you can just imagine formulas
what do you mean?
Can you upload this with the slide not in VR
Working on it :)
Going to need to reupload this, the slides are distorted.
Working on it :)
Thank you!
make normal videos please
Working on it :)
Thanks very much for this presentation. It helps a lot. Could you kindly tell me the tool you use for demo?
why does every harvard presentation have such a long preamble. just skip the first 10 minutes if you don't want to sit through it.
I'd love to have access to the data to play around with in ArcGIS Pro.
Harvard should not exist.
'Promosm' 🤪
could you share slides?
This was such an awesome video! Thank you HDSI for posting this.
I like your style!!!! #1 YT views provider -> Promo>SM!
Thanks for sharing the tutorial video. Could you share the R code and slides if possible, thanks!
p̷r̷o̷m̷o̷s̷m̷ 😑
Would it be possible to have a recorded version for those who registered?
Thanks so much indeed for this video. Helps me a lot
Are their any open datasets for fair AI ?
Great work and thanks for sharing! Is it just me or the video is a lil bit asynchronous of the voice and the pictures?
Nvm I shut the page and reentered, everything works charmingly rn.
Thank you. Great panel discussion!
Glad you enjoyed it!