Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

推荐理由

I highly recommend Sasha Rush's video on it

强烈推荐a_wild_dandan · 原文2024年3月

说明

这是2023年由Sasha Rush发布的一部课堂讲座，主题归在Machine learning & deep learning。它讲的是state space models这一支里的具体问题，而不是把整个领域扫一遍。适合想看推导、证明或实现细节的人。内容上可以概括成：Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.这些读者给出的理由，核心是：HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.它贴着当时的技术，看的时候要记得那是那个年份的判断。从2024年到2024年，有3位不同的读者在3个不同月份把它推荐出来，其中明确叫别人去看的有1次。节奏接近上课，可以暂停做笔记。它假设你愿意跟着一条推导走完。如果已经会了基本定义，就把注意力放在它怎么把state space models变成能跟着复述的例子。

视频信息

频道

Sasha Rush

时长

39 分 43 秒

发布年份

2023

同主题更多

• The spelled-out intro to neural networks and backpropagation: building micrograd
• But what is a neural network? | Deep learning chapter 1
• AlphaGo - The Movie | Full award-winning documentary
• Transformers, the tech behind LLMs | Deep Learning Chapter 5
• USENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
• Lecture 1 | Machine Learning (Stanford)