hackertube.io

LSTM is dead. Long Live Transformers!

Seattle Applied Deep Learning · lecture · Watch on YouTube

LLMs & generative AIMachine learning & deep learningComputer vision, speech & NLP

What it is

Leo Dirac explains how transformer models have replaced LSTMs in NLP. He gives background from bag of words through RNNs and LSTMs, then a technical deep dive into multi-headed self-attention and positional encoding, with sample code.

Why Hacker News recommends it

Recommended as a clear explanation of why RNNs fall short and how transformers solve it.

  • 4 people recommended it
  • 4 different months
  • 1 strong recommendations
  • first 2020, last 2023

What HN commenters said

  1. “has gotten tons of compliments and keeps people watching year after year”

    SDE-level explanation of transformers in NLP context, well complimented.

    leopd, 2023-05 · Strong recommendation · in “Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper”

  2. “This lecture by Leo Dirac was extremely helpful to finish off with”

    Lecture with pseudocode and why transformers are needed.

    FL33TW00D, 2020-10 · Recommendation · in “Rethinking Attention with Performers”

  3. “this video worked better for me”

    Worked better for commenter than Yannic Kilcher's video.

    peterfirefly, 2023-01 · Recommendation · in “The Transformer Family”

  4. “This video explains the drawbacks to RNNs and how transformers solve that”

    Explains RNN drawbacks and how transformers solve them.

    qigtyofhp, 2023-02 · Recommendation · in “Understanding and Coding the Self-Attention Mechanism of Large Language Models”

Excerpts from Hacker News comments, linked to the originals. Summaries and labels were written by a language model from those comments.

Topics: transformers, lstm, self-attention, positional encoding, nlp

Also recommended by the same people