Stanford CS224N: Self-Attention and Transformers

# Stanford CS224N: Self-Attention and Transformers **Video Category:** Natural Language Processing Tutorial / Deep Learning Lecture ## 📋 0. Video Metadata **Video Title:** Stanford ENGINEERING **YouTube Channel:** Stanford Engineering **Publication Date:** Not shown in video **Video Duration:** ~1 hour 17 minutes ## 📝 1. Core Summary (TL;DR) This lecture details the paradigm shift in Natural Language Processing from Recurrent Neural Networks (RNNs) to Transformer architectures. It explains how the sequential nature of RNNs creates bottlenecks in parallelization and struggles with long-range dependencies due to linear interaction distances. To solve this, the lecture introduces Self-Attention, a mechanism that allows every word in a sequence to directly interact with every other word in $O(1)$ sequential steps, heavily utilizing GPU parallelization. By combining self-attention with multi-head mechanisms, position embeddings, and residual connections, the Transformer model has become the foundational building block for modern, highly scalable language models. ## 2. Core Concepts & Frameworks * **Linear Interaction Distance:** -> **Meaning:** In RNNs, information must pass through every intermediate hidden state between two words. The interaction time between word $i$ and word $j$ is $O(|i-j|)$. -> **Application:** Causes problems with vanishing/exploding gradients over long sequences, making it difficult for the network to realize two distant words are syntactically or semantically related. * **Self-Attention:** -> **Meaning:** A mechanism that treats each word's representation as a "query" to perform a soft, fuzzy lookup against a set of "keys" (all other words in the sequence), pulling in information from their corresponding "values". -> **Application:** Replaces recurrence by processing the entire sequence simultaneously, allowing a word at the end of a sentence to directly incorporate information from the first word in a single computational step. * **Key, Query, Value (K, Q, V):** -> **Meaning:** Three separate representations derived from the input word embedding via learned weight matrices. The *Query* determines what information the word is looking for, the *Key* determines what information a word holds, and the *Value* is the actual content passed along if a match occurs. -> **Application:** Provides the necessary expressivity for attention; without separate matrices, the attention score would simply be a dot product of a word with itself, heavily biasing the model toward the identity function. * **Multi-Head Attention:** -> **Meaning:** Running the self-attention mechanism multiple times in parallel, where each "head" has its own set of learned Q, K, and V weight matrices mapping to a lower-dimensional space. -> **Application:** Allows the model to look at different aspects of the sequence simultaneously (e.g., one head might attend to syntactic dependencies like subject-verb, while another attends to semantic entities). * **Positional Representations:** -> **Meaning:** Because self-attention is a set operation with no inherent notion of sequence order, position vectors must be added to the input embeddings. -> **Application:** Can be implemented as learned absolute position matrices or fixed sinusoidal functions of varying periods, allowing the model to distinguish between "Dog bites man" and "Man bites dog". * **Masked Attention:** -> **Meaning:** Forcing the attention scores of future tokens to negative infinity ($-\infty$) before the softmax step. -> **Application:** Strictly required in Transformer *Decoders* (autoregressive language models) to prevent the model from "cheating" by looking ahead at the word it is supposed to predict during training. ## 3. Evidence & Examples (Hyper-Specific Details) * **Linear Locality vs. Long-Distance Dependencies Example:** - *Short distance:* In "tasty pizza", the RNN handles the relationship easily because the words are adjacent. - *Long distance:* In the sentence "The **chef** who went to the stores and picked up the ingredients and loves garlic **was**...", the RNN requires many sequential steps to apply the recurrent weight matrix and non-linearities to connect "chef" to "was", degrading the gradient signal. * **GPU Parallelization Bottleneck in RNNs:** The forward and backward passes have $O(\text{sequence length})$ unparallelizable operations. To compute the hidden state $h_3$, the GPU must wait for $h_2$, which waits for $h_1$. A GPU cannot chunk the matrix multiplications across the sequence length. * **Self-Attention as Fuzzy Lookup Demonstration:** - Standard dictionary: Query `d` matches Key `d` exactly to return Value `v4`. - Self-attention: Query `q` is compared to Keys `k1, k2, k3, k4`. It yields similarity scores between 0 and 1 via a softmax function. The output is a weighted sum: $0.1(v1) + 0.2(v2) + 0.6(v3) + 0.1(v4)$. * **Self-Attention Vector Math Example:** To represent the word "learned" in the sentence "I went to Stanford CS 224n and learned", the model computes the dot product of the Query vector for "learned" against the Key vectors for "I", "went", "to", "Stanford", "CS", "224n", "and", "learned". High dot products (e.g., with "CS" and "224n") result in higher weights in the final Value sum. * **Sinusoidal Position Matrix Visualization:** Shown on-screen as a matrix where the horizontal axis is the sequence index and the vertical axis is the dimension. The values oscillate between -1 and 1. Lower dimensions have high-frequency periods (changing rapidly word-to-word), while higher dimensions have low-frequency periods, allowing the model to triangulate absolute and relative positions. * **Transformer Performance vs. RNN Ensembles (WMT 2014):** - GNMT + RL Ensemble (RNN): 26.36 BLEU (EN-DE), $8.0 \times 10^{20}$ Training Cost (FLOPs). - Transformer (Base): 27.3 BLEU (EN-DE), $3.3 \times 10^{18}$ Training Cost (FLOPs). - Transformer (Big): **28.4** BLEU (EN-DE), $2.3 \times 10^{19}$ Training Cost (FLOPs). The Transformer achieved higher accuracy at a fraction of the computational cost. * **Residual Connections Visual Evidence:** A slide referencing the ResNet paper (He et al., 2016) shows 3D visualizations of a loss landscape. Without residuals, the landscape is chaotic with many local optima. With residuals, the landscape is smooth and convex-like, demonstrating why models train significantly faster. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Scale dot products by $\sqrt{d/h}$** - When implementing the attention formula $\text{softmax}(Q K^T)$, divide the dot products by the square root of the head dimensionality ($d/h$). As dimension size grows, dot products become exponentially larger, which pushes the softmax function into its flat tails where gradients are nearly zero. * **Rule 2: Restrict sequence lengths or use specialized variants** - Standard self-attention has a memory and compute cost of $O(n^2 \cdot d)$ where $n$ is sequence length. If you attempt to pass a 50,000-token document into a standard Transformer, the $Q K^T$ matrix multiply will cause an Out-Of-Memory (OOM) error. Use models like Linformer for ultra-long contexts. * **Rule 3: Use feed-forward networks (MLPs) after attention** - Self-attention only computes weighted averages; it contains no element-wise non-linearities. You must pass the output of every self-attention block through a Feed-Forward network (typically Linear -> ReLU -> Linear) to give the network deep learning expressivity. * **Rule 4: Do not share Q, K, V weights across attention heads** - When configuring Multi-Head Attention, initialize independent weight matrices ($W_Q, W_K, W_V$) for each head. Sharing them forces the model to look at the exact same relationships redundantly, wasting compute capacity. * **Rule 5: Apply Layer Normalization inside the block** - To stabilize training and speed up convergence, wrap the inputs to your self-attention and feed-forward layers in Layer Normalization. (Note: Modern implementations often apply LayerNorm *before* the attention/FFN operations, known as Pre-LN). * **Rule 6: Use distinct Encoder/Decoder attention patterns** - - If building a translation model (Encoder-Decoder): The Encoder uses unmasked self-attention (bidirectional). The Decoder uses *masked* self-attention for its own inputs, AND *cross-attention* where the Queries come from the Decoder, but Keys and Values come from the Encoder's final output. - If building a pure language model (like GPT): Use only a Decoder stack with masked self-attention. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Using single-head attention with large dimensions. -> **Why it fails:** The network is forced to average all relevant context into a single vector representation, washing out distinct syntactic and semantic signals. -> **Warning sign:** Sub-optimal performance compared to a multi-head setup using the same total parameter count. * **Pitfall:** Forgetting to mask the future in a Decoder. -> **Why it fails:** The autoregressive model will calculate the Query for word $t$ using the Key for word $t+1$. During training, it perfectly predicts the next word by literally looking at it. During inference (where $t+1$ doesn't exist yet), the model fails completely. -> **Warning sign:** Near-zero training loss but nonsensical text generation during inference. * **Pitfall:** Assuming learned absolute position embeddings extrapolate to longer sequences. -> **Why it fails:** If a model is trained with a maximum sequence length $n=512$, the learned position matrix $P \in \mathbb{R}^{d \times 512}$ has no values for index 513. -> **Warning sign:** The model crashes with an index out-of-bounds error if fed a sequence longer than its training context window. * **Pitfall:** Omitting Key and Query matrices (using raw embeddings). -> **Why it fails:** If $Q = X$ and $K = X$, the dot product $X X^T$ will heavily favor a word matching with itself ($x_i \cdot x_i$ is highly positive). -> **Warning sign:** The attention mechanism collapses into an identity function, ignoring the surrounding context. ## 6. Key Quote / Core Insight "The shift to self-attention allows us to look at the entire sequence at once. It abandons the recurrent left-to-right processing, eliminating the linear interaction distance bottleneck. But because it's fundamentally an operation on an unordered set, we must explicitly inject the notion of sequence order back into the model using positional representations." ## 7. Additional Resources & References * **Resource:** "Attention Is All You Need" (Vaswani et al., 2017) - **Type:** Academic Paper - **Relevance:** The original paper that introduced the Transformer architecture, establishing the standard for modern NLP. * **Resource:** "Deep Residual Learning for Image Recognition" (He et al., 2016) - **Type:** Academic Paper - **Relevance:** Introduced residual connections, essential for training the deep layer stacks found in Transformers. * **Resource:** "Layer Normalization" (Ba et al., 2016) - **Type:** Academic Paper - **Relevance:** Introduced the normalization technique used inside Transformer blocks to stabilize gradients. * **Resource:** Linformer (Wang et al., 2020) - **Type:** Academic Paper - **Relevance:** Mentioned as recent work attempting to solve the $O(n^2)$ quadratic compute cost of standard self-attention for longer sequences. * **Resource:** GLUE Benchmark (Liu et al., 2018) - **Type:** Evaluation Benchmark - **Relevance:** The standard aggregate benchmark used to demonstrate that Transformer-based models completely dominate NLP tasks.