Lecture 5: Deep Reinforcement Learning

# Lecture 5: Deep Reinforcement Learning **Video Category:** Machine Learning Lecture ## 📋 0. Video Metadata **Video Title:** Lecture 5: Deep Reinforcement Learning **YouTube Channel:** Stanford Engineering **Publication Date:** Not shown in video **Video Duration:** ~2 hours 20 minutes ## 📝 1. Core Summary (TL;DR) Deep Reinforcement Learning marries the feature extraction power of neural networks with the decision-making frameworks of reinforcement learning to solve complex, sequential problems. Rather than teaching an AI by example—which is limited by human capabilities and dataset bounds—RL trains agents through experience, optimizing for long-term discounted rewards. This paradigm shift has enabled breakthroughs ranging from superhuman gameplay in Go and StarCraft to aligning Large Language Models with human preferences via Reinforcement Learning from Human Feedback (RLHF). ## 2. Core Concepts & Frameworks * **Reinforcement Learning (RL):** -> **Meaning:** A machine learning paradigm where an agent learns to make good sequences of decisions by interacting with an environment, receiving observations ($o_t$) and delayed rewards ($r_t$). -> **Application:** Training an autonomous vehicle to navigate dynamically changing environments (like traffic lights) by maximizing safety and forward-progress rewards over time. * **Deep Q-Learning (DQN):** -> **Meaning:** Upgrading traditional Q-learning by replacing a tabular Q-value matrix (which breaks down in complex environments) with a neural network. It takes a state $s$ as input and outputs the predicted Q-values (expected future rewards) for all possible actions. -> **Application:** Playing Atari games from raw pixels, where the network evaluates the value of moving the joystick left, right, or staying idle. * **Experience Replay:** -> **Meaning:** A training stabilization technique where an agent stores its historical transitions $(s, a, r, s')$ in a memory buffer $D$, and the network is trained on randomly sampled mini-batches from this buffer. -> **Application:** Breaking the heavy sequential correlation of video frames (e.g., a ball moving across a screen) so the neural network doesn't overfit to a single trajectory and catastrophically forget previous lessons. * **Reinforcement Learning from Human Feedback (RLHF):** -> **Meaning:** A multi-step alignment pipeline that fine-tunes a model by first imitating human responses (SFT), then using a separate "Reward Model" trained on pairwise human preferences to optimize the language model using policy gradients (PPO). -> **Application:** Transforming a base next-token-prediction model into an aligned assistant like ChatGPT that understands polite, accurate, and preferred conversational formatting. * **Bellman Optimality Equation:** -> **Meaning:** The mathematical foundation of Q-learning defining the optimal value of a state-action pair: $Q^*(s, a) = r + \gamma \max_{a'} Q^*(s', a')$. It states the value of an action is the immediate reward plus the discounted maximum expected reward of the next state. -> **Application:** Calculating the target loss $y$ during the backpropagation step of a DQN. ## 3. Evidence & Examples (Hyper-Specific Details) * **Supervised Learning vs. RL in Go:** Katanforoosh notes that training a model on professional Go matches (Supervised Learning) fails because it only sees a fraction of the $13 \times 13$ board state space, and the "ground truth" is ill-defined (even professional players make sub-optimal moves). RL solves this by teaching the agent to explore and find its own optimal strategies rather than mimicking flawed, limited human data. * **The "Recycling is Good" Toy Environment:** A 5-state Markov chain used to explain Q-learning. State 1 is trash (+2 reward), State 5 is recycling (+10 reward). A discount factor ($\gamma = 0.9$) mathematically incentivizes the agent to pursue the distant but higher +10 reward rather than greedily grabbing a +1 immediate reward, effectively encoding the "value of time." * **Atari Breakout (DeepMind / Mnih et al., 2015):** The DQN agent is fed 4 stacked grayscale frames as the state input $\phi(s)$ (to capture velocity and direction, solving the problem of single-frame ambiguity). Without hardcoded rules, the agent eventually discovers the advanced "tunneling" strategy—breaking blocks on the far side to bounce the ball above the ceiling for maximum points. * **Competitive Self-Play (OpenAI Sumo):** In a continuous control environment where physics-based agents try to knock each other out of a ring, training an agent against itself forces an arms race of emergent behaviors. As they play thousands of iterations, agents naturally discover complex mechanics like tackling, lowering their center of gravity for stability, and defensive blocking. * **LLM Alignment Failure (Next-Token Prediction):** A model trained purely to predict the next word on internet text will answer the prompt "My laptop won't turn on. What should I do?" with unhelpful forum-style continuations like "Laptops sometimes don't turn on because of power issues." RLHF (Ouyang et al., 2022) fixes this by training the model against a reward function that prefers actionable, step-by-step troubleshooting. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Isolate temporal dynamics by stacking observations** - **[Action]** When your environment's single-frame state does not contain velocity or directional data (like a static screenshot of an Atari game) -> **[Mechanism]** concatenate the last $N$ frames (e.g., 4 frames) to form the state input $\phi(s)$ -> **[Result/Impact]** allowing the neural network to infer movement vectors and avoid Markov property violations. * **Rule 2: Implement Epsilon-Greedy policies to force exploration** - **[Action]** Introduce a probability threshold $\epsilon$ (e.g., 5%) where the agent ignores the Q-network and takes a completely random action -> **[Mechanism]** forcing the agent to deviate from its current known paths and visit unseen states -> **[Result/Impact]** allowing it to escape local minima and discover higher-yield long-term rewards. * **Rule 3: Use Experience Replay to break temporal correlation** - **[Action]** Store all state transitions $(s, a, r, s')$ in a replay memory buffer and train the DQN on randomly sampled mini-batches -> **[Mechanism]** mixing past and present experiences to create a diverse gradient signal -> **[Result/Impact]** preventing the neural network from catastrophically forgetting old states while navigating highly correlated sequential data. * **Rule 4: Transition to PPO for continuous action spaces** - **[Action]** If your environment requires continuous control (e.g., exact steering wheel angles or robotic joint torques) -> **[Mechanism]** abandon value-based DQN methods and implement Proximal Policy Optimization (PPO) to learn the policy distribution directly -> **[Result/Impact]** enabling stable training in complex, real-world physics environments. * **Rule 5: Define separate Reward Models for subjective tasks** - **[Action]** When the definition of a "good" output is subjective (like text generation) -> **[Mechanism]** train a separate neural network on human pairwise comparisons to output a scalar reward score -> **[Result/Impact]** creating an automated, scalable critic to guide the main policy network during RLHF. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Setting the discount factor ($\gamma$) to 1 in infinite environments. -> **Why it fails:** The agent will not value time, meaning it will delay reaching the goal indefinitely because a reward of 10 today is mathematically calculated as equal to a reward of 10 in a thousand years. -> **Warning sign:** The agent wanders aimlessly, loops, or takes unnecessarily long paths without converging on the final objective. * **Pitfall:** Treating Partial Observations as Full States. -> **Why it fails:** In games with "fog of war" (like StarCraft or Dota), treating the currently visible screen ($o_t$) as the absolute state ($s_t$) breaks the Markov property, making the agent unable to plan against hidden variables. -> **Warning sign:** The agent reacts purely reflexively to immediate threats and fails completely at long-term strategic setups. * **Pitfall:** Relying solely on Supervised Fine-Tuning (SFT) for LLM alignment. -> **Why it fails:** SFT only teaches the model to imitate the format and style of human text; it does not explicitly optimize for what humans actually *prefer* (helpfulness, accuracy, harmlessness). -> **Warning sign:** The model generates grammatically flawless but functionally useless, evasive, or misaligned answers. ## 6. Key Quote / Core Insight "In classic supervised learning, you teach by example. You show the model exactly what to do. But in reinforcement learning, you teach by experience. You let the agent experience the environment until it figures out the best decisions for itself, allowing it to surpass the very humans who built it." ## 7. Additional Resources & References * **Resource:** Human-level control through deep reinforcement learning (Mnih et al., 2015) - **Type:** Paper - **Relevance:** The foundational DeepMind paper demonstrating DQN solving Atari games. * **Resource:** Mastering the game of Go without human knowledge (Silver et al., 2017) - **Type:** Paper - **Relevance:** The DeepMind paper detailing AlphaGo Zero's architecture. * **Resource:** Proximal Policy Optimization Algorithms (Schulman et al., 2017) - **Type:** Paper - **Relevance:** Introduces PPO, the standard policy gradient algorithm used in modern LLM alignment and continuous control. * **Resource:** Training language models to follow instructions with human feedback (Ouyang et al., 2022) - **Type:** Paper - **Relevance:** The core OpenAI paper explaining the InstructGPT/ChatGPT RLHF pipeline. * **Resource:** AlphaGo (Netflix) - **Type:** Documentary - **Relevance:** Recommended by the lecturer for understanding the practical application and human impact of RL in complex strategy games.