Meta Reinforcement Learning: Adaptable Models & Policies

# Meta Reinforcement Learning: Adaptable Models & Policies **Video Category:** Machine Learning / Artificial Intelligence Lecture ## 📋 0. Video Metadata **Video Title:** Meta Reinforcement Learning Adaptable Models & Policies CS 330 **YouTube Channel:** Stanford Engineering **Publication Date:** Not shown in video **Video Duration:** ~1 hour 17 minutes ## 📝 1. Core Summary (TL;DR) Meta-Reinforcement Learning (Meta-RL) elevates standard reinforcement learning by training agents not just to solve a single task, but to "learn how to learn" across a distribution of tasks. By exposing an agent to a variety of environments or dynamics during meta-training, it develops an internal mechanism—either through recurrent memory or primed network weights—to rapidly adapt to a completely new task using only a tiny amount of new experience. This solves the chronic sample-inefficiency of standard RL, enabling applications like robots instantly adjusting to a broken leg or agents navigating unseen mazes after just one exploratory attempt. ## 2. Core Concepts & Frameworks * **Concept:** The Meta-RL Problem Formulation -> **Meaning:** Standard RL maps a state $s_t$ to an action $a_t$ to maximize return in a single Markov Decision Process (MDP). Meta-RL assumes a distribution of MDPs ($\mathcal{T}_i \sim P(\mathcal{T})$). The goal is to learn a mapping from a small, task-specific dataset of experience ($\mathcal{D}_{train}$) and the current state $s_t$ to an action $a_t$. Formally: $a_t = f(\mathcal{D}_{train}, s_t; \theta)$. -> **Application:** Designing a single neural network architecture that can act as both the learner and the policy across different physical environments. * **Concept:** Black-Box Meta-RL -> **Meaning:** This approach uses a general-purpose sequence model (like an LSTM, GRU, or Attention network) to represent the meta-learner $f$. The network takes sequences of states, actions, and rewards as input. The hidden state of the network implicitly serves as the "learned policy" for the specific task, updating dynamically as new experience is ingested. -> **Application:** Passing $(s_t, a_{t-1}, r_{t-1})$ into an RNN at each timestep, maintaining the hidden state across multiple episodes of the same task, so the agent remembers what it explored previously. * **Concept:** Optimization-Based Meta-RL -> **Meaning:** Instead of relying on an RNN's hidden state to adapt, this method embeds an actual optimization algorithm (like gradient descent) into the architecture. The meta-training process finds an optimal initialization of parameters ($\theta$) such that taking just one or a few gradient steps on the new task's data ($\mathcal{D}_{train}$) yields a highly performant adapted policy ($\phi_i$). -> **Application:** Using the MAML (Model-Agnostic Meta-Learning) algorithm to train a robot policy that can adapt to running backward or forward with a single gradient update. * **Concept:** Episodic vs. Online Meta-RL -> **Meaning:** In the Episodic variant, the agent adapts based on $k$ full rollouts (trajectories) from the task. In the Online variant, the agent adapts continuously based on the last $1 \dots k$ timesteps within the current, ongoing task. -> **Application:** Episodic is used for discrete trials (e.g., attempting a maze, failing, and trying again). Online is used for continuous adaptation (e.g., a robot suddenly losing a leg and needing to adjust its gait immediately). ## 3. Evidence & Examples (Hyper-Specific Details) * **Visual Maze Navigation (Black-Box / SNAIL):** The lecture references a study (Mishra et al., ICLR 2018) where an agent is trained on 1,000 small mazes using first-person visual inputs. The model uses an Attention + 1D Convolution architecture. At meta-test time on a new maze, an LSTM baseline takes an average of 52.4 steps in Episode 1 and drops to 39.1 steps in Episode 2. The SNAIL (attention) architecture takes 50.3 steps in Episode 1 and drastically drops to 34.8 steps in Episode 2, proving the network successfully memorizes the maze layout during the first exploratory episode. * **Simulated Continuous Control (PEARL vs. On-Policy):** Comparing off-policy to on-policy meta-RL (Rakelly et al., ICML 2019), robots (Half-Cheetah, Ant, Walker-2D) must adapt to new target velocities or directions. The evidence charts demonstrate that PEARL (off-policy) achieves maximum average return (e.g., ~600 on Ant-Goal-2D) in roughly $1 \times 10^6$ timesteps. Competing on-policy methods (RL2, ProMP, MAML) fail to reach even half that performance within the same timeframe, visually proving the massive sample-efficiency gains of using a replay buffer in meta-RL. * **Simulated Ant Direction Adaptation (MAML):** A video demonstration shows a simulated 4-legged Ant robot meta-trained with MAML. At test time (0 gradient steps), the ant merely runs in place because it does not know the target direction. After executing exactly 1 gradient step on a trajectory where the task is to run backward, the ant immediately adapts and runs smoothly backward. * **Real-World Dynamic Adaptation (VelociRoACH):** In a physical experiment (Nagabandi et al., ICLR 2019), a 6-legged robot is meta-trained to adapt its dynamics model online. When tested on a steep slope or with a missing front-right leg, a standard Model-Based RL agent drifts severely to the right off its intended path. The MAML-equipped agent, which updates its dynamics model using the last $k$ timesteps of experience continuously, corrects its trajectory and walks in a straight line despite the physical impairment. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Maintain hidden states across episodes within a task** - When building an RNN-based black-box meta-learner, never reset the hidden state at the end of an episode. The hidden state must persist across the $N$ episodes allocated for that specific task so the agent can leverage trial-and-error memory. Only reset the hidden state when switching to a completely new task. * **Rule 2: Feed previous actions and rewards as inputs** - To allow a black-box model to learn the mechanics of an environment, concatenate the current state $s_t$ with the previous action $a_{t-1}$ and previous reward $r_{t-1}$. This input vector ensures the network can correlate what it just did with the score it received. * **Rule 3: Decouple exploration from exploitation in testing** - When evaluating episodic meta-RL, structure the test phase into distinct phases: allocate the first episode strictly for environment exploration (even if it yields low reward), and expect the agent to exploit the gathered knowledge to maximize reward in the second and subsequent episodes. * **Rule 4: Use off-policy methods for sample efficiency** - If meta-training interactions are computationally expensive or require real-world robotics, use off-policy meta-RL architectures (like PEARL) that utilize experience replay buffers. Avoid on-policy methods (like standard policy gradients embedded in MAML) which require discarding data after every update step. * **Rule 5: Use attention for long-horizon meta-learning** - If the inner learning loop requires processing very long sequences of experience (e.g., thousands of timesteps) to extract task context, replace standard LSTMs with attention mechanisms (Transformers or SNAIL). This bypasses the vanishing gradient problem inherent in backpropagating through long recurrent paths. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Using sparse rewards with policy gradient inner-loops. -> **Why it fails:** Policy gradients require a non-zero reward signal to determine the direction of the gradient step. If the environment only provides a reward at the very end of a maze, and the initial unadapted policy fails to reach the end, the gradient is zero, and the model learns nothing. -> **Warning sign:** The agent's performance remains completely random across all adaptation steps, and loss metrics do not decrease. * **Pitfall:** Treating Meta-RL data as a standard supervised dataset. -> **Why it fails:** Standard supervised learning assumes i.i.d. (independent and identically distributed) data, allowing for random shuffling. Meta-RL data $\mathcal{D}_{train}$ is a temporal trajectory where actions dictate future states. Shuffling this data breaks the causal chain required for the agent to infer environment dynamics. -> **Warning sign:** The recurrent neural network fails to converge or learns a policy that ignores environment state changes. * **Pitfall:** Requiring too many gradient steps in Optimization-Based Meta-RL. -> **Why it fails:** Algorithms like MAML require backpropagating through the optimization process itself (computing Hessian-vector products). If you design the inner loop to require 50 gradient steps, the computational graph becomes prohibitively deep, leading to massive memory consumption and slow training. -> **Warning sign:** Out-of-memory (OOM) errors during training or agonizingly slow iteration times compared to standard RL. ## 6. Key Quote / Core Insight "The key idea of optimization-based meta-RL is to embed the actual optimization process—the gradient descent itself—inside the inner learning loop. You are not just training a network to act; you are training a network's initialization so that it is highly sensitive to new data, allowing it to adapt its behavior entirely after just one or two gradient steps." ## 7. Additional Resources & References * **Resource:** *Learning to Learn with Gradients* - **Type:** PhD Thesis (Chelsea Finn, 2018) - **Relevance:** The foundational document establishing the framework for Model-Agnostic Meta-Learning (MAML). * **Resource:** *RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning* (Duan et al., 2017) - **Type:** Paper - **Relevance:** Introduces the core concept of black-box meta-RL using recurrent neural networks. * **Resource:** *A Simple Neural Attentive Meta-Learner* (Mishra et al., ICLR 2018) - **Type:** Paper - **Relevance:** Details the SNAIL architecture, proving attention mechanisms outperform LSTMs in complex visual meta-RL tasks. * **Resource:** *Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables* (Rakelly et al., ICML 2019) - **Type:** Paper - **Relevance:** Introduces PEARL, crucial for understanding how to solve the sample-inefficiency problem in meta-RL using off-policy data. * **Resource:** *Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks* (Finn et al., ICML 2017) - **Type:** Paper - **Relevance:** The original paper introducing the MAML algorithm. * **Resource:** *Learning to Adapt in Dynamic Real-World Environments through Meta-RL* (Nagabandi et al., ICLR 2019) - **Type:** Paper - **Relevance:** Demonstrates the practical application of MAML with Model-Based RL on physical robots dealing with online perturbations.