Model-Based Preference Optimization: Metric Elicitation

# Model-Based Preference Optimization: Metric Elicitation **Video Category:** Machine Learning Lecture ## 📋 0. Video Metadata **Video Title:** Model-Based Preference Optimization: Metric Elicitation **YouTube Channel:** Stanford Engineering **Publication Date:** Autumn 2024 **Video Duration:** ~124 minutes ## 📝 1. Core Summary (TL;DR) This lecture introduces the concept of "Metric Elicitation," which addresses the critical problem of selecting the right evaluation metric for machine learning models, particularly in cost-sensitive classification scenarios. Instead of defaulting to standard metrics like accuracy or RMSE, metric elicitation treats the determination of the evaluation metric itself as a machine learning problem. By efficiently querying human stakeholders with pairwise comparisons between different model outcomes, practitioners can systematically uncover the true underlying utility function and build models that align with real-world costs and values. ## 2. Core Concepts & Frameworks * **Metric Elicitation:** -> **Meaning:** The process of determining the appropriate evaluation metric for a machine learning task by interacting with individual stakeholders or groups to understand their preferences regarding different types of errors. -> **Application:** Used when the default metric (like accuracy) does not accurately reflect the real-world cost of model mistakes, such as in medical diagnosis or criminal justice. * **Cost-Sensitive Classification:** -> **Meaning:** A classification paradigm where different types of errors (e.g., false positives vs. false negatives) carry different asymmetric penalties or costs. -> **Application:** In a cancer diagnosis model, a false negative (missing cancer) is significantly more costly than a false positive (a false alarm). The model must be optimized for this specific cost asymmetry. * **Confusion Matrix Space (Feasible Set):** -> **Meaning:** The set of all possible confusion matrices that can be achieved by applying different decision thresholds to a probabilistic classifier over a specific data distribution. This forms a compact convex set. -> **Application:** By mapping out this space, you can identify the Pareto frontier (the boundary) of optimal models. The goal of metric elicitation is to find the specific point on this boundary that maximizes the stakeholder's utility. * **Active Learning for Elicitation:** -> **Meaning:** Using a sequential query strategy to explicitly seek out the most informative examples to present to a human oracle, minimizing the number of questions needed to learn their preferences. -> **Application:** Instead of asking a user to evaluate random pairs of models, the system uses binary search (or probabilistic bisection) along the Pareto frontier to rapidly hone in on the exact trade-off parameter the user prefers. * **Cooperative Inverse Decision Theory (CIDT):** -> **Meaning:** A framework where an imitator (the machine) seeks to learn the decision rule matching a demonstrator's (the human's) preferences by querying them. -> **Application:** Used to formalize the process of an AI system learning the preferred threshold for a classification task by actively querying a human expert with specific data points or model comparisons. ## 3. Evidence & Examples (Hyper-Specific Details) * **Cancer Diagnosis (False Positives vs. False Negatives):** The speaker presents a scenario with two models. Model A has 94.1% accuracy but a 5% false negative rate. Model B has 89.6% accuracy but a 1% false negative rate. Despite lower overall accuracy, Model B is preferable in a medical context because false negatives (missing a cancer diagnosis) result in severe harm, whereas false positives (false alarms), while expensive, are less catastrophic. * **Criminal Justice & COMPAS Algorithm (ProPublica Analysis):** The lecture highlights the COMPAS recidivism prediction algorithm. ProPublica's 2016 analysis showed that while the model might have seemed fair on some metrics, it had a significantly higher false positive rate for Black defendants (they were twice as likely to be incorrectly labeled as high risk compared to white defendants). This demonstrates that picking a metric without considering the specific costs of false positives (unjust incarceration) leads to harmful, biased real-world outcomes. * **The Netflix Prize (Metric Mismatch):** Netflix offered $1M to improve their recommendation algorithm. The competition was structured to optimize for Root Mean Square Error (RMSE) on predicting exact star ratings (1 to 5). However, it was later discovered that improvements in RMSE did not translate into better "Top-N ranking" accuracy (which is what users actually experience on the dashboard). Netflix ultimately did not deploy the winning ensemble algorithm because it was too computationally expensive and the metric (RMSE) didn't align with the actual business goal (providing a good top-5 list). * **Linear Binary Classification Metric Equation:** The lecture defines the target metric to elicit as: $\phi^*(C(h)) = 1 - (a_1^* FP(h) + a_2^* FN(h))$. The goal of the elicitation process is to find the optimal ratio of the weights $a_1^*$ and $a_2^*$, which dictates how much more a false positive costs relative to a false negative. * **Probabilistic Bisection Algorithm (PBA):** To handle noisy human feedback (where an oracle might occasionally pick the wrong model during a pairwise comparison), the speaker introduces PBA. This algorithm maintains a belief distribution over the optimal trade-off parameter $\tau^*$ and updates this distribution using Bayesian updates after every query, ensuring the system converges to the true metric even if the human makes mistakes. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Stop defaulting to accuracy or RMSE.** - Before training a model, explicitly map out the real-world costs of different error types. If a false positive costs more than a false negative, standard accuracy is invalid. You must define a cost-sensitive metric. * **Rule 2: Do not ask humans for abstract weights.** - Humans are terrible at answering questions like "What is the relative weight of a false positive versus a false negative?" Instead, use pairwise comparisons. Present the human with the outcomes of two different models (e.g., Model A has 10 FPs and 2 FNs; Model B has 5 FPs and 8 FNs) and ask: "Which model's outcome do you prefer?" * **Rule 3: Search along the Pareto frontier.** - When presenting options to stakeholders, only present models that lie on the optimal boundary of the confusion matrix space (the ROC curve equivalent). Do not waste human query budget comparing sub-optimal models. * **Rule 4: Use binary search for linear metrics.** - If you are eliciting a linear combination of false positives and false negatives, the optimal models lie on a 1D curve. You can use binary search to find the optimal trade-off parameter in $O(\log(1/\epsilon))$ queries. * **Rule 5: Implement noise-tolerant search for human feedback.** - Because humans make mistakes or provide inconsistent feedback, replace standard binary search with the Probabilistic Bisection Algorithm (PBA) to maintain a robust belief distribution over the optimal metric threshold. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Assuming metrics are exchangeable. -> **Why it fails:** Optimizing for a proxy metric (like RMSE) does not guarantee improvements in the actual task metric (like Top-N ranking). -> **Warning sign:** You achieve state-of-the-art performance on your loss function, but user experience, engagement, or business KPIs do not improve. * **Pitfall:** Asking stakeholders to mathematically define their utility function. -> **Why it fails:** Stakeholders lack the technical vocabulary and cognitive calibration to assign precise numerical weights to abstract error rates. -> **Warning sign:** Stakeholders give conflicting weights, or the resulting model behaves in ways the stakeholders immediately reject upon deployment. * **Pitfall:** Randomly sampling models for pairwise comparison. -> **Why it fails:** Random sampling is highly sample-inefficient and frustrates the human oracle by asking them to evaluate clearly inferior or redundant models. -> **Warning sign:** The human labeler gets bored or fatigued quickly, and the algorithm requires thousands of queries to converge on a metric. ## 6. Key Quote / Core Insight "The choice of your evaluation or utility function can be just as critical as every other algorithmic choice you make in your model design. If you pick the wrong metric, even a perfectly optimized model will deliver a bad real-world outcome." ## 7. Additional Resources & References * **Resource:** "Performance metric elicitation from pairwise classifier comparisons" by Hiranandani et al. - **Type:** Paper - **Relevance:** Foundational paper for the core methodology discussed in the lecture regarding eliciting metrics via pairwise comparisons. * **Resource:** "Machine Bias" by Julia Angwin et al. (ProPublica, 2016) - **Type:** Article/Analysis - **Relevance:** The primary case study used to demonstrate the real-world harm of ignoring false positive rates in criminal justice algorithms. * **Resource:** "Cooperative Inverse Decision Theory for Uncertain Preferences" by Robertson et al. (AISTATS 2023) - **Type:** Paper - **Relevance:** Details the methodology for eliciting thresholds using probabilistic bisection algorithms to handle noisy feedback. * **Resource:** "Results show that improvements in RMSE often do not translate into [top-N ranking] accuracy improvements." by Cremonesi, Koren, Turrin (2010) - **Type:** Paper - **Relevance:** The core citation proving the failure of RMSE as a proxy metric in the Netflix Prize scenario.