Scaling Social Impact: Applying Machine Learning to Targeted Email Campaigns at Change.org

# Scaling Social Impact: Applying Machine Learning to Targeted Email Campaigns at Change.org **Video Category:** Data Science & Engineering ## 📋 0. Video Metadata **Video Title:** Real world application of Machine Learning for Social Change **YouTube Channel:** Stanford **Publication Date:** October 10, 2014 **Video Duration:** ~1 hour ## 📝 1. Core Summary (TL;DR) Change.org faced a critical scaling problem: matching a rapidly growing user base with thousands of new weekly petitions without burning out their email lists through irrelevant spam. By transitioning from manual, category-based email targeting to a Machine Learning approach utilizing Random Forest classifiers, they could accurately predict individual user signing behavior based on past actions rather than broad demographics. This systemic shift decoupled email targeting from manual human curation, resulting in a 30% increase in email conversion rates and allowing a small engineering team to facilitate massive global social impact. ## 2. Core Concepts & Frameworks * **Concept:** Algorithmic Behavioral Targeting -> **Meaning:** The shift from manually grouping users into broad, static categories (e.g., "Animal Rights" or "Economic Justice") to predicting an individual's likelihood to take action based on their specific historical behavior. -> **Application:** Using metrics like past petition signatures, email open rates, and login frequency to decide exactly which campaigns land in a user's inbox. * **Concept:** Random Forest Classification -> **Meaning:** An ensemble machine learning algorithm that constructs a multitude of decision trees at training time and outputs the mode of the classes. -> **Application:** Used by Change.org to predict the binary outcome of whether a user will sign a specific petition, specifically chosen because it handles highly unbalanced datasets (where only ~2% of users take action) and mixed data types natively. * **Concept:** Out-of-Band (Stateless) Processing Pipeline -> **Meaning:** An architectural pattern where individual computational modules (data extraction, feature matrix generation, prediction) do not communicate directly, but instead read and write their state to a highly available central storage system. -> **Application:** Using Amazon S3 as the intermediary state holder allows different AWS modules (like Elastic MapReduce and Simple Workflow) to run asynchronously at their own pace without blocking the overall pipeline. ## 3. Evidence & Examples (Hyper-Specific Details) * **The Boy Scouts Policy Reversal:** Jennifer Tyrrell (a den leader kicked out for being gay) started a petition that gained over 300,000 signatures (growing to over 600,000). This sparked a movement, leading to Ryan Andresen (an 18-year-old denied his Eagle Scout award for being gay) getting over 1 million signatures. This targeted escalation culminated in the Boy Scouts of America ending their ban on gay youth 8 months later. * **Janis Shinwari Visa Campaign:** Matt Zeller, a US Army veteran, started a petition for his Afghan interpreter, Janis Shinwari, who was receiving death threats from the Taliban and was denied a US visa. The petition gained 113,861 signatures, resulting in the visa being approved and sparking a broader movement for allied interpreters. * **The Scale of the Targeting Problem:** By 2014, Change.org had 78 million global users, sending over 514,188,496 emails in the US alone, with over 6,000 new petitions started in the US every week. The email team consisted of only 4 people managing a list of 23 million US users. * **The Failure of Manual Targeting (Yale Sexual Assault Case):** Alexandra, a Yale sexual assault survivor, started a petition against the university. Because of its disturbing content, manual targeters struggled to fit it into their 12 broad "cause" buckets. When sent to a generic list of 500,000 users, it yielded a very poor 3-4% action rate. * **Machine Learning Rescue of the Yale Petition:** Once the Machine Learning algorithm was deployed, it identified an audience of over 1 million users for Alexandra's petition with a predicted 12% to 16% conversion rate. This accurate targeting elevated the campaign to national media attention and contributed to the Obama administration launching a Title IX task force. * **Model Evaluation & Selection:** Change.org evaluated Logistic Regression, Support Vector Machines (SVM), SV Data Description (SVDD), and Random Forests. SVDD resulted in low prediction scores and high false positives/negatives. SVM and Logistic Regression required "binarizing" (creating dummy variables for) all 500+ features, heavily increasing computational load. Random Forest won because it handled mixed data, performed its own feature selection, and easily interpreted the heavily unbalanced data (~2% positive conversion rate). * **The 30% Performance Boost:** By deploying the Random Forest model and utilizing a "sample, predict, threshold, and send" workflow, Change.org improved their overall email sign/send conversion rate by approximately 30% at no additional sending cost. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Prioritize behavioral data over demographic data.** - Do not rely primarily on user location, age, or gender for targeting. Base predictive models heavily on user activity metrics (e.g., email open rates, login frequency, and specific past actions) as they are vastly superior predictors of future conversion. * **Rule 2: Adopt Random Forest for highly imbalanced, mixed-data sets.** - If your positive outcome rate is exceptionally low (e.g., 2%) and your features include a mix of continuous and categorical data, use Random Forest classifiers. This avoids the computational penalty of binarizing hundreds of features required by SVM or Logistic Regression models. * **Rule 3: Implement a "Sample and Predict" campaign workflow.** - Before sending a mass email, send the campaign to a small, random sample set. Collect engagement data over 24 hours, feed that data into your predictive model to generate a conversion curve, and set a hard threshold (e.g., only send to users predicted to convert at >10%) to protect list health. * **Rule 4: Decouple data pipeline modules using cloud object storage.** - Prevent system bottlenecks by ensuring your extraction, feature generation, and prediction modules do not wait on each other. Use a central storage layer (like Amazon S3) to house state data, allowing independent orchestration tools (like AWS Simple Workflow) to spin up compute nodes only when preceding data is verified and ready. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Using manual, broad-category clustering for user segmentation. -> **Why it fails:** It lacks nuance, causing highly active users to get spammed with semi-relevant content (leading to list fatigue) while ignoring users with niche, hyper-specific interests that don't fit into generic buckets. -> **Warning sign:** High unsubscribe rates from active users, and massive lists yielding low single-digit conversion rates (e.g., the initial 3-4% on the Yale petition). * **Pitfall:** Attempting to run complex targeting queries directly on the primary transactional database. -> **Why it fails:** As data size and user load grow, analytical queries tie up database resources, causing the system to crash or significantly degrade the core user experience. -> **Warning sign:** Your MySQL (or primary SQL) database "falls over" or times out when marketing teams attempt to pull targeting lists. * **Pitfall:** Using Support Vector Data Description (SVDD) for non-circular data distributions. -> **Why it fails:** SVDD attempts to group data into spherical boundaries with a defined radius. If user behavior data does not cluster neatly in this geometric shape, the model generates excessive false positives and false negatives. -> **Warning sign:** Low prediction scores and poor real-world conversion when A/B testing the model. ## 6. Key Quote / Core Insight "Behavioral data fundamentally trumps demographic data. A small team leveraging cloud infrastructure and machine learning can achieve massive global impact by dynamically listening to what users actually do, rather than manually guessing what broad categories they belong to." ## 7. Additional Resources & References * **Resource:** Amazon Web Services (AWS) ecosystem - **Type:** Cloud Infrastructure - **Relevance:** The architecture heavily relied on specific AWS tools to scale: Elastic MapReduce (EMR) with Cascading for feature matrices, Simple Workflow (SWF) for dynamically orchestrating prediction nodes, S3 for stateless data handoffs, and Redshift for offline BI querying. * **Resource:** Engineers for Change (Code for Good) - **Type:** Internship/Fellowship Program - **Relevance:** An explicit program offered by Change.org for Stanford juniors, seniors, and grad students to apply technology and data science to real-world social impact problems.