Stanford CS224U: Natural Language Understanding - Sentiment Analysis and the SST

# Stanford CS224U: Natural Language Understanding - Sentiment Analysis and the SST **Video Category:** Natural Language Processing Tutorial ## 📋 0. Video Metadata **Video Title:** Stanford CS224U: Natural Language Understanding | Spring 2019 | Lecture 3 - Sentiment Analysis **YouTube Channel:** Stanford Engineering **Publication Date:** Spring 2019 (Slides indicate April 15 and 17) **Video Duration:** ~1 hour 11 minutes ## 📝 1. Core Summary (TL;DR) This lecture establishes sentiment analysis not as a superficial text classification task, but as a deep and complex Natural Language Understanding (NLU) challenge. It demonstrates how seemingly minor preprocessing decisions—like tokenization and stemming—can drastically alter a model's ability to interpret emotion. By introducing the Stanford Sentiment Treebank (SST) and providing a framework for robust experimentation, the lecture equips practitioners to build models that capture the nuances, negations, and compositional nature of human sentiment. ## 2. Core Concepts & Frameworks * **Sentiment Analysis:** -> **Meaning:** The computational task of determining the emotional tone, polarity (positive, negative, neutral), or affective state expressed in text. -> **Application:** Used in business for product review mining, social media monitoring, and customer feedback analysis. * **Stanford Sentiment Treebank (SST):** -> **Meaning:** A sentence-level corpus based on movie reviews where every sub-phrase (node) in the parse tree is annotated with a 5-way sentiment label (from very negative to very positive). -> **Application:** Training models to understand compositional semantics—how the sentiment of individual words combines to form the sentiment of a phrase or sentence (e.g., handling negations or complex clauses). * **Sentiment-Aware Tokenization:** -> **Meaning:** The process of segmenting text into words while explicitly preserving elements that carry emotional weight, such as emoticons, capitalization, and hashtags, rather than treating them as disposable punctuation. -> **Application:** Preprocessing raw social media text (like tweets) before feature extraction to prevent the loss of critical signals like `>:-D` or `YAAAAAAY!!!`. * **Stemming:** -> **Meaning:** A heuristic process of stripping prefixes and suffixes from words to collapse them into a base root or "stem" (e.g., using Porter or Lancaster algorithms) to reduce feature space sparsity. -> **Application:** Generally useful in information retrieval, but **highly discouraged** in sentiment analysis because it destroys morphological distinctions that carry opposite polarities (e.g., collapsing "defense" and "defensive"). * **Macro-Averaging:** -> **Meaning:** An evaluation metric that calculates the performance (like F1-score) independently for each class and then takes the unweighted mean, treating all classes equally regardless of their frequency in the data. -> **Application:** The default evaluation method for sentiment tasks, which often suffer from class imbalance (e.g., many more neutral reviews than extreme ones), ensuring the model doesn't just optimize for the majority class. * **Model Wrappers / Experiment Frameworks:** -> **Meaning:** Code structures (like `sst.experiment`) that package data loading, feature extraction, model fitting, and evaluation into a single, repeatable pipeline. -> **Application:** Enables rapid, reproducible iteration over different hyperparameter or feature combinations while preserving artifacts (like the vectorizer and raw text) necessary for deep error analysis. ## 3. Evidence & Examples (Hyper-Specific Details) * **Bake-off 1 Procedure:** The speaker details the strict honor-code process for evaluating the first assignment. Students download test data (`http://web.stanford.edu/class/cs224u/data/bakeoff1-wordsim-test-data.zip`), place it in `WORDSIM_HOME`, and run a specific evaluation cell against the `mturk287` and `simlex999` datasets. The rule is explicit: `full_word_similarity_evaluation(custom_df, readers=BAKEOFF)` must be run exactly once. Tuning the `custom_df` model based on this final score is a violation of the rules. * **Conceptual Edge Cases in Sentiment:** * *"There was an earthquake in California."* - Typically negative, but potentially positive for a seismologist. * *"The team failed to complete the physical challenge."* - Sentiment depends entirely on whether it's your team or the opposing team. * *"They said it would be great, and they were wrong."* - Demonstrates implicit negation where the final clause overrides the positive expectation. * *"The party fat-cats are sipping their expensive imported wines."* - Contains positive words ("expensive", "imported") but carries a negative, mimicking, or sarcastic affect. * *"Oh, you're terrible!"* - Negative literal meaning, but often used positively as a term of endearment or social bonding among friends. * *"Many consider the masterpiece bewildering, boring, slow-moving or annoying..."* (2001 Movie Review) - Packed with negative lexicon, but the sentence structure implies the author disagrees with the critics and actually likes the film. * **The Business Pitfall of Aggregate Sentiment:** A slide shows a pie chart comparing Q1 (30% negative, 70% positive) to Q2 (35% negative, 65% positive). The speaker emphasizes that this high-level aggregation is practically useless for decision-makers because it hides the latent phenomena—the "why"—driving the 5% shift. * **Tokenization Failure Example:** A tweet is shown: `@NLUers: can't wait for the Jun 9 #projects! YAAAAAAY!!! >:-D http://stanford.edu/class/cs224u/` * The standard Treebank tokenizer shatters the emoticon into `>`, `:`, `-`, `D` and splits `#projects` into `#` and `projects`. * A Sentiment-aware tokenizer (`nltk.tokenize.casual.TweetTokenizer`) perfectly preserves the emoticon `>:-D`, the hashtag `#projects`, and the repeated letters in `YAAAAAAY!!!`. * **Tokenization Performance Data:** A plot showing 10-fold cross-validated mean accuracy on 6,000 OpenTable reviews. The Sentiment-aware tokenizer (orange line, ~0.884 accuracy) consistently outperforms the Treebank tokenizer (green line, ~0.873) and Whitespace tokenizer (gray line) across all training set sizes from 250 to 6,000 examples. * **Stemming Destruction Evidence (Harvard Inquirer):** * The Porter stemmer collapses "extravagance" (Positive) and "extravagant" (Negative) into `extravag`. * The Lancaster stemmer collapses "compliment" (Positive), "dependability" (Positive), "complicate" (Negative), and "depend" (Negative) into just two stems: `comply` and `depend`. * A performance plot on OpenTable data shows that using Porter or Lancaster stemming results in strictly lower accuracy (around 0.871) compared to the sentiment-aware unstemmed baseline (0.884). * **Simple Negation Marking Mechanism:** Based on Das & Chen (2001), a heuristic appends `_NEG` to every word between a negation and a punctuation mark. * Example: *"I don't think I will enjoy it, but I might."* -> `i don't think_NEG i_NEG will_NEG enjoy_NEG it_NEG , but i might.` * A chart demonstrates that adding this heuristic to the sentiment-aware tokenizer provides a consistent accuracy boost across all training data sizes. * **SST Compositional Parsing:** A slide shows a fully labeled tree for "NLU is amazing". The node for "is amazing" is labeled 4 (very positive), which projects up to label the entire sentence "NLU is amazing" as 4. Another tree shows how "they were wrong" (labeled 1, negative) overrides the preceding context to make the root node negative. * **Part-of-Speech Tagging for Disambiguation:** The lecture shows how POS tags can separate sentiment. For example, "arrest" as an adjective ("an arresting performance") is Positive in the Harvard Inquirer, but "arrest" as a verb is Negative. "Fine" as a noun ("a parking fine") is Negative, but as an adjective or adverb, it is Positive. * **Code Implementation (`sst.py`):** * `unigrams_phi(tree)` function: Takes an NLTK tree, calls `tree.leaves()` to get the words, and returns a `collections.Counter` dictionary to create a bag-of-words feature set. * `fit_softmax_classifier(X, y)`: Uses `sklearn.linear_model.LogisticRegression` with `fit_intercept=True`, `solver='liblinear'`, and `multi_class='auto'`. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Never optimize hyperparameters on the test set.** - When conducting an evaluation (like Bake-off 1), run the evaluation script (`full_word_similarity_evaluation`) on the test data exactly once. If you look at the test score, change your model's parameters, and run it again, you invalidate your results. * **Rule 2: Ditch standard tokenizers for sentiment tasks.** - Do not use the Treebank or standard whitespace tokenizers on social media or review data. Implement a sentiment-aware tokenizer (like `nltk.tokenize.casual.TweetTokenizer` or similar) that explicitly preserves emoticons, URLs, hashtags, and consecutive punctuation (`!!!`). * **Rule 3: Do not stem your text.** - Avoid using Porter or Lancaster stemmers when extracting features for sentiment analysis. They aggressively destroy morphological differences that carry opposite emotional polarities. Let your model handle the raw word forms or use context-aware embeddings instead. * **Rule 4: Implement simple negation scope marking.** - As a strong baseline feature, append a suffix like `_NEG` to all tokens occurring between a negation word (not, never, didn't) and the next clause-level punctuation mark. This immediately gives linear classifiers context about flipped sentiment. * **Rule 5: Use Macro F1 for imbalanced sentiment data.** - Always evaluate sentiment models using macro-averaged F1 scores rather than raw accuracy. Sentiment datasets are usually heavily skewed toward neutral or mildly positive classes; macro-averaging ensures the model is penalized if it fails on the rare, extreme classes. * **Rule 6: Package experiments in a unified wrapper.** - Build a function (similar to `sst.experiment`) that takes a feature extraction function (`phi`) and a model fitting function, runs the training pipeline, and returns a dictionary containing the trained model, the vectorizer, the predictions, and the raw text examples. This is essential for conducting deep error analysis later. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Delivering aggregate pie charts to business leaders. -> **Why it fails:** Aggregating sentiment into pure positive/negative percentages strips away the causal factors. It tells a business that sentiment dropped 5%, but offers no latent variables or text evidence explaining *why* it dropped. -> **Warning sign:** Dashboards that only show "Sentiment Score: 70%" without linked topics, feature extraction, or root cause text examples. * **Pitfall:** Treating sentiment as a binary classification problem. -> **Why it fails:** Human language utilizes sarcasm, mimicry, humor, and complex social bonding (e.g., affectionately calling someone a "bastard"). A simple positive/negative binary forces highly contextual affective states into rigid boxes. -> **Warning sign:** Models exhibiting high error rates on text containing jokes, slang, or deep domain-specific jargon. * **Pitfall:** Using the Treebank Tokenizer on Twitter data. -> **Why it fails:** The Treebank tokenizer is designed for formal text. It rips apart emoticons (turning `>:-D` into four meaningless punctuation tokens) and splits hashtags, destroying the primary signals of emotion in short-form text. -> **Warning sign:** A bag-of-words model that has high counts for isolated colons and dashes, but zero features representing actual smileys. * **Pitfall:** Relying solely on raw lexicons without context. -> **Why it fails:** Lexicons (like Bing Liu's) classify words in isolation. A model relying only on a lexicon will see the words "bewildering, boring, slow-moving" in a movie review and predict "Negative", missing the syntactical context that the reviewer is actually mocking the critics who hold that view. -> **Warning sign:** The model fails on sentences featuring complex syntax, reporting speech ("they said it would be bad"), or implicit negation. ## 6. Key Quote / Core Insight "Sentiment analysis is not just a superficial categorization task; it is a deep NLU challenge. Seemingly simple tasks—like figuring out if 'Oh, you're terrible!' is an insult or a term of endearment—require a model to uncover the latent, multidimensional nature of human emotion, making sentiment a perfect microcosm for everything we do in Natural Language Understanding." ## 7. Additional Resources & References * **Resource:** Stanford Sentiment Treebank (SST) - **Type:** Dataset - **Relevance:** The core dataset for the lecture, featuring 11,855 sentences from movie reviews with fully labeled parse trees (over 318,000 annotated nodes) using a 5-way sentiment scale. Available at `nlp.stanford.edu/sentiment/`. * **Resource:** `nltk.tokenize.casual.TweetTokenizer` - **Type:** Tool - **Relevance:** A built-in Python NLTK tokenizer recommended for preserving emoticons, hashtags, and sentiment-heavy elements in casual text. * **Resource:** Socher et al. (2013) - **Type:** Paper - **Relevance:** The foundational paper that introduces the Stanford Sentiment Treebank and recursive neural models for compositional sentiment. * **Resource:** Pang & Lee (2008) - **Type:** Paper - **Relevance:** A comprehensive, highly recommended compendium and literature review covering early ideas and methods in sentiment analysis. * **Resource:** Goldberg (2015) - **Type:** Paper - **Relevance:** Recommended as an excellent primer for applying deep learning techniques to NLP problems. * **Resource:** Harvard General Inquirer - **Type:** Lexicon - **Relevance:** A massive, hand-curated spreadsheet mapping words to various affective and social dimensions, useful for baseline feature extraction. * **Resource:** SentiWordNet - **Type:** Lexicon - **Relevance:** A project that assigns sentiment polarity scores to WordNet synsets, available via NLTK. * **Resource:** Bing Liu's Opinion Lexicon - **Type:** Lexicon - **Relevance:** A straightforward, word-level positive/negative dictionary built into NLTK.