Bias and Representation in Sociotechnical Systems: Algorithm Auditing and Ambient Belonging

# Bias and Representation in Sociotechnical Systems: Algorithm Auditing and Ambient Belonging **Video Category:** Technology / Human-Computer Interaction ## 📋 0. Video Metadata **Video Title:** Bias and Representation in Sociotechnical Systems (Danaë Metaxa) **YouTube Channel:** Stanford Online **Publication Date:** May 14, 2021 **Video Duration:** ~1 hour 24 minutes ## 📝 1. Core Summary (TL;DR) This presentation explores how algorithmic content in sociotechnical systems—such as search engines and web interfaces—conveys biases that significantly impact users' sense of belonging and real-world behavior. By combining computational algorithm audits with behavioral science experiments, the research demonstrates that platforms like Google Search systematically misrepresent demographic realities and function as "Search Media" that users consume passively. Ultimately, the work proves that while visual representation can alter users' expectations and inclusivity, purely technical fixes are insufficient to solve deeply ingrained social inequities, requiring robust, user-consented auditing infrastructure to measure and intervene effectively. ## 2. Core Concepts & Frameworks * **Ambient Belonging:** -> **Meaning:** A psychological concept (drawn from Abraham Maslow's hierarchy of needs and Cheryan et al., 2009) describing the feeling of fitting in with a culture or community, passively elicited by the surrounding environment. -> **Application:** Used to evaluate how the visual design and thematic choices of a website (e.g., using *Star Trek* or *The Matrix* imagery) subtly signal to marginalized groups whether they are welcome in a particular academic or professional field. * **Sociotechnical Systems:** -> **Meaning:** Complex platforms that shape and are continuously shaped by user behavior in a closed loop (e.g., search engines, social media). -> **Application:** Evaluating these systems requires moving beyond just verifying that the software code "functions" correctly; it necessitates measuring the content these systems produce at scale and analyzing the psychological impact that content has on the human users integrated into the loop. * **Algorithm Auditing:** -> **Meaning:** A methodology adapted from the social sciences (such as testing housing or lending discrimination) that involves repeatedly querying a "black box" algorithmic system with controlled inputs and analyzing the outputs to draw inferences about its inner workings and potential biases. -> **Application:** Deploying web scrapers across Amazon EC2 instances to programmatically search Google for thousands of political candidates to measure systemic partisan bias. * **Search Media:** -> **Meaning:** The concept that users do not scrutinize individual search results link-by-link to trace their origins. Instead, they consume the combination of headlines, text snippets, and image orderings as a single, unified media object. -> **Application:** Treating Google not just as a neutral index, but as an active publisher whose curation choices have psychological impacts equivalent to a newspaper's front page. * **Occupational Feminization:** -> **Meaning:** A sociological theory (Levanon, England, and Allison, 2009) noting that as the proportion of women increases in a specific professional field, the pay, prestige, and other status markers associated with that field tend to decrease. -> **Application:** Explains the counter-intuitive experimental finding where artificially increasing the visual representation of women in certain jobs actually led to a slight *decrease* in overall user interest in those careers. ## 3. Evidence & Examples (Hyper-Specific Details) * **Web Interface Ambient Belonging Experiment (Stanford CS106A):** * Prompted by intense pop-culture course themes at Brown University (e.g., *CS053 Reloaded: The Matrix in Computer Science*, *CS166 Computer Systems Security* using *James Bond* and *Star Trek*). * Designed two identical syllabi for Stanford's CS106A intro course: one using "Neutral" nature imagery and one using "Stereotypical" hacker/sci-fi imagery. * Tested on crowd-workers matching US undergraduate age using 7-point Likert scales. * *Result:* Men's responses were identical across both sites. Women exposed to the stereotypical site reported statistically significantly lower intention to enroll, lower anticipated success, lower future CS intentions, and higher stereotype anxiety (p < 0.01 and p < 0.001). * **Google Image Search Diversity Audit (vs. BLS Data):** * Scraped Google Image results for 100 common occupations and compared them to ground-truth US Bureau of Labor Statistics (BLS) workforce demographics. * Used Amazon Mechanical Turk (3 workers per image, 2/3 consensus required) to label gender (woman/not) and race (person of color/not). * *Findings:* Images systematically exaggerated gender and racial disparities. Receptionists are 88% women in BLS, but nearly 100% in Google Images. CEOs are 30% women in BLS, but only 11% in Google. Bartenders are >50% women in BLS, but 25% in Google. * *Generalized Linear Model (GLM):* If an occupation is 50% women, Google Images predicted to show only 42% women. If an occupation is 22% POC, Google predicted to show only 16% POC (p < 0.001). * **Synthetic Search Results Intervention Study:** * Selected 10 occupations with severe underrepresentation (e.g., Electrician [2% women], Construction worker [4% women], Pilot [8% women, 6% POC], CEO [11% POC]). * Created synthetic search pages at three representation levels: Low (~10%), Medium (50%), High (90%). * *Inclusivity:* Higher marginalized representation made occupations seem more inclusive across the board (p < 0.001). * *Belonging:* For gender, women's sense of belonging improved with representation, while men's declined. For race, white participants consistently reported higher belonging than POC, and altering visual representation did *not* significantly change racial belonging gaps. * **2018 Midterm Elections Political Search Audit:** * Prompted by President Trump's 2018 tweets accusing Google of burying conservative news. * Scraped Google daily for 6 months for >3,000 House/Senate candidates, generating ~4 million URLs. * Used 5 Amazon EC2 scrapers rotating IP addresses, User-Agent strings, implementing a 1-minute random wait between queries, and using the `pws=0` parameter to depersonalize results. * *Finding:* No systemic bias against conservative sources. Search media drew extensively from conservative sources regardless of the query. * *Incumbency Effect:* Election winners (largely incumbents) had more moderate/centrist search media than losers (challengers), perfectly aligning with Groseclose et al. (2001) political science theory regarding primary vs. general election positioning. * **The Intervenr System Architecture:** * Built to solve the limitations of one-off scraping scripts, expensive proprietary panels (Comscore, YouGov Pulse), and light volunteer interventions (Mozilla Rally). * Consists of a Django web app (Front end/admin), a Chrome browser extension (Data collection/intervention), and an AWS RDS database (Secure storage). * Allows researchers to conduct longitudinal, cross-site tracking and inject controlled visual interventions directly into a consenting user's organic browsing experience. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Eliminate Stereotypical "Geek" Branding in Onboarding** - When designing introductory material, course syllabi, or recruitment pages, strictly utilize neutral imagery. Do not use hyper-specific pop-culture references (e.g., *The Matrix*, hacking green-text) as they reliably trigger ambient non-belonging and stereotype anxiety in women without providing any measurable benefit to men. * **Rule 2: Implement Depersonalization for Valid Algorithm Audits** - When scraping search engines to test for systemic bias, append depersonalization parameters (e.g., `pws=0` on Google) to strip out user-history bias. Rotate IP addresses and User-Agent strings, and implement randomized wait times (e.g., 60 seconds) to prevent the platform from blocking the scraper or serving anomaly-detection results. * **Rule 3: Optimize for "Search Media," Not Just Search Ranking** - When managing brand or political reputation, operate under the assumption that users will not click the links. Curate the holistic "Search Media" experience—the visual combination of headlines, snippets, and image carousels on the first page—as a unified, single-consumption object. * **Rule 4: Do Not Assume "More Representation" Yields Universal Benefits** - Be cautious when artificially inflating visual representation in marketing or search. While increasing representation generally improves perceived inclusivity, it can trigger negative reactions (e.g., decreased overall interest) due to systemic biases like occupational feminization, and it may lower the sense of belonging for the dominant demographic group. * **Rule 5: Use Consenting-User Extensions for Ecological Validity** - To measure the true impact of algorithms (like selective avoidance of misinformation or highly personalized ad targeting), deploy browser extensions on compensated, consenting panels rather than relying on synthetic lab environments or passive, one-off web scraping. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Relying exclusively on technical fixes for social problems. -> **Why it fails:** Algorithmic systems absorb and reflect deeper societal inequities. For example, simply increasing the number of images of people of color for a specific job did not close the "belonging gap" because systemic racial inequities override surface-level visual tweaks. -> **Warning sign:** You change the UI/algorithm, but downstream metrics (hiring, enrollment, retention of marginalized groups) remain stagnant. * **Pitfall:** Using binary categorization for complex identities. -> **Why it fails:** Simplifying gender (Woman/Not Woman) and race (POC/White) into binary metrics for data annotation forces complex, intersectional, and non-normative identities into rigid boxes. -> **Warning sign:** Annotators struggle to reach consensus on images, or the data fails to capture the distinct experiences of non-binary or multi-racial individuals. * **Pitfall:** Over-correcting visual representation without context. -> **Why it fails:** Aggressively pushing imagery of marginalized groups into fields where they are severely underrepresented can cause the dominant group to feel alienated, or trigger sociological penalties (occupational feminization) where the perceived prestige of the role drops. -> **Warning sign:** Inclusivity scores rise, but actual interest or application rates for the role decrease. * **Pitfall:** Believing search algorithms are purely neutral indexes. -> **Why it fails:** Algorithms actively curate and arrange data, functioning as publishers. Failing to recognize this means you miss the psychological impact the arrangement has on users who consume the results page as a single news source. -> **Warning sign:** You only track click-through rates (CTR) and ignore the impression-level impact of headlines and image carousels. ## 6. Key Quote / Core Insight "By closing the loop between algorithmic content and its effects on people, we can transition from merely identifying flaws in sociotechnical systems to actively building platforms that generate positive, measurable impacts on ourselves and our communities." ## 7. Additional Resources & References * **Resource:** "Ambient belonging: How stereotypical cues impact gender participation in computer science" (Cheryan et al., 2009) - **Type:** Research Paper - **Relevance:** Foundational study on how physical and digital environments passively signal inclusion or exclusion. * **Resource:** "Psychologically Inclusive Design" (Kizilcec and Saltarelli, 2019) - **Type:** Research Paper - **Relevance:** Referenced regarding the impact of visual cues on STEM education participation. * **Resource:** "Unequal representation and gender stereotypes in image search results for occupations" (Kay et al., 2015) - **Type:** Research Paper - **Relevance:** The prior work demonstrating that Google Images exaggerates gender stereotypes, prompting the updated race/gender audit. * **Resource:** "Occupational Feminization and Pay" (Levanon, England, and Allison, 2009) - **Type:** Research Paper - **Relevance:** Explains the sociological phenomenon where increasing female representation in a field lowers its perceived status and pay. * **Resource:** "Electoral Studies" (Groseclose et al., 2001) - **Type:** Research Paper - **Relevance:** Explains the political theory regarding why incumbents adopt more centrist/moderate positioning compared to challengers. * **Resource:** "Auditing Algorithms: Understanding Algorithmic Systems from the Outside In" (Metaxa et al., 2021) - **Type:** Review Paper (Foundations and Trends in HCI) - **Relevance:** An authoritative guide on the history, best practices, ethics, and norms of algorithm auditing. * **Resource:** The Intervenr System - **Type:** Software Infrastructure (Django/Chrome/AWS) - **Relevance:** The custom toolset developed to allow researchers to conduct active, longitudinal audits and interventions on consenting users' browsers.