Interactivity as a Medium: Designing Alternate Perceptual Realities

# Interactivity as a Medium: Designing Alternate Perceptual Realities **Video Category:** Human-Computer Interaction / Interaction Design ## 📋 0. Video Metadata **Video Title:** Interactivity as a Medium **YouTube Channel:** StanfordCenter for Professional Development **Publication Date:** September 28, 2012 **Video Duration:** ~1 hour 10 minutes ## 📝 1. Core Summary (TL;DR) This presentation explores how interactivity functions as a distinct creative medium, evolving from abstract cinema and mechanical art into modern digital applications. It argues that computers should not be constrained by physical metaphors like the "desktop," but rather embraced as infinite, time-varying planes of light and color capable of generating alternate perceptual realities. By designing "meta-programs" and augmenting human intelligence, creators can build deeply engaging, social, and emotional experiences that transcend traditional utility to become forms of interactive art. ## 2. Core Concepts & Frameworks * **Concept:** Perceptual Reality as a Construct -> **Meaning:** Human beings do not perceive reality directly; instead, our brains reconstruct a model of reality from five time-varying sensory streams. We possess a "sixth sense"—the brain's ability to generate its own sensory stimuli through dreams, hallucinations, and imagination. -> **Application:** Interaction designers can use computers to artificially generate these "super stimuli," creating alternate perceptual realities that users can explore and manipulate. * **Concept:** The Computer as an Infinite Canvas -> **Meaning:** A computer screen is fundamentally different from a typewriter or a physical desk; it is an infinitely varying plane of time-varying light and color where any visual or interactive concept is possible. -> **Application:** Instead of relying on mundane physical metaphors (like folders and trash cans), designers should create entirely new visual and interactive paradigms that leverage the screen's boundless potential. * **Concept:** Intelligence Augmentation (IA) vs. Artificial Intelligence (AI) -> **Meaning:** A philosophical divide in computing identified by comparing John McCarthy (AI) and Doug Engelbart (IA). AI seeks to replace human intelligence with autonomous machines (e.g., IBM's Watson), while IA seeks to use computers to augment and expand human creativity and communication. -> **Application:** Building tools, instruments, and interactive experiences that empower users to create and connect (like Engelbart's "Mother of All Demos"), rather than building isolated, "smart" black boxes. * **Concept:** Meta-Programs -> **Meaning:** Software designed not to deliver a fixed experience, but to provide a toolset that allows the user to construct their own experiences, games, or art within a creative system. -> **Application:** Developing applications where the primary output is user-generated structures, effectively making the user a co-creator of the software's value. ## 3. Evidence & Examples (Hyper-Specific Details) * **Joseph Cornell Box (1940):** An early, physical example of interactive art roughly the size of an iPad. It contains loose elements like sand, a pearl, and musical paper that the user must physically manipulate to experience an alternate reality. * **"Orpheus" Film by Jean Cocteau (1950):** Demonstrated alternate realities through practical effects rather than digital ones. A scene where a character pushes his hands through a "mirror" was achieved practically by filming a pool of mercury sideways, heavily influencing modern sci-fi like *The Matrix*. * **Pinball Construction Set (1983) by Bill Budge:** A software "meta-program" that provided an empty pinball cabinet and a palette of parts (bumpers, flippers). Snibbe used it to place 100 flippers on a board, learning that software could serve as an open-ended creative system rather than a fixed game. * **Music Construction Set (1984):** An early music notation software. Snibbe used it to transcribe sheet music from New Order, The Beatles, and The Clash, proving that open-ended digital tools could inadvertently serve as powerful, self-driven educational platforms. * **Oskar Fischinger’s "Study No. 8" (1931):** An abstract film that visualized music through motion. It demonstrated the concept of "visual music," where visuals do not mechanically mimic the audio, but interpret it thoughtfully—akin to jazz improvisation. * **Thomas Wilfred's Clavilux (1920s):** An early, pre-television mechanical instrument designed to perform abstract light and color compositions in real-time, functioning as an early precursor to interactive screens. * **Motion Phone (1995):** An abstract animation communication system built on SGI computers. It allowed multiple users across a network to collaboratively draw and animate abstract shapes in real-time, demonstrating an early form of non-verbal, visual social interaction. * **Boundary Functions (1998):** A physical installation where a projected floor draws Voronoi diagrams (computational geometry) between people standing on it. It dynamically maps personal space. When one person is present, nothing happens; when multiple people enter, it forces them to navigate shared boundaries, revealing cultural differences in personal space (e.g., Austrian vs. Japanese interactions). * **Deep Walls (2002):** A shadow-recording installation that captures participants' silhouettes and loops them across 16 projected rectangles. Initially designed with a clunky "photo booth" button interface, it failed until redesigned to automatically record upon entry and exit, making the system subservient to natural human behavior. * **Blow Up (2005):** An installation featuring a wall of fans controlled by a small impeller (breath amplifier). Participants blow into the device, and their breath is translated into massive wind patterns on the fans, turning an intimate biological function into a public spectacle. * **Avatar / SocialScreen / SocialTable (2009):** Work done for James Cameron to translate *Avatar* film assets into interactive museum exhibits. Included multi-touch tables for exploring Pandora's creatures and "SocialScreen" shadow walls where digital Woodsprites interacted with visitors' physical silhouettes. * **Björk’s "Biophilia" - Crystalline App (2011):** An interactive album app where the song's structure is represented as a spatial tunnel. Users catch crystals to physically restructure the song's arrangement, turning music listening into an active, spatial game. * **Björk’s "Biophilia" - Virus App (2011):** A "virus fatale" interactive song. Viruses visually attack a cell on screen. To hear the entire song, the user must stop interacting and allow the cell to be destroyed; if the user fights off the viruses to save the cell, the song stops, subverting traditional game win-states. * **Björk’s "Biophilia" - Thunderbolt App (2011):** An instrument app where users draw lightning between two fingers to create musical arpeggios. The distance and angle of the fingers dictate the tempo and sequence of the notes. * **Björk’s "Biophilia" - Moon App (2011):** A circular harp sequencer based on pearls and lunar phases. Rotating the moon changes the "spinal fluid" (the musical mode/scale), allowing users to intuitively experiment with complex musicology (e.g., switching from Pentatonic to Hungarian Minor scales) without formal training. * **Theodore Gray's "The Elements" & "The Waste Land" Apps:** Highlighted as prime examples of interactive publishing where non-fiction data (the periodic table) and classic poetry (T.S. Eliot) are transformed into highly successful, immersive digital experiences, proving that interactive media can elevate "boring" subject matter. ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Build Meta-Programs, Not Just Content** - Design interactive systems that give users a palette of tools and an empty canvas to create their own experiences, rather than forcing them down a rigid, pre-authored path. * **Rule 2: Make Technology Subservient to Natural Behavior** - When designing physical interactive installations, remove explicit controls (like buttons or countdowns) that make the human a slave to the computer. Design systems that automatically react to natural human presence and movement. * **Rule 3: Design for Social Dependency** - Create interactive experiences that remain dormant or incomplete when used by a single person. Force interaction by requiring two or more people to activate the system (e.g., Voronoi diagrams requiring multiple bodies). * **Rule 4: Avoid Mechanical Audio-Visual Mapping** - When creating visual representations of audio, do not map graphics directly to the beat or amplitude (referred to as "Mickey Mousing"). Give the visual system its own internal logic and "soul" so it interprets the music rather than just measuring it. * **Rule 5: Subvert the "Win State"** - Challenge traditional game mechanics by making the "failure" state the only way to experience the full narrative or content, forcing the user to reflect on their actions (as seen in the *Virus* app). * **Rule 6: Use Abstraction to Teach Complexity** - Use interactive, visual models to teach complex concepts (like musical scales or song structures) intuitively, removing the barrier of formal notation and terminology. ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Over-reliance on the "Desktop Metaphor." -> **Why it fails:** It collapses the infinite, creative potential of a computer screen into the most mundane and boring elements of a physical office (trash cans, folders). -> **Warning sign:** Interfaces that strictly mimic physical office supplies and fail to utilize time-varying light, color, and abstraction. * **Pitfall:** "Mickey Mousing" in visual music. -> **Why it fails:** It treats music as pure data to be mechanically translated into bouncing visuals, resulting in an experience devoid of human interpretation, emotion, or aesthetic logic. -> **Warning sign:** Graphics that simply pulse larger or brighter exactly when a bass drum hits. * **Pitfall:** Developing "AI" that ignores human context. -> **Why it fails:** Creating highly intelligent machines (like IBM's Watson) that can win trivia games but lack the situational awareness to know they are in a room with humans represents a weak, isolated view of intelligence. -> **Warning sign:** Technology that performs complex tasks but fails to augment human creativity or facilitate human-to-human connection. * **Pitfall:** Designing interactive art with rigid "Photo Booth" interfaces. -> **Why it fails:** Forcing users to press a button and wait for a countdown shifts control to the machine, making users feel awkward and resulting in unnatural interactions. -> **Warning sign:** The system requires explicit, unnatural user commands to initiate an experience rather than detecting natural presence. ## 6. Key Quote / Core Insight "The computer screen is not a typewriter; it is a movie screen. It is an infinitely varying plane of time-varying light and color where absolutely anything is possible, and we limit ourselves when we only build digital versions of physical desks." ## 7. Additional Resources & References * **Resource:** *Count Zero* by William Gibson - **Type:** Book - **Relevance:** Sci-fi novel featuring AI creating counterfeit Joseph Cornell boxes, used to illustrate the intersection of AI, art, and meta-creation. * **Resource:** *Orpheus* (directed by Jean Cocteau) - **Type:** Film - **Relevance:** Demonstrates creating alternate realities through practical visual effects (the mercury mirror scene), serving as inspiration for immersive visual design. * **Resource:** *Pinball Construction Set* (by Bill Budge) - **Type:** Software - **Relevance:** Cited as a foundational example of a "meta-program" that empowers user creativity. * **Resource:** *Music Construction Set* - **Type:** Software - **Relevance:** Example of how open-ended creation tools act as accidental educational platforms. * **Resource:** *Study No. 8* (by Oskar Fischinger) - **Type:** Film - **Relevance:** A primary reference for "visual music" and abstract animation that interprets, rather than mechanically mimics, sound. * **Resource:** *Biophilia* (by Björk) - **Type:** App / Album - **Relevance:** The definitive case study for turning passive media (an album) into an interactive, educational, and spatial app suite. * **Resource:** *The Elements* (by Theodore Gray) - **Type:** App - **Relevance:** Proof that interactive publishing can make scientific data highly engaging and commercially successful. * **Resource:** *The Waste Land* (T.S. Eliot iPad edition) - **Type:** App - **Relevance:** Proof that interactive formatting can elevate traditional literature into a modern interactive experience.