Stanford CS224W: Setting-up GNN Prediction Tasks

# Stanford CS224W: Setting-up GNN Prediction Tasks **Video Category:** Machine Learning with Graphs / Programming Tutorial ## 📋 0. Video Metadata **Video Title:** Stanford CS224W: Setting-up GNN Prediction Tasks **YouTube Channel:** Stanford ENGINEERING **Publication Date:** 2/5/21 (visible on slide footers) **Video Duration:** ~17.5 minutes ## 📝 1. Core Summary (TL;DR) This lecture explains how to properly split graph datasets into training, validation, and test sets to evaluate Graph Neural Network (GNN) performance accurately. Unlike standard image or text data where data points are independent, nodes in a graph are interconnected, meaning a naive data split will cause test nodes to "leak" information from training nodes via message passing. To solve this, the video introduces "transductive" and "inductive" splitting methodologies across three primary tasks: node classification, graph classification, and the notably complex task of link prediction. Mastering these specific dataset splits prevents model cheating and ensures GNNs can actually generalize to unseen graph structures. ## 2. Core Concepts & Frameworks * **Concept:** Graph Information Leakage -> **Meaning:** A phenomenon occurring when training and testing nodes share edges in a graph. Because GNNs use message passing, a node in the test set will aggregate features from its neighboring nodes, which might be in the training set, breaking the independence of the evaluation data. -> **Application:** Necessitates the use of specialized transductive or inductive splitting strategies rather than standard random row-splitting used in traditional machine learning. * **Concept:** Transductive Setting -> **Meaning:** A dataset split where the entire graph structure (all nodes and all edges) is observable across all phases (training, validation, and testing), but only the *labels* of the nodes belonging to the specific split are revealed. -> **Application:** Ideal for tasks within a single, static network, such as predicting the roles of unlabelled users within a known, massive social media graph. * **Concept:** Inductive Setting -> **Meaning:** A dataset split where the graph is physically broken down into multiple, completely independent and disconnected sub-graphs. The model is trained on one graph and evaluated on entirely different, unseen graphs. -> **Application:** Crucial for testing a model's ability to generalize to novel structures, such as training on a set of known protein molecules and predicting the properties of newly discovered, separate protein molecules. * **Concept:** Message Passing Edges vs. Supervision Edges -> **Meaning:** In link prediction, edges are categorized into two roles. "Message passing edges" are fed into the GNN to compute node embeddings. "Supervision edges" are hidden from the GNN's input and are used solely as the ground-truth targets to calculate the loss function. -> **Application:** Ensures the GNN actually learns to infer missing connections based on graph structure rather than trivially "predicting" an edge just because it was given that edge as an input feature. ## 3. Evidence & Examples (Hyper-Specific Details) * **Image vs. Graph Independence Comparison:** The speaker contrasts an image dataset (where Image 5 is entirely independent of Image 1) with a graph dataset. A diagram shows Node 5 connected via edges to Nodes 1 and 2. If Nodes 1 and 2 are placed in the training set and Node 5 is in the test set, the prediction for Node 5 will inherently incorporate the features of Nodes 1 and 2 due to the GNN's neighborhood aggregation mechanism, illustrating how standard splitting fails. * **Inductive Node Classification Demonstration:** To create an inductive split from a single graph, a visual example shows a graph chopped into three pieces. The dotted edges connecting the three sub-graphs are literally deleted. This yields a Training Graph (nodes 1, 2), Validation Graph (nodes 3, 4), and Test Graph (nodes 5, 6), ensuring predictions on test node 5 are mathematically isolated from training nodes 1 and 2. * **Graph Classification Constraint:** The lecture explicitly notes that for Graph Classification (e.g., predicting if a molecule is toxic), only the Inductive setting is well-defined. The dataset inherently must consist of multiple independent graphs, as you cannot evaluate generalization to a "new graph" if you only have one large connected network. * **Transductive Link Prediction Evolving Graph Example:** A visual step-by-step breakdown illustrates the complex transductive link prediction split. * *Step 1 (Training):* The GNN receives only "training message edges" to predict "training supervision edges" (e.g., a hidden red edge between nodes 1 and 3). * *Step 2 (Validation):* The graph context grows. The GNN now receives "training message edges" AND the revealed "training supervision edges" to predict new, hidden "validation edges" (e.g., a red edge between 5 and 4). * *Step 3 (Testing):* The context grows again. The GNN receives "training message edges" + "training supervision edges" + "validation edges" to predict the final missing "test edges". ## 4. Actionable Takeaways (Implementation Rules) * **Rule 1: Use Fixed Splits for Benchmarking, Random Splits for Robustness** -> When evaluating GNNs, use a "Fixed Split" (split the dataset once) to establish a standard baseline. However, to guarantee the model's reliability, use a "Random Split" by re-splitting the dataset multiple times using different random seeds and reporting the average performance. * **Rule 2: Apply Transductive Splits for Single-Network Problems** -> If your objective is to infer missing attributes within one cohesive system (like a citation network or recommendation engine), use the Transductive setting. Keep the graph structure intact but mask the labels of the validation and test nodes during training. * **Rule 3: Cut Edges to Force Inductive Generalization** -> If you have a single graph but need the model to generalize to unseen networks, you must implement an Inductive split by physically deleting the edges that connect your designated train, validation, and test sub-graphs, accepting the loss of some structural data to guarantee strict independence. * **Rule 4: Strictly Segregate Edges for Link Prediction** -> When building a link prediction task, you must assign every edge as either a "message passing edge" (the input) or a "supervision edge" (the label). Never pass a supervision edge into the GNN's forward pass, or the model will simply learn an identity function. * **Rule 5: Expand Graph Context Sequentially in Transductive Link Prediction** -> To give the model maximum structural context during evaluation, feed previously predicted supervision edges back into the model as message-passing inputs for subsequent phases (i.e., use training targets as validation inputs, and validation targets as testing inputs). ## 5. Pitfalls & Limitations (Anti-Patterns) * **Pitfall:** Naively assigning connected nodes into separate train/test splits. -> **Why it fails:** GNNs compute node embeddings by aggregating information from neighboring nodes. If a test node is connected to a training node, it will pull in the training node's features during the forward pass, violating the independence of the test set (information leakage). -> **Warning sign:** The model achieves unusually high accuracy on the test set but fails catastrophically when deployed on a genuinely disconnected, new graph. * **Pitfall:** Using an Inductive split on a very small, dense graph. -> **Why it fails:** To create independent sub-graphs from a single network, you must permanently delete the edges that cross between your train/val/test splits. In a small graph, this "chops up" too much of the fundamental structure, throwing away critical data and severely handicapping the model's ability to learn. -> **Warning sign:** The resulting sub-graphs are highly fragmented or disconnected, and model performance is poorer than a transductive baseline. * **Pitfall:** Feeding target edges into the GNN input during link prediction. -> **Why it fails:** If the edge the model is supposed to predict is included in the graph structure passed to the GNN layers, the model does not learn graph topology. It simply learns to output "True" when it sees the edge in its own input data. -> **Warning sign:** Training loss drops to zero almost instantly, but validation/test metrics on hidden edges are equivalent to random guessing. ## 6. Key Quote / Core Insight "The fundamental challenge of evaluating graphs is that nodes are not isolated entities. If you put Node 1 in training and Node 5 in testing, but they share an edge, Node 5 will mathematically absorb Node 1's data during message passing. To prevent this leakage, we must fundamentally alter how we split our datasets compared to traditional machine learning, utilizing strict transductive or inductive boundaries." ## 7. Additional Resources & References * **Resource:** DeepSNAP - **Type:** Software Library - **Relevance:** Identified as a core Python module that provides the necessary infrastructure to handle complex graph dataset splitting, preventing manual implementation errors. * **Resource:** GraphGym - **Type:** Software Framework - **Relevance:** Mentioned as a higher-level tool that implements the full GNN pipeline discussed in the lecture, specifically facilitating the complicated edge-splitting logic required for link prediction tasks.