Large-Scale & Scalable Training
Sampling, partitioning, and mini-batching techniques for training GNNs on graphs too big for one step.
-
A simple GCN baseline for node property prediction
PyG's example trains a one-layer GCN with a small MLP head on the GraphLand benchmark, a collection of industrial graphs with tabular node features.
Photo (Amazon)·GCNConv -
Classifying nodes from their own subgraphs (ShaDow-GNN)
ShaDow-GNN (Zeng et al., 2021) decouples the depth of a GNN from the size of its receptive field: for every target node it extracts a small subgraph (here the 2-hop neighborhood, with up to 5 neighbors per node) and runs a GNN on that subgraph only.
PubMed (Planetoid)·SAGEConv -
Faster neighbor sampling with hierarchical trimming
With neighbor sampling, the nodes sampled in the last hop only matter for the first GNN layer, the nodes of the second-to-last hop only for the first two layers, and so on.
PubMed (Planetoid)·GraphSAGE -
Memory-efficient deep GNNs with reversible layers (RevGNN)
RevGNN (Li et al., 2021) builds deep GNNs from grouped reversible blocks: the features are split into groups, and each group is updated from the others, so every block's input can be recomputed from its output.
PubMed (Planetoid)·SAGEConv · GroupAddRev -
Mini-batch training of GNNs and graph transformers
Train a node classifier on mini-batches sampled with
NeighborLoader.PubMed (Planetoid)·GAT · GraphSAGE -
Mini-batch training on graph clusters (Cluster-GCN)
Cluster-GCN (Chiang et al., 2019) partitions the graph into many small clusters of densely connected nodes.
PubMed (Planetoid)·SAGEConv -
Mini-batch training on sampled subgraphs (GraphSAINT)
GraphSAINT (Zeng et al., 2020) trains on small subgraphs sampled from the full graph: here, the nodes visited by short random walks.
PubMed (Planetoid)·GraphConv -
Mini-batch training with neighbor sampling (GraphSAGE)
Train a GraphSAGE model (Hamilton et al., 2017) on mini-batches: for every batch of training nodes,
NeighborLoadersamples up to 25 of their neighbors and 10 of each neighbor's neighbors.PubMed (Planetoid)·SAGEConv -
Using known labels as features (UniMP)
UniMP (Shi et al., 2020) feeds part of the known training labels into the model, added to the node features, and learns to predict the labels of the other nodes with graph transformer layers.
PubMed (Planetoid)·TransformerConv · MaskLabel -
Very deep GNNs for protein function prediction (DeeperGCN)
DeeperGCN (Li et al., 2020) trains GNNs with 28 layers using pre-activation residual blocks (
DeepGCNLayer, "res+") andGENConvlayers with a learnable softmax aggregation.PPI·GENConv · DeepGCNLayer