AI FOR HIGH-PERFORMANCE COMPUTING

The Power of 3D Tensor Parallelism

Plexus addresses the challenges of distributed GNN training by adopting a three-dimensional (3D) tensor parallel algorithm, inspired by Agarwal et al.'s work in matrix multiplication. This approach efficiently distributes all matrices (adjacency, features, weights) and parallelizes computations across a virtual GPU grid.

In a 3D grid of Gx × Gy × Gz GPUs, matrices are sharded across different planes to ensure compatibility for local computations and minimize communication. For example, the adjacency matrix `A` is sharded across the ZX-plane and replicated across the Y-plane, while input features `F` are sharded across XY and Z planes. This strategic distribution ensures that each GPU only holds a portion of the data necessary for its local calculations, significantly reducing memory burden.

The forward pass involves critical communication steps:

• An all-gather of feature matrix shards across the Z-parallel group.
• An all-reduce of aggregation output across the X-parallel group.
• An all-gather of weight matrix shards across the Z-parallel group.
• An all-reduce of combination output across the Y-parallel group.

This systematic approach ensures local computation compatibility and optimizes communication, making it possible to scale GNN training to unprecedented graph sizes.

Strategic Optimizations for Peak Performance

Plexus incorporates several key optimizations to enhance performance and mitigate common issues in distributed GNN training:

• Double Permutation for Load Balancing: The sparse and uneven distribution of nonzeros in adjacency matrices can cause significant load imbalance across GPUs. Plexus employs a double permutation scheme (separate permutations for rows and columns) that ensures a near-perfect even distribution of nonzeros, effectively eliminating computational stragglers and improving overall training stability. This contrasts with graph partitioners that focus on edge cuts, which are less suitable for dense aggregation outputs.
• Blocked Aggregation: To reduce performance variability observed in SpMM, especially on larger datasets and modest GPU counts, Plexus blocks the sparse adjacency matrix into smaller row-blocks. After each block's SpMM, an all-reduce is performed, and blocks are concatenated. This mitigates performance fluctuations and reduces communication overheads.
• Dense Matrix Multiplication Tuning: Although dense matrix multiplications consume a small portion of the overall time, they can still impact scaling on high GPU counts. Plexus optimizes these kernels by reversing the multiplication order (e.g., for gradient computations), significantly reducing their execution time.
• Parallel Data Loading: Traditional methods often load entire datasets into CPU memory before transfer, which is unsustainable for massive graphs. Plexus implements a parallel data loader that shards processed data into 2D files offline. Each GPU then loads, merges, and extracts only the shards it needs, drastically cutting CPU memory usage and data loading time.

Unprecedented Scaling on Modern Supercomputers

Plexus demonstrates groundbreaking strong scaling capabilities across a variety of graph datasets on both Perlmutter (NERSC) and Frontier (OLCF). Our experiments show consistent performance improvements as GPU counts increase, distinguishing Plexus from state-of-the-art frameworks like SA and BNS-GCN, which often struggle with scalability beyond a limited number of GPUs.

• Perlmutter Performance: Plexus scales effectively up to 128 GPUs on Reddit, achieving 9x speedup over SA and 6x over BNS-GCN on 32 GPUs. On larger datasets like Isolate-3-8M and products-14M, Plexus maintains strong scaling up to 1024 GPUs, significantly outperforming BNS-GCN by 3.8x at 256 GPUs and SA by 2.3x at 128 GPUs, respectively.
• Frontier Performance: On Frontier, Plexus shows even better trends, with superior SpMM times on AMD GPUs contributing to improved scaling. It demonstrates impressive performance for ogbn-papers100M, the largest graph dataset tested, scaling up to 2048 GCDs (Graphics Compute Dies).
• Communication Efficiency: Unlike BNS-GCN, which relies on all-to-all collectives that become inefficient at scale, Plexus's ring-based collectives communicate only with neighbors, leading to lower latency and better scalability, especially at higher GPU counts.