Toward an extreme-scale electronic structure system
Jorge Luis Galvez Vallejo, Calum Snowdon, Ryan Stocks, Fazeleh Kazemian, Fiona Yu, Christopher Seidl, Zoe L. Seeger, Melisa Alkan, David Poole, Bryce M. Westheimer, Mehaboob Basha, Marco De La Pierre, Alistair P. L. Rendell, Ekaterina I. Izgorodina, Mark S. Gordon, Giuseppe M. J. Barca
Electronic structure calculations have the potential to predict key matter transformations for applications of strategic technological importance, from drug discovery to material science and catalysis. However, a predictive physicochemical characterization of these processes often requires accurate quantum chemical modeling of complex molecular systems with hundreds to thousands of atoms. Due to the computationally demanding nature of electronic structure calculations and the complexity of modern high-performance computing hardware, quantum chemistry software has historically failed to operate at such large molecular scales with accuracy and speed that are useful in practice. In this paper, novel algorithms and software are presented that enable extreme-scale quantum chemistry capabilities with particular emphasis on exascale calculations. This includes the development and application of the multi-Graphics Processing Unit (GPU) library LibCChem 2.0 as part of the General Atomic and Molecular Electronic Structure System package and of the standalone Extreme-scale Electronic Structure System (EXESS), designed from the ground up for scaling on thousands of GPUs to perform high-performance accurate quantum chemistry calculations at unprecedented speed and molecular scales. Among various results, we report that the EXESS implementation enables Hartree-Fock/cc-pVDZ plus RI-MP2/cc-pVDZ/cc-pVDZ-RIFIT calculations on an ionic liquid system with 623 016 electrons and 146 592 atoms in less than 45 min using 27 600 GPUs on the Summit supercomputer with a 94.6% parallel efficiency.
Electronic structure calculations have the potential to predict key matter transformations for applications of strategic technological importance, from drug discovery to material science and catalysis. However, a predictive physicochemical characterization of these processes often requires accurate quantum chemical modeling of complex molecular systems with hundreds to thousands of atoms. Due to the computationally demanding nature of electronic structure calculations and the complexity of modern high-performance computing hardware, quantum chemistry software has historically failed to operate at such large molecular scales with accuracy and speed that are useful in practice. In this paper, novel algorithms and software are presented that enable extreme-scale quantum chemistry capabilities with particular emphasis on exascale calculations. This includes the development and application of the multi-Graphics Processing Unit (GPU) library LibCChem 2.0 as part of the General Atomic and Molecular Electronic Structure System package and of the standalone Extreme-scale Electronic Structure System (EXESS), designed from the ground up for scaling on thousands of GPUs to perform high-performance accurate quantum chemistry calculations at unprecedented speed and molecular scales. Among various results, we report that the EXESS implementation enables Hartree-Fock/cc-pVDZ plus RI-MP2/cc-pVDZ/cc-pVDZ-RIFIT calculations on an ionic liquid system with 623 016 electrons and 146 592 atoms in less than 45 min using 27 600 GPUs on the Summit supercomputer with a 94.6% parallel efficiency.
Second-order Møller-Plesset perturbation theory using the Resolution-of-the-Identity approximation (RI-MP2) is a state-of-the-art approach to accurately estimate many-body electronic correlation effects. This is critical for predicting the physicochemical properties of complex molecular systems; however, the scale of these calculations is limited by their extremely high computational cost. In this paper, a novel many-GPU algorithm and implementation of a molecular-fragmentation-based RI-MP2 method are presented that enable correlated calculations on over 180,000 electrons and 45,000 atoms using up to the entire Summit supercomputer in 12 minutes. The implementation demonstrates remarkable speedups with respect to other current GPU and CPU codes, excellent strong scalability on Summit achieving 89.1% parallel efficiency on 4600 nodes, and shows nearly-ideal weak scaling up to 612 nodes. This work makes feasible ab initio correlated quantum chemistry calculations on significantly larger molecular scales than before on both large supercomputing systems and on commodity clusters, with a potential for major impact on progress in chemical, physical, biological and engineering sciences.
Predicting the chemico-physical properties of large molecular systems is a formidable challenge in chemistry and materials science. Traditional quantum mechanical methods, while accurate, have impractical scaling for large molecules with thousands of atoms, which are crucial in the development of novel therapeutics, catalysts, and nanomaterials. To address this, molecular fragmentation algorithms have been proposed to improve scalability and enable extensive parallelism. In this article, we introduce a significant enhancement to the Fragment Molecular Orbital (FMO) method, termed the Coulomb-Perturbed Fragmentation (CPF) method. CPF incorporates algorithmic improvements and implementation enhancements to optimize performance on heterogeneous computing systems equipped with a large number of GPUs. Key developments include a significant simplification of iteratitve self-consistent field (SCF) algorithm, advanced data management through a one-sided communication model, topology-aware optimizations, and a hybrid communication strategy for intra-group exchanges. Moreover, CPF integrates a distributed dynamic multi-layer load balancing scheme to optimise fragment distribution and workload management across nodes and GPUs. Performance evaluations on a 420-atom benzene molecule system comprising 35 fragments reveal that CPF outperforms existing GPU/CPU-based FMO algorithms in both efficiency and accuracy. When deployed on the Gadi supercomputer, CPF achieves over 97% parallel efficiency on 20 nodes, with scalability maintaining above 98% and 90% efficiency in weak-scaling tests for smaller and larger systems, respectively. Notably, CPF matches or exceeds the computational accuracy of conventional FMO methods, marking a substantial progress in the field of computational chemistry for fragmentation-based large-scale molecular modelling.
Correlated electronic structure calculations enable an accurate prediction of the physicochemical properties of complex molecular systems; however, the scale of these calculations is limited by their extremely high computational cost. The Fragment Molecular Orbital (FMO) method is arguably one of the most effective ways to lower this computational cost while retaining predictive accuracy. In this paper, a novel distributed many-GPU algorithm and implementation of the FMO method are presented. When applied in tandem with the Hartree-Fock and RI-MP2 methods, the new implementation enables correlated calculations on 623,016 electrons and 146,592 atoms in less than 45 minutes using 99.8% of the Summit supercomputer (27,600 GPUs). The implementation demonstrates remarkable speedups with respect to other current GPU and CPU codes, and excellent strong scalability on Summit achieving 94.6 % parallel efficiency on 4600 nodes. This work makes feasible correlated quantum chemistry calculations on significantly larger molecular systems than before and with higher accuracy.
Graph-based electronic structure theory offers a scalable approach to study large, complex atomistic systems using distributed and hybrid computational platforms. We demonstrate the coupling of graph-based linear scaling electronic structure theory, as implemented in the Scalable Ecosystem, Driver, and Analyzer for Complex Chemistry Simulations (SEDACS), with semiempirical quantum chemistry methods as implemented in the PySEQM code, with Graphics Processing Unit (GPU) acceleration. This powerful combination enables efficient, scalable electronic structure calculations over many nodes, significantly reducing computational cost while naturally harnessing parallelism. Detailed analyses of parallelization efficiency, computational accuracy, and communication overheads are provided, highlighting an order-of-magnitude speedup for systems of up to 10,000 atoms.
Advances in computer hardware have pushed forward the size of chemical investigations that are accessible using quantum chemistry. However, because state-of-the-art quantum chemistry methods scale as N 5 - N7, the hardware advances can only take us so far. A key advance in the past decade has been the development of scalable (parallel) codes for highly correlated wavefunctions. Following a summary of the methods used in our approach to parallelism in GAMESS (General Atomic and Molecular Electronic Structure System), several applications that emphasize the efficiacy of scalable codes will be discussed. These include the prediction of potential new fuels, the study of new materials with desirable properties and the study of species of biological importance
Evaluating the Coulomb interaction remains a fundamental computational bottleneck in electronic structure theory, dictating the scaling and efficiency of quantum chemical simulations. While recent advancements in graphics processing units (GPUs) offer immense theoretical peak performance, translating traditional, memory-intensive CPU algorithms for evaluating electron repulsion integrals to GPU architectures presents significant challenges due to memory bandwidth limitations and warp divergence. This thesis presents the development of algorithms to efficiently evaluate Coulomb interactions across diverse chemical systems. Chapter 2 describes the GPU implementation of Coulomb interaction for molecular systems at Hartree-Fock level of theory. Chapter 3 shows how multigrid Gaussian-Plane-Wave algorithm can better utilize GPU compute performance by a two-level parallelization scheme and grid-based screening of orbital pairs in the real space. Chapter 4 illustrates the issues of the original modified Coulomb potential from GFN-xTB in periodic systems, and the effectiveness of a new generalized Ewald summation scheme for the potential in X23 molecular crystal dataset and a range of prototypical bulk semiconductors.