International Conference for High Performance Computing, Networking, Storage and Analysis (SC)

Enabling large-scale correlated electronic structure calculations

Giuseppe M. J. Barca, Jorge Luis Galvez Vallejo, David Poole, Melisa Alkan, Ryan Stocks, Alistair P. L. Rendell, Mark S. Gordon

Published Oct 21, 2021• 18 citations

Publisher

Abstract

Second-order Møller-Plesset perturbation theory using the Resolution-of-the-Identity approximation (RI-MP2) is a state-of-the-art approach to accurately estimate many-body electronic correlation effects. This is critical for predicting the physicochemical properties of complex molecular systems; however, the scale of these calculations is limited by their extremely high computational cost. In this paper, a novel many-GPU algorithm and implementation of a molecular-fragmentation-based RI-MP2 method are presented that enable correlated calculations on over 180,000 electrons and 45,000 atoms using up to the entire Summit supercomputer in 12 minutes. The implementation demonstrates remarkable speedups with respect to other current GPU and CPU codes, excellent strong scalability on Summit achieving 89.1% parallel efficiency on 4600 nodes, and shows nearly-ideal weak scaling up to 612 nodes. This work makes feasible ab initio correlated quantum chemistry calculations on significantly larger molecular scales than before on both large supercomputing systems and on commodity clusters, with a potential for major impact on progress in chemical, physical, biological and engineering sciences.

DOI: 10.1145/3458817.3476222

Related papers

Scaling Correlated Fragment Molecular Orbital Calculations on Summit

Giuseppe M. J. Barca, Calum Snowdon, Jorge Luis Galvez Vallejo +3 more

Correlated electronic structure calculations enable an accurate prediction of the physicochemical properties of complex molecular systems; however, the scale of these calculations is limited by their extremely high computational cost. The Fragment Molecular Orbital (FMO) method is arguably one of the most effective ways to lower this computational cost while retaining predictive accuracy. In this paper, a novel distributed many-GPU algorithm and implementation of the FMO method are presented. When applied in tandem with the Hartree-Fock and RI-MP2 methods, the new implementation enables correlated calculations on 623,016 electrons and 146,592 atoms in less than 45 minutes using 99.8% of the Summit supercomputer (27,600 GPUs). The implementation demonstrates remarkable speedups with respect to other current GPU and CPU codes, and excellent strong scalability on Summit achieving 94.6 % parallel efficiency on 4600 nodes. This work makes feasible correlated quantum chemistry calculations on significantly larger molecular systems than before and with higher accuracy.

International Conference for High Performance Computing, Networking, Storage and Analysis (SC)Nov 1, 202224 citations

An Efficient RI-MP2 Algorithm for Distributed Many-GPU Architectures

Calum Snowdon, Giuseppe M. J. Barca

Second-order Møller-Plesset perturbation theory (MP2) using the Resolution of the Identity approximation (RI-MP2) is a widely used method for computing molecular energies beyond the Hartree-Fock mean-field approximation. However, its high computational cost and lack of efficient algorithms for modern supercomputing architectures limit its applicability to large molecules. In this paper, we present the first distributed-memory many-GPU RI-MP2 algorithm explicitly designed to utilize hundreds of GPU accelerators for every step of the computation. Our novel algorithm achieves near-peak performance on GPU-based supercomputers through the development of a distributed memory algorithm for forming RI-MP2 intermediate tensors with zero inter-node communication, except for a single O(N^2) asynchronous broadcast, and a distributed memory algorithm for the O(N^5) energy reduction step, capable of sustaining near-peak performance on clusters with several hundred GPUs. Comparative analysis shows our implementation outperforms state-of-the-art quantum chemistry software by over 3.5 times in speed while achieving an eightfold reduction in computational power consumption. Benchmarking on the Perlmutter supercomputer, our algorithm achieves 11.8 PFLOP/s (83% of peak performance) performing and the RI-MP2 energy calculation on a 314-water cluster with 7,850 primary and 30,144 auxiliary basis functions in 4 minutes on 180 nodes and 720 A100 GPUs. This performance represents a substantial improvement over traditional CPU-based methods, demonstrating significant time-to-solution and power consumption benefits of leveraging modern GPU-accelerated computing environments for quantum chemistry calculations.

Journal of Chemical Theory and ComputationOct 18, 20247 citations

High-Performance Multi-GPU Analytic RI-MP2 Energy Gradients

Ryan Stocks, Elise Palethorpe, Giuseppe M. J. Barca

This article presents a novel algorithm for the calculation of analytic energy gradients from second-order Møller-Plesset perturbation theory within the Resolution-of-the-Identity approximation (RI-MP2), which is designed to achieve high performance on clusters with multiple graphical processing units (GPUs). The algorithm uses GPUs for all major steps of the calculation, including integral generation, formation of all required intermediate tensors, solution of the Z-vector equation and gradient accumulation. The implementation in the EXtreme Scale Electronic Structure System (EXESS) software package includes a tailored, highly efficient, multistream scheduling system to hide CPU-GPU data transfer latencies and allows nodes with 8 A100 GPUs to operate at over 80% of theoretical peak floating-point performance. Comparative performance analysis shows a significant reduction in computational time relative to traditional multicore CPU-based methods, with our approach achieving up to a 95-fold speedup over the single-node performance of established software such as Q-Chem and ORCA. Additionally, we demonstrate that pairing our implementation with the molecular fragmentation framework in EXESS can drastically lower the computational scaling of RI-MP2 gradient calculations from quintic to subquadratic, enabling further substantial savings in runtime while retaining high numerical accuracy in the resulting gradients.

Journal of Chemical Theory and ComputationMar 8, 202420 citations

Lower-scaling advanced correlation methods with local Resolution of Identity strategies

Arvid Conrad Ihrig

First-principles methods are a valuable tool to study the electronic structure of systems and explore the quantum-mechanical mechanisms driving them on the microscopic scale. Semi-local Density-Functional Approximations (DFAs) are still very popular due to the good balance between accuracy and computational efficiency. However, the instantaneous electronic interactions are described with an averaged potential in these methods, which is often insufficient to describe charge transfers or subtle many-body correlations. A promising way to improve the description of such interactions is the usage of advanced correlation methods which explicitly treat the electronic interactions in a many-body picture. Well known methods form this class are e.g. the Random- Phase Approximation (RPA), 2nd Order Møller-Plesset Perturbation Theory (MP2) and Coupled-Cluster Theory (CC). Unfortunately, the time and memory requirements for all these methods suffer from a steep scaling with system size in their canonical formulations. Several lower-scaling reformulations exist for these methods, but the at least cubic memory requirements and the increased algorithmic complexity prevented any efficient parallelization of these methods so far. In my thesis I developed a new lower-scaling technique for advanced correlation methods to address this challenge. The key components of this technique are the Laplace-Transformation (LT) and a new local Resolution of Identity (RI) strategy, named RI-LVL. Together, they significantly reduce the memory requirement without sacrificing accuracy and allow for an efficient parallelization by means of à priori known sparsity patterns. In combination with established integral screening techniques, the computational scaling can also be reduced considerably. The potential of this new approach is demonstrated at the example of an in-memory LT-MP2 with a MPI/OpenMP hybrid parallelization. It features an at worst cubic computational scaling, a quadratic memory scaling and a high parallel efficiency. Based on the Numeric Atom-Centered Orbital (NAO) framework used in the Fritz Haber Institute ab initio molecular simulations (FHI-aims) code, this implementation can handle both molecular and periodic systems with the same numerical footing. I will also discuss in detail how the combination of the LT and RI- LVL can be applied to RPA calculations. The accuracy and performance of my LT- MP2 is demonstrated in detail for both molecular and periodic systems. In addition, the convergence of the MP2 level adsorption energy of water on a rutile titanium dioxide (TiO2) surface in the low coverage limit is explored. This new LT-MP2 recovers more than 99.9% of the canonical MP2 correlation energy and can handle TiO2 surfaces with up to 270 atoms in the unit cell.

Jan 1, 20170 citations

Accelerating Resolution-of-the-Identity Second-Order Møller−Plesset Quantum Chemistry Calculations with Graphical Processing Units

Leslie Vogt-Maranto, Roberto Olivares‐Amaya, Sean Kermes +3 more

The modification of a general purpose code for quantum mechanical calculations of molecular properties (Q-Chem) to use a graphical processing unit (GPU) is reported. A 4.3x speedup of the resolution-of-the-identity second-order Møller-Plesset perturbation theory (RI-MP2) execution time is observed in single point energy calculations of linear alkanes. The code modification is accomplished using the compute unified basic linear algebra subprograms (CUBLAS) library for an NVIDIA Quadro FX 5600 graphics card. Furthermore, speedups of other matrix algebra based electronic structure calculations are anticipated as a result of using a similar approach.

The Journal of Physical Chemistry AJan 30, 2008156 citations

Toward an extreme-scale electronic structure system

Jorge Luis Galvez Vallejo, Calum Snowdon, Ryan Stocks +13 more

Electronic structure calculations have the potential to predict key matter transformations for applications of strategic technological importance, from drug discovery to material science and catalysis. However, a predictive physicochemical characterization of these processes often requires accurate quantum chemical modeling of complex molecular systems with hundreds to thousands of atoms. Due to the computationally demanding nature of electronic structure calculations and the complexity of modern high-performance computing hardware, quantum chemistry software has historically failed to operate at such large molecular scales with accuracy and speed that are useful in practice. In this paper, novel algorithms and software are presented that enable extreme-scale quantum chemistry capabilities with particular emphasis on exascale calculations. This includes the development and application of the multi-Graphics Processing Unit (GPU) library LibCChem 2.0 as part of the General Atomic and Molecular Electronic Structure System package and of the standalone Extreme-scale Electronic Structure System (EXESS), designed from the ground up for scaling on thousands of GPUs to perform high-performance accurate quantum chemistry calculations at unprecedented speed and molecular scales. Among various results, we report that the EXESS implementation enables Hartree-Fock/cc-pVDZ plus RI-MP2/cc-pVDZ/cc-pVDZ-RIFIT calculations on an ionic liquid system with 623 016 electrons and 146 592 atoms in less than 45 min using 27 600 GPUs on the Summit supercomputer with a 94.6% parallel efficiency.

The Journal of Chemical PhysicsJul 27, 202334 citations