TeraChem: Accelerating electronic structure and ab initio molecular dynamics with graphical processing units
Stefan Seritan, Christoph Bannwarth, B. Scott Fales, Edward G. Hohenstein, Sara I. L. Kokkila-Schumacher, Nathan Luehr, James W. Snyder, Chenchen Song, Alexey V. Titov, Ivan S. Ufimtsev, Todd J. Martı́nez
Developed over the past decade, TeraChem is an electronic structure and ab initio molecular dynamics software package designed from the ground up to leverage graphics processing units (GPUs) to perform large-scale ground and excited state quantum chemistry calculations in the gas and the condensed phase. TeraChem's speed stems from the reformulation of conventional electronic structure theories in terms of a set of individually optimized high-performance electronic structure operations (e.g., Coulomb and exchange matrix builds, one- and two-particle density matrix builds) and rank-reduction techniques (e.g., tensor hypercontraction). Recent efforts have encapsulated these core operations and provided language-agnostic interfaces. This greatly increases the accessibility and flexibility of TeraChem as a platform to develop new electronic structure methods on GPUs and provides clear optimization targets for emerging parallel computing architectures.
Developed over the past decade, TeraChem is an electronic structure and ab initio molecular dynamics software package designed from the ground up to leverage graphics processing units (GPUs) to perform large-scale ground and excited state quantum chemistry calculations in the gas and the condensed phase. TeraChem's speed stems from the reformulation of conventional electronic structure theories in terms of a set of individually optimized high-performance electronic structure operations (e.g., Coulomb and exchange matrix builds, one- and two-particle density matrix builds) and rank-reduction techniques (e.g., tensor hypercontraction). Recent efforts have encapsulated these core operations and provided language-agnostic interfaces. This greatly increases the accessibility and flexibility of TeraChem as a platform to develop new electronic structure methods on GPUs and provides clear optimization targets for emerging parallel computing architectures.
We present libterachem.py, a Python interface and method framework that facilitates development and prototyping of electronic structure methods while leveraging TeraChem's optimized graphical processing unit (GPU) operations. Direct access to these building blocks and their intermediate quantities allows users to assemble and modify methods within Python while reusing the underlying GPU implementations. A layered architecture, programmable at both the operation level and the method level, supports numerical experimentation, composition of complete electronic structure methods with interchangeable components, and integration with external scientific libraries. We illustrate operation-level use with grid pruning for least-squares tensor hypercontraction and with exchange--correlation functionals supplied through LibXC and by the machine-learned Skala functional. We illustrate method-level use with dispersion corrections, periodic calculations, and geometry optimization with an external optimizer. Python-driven self-consistent-field calculations using libterachem.py incur no significant computational overhead compared to native TeraChem and preserve its multi-GPU scaling. A photoactive yellow protein QM/MM molecular dynamics application further demonstrates throughput comparable to native TeraChem in a workflow composed with external simulation libraries. These results establish libterachem.py as a platform for method development and interoperable molecular simulation that retains the performance of TeraChem's GPU operations.
Graphical Processing Units (GPUs) are highly parallel, programmable accelerators boasting high peak floating point performance. Over the last couple of years the use of GPUs for general purpose computing have revolutionized quantum chemistry. The computational bottleneck in an ab-initio quantum method is the calculation of a large number of twoelectron integrals. To date, a number of GPU accelerated two-electron integral implementations have been developed significantly improving the performance of a static quantum mechanical (QM) calculation. However, when performing an ab-initio QM gradient calculation, optimization, QM or Hybrid Quantum Mechanical/Molecular Mechanical (QM/MM) dynamics simulation the twoelectron integral derivatives arise as an additional bottleneck. Hybrid QM/MM methods particularly dynamics methods are commonly used to study large chemical/biological systems. These methods are a popular choice used for studying reaction mechanisms, conformational and configurational structures important in glycobiology. Usually a semiempirical QM method is used however, these have shown variable accuracy for the study of carbohydrate conformation and prevents an analytical investigation of electronic structure. The use of a higher level of theory, such as an ab initio method, is desirable however this comes at a much greater computational cost. Together with the bottlenecks above this cost results from the polarization of the QM region within an electrostatic embedding scheme, which requires the calculation of a large number of one-electron integral derivatives. Thus for QM/MM calculations the one-electron integral derivatives becomes a third bottleneck together with the two mentioned above. Recently, the above bottlenecks have become popular GPU acceleration targets. This thesis describes an extension of the GPU based Quantum Supercharger Library (QSL) to perform the above calculations/simulations. In contrast to GPU packages developed from the ground up, the QSL is a library of routines aimed at accelerating legacy codes, such as GAMESS-UK, GAMESS-US and NWChem, used in electronic structure calculations. Algorithms are presented for accelerating the one- and two-electron integral derivatives on a GPU. In addition to the derivatives, the one-electron integral calculation was ported to the GPU in order to remove a small additional cost arising for large QM/MM systems. Furthermore, the use of automatic code generation for generating GPU kernels was explored and compared to the original approach. Using the QSL library implemented in GAMESS-UK several benchmark calculations and simulations were performed. These were performed in double precision on a single GPU (Kepler K20) and compared to a single CPU using the 6-31G basis set. A speedup of up to 9.3X is achieved for an ab initio gradient calculation compared to the CPU running an optimized serial version of GAMESS-UK using the Schlegel method. For a single point QM/MM calculation of cellobiose in different sized water spheres (3267-24843 point charges) speedups of between 13X and 34X is achieved. QSL/GAMESS-UK coupled to the CHARMM molecular dynamics package was then used in order to perform accelerated molecular dynamics simulations. Benchmark QM and QM/MM molecular dynamics simulations were performed on cellobiose in vacuo and in a water sphere (45 QM atoms and 24843 point charges respectively). The QSL is able to perform 9.7 ps/day of ab initio QM dynamics and 6.4 ps/day of QM/MM dynamics. Testing of the auto-generated version of the QSL showed better performance for lower angular momentum classes but reduced performance for higher angular momentum classes. The efficiency of the integral and derivative routines within the QSL library was tested on a computationally intense realistic glycobiological condensed phase free energy computation. Ab-initio pucker free energy surfaces/volumes of !-Ribofuranose and !-Glucopyranose in vacuum and in water were computed. These are the first converged Hartree-Fock/MM free energy simulations for these carbohydrates performed in solution. The value of the ab initio Free energy surfaces/volumes was demonstrated through an analysis of solvent polarization effects that are evident through a comparison of the vacuum and solution minimum free energy pathways. In particular the water structure around the pucker conformers, analysis of the primary alcohol distribution as well as analysis of the electronic structure of these conformers reveals that water significantly affects the free energy pathways, primary alcohol distribution and the barriers for inter-conversion between pucker conformers.
Coupled cluster with singles, doubles, and perturbative triples (CCSD(T)) often provides ground state correlation energies within ″chemical accuracy", but it suffers from high computational cost and steep scaling with system size. We present a GPU-accelerated implementation of CCSD(T) in the TeraChem software package. The new implementation achieves state-of-the-art performance, enabling the calculation of the (T) correction for a system with 63 atoms and more than 1000 basis functions in a little under 8 h on a single node. Additionally, we demonstrate the utility of our optimized implementation for the rapid calculation of full CCSD(T)/CBS stacking energies for all ten unique DNA base pair stacked tetramers. We expect that the TeraChem CCSD(T) implementation will enable the rapid calculation of high-level data that was not previously accessible in a reasonable time frame.
Electronic structure calculations have the potential to predict key matter transformations for applications of strategic technological importance, from drug discovery to material science and catalysis. However, a predictive physicochemical characterization of these processes often requires accurate quantum chemical modeling of complex molecular systems with hundreds to thousands of atoms. Due to the computationally demanding nature of electronic structure calculations and the complexity of modern high-performance computing hardware, quantum chemistry software has historically failed to operate at such large molecular scales with accuracy and speed that are useful in practice. In this paper, novel algorithms and software are presented that enable extreme-scale quantum chemistry capabilities with particular emphasis on exascale calculations. This includes the development and application of the multi-Graphics Processing Unit (GPU) library LibCChem 2.0 as part of the General Atomic and Molecular Electronic Structure System package and of the standalone Extreme-scale Electronic Structure System (EXESS), designed from the ground up for scaling on thousands of GPUs to perform high-performance accurate quantum chemistry calculations at unprecedented speed and molecular scales. Among various results, we report that the EXESS implementation enables Hartree-Fock/cc-pVDZ plus RI-MP2/cc-pVDZ/cc-pVDZ-RIFIT calculations on an ionic liquid system with 623 016 electrons and 146 592 atoms in less than 45 min using 27 600 GPUs on the Summit supercomputer with a 94.6% parallel efficiency.
The increasing availability of graphics processing units (GPUs) for scientific computing has prompted interest in accelerating quantum chemical calculations through their use. However, the complexity of integral kernels for high angular momentum basis functions often limits the utility of GPU implementations with large basis sets or for metal containing systems. In this work, we report the implementation of f function support in the GPU-accelerated TeraChem software package through the development of efficient kernels for the evaluation of Hamiltonian integrals. The high efficiency of the resulting code is demonstrated through density functional theory (DFT) calculations on increasingly large organic molecules and transition metal complexes, as well as coupled cluster singles and doubles calculations on water clusters. Preliminary investigations into Ni(I) catalysis with DFT and the photochemistry of MnH(CH3) with complete active space self-consistent field are also carried out. Overall, our GPU-accelerated software appears to be well-suited for fast simulation of large transition metal containing systems, as well as organic molecules.
The dynamic rearrangement of a molecule at finite temperature is not captured by a single static input geometry to a quantum chemical calculation. Multilevel workflows have established themselves as standard procedures to capture the dependence of the molecule’s configuration on computed properties. Each step within this workflow uses an electronic structure theory method that features the optimal computational time to accuracy ratio. For exploratory steps the computational expense of a method needs to be minimal as several thousand energy and nuclear gradient evaluations are needed. Semiempirical tight binding methods, such as GFN2-xTB are often used for such purposes. However, these methods usually lack the accuracy to rank conformers relative to each other within chemical accuracy (2 kcal/mol). Therefore, so-called “low-cost” DFT methods are commonly used for geometry optimizations and conformer ranking. The combination of (semi)empirical correction models with small basis sets leads to good accuracy at moderate computational cost. However, even these highly efficient methods reach the limits of feasibility for very large systems (> 200 atoms). This work proposes solutions to partially alleviate those restrictions and allow routine calculations of these system sizes outside of large high-performance computing facilities. The key to unlock those calculations is to combine tailored algorithmic developments with consumer-grade GPUs. A GPU-enabled linear algebra library is developed to accelerate further development and make the entry to GPU-accelerated software development easier. To demonstrate the ease of implementation, the addition of Mulliken-approximated exchange to GFN2-xTB will be shown. In most semiempirical methods, the general eigenvalue solver algorithm becomes the computational bottleneck, dominating the walltime with increasing system size. A density matrix purification algorithm is tailored to give up to six-fold acceleration of GFN2-xTB calculations using a mixed-precision approach. Lastly, 3c-methods are added to the GPU-accelerated TeraChem software package. In combination with the “mixed-precision” scheme for computing the electron repulsion integrals, this addition decreases the runtime of DFT calculations by up to an order of magnitude on mid-level consumer hardware. The overall goal of these projects is to make GPU-accelerated calculations in quantum chemistry more available. This achieve by exposing algorithmic developments that are performant on reasonably priced hardware and making GPU computing more accessible to non-experts.