A parallel algorithm is described for the coupled-cluster singles and doubles method augmented with a perturbative correction for triple excitations [CCSD(T)] using the resolution-of-the-identity (RI) approximation for two-electron repulsion integrals (ERIs). The algorithm bypasses the storage of four-center ERIs by adopting an integral-direct strategy. The CCSD amplitude equations are given in a compact quasi-linear form by factorizing them in terms of amplitude-dressed three-center intermediates. A hybrid MPI/OpenMP parallelization scheme is employed, which uses the OpenMP-based shared memory model for intranode parallelization and the MPI-based distributed memory model for internode parallelization. Parallel efficiency has been optimized for all terms in the CCSD amplitude equations. Two different algorithms have been implemented for the rate-limiting terms in the CCSD amplitude equations that entail and -scaling computational costs, where N O and N V denote the number of correlated occupied and virtual orbitals, respectively. One of the algorithms assembles the four-center ERIs requiring N V 4 and N O 2 N V 2 -scaling memory costs in a distributed manner on a number of MPI ranks, while the other algorithm completely bypasses the assembling of quartic memory-scaling ERIs and thus largely reduces the memory demand. It is demonstrated that the former memory-expensive algorithm is faster on a few hundred cores, while the latter memory-economic algorithm shows a better strong scaling in the limit of a few thousand cores. The program is shown to exhibit a near-linear scaling, in particular for the compute-intensive triples correction step, on up to 8000 cores. The performance of the program is demonstrated via calculations involving molecules with 24–51 atoms and up to 1624 atomic basis functions. As the first application, the complete basis set (CBS) limit for the interaction energy of the π-stacked uracil dimer from the S66 data set has been investigated. This work reports the first calculation of the interaction energy at the CCSD(T)/aug-cc-pVQZ level without local orbital approximation. The CBS limit for the CCSD correlation contribution to the interaction energy was found to be −8.01 kcal/mol, which agrees very well with the value −7.99 kcal/mol reported by Schmitz, Hättig, and Tew [ Phys. Chem. Chem. Phys. 2014, 16, 22167−22178]. The CBS limit for the total interaction energy was estimated to be −9.64 kcal/mol.
An algorithm is presented for the coupled-cluster singles, doubles, and perturbative triples correction [CCSD(T)] method based on the density fitting or the resolution-of-the-identity (RI) approximation for performing calculations on heterogeneous computing platforms composed of multicore CPUs and graphics processing units (GPUs). The directive-based approach to GPU offloading offered by the OpenMP application programming interface has been employed to adapt the most compute-intensive terms in the RI-CCSD amplitude equations with computational costs scaling as O ( N O 2 N V 4 ), O ( N O 3 N V 3 ), and O ( N O 4 N V 2 ) (where N O and N V denote the numbers of correlated occupied and virtual orbitals, respectively) and the perturbative triples correction to execute on GPU architectures. The pertinent tensor contractions are performed using an accelerated math library such as cuBLAS or hipBLAS. Optimal strategies are discussed for splitting large data arrays into tiles to fit them into the relatively small memory space of the GPUs, while also minimizing the low-bandwidth CPU–GPU data transfers. The performance of the hybrid CPU–GPU RI-CCSD(T) code is demonstrated on pre-exascale supercomputers composed of heterogeneous nodes equipped with NVIDIA Tesla V100 and A100 GPUs and on the world’s first exascale supercomputer named “Frontier”, the nodes of which consist of AMD MI250X GPUs. Speedups within the range 4–8× relative to the recently reported CPU-only algorithm are obtained for the GPU-offloaded terms in the RI-CCSD amplitude equations. Applications to polycyclic aromatic hydrocarbons containing 16–66 carbon atoms demonstrate that the acceleration of the hybrid CPU–GPU code for the perturbative triples correction relative to the CPU-only code increases with the molecule size, attaining a speedup of 5.7× for the largest circumovalene molecule (C 66 H 20 ). The GPU-offloaded code enables the computation of the perturbative triples correction for the C 60 molecule using the cc-pVDZ/aug-cc-pVTZ-RI basis sets in 7 min on Frontier when using 12,288 AMD GPUs with a parallel efficiency of 83.1%.
Accurate correlated electronic structure methods involve a significant amount of computations and can be only employed to small molecular systems. For example, the coupled-cluster singles, doubles, and perturbative triples model (CCSD(T)), which is known as the ``gold standard" of quantum chemistry for its accuracy, usually can treat molecules with 20-30 atoms. To extend the reach of accurate correlated electronic structure methods to larger molecular systems, we work towards two directions: parallel computing and reduced-cost/scaling approaches. Parallel computing can utilize more computational resources to handle systems that demand more substantial computational efforts. Reduced-cost/scaling approaches, which introduce approximations to the existing electronic structure methods, can significantly reduce the amount of computation and storage requirements. In this work, we introduce a new distributed-memory massively parallel implementation of standard and explicitly correlated (F12) coupled-cluster singles and doubles (CCSD) with canonical bigO{N^6} computational complexity ( C. Peng, J. A. Calvin, F. Pavov{s}evi'c, J. Zhang, and E. F. Valeev, textit{J. Phys. Chem. A} 2016, textbf{120}, 10231.), based on the TiledArray tensor framework. Excellent strong scaling is demonstrated on a multi-core shared-memory computer, a commodity distributed-memory computer, and a national-scale supercomputer. We also present a distributed-memory implementation of the density-fitting (DF) based CCSD(T) method. (C. Peng, J. A. Calvin, and E. F. Valeev, textit{in preparation for submission}) An improved parallel DF-CCSD is presented utilizing lazy evaluation for tensors with more than two unoccupied indices, which makes the DF-CCSD storage requirements always smaller than those of the non-iterative triples correction (T). Excellent strong scaling is observed on both shared-memory and distributed-memory computers equipped with conventional Intel Xeon processors and the Intel Xeon Phi (Knights Landing) processors. With the new implementation, the CCSD(T) energies can be evaluated for systems containing 200 electrons and 1000 basis functions in a few days using a small size commodity cluster, with even more massive computations possible on leadership-class computing resources. The inclusion of F12 correction to the CCSD(T) method makes it converge to basis set limit much more rapidly. The large-scale parallel explicitly correlated coupled-cluster program makes the accurate estimation of the coupled-cluster basis set limit for molecules with 20 or more atoms a routine. Thus, it can be used rigorously to test the emerging reduced-scaling coupled-cluster approaches. Moreover, we extend the pair natural orbital (PNO) approach to excited states through the equation-of-motion coupled cluster singles and doubles (EOM-CCSD) method. (C. Peng, M. C. Clement, and E. F. Valeev, textit{submitted}) We simulate the PNO-EOM-CCSD method using an existing massively parallel canonical EOM-CCSD program. We propose the use of state-averaged PNOs, which are generated from the average of the pair density of excited states, to span the PNO space of all the excited states. The doubles amplitudes in the CIS(D) method are used to compute the state-averaged pair density of excited states. The issue of incorrect states in the state-averaged pair density, caused by an energy reordering of excited states between the CIS(D) and EOM-CCSD, is resolved by simply computing more states than desired. We find that with a truncation threshold of
10−7, the truncation error for the excitation energy is already below 0.02 eV for the systems tested, while the average number of PNOs is reduced to 50-70 per pair. The accuracy of the PNO-EOM-CCSD method on local, Rydberg and charge transfer states is also investigated.
Efficient scalable implementations are presented for two single-reference coupled-cluster (CC) methods with non-iterative corrections for triple excitations, which accurately describe single bond dissociations in molecules in contrast to the standard CCSD(T) approach. These methods are (a) the completely renormalized coupled-cluster theory, CR-CC(2,3), and (b) the ΛCCSD(T) approach, the triples energy corrections in both of which are formulated using the left eigenvector of the CC similarity transformed Hamiltonian. The density fitting approximation is employed for factorizing the two-electron repulsion integrals (ERIs), which eliminates memory bottlenecks associated with the storage of the four-index ERIs by allowing an integral-direct algorithm. Massively parallel algorithms have been developed for the above methods within the GAMESS quantum chemistry program using a hybrid MPI and OpenMP based parallelization model. All compute-intensive steps in the CCSD amplitude and the left eigenvector equations, as well as the triples corrections in both methods, have been adapted to execute on graphical processing units (GPUs) using the OpenMP target directives. Performance of these codes is demonstrated on petascale supercomputers equipped with NVIDIA A100 GPUs. While the CR-CC(2,3) and ΛCCSD(T) computations take roughly twice as much time as the standard CCSD(T) method, the GPU-offloaded codes are shown to impart 3.5-5× accelerations relative to the CPU-only parallel codes, thereby significantly reducing the time-to-solution for the CR-CC(2,3) and ΛCCSD(T) methods.
We have developed a graphical processing unit (GPU) accelerated implementation of our recently introduced rank-reduced coupled-cluster singles and doubles (RR-CCSD) method. RR-CCSD introduces a low-rank approximation of the doubles amplitudes. This is combined with a low-rank approximation of the electron repulsion integrals via Cholesky decomposition. The result of these two low-rank approximations is the replacement of the usual fourth-order CCSD tensors with products of second- and third-order tensors. In our implementation, only a single fourth-order tensor must be constructed as an intermediate during the solution of the amplitude equations. Owing in large part to the compression of the doubles amplitudes, the GPU-accelerated implementation shows excellent parallel efficiency (95% on eight GPUs). Our implementation can solve the RR-CCSD equations for up to 400 electrons and 1550 basis functions—roughly 50% larger than the largest canonical CCSD computations that have been performed on any hardware. In addition to increased scalability, the RR-CCSD computations are faster than the corresponding CCSD computations for all but the smallest molecules. We test the accuracy of RR-CCSD for a variety of chemical systems including up to 1000 basis functions and determine that accuracy to better than 0.1% error in the correlation energy can be achieved with roughly 95% compression of the ov space for the largest systems considered. We also demonstrate that conformational energies can be predicted to be within 0.1 kcal mol−1 with efficient compression applied to the wavefunction. Finally, we find that low-rank approximations of the CCSD doubles amplitudes used in the similarity transformation of the Hamiltonian prior to a conventional equation-of-motion CCSD computation will not introduce significant errors (on the order of a few hundredths of an electronvolt) into the resulting excitation energies.
We consider the rank-reduced coupled-cluster theory with single and double (RR-CCSD) excitations introduced recently [Parrish et al., J. Chem. Phys. 150, 164118 (2019)]. The main feature of this method is the decomposed form of doubly excited amplitudes, which are expanded in the basis of largest magnitude eigenvectors of MP2 or MP3 amplitudes. This approach enables a substantial compression of amplitudes with only minor loss of accuracy. However, the formal scaling of the computational costs with the system size (N) is unaffected in comparison with the conventional CCSD theory (∝N6) due to the presence of some terms quadratic in amplitudes, which do not naturally factorize to a simpler form even within the rank-reduced framework. We show how to solve this problem, exploiting the fact that their effective rank increases only linearly with the system size. We provide a systematic way to approximate the problematic terms using the singular value decomposition and reduce the scaling of the RR-CCSD iterations down to the level of N5. This is combined with an iterative method of finding dominant eigenpairs of MP2 or MP3 amplitudes, which eliminates the necessity to perform the complete diagonalization, making the cost of this step proportional to the fifth power of the system size, as well. Next, we consider the evaluation of perturbative corrections to CCSD energies resulting from triply excited configurations. The triply excited amplitudes present in the CCSD(T) method are decomposed to the Tucker-3 format using the higher-order orthogonal iteration procedure. This enables us to compute the energy correction due to triple excitations non-iteratively with N6 cost. The accuracy of the resulting rank-reduced CCSD(T) method is studied for both total and relative correlation energies of a diverse set of molecules. Accuracy levels better than 99.9% can be achieved with a substantial reduction of the computational costs. Concerning the computational timings, the break-even point between the rank-reduced and conventional CCSD implementations occurs for systems with about 30–40 active electrons.
Abstract A high‐performance implementation of the coupled‐cluster singles, doubles, and perturbative triples [CCSD(T)] is developed in the Massively Parallel Quantum Chemistry program. Novel features include: (1) reduced memory requirements via a density‐fitting (DF) CCSD implementation utilizing distributed lazy evaluation for tensors with more than two unoccupied indices and (2) the ability to utilize efficiently many‐core nodes (Intel Xeon Phi) and heterogeneous nodes with multiple NVIDIA GPUs on each node. All data that are greater than quadratic in the system size are distributed among processes. Excellent strong scaling is observed on distributed‐memory computers equipped with conventional CPUs, Intel Xeon Phi processors, and heterogeneous nodes with multiple NVIDIA GPUs Canonical CCSD(T) energies can be evaluated for systems containing 200 electrons and 1000 basis functions in a few days using a small size commodity cluster, with even larger computations possible on leadership‐class computing resources.