arXiv:2211.05504 · Physical Review Letters

Transformer Variational Wave Functions for Frustrated Quantum Spin Systems

Luciano Loris Viteritti, Riccardo Rende, Federico Becca

Published Jun 5, 2023• 85 citations

Abstract

The transformer architecture has become the state-of-art model for natural language processing tasks and, more recently, also for computer vision tasks, thus defining the vision transformer (ViT) architecture. The key feature is the ability to describe long-range correlations among the elements of the input sequences, through the so-called self-attention mechanism. Here, we propose an adaptation of the ViT architecture with complex parameters to define a new class of variational neural-network states for quantum many-body systems, the ViT wave function. We apply this idea to the one-dimensional J_{1}-J_{2} Heisenberg model, demonstrating that a relatively simple parametrization gets excellent results for both gapped and gapless phases. In this case, excellent accuracies are obtained by a relatively shallow architecture, with a single layer of self-attention, thus largely simplifying the original architecture. Still, the optimization of a deeper structure is possible and can be used for more challenging models, most notably highly frustrated systems in two dimensions. The success of the ViT wave function relies on mixing both local and global operations, thus enabling the study of large systems with high accuracy.

DOI: 10.1103/physrevlett.130.236401

Related papers

Vision transformer neural quantum states for impurity models

Xiaodong Cao, Zhicheng Zhong, Yi Lu

Transformer neural networks, known for their ability to recognize complex patterns in high-dimensional data, offer a promising framework for capturing many-body correlations in quantum systems. We employ an adapted vision transformer (ViT) architecture to model quantum impurity models, optimizing it with a subspace expansion scheme that surpasses conventional variational Monte Carlo in both accuracy and efficiency. Benchmarks against matrix product states in single- and three-orbital Anderson impurity models show that these ViT-based neural quantum states achieve comparable or superior accuracy with significantly fewer variational parameters. We further extend our approach to compute dynamical quantities by constructing a restricted excitation space that effectively captures relevant physical processes, yielding accurate core-level x-ray absorption spectra. These findings highlight the potential of ViT-based neural quantum states for accurate and efficient modeling of quantum impurity models.

arXiv:2408.13050Physical review. B./Physical review. BNov 24, 20253 citations

Transformer wave function for quantum long-range models

Sebastián Roca-Jerat, Manuel Gallego, Fernando M. Luis +2 more

We employ a neural-network architecture based on the vision transformer (ViT) architecture to find the ground states of quantum long-range models, specifically the transverse-field Ising model for spin-1/2 chains across different interaction regimes. Harnessing the transformer's capacity to capture long-range correlations, we compute the full phase diagram and critical properties of the model in both the ferromagnetic and antiferromagnetic cases. Our findings show that the ViT maintains high accuracy across the full phase diagram. We compare these results with previous numerical studies in the literature and, in particular, show that the ViT has a superior performance than a restricted-Boltzmann-machine-like ansatz.

arXiv:2407.04773Physical review. B./Physical review. BNov 25, 20247 citations

Design principles of deep translationally symmetric neural quantum states for frustrated magnets

Rajah P. Nutakki, Ahmedeo Shokry, Filippo Vicentini

Deep neural-network quantum states have emerged as a leading method for studying the ground states of quantum magnets. Successful architectures exploit translational symmetry, but the root of their effectiveness and differences between architectures remain unclear. Here, we apply the ConvNext architecture, designed to incorporate elements of transformers into convolutional networks, to quantum many-body ground states. We find that it is remarkably similar to the factored vision transformer, which has been employed successfully for several frustrated spin systems, allowing us to relate this architecture to more conventional convolutional networks. Through a series of numerical experiments, we design the ConvNext to achieve greatest performance at lowest computational cost, and then apply this network to the Shastry-Sutherland and J 1 − J 2 models, obtaining variational energies comparable to the state of the art, providing a blueprint for network design choices of translationally symmetric architectures to tackle challenging ground-state problems in frustrated magnetism.

arXiv:2505.03466Physical Review ResearchSep 15, 20253 citations

Variational Monte Carlo with large patched transformers

Kyle Sprague, Stefanie Czischek

Abstract Large language models, like transformers, have recently demonstrated immense powers in text and image generation. This success is driven by the ability to capture long-range correlations between elements in a sequence. The same feature makes the transformer a powerful wavefunction ansatz that addresses the challenge of describing correlations in simulations of qubit systems. Here we consider two-dimensional Rydberg atom arrays to demonstrate that transformers reach higher accuracies than conventional recurrent neural networks for variational ground state searches. We further introduce large, patched transformer models, which consider a sequence of large atom patches, and show that this architecture significantly accelerates the simulations. The proposed architectures reconstruct ground states with accuracies beyond state-of-the-art quantum Monte Carlo methods, allowing for the study of large Rydberg systems in different phases of matter and at phase transitions. Our high-accuracy ground state representations at reasonable computational costs promise new insights into general large-scale quantum many-body systems.

arXiv:2306.03921Communications PhysicsMar 11, 202432 citations

Transformer Neural-Network Quantum States for Lattice Models of Spins and Fermions: Application to the Ancilla Layer Model

Riccardo Rende, Alexander Nikolaenko, Luciano Loris Viteritti +2 more

We introduce a variational wave function based on neural-network quantum states (NQS) to study lattice systems whose local Hilbert space contains both spin and fermionic degrees of freedom. Our approach is based on the use of the transformer architecture, which can naturally handle composite local Hilbert spaces through a tokenization procedure closely inspired by techniques from natural language processing. The key methodological innovation is that the neural network predicts a set of fermionic orbitals that depend on the spin configuration in a backflow-inspired manner. We apply the method to the one-dimensional ancilla layer model, consisting of a chain of mobile spin- 1 / 2 fermions coupled to a two-leg spin- 1 / 2 ladder. For open boundary conditions, we achieve excellent quantitative agreement with density matrix renormalization group results across the full range of parameters considered. We find a phase in which the chain forms an effectively decoupled Luttinger liquid (LL), and an LL * phase with a distinct Fermi wave vector in which the mobile fermions are Kondo screened by one leg of the ladder, while the other leg forms the critical Bethe spin liquid. The LL * is the analog of the FL * phase in two dimensions proposed to describe the cuprate pseudogap. We also find a Luther-Emery phase, where the LL * state becomes unstable toward the formation of a spin gap. The transformer ansatz maintains comparable accuracy for periodic boundary conditions, where tensor-network methods are computationally more demanding. Moreover, we show that the approach extends naturally to fully periodic two-dimensional clusters, a regime beyond the reach of current tensor-network methods. Together, these findings establish transformer-based NQS as an accurate and scalable variational framework for correlated lattice systems with composite local Hilbert spaces and highlight its potential for studying higher-dimensional models where boundary effects and heterogeneous local structures pose significant challenges.

arXiv:2603.02316PRX IntelligenceOct 22, 20260 citations

Theory of Transformers and their application to Neural Quantum States

Riccardo Rende

My PhD research focused on the Transformer architecture, a powerful deep neural network model that has emerged as a cornerstone for solving complex problems in natural language processing, image analysis, signal processing, and beyond. In particular, we studied the learning dynamics of this architecture, and its application to the representation of many-body wavefunctions, the so-called Neural Quantum States. Initially, we investigated the representational capabilities of Transformers by characterizing the statistical structures that a simplified Transformer layer, utilizing the so-called factored attention, is capable of learning. Building on these results, we utilized factored attention in deep Transformers to develop an accurate ansatz for approximating the ground states of quantum many-body Hamiltonians within the variational Monte Carlo framework. In this specific application, factored attention is crucial for achieving accurate results, demonstrating superior performance compared to the standard attention mechanism used in most of the other applications of the Transformers, and in particular in natural language processing. Alongside the development of an efficient optimization method for large-scale neural networks, we achieved state-of-the-art results on the most popular benchmark in Neural Quantum States and addressed complex physical problems that are subjects of ongoing debate. Finally, we developed a framework to train Foundation Neural Quantum States, which are versatile neural network models that approximate quantum wave functions of multiple systems simultaneously, enabling accurate estimates of challenging quantities such as disorder averages and fidelity susceptibility. We envision numerous future directions for this approach, including its extension to quantum dynamics by explicitly modeling time-dependent variational states, as well as its application to the design of novel materials in fermionic systems.

Sep 11, 20250 citations