Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

Theory of Transformers and their application to Neural Quantum States

Riccardo Rende

Published Sep 11, 2025• 0 citations
OpenAlex

Abstract

My PhD research focused on the Transformer architecture, a powerful deep neural network model that has emerged as a cornerstone for solving complex problems in natural language processing, image analysis, signal processing, and beyond. In particular, we studied the learning dynamics of this architecture, and its application to the representation of many-body wavefunctions, the so-called Neural Quantum States. Initially, we investigated the representational capabilities of Transformers by characterizing the statistical structures that a simplified Transformer layer, utilizing the so-called factored attention, is capable of learning. Building on these results, we utilized factored attention in deep Transformers to develop an accurate ansatz for approximating the ground states of quantum many-body Hamiltonians within the variational Monte Carlo framework. In this specific application, factored attention is crucial for achieving accurate results, demonstrating superior performance compared to the standard attention mechanism used in most of the other applications of the Transformers, and in particular in natural language processing. Alongside the development of an efficient optimization method for large-scale neural networks, we achieved state-of-the-art results on the most popular benchmark in Neural Quantum States and addressed complex physical problems that are subjects of ongoing debate. Finally, we developed a framework to train Foundation Neural Quantum States, which are versatile neural network models that approximate quantum wave functions of multiple systems simultaneously, enabling accurate estimates of challenging quantities such as disorder averages and fidelity susceptibility. We envision numerous future directions for this approach, including its extension to quantum dynamics by explicitly modeling time-dependent variational states, as well as its application to the design of novel materials in fermionic systems.

Related papers

Neural networks and quantum many-body physics: exploring reciprocal benefits.

Anna Golubeva

One of the main reasons why the physics of quantum many-body systems is hard \nlies in the curse of dimensionality: \nThe number of states of such systems increases exponentially with the number of degrees of freedom involved. \nAs a result, computations for realistic systems become intractable, \nand even numerical methods are limited to comparably small system sizes. \nMany efforts in modern physics research are therefore concerned with \nfinding efficient representations of quantum states and \nclever approximations schemes that would allow them to characterize physical systems of interest. \n \nMeanwhile, Deep Learning (DL) has solved many non-scientific problems \nthat have been unaccessible to conventional methods for a similar reason. \nThe concept underlying DL is to extract knowledge from data \nby identifying patterns and regularities. \nThe remarkable success of DL has excited many physicists about \nthe prospect of leveraging its power to solve intractable problems in physics. \nAt the same time, DL turned out to be an interesting complex many-body \nproblem in itself. \nIn contrast to its widespread empirical applications, \nthe theoretical foundation of DL is strongly underdeveloped. \nIn particular, as long as its decision-making process and result interpretability remain opaque, \nDL can not claim the status of a scientific tool. \n \nIn this thesis, I explore the interface between DL and quantum many-body physics, \nand investigate DL both as a tool and as a subject of study. \nThe first project presented here is a theory-based study of \na fundamental open question about the role of width and the number of parameters in deep neural networks. \nIn this work, we consider a DL setup for the image recognition task on standard benchmarking datasets. \nWe combine controlled experiments with a theoretical analysis, including analytical calculations for a toy model. \nThe other three works focus on the application of Restricted Boltzmann Machines as generative models \nfor the task of wavefunction reconstruction from measurement data on a quantum many-body system. \nFirst, we implement this approach as a software package, making it available as a tool for experimentalists. \nFollowing the idea that physics problems can be used to characterize DL tools, \nwe then use our extensive knowledge of this setup to conduct a systematic study of \nhow the RBM complexity scales with the complexity of the physical system. \nFinally, in a follow-up study we focus on the effects of parameter pruning techniques \non the RBM and its scaling behavior.

•Aug 3, 2021•0 citations

Efficient representation of quantum many-body states with deep neural networks

Xun Gao, Lu-Ming Duan

Part of the challenge for quantum many-body problems comes from the difficulty of representing large-scale quantum states, which in general requires an exponentially large number of parameters. Neural networks provide a powerful tool to represent quantum many-body states. An important open question is what characterizes the representational power of deep and shallow neural networks, which is of fundamental interest due to the popularity of deep learning methods. Here, we give a proof that, assuming a widely believed computational complexity conjecture, a deep neural network can efficiently represent most physical states, including the ground states of many-body Hamiltonians and states generated by quantum dynamics, while a shallow network representation with a restricted Boltzmann machine cannot efficiently represent some of those states.One of the challenges in studies of quantum many-body physics is finding an efficient way to record the large system wavefunctions. Here the authors present an analysis of the capabilities of recently-proposed neural network representations for storing physically accessible quantum states.

arXiv:1701.05039•Nature Communications•Sep 18, 2017•345 citations

First-principles Deep Learning for Quantum Many-body Physics

Khachatur G. Nazaryan

This thesis investigates emergent phenomena in strongly correlated quantum matter, with a central focus on developing first-principles deep learning methods for quantum many-body systems. A common theme throughout this work is the search for microscopic mechanisms that generate unexpected collective behavior, together with computational frameworks capable of describing such behavior in regimes where conventional approaches become inadequate. The first part of the thesis studies a doped Mott insulator in a magnetic field and uncovers a new superconducting state, which we call \emph{magnonic superconductivity}. In this state, magnons mediate pairing, and the superconducting order is carried by composite objects binding electron pairs to spin excitations. This mechanism gives rise to a partially spin-polarized superconducting phase beyond the conventional paradigm of electron pairing and further suggests the possibility of higher-charge bound states, including charge-$6e$ superconductivity. The main thrust of the thesis is the use of deep neural networks as variational, first-principles representations of many-body wave functions. In contrast to the usual data-driven role of machine learning, the neural-network ans\"atze developed here do not learn from labeled datasets; instead, they are optimized directly from the microscopic Hamiltonian through variational Monte Carlo. Within this framework, we first develop self-attention-based neural-network wave functions for correlated electrons in solids and show that they accurately describe moir\'e systems with minimal human bias, capture interaction-driven phase transitions, and exhibit favorable scaling of representational complexity with system size. We then introduce a general physics-informed initialization strategy that reconstructs complex many-body wave functions from probability density and probability current. This approach enables highly accurate pretraining of neural wave functions, achieves near-exact overlaps for topological and paired quantum states, and allows efficient simulations of fractional quantum Hall systems with realistic interactions at unprecedented sizes. Finally, we introduce QERNEL, an efficient foundational neural variational ansatz designed to learn a family of many-body ground states within a single conditional model. By combining efficient context conditioning with modern permutation-equivariant building blocks, QERNEL achieves substantially improved expressivity at low computational cost. Applied to interacting electrons in semiconductor moir\'e heterobilayers, it captures distinct many-body phases and resolves the sharp phase transition between them in systems as large as 150 electrons. These results show that context-aware neural wave functions can move beyond Hamiltonian-by-Hamiltonian solution toward genuinely foundational models of quantum matter. Taken together, the results of this thesis show that first-principles deep learning can serve not merely as a numerical tool, but as a general framework for quantum many-body physics: one that bridges microscopic Hamiltonians, expressive wave-function ansätze, and scalable computation, while opening new paths to the discovery and characterization of strongly correlated quantum matter.

•May 1, 2026•0 citations

Visualizing Quantum Dynamics

Tyler N. Thompson

My research has three main foci, one which is of interest to the fields of quantum information and quantum optics is to characterize the effects of coupling between quantum systems. Advances in technology made since the conception of quantum mechanics spurred the development of quantum technologies like quantum optics and quantum computing. These applications require a good understanding of the quantum systems that compose them, and the clever ways in which the information in these systems is stored and manipulated. By understanding the ways which quantum systems interact with their environment, their usefulness can be further utilized. The second is to develop methods of calculation which simplify the calculational and computational side of quantum dynamics. Understanding how quantum systems evolve in time is very important, but traditional methods of solving for the dynamics of quantum systems struggle to deal with time dependent parameters. Using results from the theory of Lie algebras, I will show how quantum dynamics problems can be made less computationally intensive by avoiding the partial differential equations typically associated with such problems and reducing the problem to a system of coupled ordinary differential equations. The third focus is to develop methods of visualizing properties of quantum systems. I think this is important as an educational tool and as a way to develop intuition about how quantum systems evolve considering how few opportunities there are to visualize quantum effects on a macroscopic scale. This problem is particularly challenging because quantum mechanics is non-deterministic, meaning that the only nontrivial definitive statements that can be made regarding quantum systems deal with probabilities. This coupled with the fact that many of the systems of interest depend on upwards of six variables and also evolve in time make creating useful visual representations challenging. Despite the challenges, I feel that useful insights can be gained from the visualizations, and will illustrate a few examples.

•Jan 1, 2018•0 citations

Foundation neural-networks quantum states as a unified Ansatz for multiple hamiltonians

Riccardo Rende, Luciano Loris Viteritti, Federico Becca +3 more

Foundation models are highly versatile neural-network architectures capable of processing different data types, such as text and images, and generalizing across various tasks like classification and generation. Inspired by this success, we propose Foundation Neural-Network Quantum States (FNQS) as an integrated paradigm for studying quantum many-body systems. FNQS leverage key principles of foundation models to define variational wave functions based on a single, versatile architecture that processes multimodal inputs, including spin configurations and Hamiltonian physical couplings. Unlike specialized architectures tailored for individual Hamiltonians, FNQS can generalize to physical Hamiltonians beyond those encountered during training, offering a unified framework adaptable to various quantum systems and tasks. FNQS enable the efficient estimation of quantities that are traditionally challenging or computationally intensive to calculate using conventional methods, particularly disorder-averaged observables. Furthermore, the fidelity susceptibility can be easily obtained to uncover quantum phase transitions without prior knowledge of order parameters. These pretrained models can be efficiently fine-tuned for specific quantum systems. The architectures trained in this paper are publicly available at https://huggingface.co/nqs-models , along with examples for implementing these neural networks in NetKet.

arXiv:2502.09488•Nature Communications•Aug 5, 2025•17 citations

Simulation, benchmarking and control of quantum many-qubit systems

Mehran Khosrojerdi

This thesis lies at the intersection of quantum machine learning and tensor network methods, focusing on the simulation, benchmarking, and control of many-body quantum systems.

•Jan 1, 2026•0 citations
My PhD research focused on the Transformer architecture, a powerful deep neural network model that has emerged as a cornerstone for solving complex problems in natural language processing, image analysis, signal processing, and beyond. In particular, we studied the learning dynamics of this architecture, and its application to the representation of many-body wavefunctions, the so-called Neural Quantum States. Initially, we investigated the representational capabilities of Transformers by characterizing the statistical structures that a simplified Transformer layer, utilizing the so-called factored attention, is capable of learning. Building on these results, we utilized factored attention in deep Transformers to develop an accurate ansatz for approximating the ground states of quantum many-body Hamiltonians within the variational Monte Carlo framework. In this specific application, factored attention is crucial for achieving accurate results, demonstrating superior performance compared to the standard attention mechanism used in most of the other applications of the Transformers, and in particular in natural language processing. Alongside the development of an efficient optimization method for large-scale neural networks, we achieved state-of-the-art results on the most popular benchmark in Neural Quantum States and addressed complex physical problems that are subjects of ongoing debate. Finally, we developed a framework to train Foundation Neural Quantum States, which are versatile neural network models that approximate quantum wave functions of multiple systems simultaneously, enabling accurate estimates of challenging quantities such as disorder averages and fidelity susceptibility. We envision numerous future directions for this approach, including its extension to quantum dynamics by explicitly modeling time-dependent variational states, as well as its application to the design of novel materials in fermionic systems.
One of the main reasons why the physics of quantum many-body systems is hard \nlies in the curse of dimensionality: \nThe number of states of such systems increases exponentially with the number of degrees of freedom involved. \nAs a result, computations for realistic systems become intractable, \nand even numerical methods are limited to comparably small system sizes. \nMany efforts in modern physics research are therefore concerned with \nfinding efficient representations of quantum states and \nclever approximations schemes that would allow them to characterize physical systems of interest. \n \nMeanwhile, Deep Learning (DL) has solved many non-scientific problems \nthat have been unaccessible to conventional methods for a similar reason. \nThe concept underlying DL is to extract knowledge from data \nby identifying patterns and regularities. \nThe remarkable success of DL has excited many physicists about \nthe prospect of leveraging its power to solve intractable problems in physics. \nAt the same time, DL turned out to be an interesting complex many-body \nproblem in itself. \nIn contrast to its widespread empirical applications, \nthe theoretical foundation of DL is strongly underdeveloped. \nIn particular, as long as its decision-making process and result interpretability remain opaque, \nDL can not claim the status of a scientific tool. \n \nIn this thesis, I explore the interface between DL and quantum many-body physics, \nand investigate DL both as a tool and as a subject of study. \nThe first project presented here is a theory-based study of \na fundamental open question about the role of width and the number of parameters in deep neural networks. \nIn this work, we consider a DL setup for the image recognition task on standard benchmarking datasets. \nWe combine controlled experiments with a theoretical analysis, including analytical calculations for a toy model. \nThe other three works focus on the application of Restricted Boltzmann Machines as generative models \nfor the task of wavefunction reconstruction from measurement data on a quantum many-body system. \nFirst, we implement this approach as a software package, making it available as a tool for experimentalists. \nFollowing the idea that physics problems can be used to characterize DL tools, \nwe then use our extensive knowledge of this setup to conduct a systematic study of \nhow the RBM complexity scales with the complexity of the physical system. \nFinally, in a follow-up study we focus on the effects of parameter pruning techniques \non the RBM and its scaling behavior.
Part of the challenge for quantum many-body problems comes from the difficulty of representing large-scale quantum states, which in general requires an exponentially large number of parameters. Neural networks provide a powerful tool to represent quantum many-body states. An important open question is what characterizes the representational power of deep and shallow neural networks, which is of fundamental interest due to the popularity of deep learning methods. Here, we give a proof that, assuming a widely believed computational complexity conjecture, a deep neural network can efficiently represent most physical states, including the ground states of many-body Hamiltonians and states generated by quantum dynamics, while a shallow network representation with a restricted Boltzmann machine cannot efficiently represent some of those states.One of the challenges in studies of quantum many-body physics is finding an efficient way to record the large system wavefunctions. Here the authors present an analysis of the capabilities of recently-proposed neural network representations for storing physically accessible quantum states.
This thesis investigates emergent phenomena in strongly correlated quantum matter, with a central focus on developing first-principles deep learning methods for quantum many-body systems. A common theme throughout this work is the search for microscopic mechanisms that generate unexpected collective behavior, together with computational frameworks capable of describing such behavior in regimes where conventional approaches become inadequate. The first part of the thesis studies a doped Mott insulator in a magnetic field and uncovers a new superconducting state, which we call \emph{magnonic superconductivity}. In this state, magnons mediate pairing, and the superconducting order is carried by composite objects binding electron pairs to spin excitations. This mechanism gives rise to a partially spin-polarized superconducting phase beyond the conventional paradigm of electron pairing and further suggests the possibility of higher-charge bound states, including charge-6e6e6e superconductivity. The main thrust of the thesis is the use of deep neural networks as variational, first-principles representations of many-body wave functions. In contrast to the usual data-driven role of machine learning, the neural-network ans\"atze developed here do not learn from labeled datasets; instead, they are optimized directly from the microscopic Hamiltonian through variational Monte Carlo. Within this framework, we first develop self-attention-based neural-network wave functions for correlated electrons in solids and show that they accurately describe moir\'e systems with minimal human bias, capture interaction-driven phase transitions, and exhibit favorable scaling of representational complexity with system size. We then introduce a general physics-informed initialization strategy that reconstructs complex many-body wave functions from probability density and probability current. This approach enables highly accurate pretraining of neural wave functions, achieves near-exact overlaps for topological and paired quantum states, and allows efficient simulations of fractional quantum Hall systems with realistic interactions at unprecedented sizes. Finally, we introduce QERNEL, an efficient foundational neural variational ansatz designed to learn a family of many-body ground states within a single conditional model. By combining efficient context conditioning with modern permutation-equivariant building blocks, QERNEL achieves substantially improved expressivity at low computational cost. Applied to interacting electrons in semiconductor moir\'e heterobilayers, it captures distinct many-body phases and resolves the sharp phase transition between them in systems as large as 150 electrons. These results show that context-aware neural wave functions can move beyond Hamiltonian-by-Hamiltonian solution toward genuinely foundational models of quantum matter. Taken together, the results of this thesis show that first-principles deep learning can serve not merely as a numerical tool, but as a general framework for quantum many-body physics: one that bridges microscopic Hamiltonians, expressive wave-function ansätze, and scalable computation, while opening new paths to the discovery and characterization of strongly correlated quantum matter.
My research has three main foci, one which is of interest to the fields of quantum information and quantum optics is to characterize the effects of coupling between quantum systems. Advances in technology made since the conception of quantum mechanics spurred the development of quantum technologies like quantum optics and quantum computing. These applications require a good understanding of the quantum systems that compose them, and the clever ways in which the information in these systems is stored and manipulated. By understanding the ways which quantum systems interact with their environment, their usefulness can be further utilized. The second is to develop methods of calculation which simplify the calculational and computational side of quantum dynamics. Understanding how quantum systems evolve in time is very important, but traditional methods of solving for the dynamics of quantum systems struggle to deal with time dependent parameters. Using results from the theory of Lie algebras, I will show how quantum dynamics problems can be made less computationally intensive by avoiding the partial differential equations typically associated with such problems and reducing the problem to a system of coupled ordinary differential equations. The third focus is to develop methods of visualizing properties of quantum systems. I think this is important as an educational tool and as a way to develop intuition about how quantum systems evolve considering how few opportunities there are to visualize quantum effects on a macroscopic scale. This problem is particularly challenging because quantum mechanics is non-deterministic, meaning that the only nontrivial definitive statements that can be made regarding quantum systems deal with probabilities. This coupled with the fact that many of the systems of interest depend on upwards of six variables and also evolve in time make creating useful visual representations challenging. Despite the challenges, I feel that useful insights can be gained from the visualizations, and will illustrate a few examples.
Foundation models are highly versatile neural-network architectures capable of processing different data types, such as text and images, and generalizing across various tasks like classification and generation. Inspired by this success, we propose Foundation Neural-Network Quantum States (FNQS) as an integrated paradigm for studying quantum many-body systems. FNQS leverage key principles of foundation models to define variational wave functions based on a single, versatile architecture that processes multimodal inputs, including spin configurations and Hamiltonian physical couplings. Unlike specialized architectures tailored for individual Hamiltonians, FNQS can generalize to physical Hamiltonians beyond those encountered during training, offering a unified framework adaptable to various quantum systems and tasks. FNQS enable the efficient estimation of quantities that are traditionally challenging or computationally intensive to calculate using conventional methods, particularly disorder-averaged observables. Furthermore, the fidelity susceptibility can be easily obtained to uncover quantum phase transitions without prior knowledge of order parameters. These pretrained models can be efficiently fine-tuned for specific quantum systems. The architectures trained in this paper are publicly available at https://huggingface.co/nqs-models , along with examples for implementing these neural networks in NetKet.