Abstract Analytically continuing the von Neumann entropy from Rényi entropies is a challenging task in quantum field theory. While the n -th Rényi entropy can be computed using the replica method in the path integral representation of quantum field theory, the analytic continuation can only be achieved for some simple systems on a case-by-case basis. In this work, we propose a general framework to tackle this problem using classical and quantum neural networks with supervised learning. We begin by studying several examples with known von Neumann entropy, where the input data is generated by representing $${\text {Tr}}\rho _A^n$$ Tr ρ A n with a generating function. We adopt KerasTuner to determine the optimal network architecture and hyperparameters with limited data. In addition, we frame a similar problem in terms of quantum machine learning models, where the expressivity of the quantum models for the entanglement entropy as a partial Fourier series is established. Our proposed methods can accurately predict the von Neumann and Rényi entropies numerically, highlighting the potential of deep learning techniques for solving problems in quantum information theory.
Abstract Analytically continuing the von Neumann entropy from Rényi entropies is a challenging task in quantum field theory. While the n -th Rényi entropy can be computed using the replica method in the path integral representation of quantum field theory, the analytic continuation can only be achieved for some simple systems on a case-by-case basis. In this work, we propose a general framework to tackle this problem using classical and quantum neural networks with supervised learning. We begin by studying several examples with known von Neumann entropy, where the input data is generated by representing TrρAn Tr ρ A n with a generating function. We adopt KerasTuner to determine the optimal network architecture and hyperparameters with limited data. In addition, we frame a similar problem in terms of quantum machine learning models, where the expressivity of the quantum models for the entanglement entropy as a partial Fourier series is established. Our proposed methods can accurately predict the von Neumann and Rényi entropies numerically, highlighting the potential of deep learning techniques for solving problems in quantum information theory.
Entanglement entropy is an essential metric for characterizing quantum many-body systems, but its numerical evaluation for neural network representations of quantum states has so far been inefficient and demonstrated only for the restricted Boltzmann machine architecture. Here we estimate generalized R\'enyi entropies of autoregressive neural quantum states with up to N=256 spins using quantum Monte Carlo methods. A naive ``direct sampling'' approach performs well for low-order R\'enyi entropies but fails for larger orders when benchmarked on a one-dimensional Heisenberg model. We therefore propose an improved ``conditional sampling'' method exploiting the autoregressive structure of the network ansatz, which outperforms direct sampling and facilitates calculations of higher-order R\'enyi entropies in both one- and two-dimensional Heisenberg models. Access to higher-order R\'enyi entropies allows for an approximation of the von Neumann entropy as well as extraction of the single-copy entanglement. Both methods elucidate the potential of neural network quantum states in quantum Monte Carlo studies of entanglement entropy for many-body systems.
The remarkable generalization capabilities of overparameterized neural networks remain one of the most profound mysteries in modern machine learning. Despite having sufficient capacity to simply memorize training data, these networks consistently achieve superior performance on unseen test examples. Traditional analyses focusing on weight distributions, loss landscapes, and optimization dynamics have provided partial insights but fail to capture the complex correlational structure that emerges between neurons during learning. This paper introduces a comprehensive quantum information framework for analyzing neural network representations, treating hidden layer activations as quantum states and characterizing them through density matrices, von Neumann entropy, purity, and related quantum information metrics. Through systematic experiments on synthetic classification tasks and algorithmic reasoning problems spanning over 3000 cumulative training epochs and 5000 Wang-Landau sampling iterations, we demonstrate that these quantum metrics reveal structure completely invisible to classical analysis. Our results show that stochastic gradient descent systematically reduces von Neumann entropy while increasing purity as networks learn, transitioning from highly mixed quantum states with entropy approximately 1.42 and purity approximately 0.29 to purer configurations with entropy approximately 1.19 and purity approximately 0.37 that achieve superior test accuracy of 99.5%. Using Wang-Landau sampling to explore the equilibrium entropy landscape, we discover that neural networks naturally favor high-entropy states with entropy approximately 1.42 and moderate accuracy of 62.0%, and that optimization actively purifies the quantum state to achieve task performance. On modular arithmetic tasks exhibiting the grokking phenomenon, we observe that von Neumann entropy increases dramatically from 1.55 to 3.29, representing a 112% increase, during an extended 2000-epoch memorization phase before any improvement in generalization, while purity decreases correspondingly from 0.444 to 0.042, a 90.5% decrease, approaching the maximally mixed limit of 0.01. This entropy accumulation phase appears necessary for eventual generalization, with the network building quantum correlations at a rate of approximately 0.00097 nats per epoch during the plateau. Weight entropy, in contrast, shows minimal change of only 11% increase over the same period. Correlation analysis reveals that von Neumann entropy and purity are almost perfectly anticorrelated with a correlation coefficient of 1 -0.99, while weight entropy shows weak correlation with quantum metrics with absolute correlation coefficients less than 0.3. The effective rank of the density matrix stabilizes near 2.7 to 3.0 for the 6-neuron classifier, indicating that the network effectively uses approximately three independent activation modes. These findings establish quantum information metrics as powerful tools for understanding neural network learning dynamics, provide the first quantitative evidence for the entropy accumulation interpretation of grokking, and suggest that neural networks undergo a quantum-inspired phase transition during learning. We discuss implications for generalization theory, optimization algorithm design, and the statistical physics of deep learning, and outline a comprehensive research agenda for extending these methods to larger architectures and more complex tasks.
The remarkable generalization capabilities of overparameterized neural networks remain one of the most profound mysteries in modern machine learning. Despite having sufficient capacity to simply memorize training data, these networks consistently achieve superior performance on unseen test examples. Traditional analyses focusing on weight distributions, loss landscapes, and optimization dynamics have provided partial insights but fail to capture the complex correlational structure that emerges between neurons during learning. This paper introduces a comprehensive quantum information framework for analyzing neural network representations, treating hidden layer activations as quantum states and characterizing them through density matrices, von Neumann entropy, purity, and related quantum information metrics. Through systematic experiments on synthetic classification tasks and algorithmic reasoning problems spanning over 3000 cumulative training epochs and 5000 Wang-Landau sampling iterations, we demonstrate that these quantum metrics reveal structure completely invisible to classical analysis. Our results show that stochastic gradient descent systematically reduces von Neumann entropy while increasing purity as networks learn, transitioning from highly mixed quantum states with entropy approximately 1.42 and purity approximately 0.29 to purer configurations with entropy approximately 1.19 and purity approximately 0.37 that achieve superior test accuracy of 99.5%. Using Wang-Landau sampling to explore the equilibrium entropy landscape, we discover that neural networks naturally favor high-entropy states with entropy approximately 1.42 and moderate accuracy of 62.0%, and that optimization actively purifies the quantum state to achieve task performance. On modular arithmetic tasks exhibiting the grokking phenomenon, we observe that von Neumann entropy increases dramatically from 1.55 to 3.29, representing a 112% increase, during an extended 2000-epoch memorization phase before any improvement in generalization, while purity decreases correspondingly from 0.444 to 0.042, a 90.5% decrease, approaching the maximally mixed limit of 0.01. This entropy accumulation phase appears necessary for eventual generalization, with the network building quantum correlations at a rate of approximately 0.00097 nats per epoch during the plateau. Weight entropy, in contrast, shows minimal change of only 11% increase over the same period. Correlation analysis reveals that von Neumann entropy and purity are almost perfectly anticorrelated with a correlation coefficient of 1-0.99, while weight entropy shows weak correlation with quantum metrics with absolute correlation coefficients less than 0.3. The effective rank of the density matrix stabilizes near 2.7 to 3.0 for the 6-neuron classifier, indicating that the network effectively uses approximately three independent activation modes. These findings establish quantum information metrics as powerful tools for understanding neural network learning dynamics, provide the first quantitative evidence for the entropy accumulation interpretation of grokking, and suggest that neural networks undergo a quantum-inspired phase transition during learning. We discuss implications for generalization theory, optimization algorithm design, and the statistical physics of deep learning, and outline a comprehensive research agenda for extending these methods to larger architectures and more complex tasks.
Abstract We propose a method of quantum machine learning called quantum mutual information neural estimation (QMINE) for estimating von Neumann entropy and quantum mutual information, which are fundamental properties in quantum information theory. The QMINE proposed here basically utilizes a technique of quantum neural networks (QNNs), to minimize a loss function that determines the von Neumann entropy, and thus quantum mutual information, which is believed more powerful to process quantum datasets than conventional neural networks due to quantum superposition and entanglement. To create a precise loss function, we propose a quantum Donsker-Varadhan representation (QDVR), which is a quantum analog of the classical Donsker-Varadhan representation. By exploiting a parameter shift rule on parameterized quantum circuits, we can efficiently implement and optimize the QNN and estimate the quantum entropies using the QMINE technique. Furthermore, numerical observations support our predictions of QDVR and demonstrate the good performance of QMINE.
Abstract An alternative method is presented for extracting the von Neumann entropy − Tr(ρ ln ρ) from Tr(ρn) for integer n in a quantum system with density matrix ρ. Instead of relying on direct analytic continuation in n, the method uses a generating function − Tr{ρ ln[(1 − zρ)/(1 − z)]} of an auxiliary complex variable z. The generating function has a Taylor series that is absolutely convergent within |z| < 1, and may be analytically continued in z to z = −∞ where it gives the von Neumann entropy. As an example, we use the method to calculate analytically the CFT entanglement entropy of two intervals in the small cross ratio limit, reproducing a result that Calabrese et al. obtained by direct analytic continuation in n. Further examples are provided by numerical calculations of the entanglement entropy of two intervals for general cross ratios, and of one interval at finite temperature and finite interval length.
Abstract Calculating the von Neumann entanglement entropy from experimental data is challenging due to its dependence on the complete wavefunction, forcing reliance on approximations such as classical mutual information (MI). We propose a machine learning approach using a graph neural network to predict the von Neumann entropy directly from experimentally accessible bitstrings. We test this approach on a Rydberg ladder system and achieve a mean absolute error of 3.6 × 10 − 3 when evaluating within the training range on a dataset with entropy values ranging from 0 to 1.9. The model achieves a mean absolute percentage error of 1.44% and outperforms MI-based bounds. When tested beyond the training range, the model maintains reasonable accuracy. Furthermore, we demonstrate that fine-tuning the model with small datasets significantly improves performance on data outside the original training range.