We demonstrate that conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics—also known as the edge of chaos—exhibit universal scaling laws of absorbing phase transitions in nonequilibrium statistical mechanics. We exploit the fully deterministic nature of the propagation dynamics to elucidate an analogy between a signal collapse in the neural networks and an absorbing state (a state that the system can enter but cannot escape from). Our numerical results indicate that the multilayer perceptrons and the convolutional neural networks belong to the mean-field and the directed percolation universality classes, respectively. Also, the finite-size scaling is successfully applied, suggesting a potential connection to the depth-width trade-off in deep learning. Furthermore, our analysis of the training dynamics under the gradient descent reveals that hyperparameter tuning to the phase boundary is necessary but insufficient for achieving optimal generalization in deep networks. Remarkably, nonuniversal metric factors associated with the scaling laws are shown to play a significant role in concretizing the above observations. These findings highlight the usefulness of the notion of criticality for analyzing the behavior of artificial deep neural networks and offer new insights toward a unified understanding of the essential relationship between criticality and intelligence.
We demonstrate that conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics—also known as the edge of chaos—exhibit universal scaling laws of absorbing phase transitions in nonequilibrium statistical mechanics. We exploit the fully deterministic nature of the propagation dynamics to elucidate an analogy between a signal collapse in the neural networks and an absorbing state (a state that the system can enter but cannot escape from). Our numerical results indicate that the multilayer perceptrons and the convolutional neural networks belong to the mean-field and the directed percolation universality classes, respectively. Also, the finite-size scaling is successfully applied, suggesting a potential connection to the depth-width trade-off in deep learning. Furthermore, our analysis of the training dynamics under the gradient descent reveals that hyperparameter tuning to the phase boundary is necessary but insufficient for achieving optimal generalization in deep networks. Remarkably, nonuniversal metric factors associated with the scaling laws are shown to play a significant role in concretizing the above observations. These findings highlight the usefulness of the notion of criticality for analyzing the behavior of artificial deep neural networks and offer new insights toward a unified understanding of the essential relationship between criticality and intelligence.
Non-equilibrium critical phenomena have attracted a lot of research interest in the recent decades. Similar to equilibrium critical phenomena, the concept of universality remains the major tool to order the great variety of non-equilibrium phase transitions systematically. All systems belonging to a given universality class share the same set of critical exponents, and certain scaling functions become identical near the critical point. It is known that the scaling functions vary more widely between different universality classes than the exponents. Thus, universal scaling functions offer a sensitive and accurate test for a system's universality class. On the other hand, universal scaling functions demonstrate the robustness of a given universality class impressively. Unfortunately, most studies focus on the determination of the critical exponents, neglecting the universal scaling functions. In this work a particular class of non-equilibrium critical phenomena is considered, the so-called absorbing phase transitions. Absorbing phase transitions are expected to occur in physical, chemical as well as biological systems, and a detailed introduction is presented. The universal scaling behavior of two different universality classes is analyzed in detail, namely the directed percolation and the Manna universality class. Especially, directed percolation is the most common universality class of absorbing phase transitions. The presented picture gallery of universal scaling functions includes steady state, dynamical as well as finite size scaling functions. In particular, the effect of an external field conjugated to the order parameter is investigated. Incorporating the conjugated field, it is possible to determine the equation of state, the susceptibility, and to perform a modified finite-size scaling analysis appropriate for absorbing phase transitions. Focusing on these equations, the obtained results can be applied to other non-equilibrium continuous phase transitions observed in numerical simulations or experiments. Thus, we think that the presented picture gallery of universal scaling functions is valuable for future work. Additionally to the manifestation of universality classes, universal scaling functions are useful in order to check renormalization group results quantitatively. Since the renormalization group theory is the basis of our understanding of critical phenomena, it is of fundamental interest to examine the accuracy of the obtained results. Due to the continuing improvement of computer hardware, accurate numerical data have become available, resulting in a fruitful interplay between numerical investigations and renormalization group analyzes.
Over the past several years, there have been many studies demonstrating the ability of deep neural networks to identify phase transitions in many physical systems, notably in classical statistical physics systems. One often finds that the prediction of deep learning methods trained on many ensembles below and above the critical temperature T_{c} behaves similarly to an order parameter, and this analogy has been successfully used to locate T_{c} and estimate universal critical exponents. In this work, we pay particular attention to the ability of a convolutional neural network to capture these critical parameters for the two-dimensional Ising model when the network is trained on configurations at T=0 and T=∞ only. We directly compare its output to the same network trained at multiple temperatures below and above T_{c} to gain understanding of how this extreme restriction of training data can impact a neural network's ability to classify phases. We find that the network trained on two temperatures is still able to identify T_{c} and ν, while the extraction of γ becomes more challenging.
We explicitly construct the quantum field theory corresponding to a general class of deep neural networks encompassing both recurrent and feedforward architectures. We first consider the mean-field theory (MFT) obtained as the leading saddlepoint in the action, and derive the condition for criticality via the largest Lyapunov exponent. We then compute the loop corrections to the correlation function in a perturbative expansion in the ratio of depth T to width N, and find a precise analogy with the well-studied O(N) vector model, in which the variance of the weight initializations plays the role of the 't Hooft coupling. In particular, we compute both the O(1) corrections quantifying fluctuations from typicality in the ensemble of networks, and the subleading O(T/N) corrections due to finite-width effects. These provide corrections to the correlation length that controls the depth to which information can propagate through the network, and thereby sets the scale at which such networks are trainable by gradient descent. Our analysis provides a first-principles approach to the rapidly emerging NN-QFT correspondence, and opens several interesting avenues to the study of criticality in deep neural networks.
Non-equilibrium critical phenomena have attracted a lot of research interest in the recent decades. Similar to equilibrium critical phenomena, the concept of universality remains the major tool to order the great variety of non-equilibrium phase transitions systematically. All systems belonging to a given universality class share the same set of critical exponents, and certain scaling functions become identical near the critical point. It is known that the scaling functions vary more widely between different universality classes than the exponents. Thus, universal scaling functions offer a sensitive and accurate test for a system's universality class. On the other hand, universal scaling functions demonstrate the robustness of a given universality class impressively. Unfortunately, most studies focus on the determination of the critical exponents, neglecting the universal scaling functions. In this work a particular class of non-equilibrium critical phenomena is considered, the so-called absorbing phase transitions. Absorbing phase transitions are expected to occur in physical, chemical as well as biological systems, and a detailed introduction is presented. The universal scaling behavior of two different universality classes is analyzed in detail, namely the directed percolation and the Manna universality class. Especially, directed percolation is the most common universality class of absorbing phase transitions. The presented picture gallery of universal scaling functions includes steady state, dynamical as well as finite size scaling functions. In particular, the effect of an external field conjugated to the order parameter is investigated. Incorporating the conjugated field, it is possible to determine the equation of state, the susceptibility, and to perform a modified finite-size scaling analysis appropriate for absorbing phase transitions. Focusing on these equations, the obtained results can be applied to other non-equilibrium continuous phase transitions observed in numerical simulations or experiments. Thus, we think that the presented picture gallery of universal scaling functions is valuable for future work. Additionally to the manifestation of universality classes, universal scaling functions are useful in order to check renormalization group results quantitatively. Since the renormalization group theory is the basis of our understanding of critical phenomena, it is of fundamental interest to examine the accuracy of the obtained results. Due to the continuing improvement of computer hardware, accurate numerical data have become available, resulting in a fruitful interplay between numerical investigations and renormalization group analyzes.
Abstract Phase transitions play a central role in statistical physics and are well understood in systems at thermal equilibrium. However, their nonequilibrium counterparts remain less explored because of the lack of a unified theoretical framework. In this study, we examine the potential of deep learning, specially convolutional neural networks (CNN), to detect nonequilibrium phase transitions (NEPT) in a 2D Ising model-like square lattice ferromagnetic system under effective parameter h . Without this parameter ( h = 0 ), we generate equilibrium spin configurations using the standard Metropolis algorithm as usual. With h ≠ 0 , we generate nonequilibrium spin configurations using an effective Glauber update rule which breaks detailed balance. While the former is used to train CNN, the later is not subsequently described by the usual Boltzmann-Gibbs distribution rather used to test the model. The study shows that the trained CNN can distinguish between ordered and disordered nonequilibrium steady states and accurately predict the critical temperature of the system undergoing NEPT. Finite size scaling analysis of the model’s output (perdition) reproduces critical exponents that agree with theoretical expectations, indicating that this method provides a strong, data driven approach for examining nonequilibrium critical phenomena.
This paper provides a comprehensive review and synthesis of phase transitions and criticality in random linear systems. Such systems, characterized by matrices with stochastic entries, are ubiquitous across statistical physics, machine learning, signal processing, and control theory. We examine how their macroscopic properties undergo abrupt qualitative changes as underlying parameters vary, analogous to physical phase transitions. Criticality, the state at these phase boundaries, exhibits unique statistical properties like scaling laws and universal behavior, often defying characterization by conventional perturbation theories. The paper reviews foundational concepts from Random Matrix Theory and statistical mechanics of disordered systems, connecting them to modern high-dimensional data analysis and artificial intelligence. We discuss theoretical frameworks, including the replica and cavity methods, and numerical approaches employed to uncover these transitions. Furthermore, we highlight the practical implications of understanding criticality for designing robust algorithms, predicting system performance limits, and comprehending emergent complexity. Our aim is to offer an interdisciplinary synthesis, bridging classical insights with cutting-edge developments, to provide a unified perspective on these profound phenomena across diverse research fields.