The analysis of complex physical systems hinges on the ability to extract the relevant degrees of freedom from among the many others. Though much hope is placed in machine learning, it also brings challenges, chief of which is interpretability. It is often unclear what relation, if any, the architecture- and training-dependent learned "relevant" features bear to standard objects of physical theory. Here we report on theoretical results which may help to systematically address this issue: we establish equivalence between the field-theoretic relevance of the renormalization group, and an information-theoretic notion of relevance we define using the information bottleneck (IB) formalism of compression theory. We show analytically that for statistical physical systems described by a field theory the relevant degrees of freedom found using IB compression indeed correspond to operators with the lowest scaling dimensions. We confirm our field theoretic predictions numerically. We study dependence of the IB solutions on the physical symmetries of the data. Our findings provide a dictionary connecting two distinct theoretical toolboxes, and an example of constructively incorporating physical interpretability in applications of deep learning in physics.
The analysis of complex physical systems hinges on the ability to extract the relevant degrees of freedom from among the many others. Though much hope is placed in machine learning, it also brings challenges, chief of which is interpretability. It is often unclear what relation, if any, the architecture- and training-dependent learned "relevant" features bear to standard objects of physical theory. Here we report on theoretical results which may help to systematically address this issue: we establish equivalence between the field-theoretic relevance of the renormalization group, and an information-theoretic notion of relevance we define using the information bottleneck (IB) formalism of compression theory. We show analytically that for statistical physical systems described by a field theory the relevant degrees of freedom found using IB compression indeed correspond to operators with the lowest scaling dimensions. We confirm our field theoretic predictions numerically. We study dependence of the IB solutions on the physical symmetries of the data. Our findings provide a dictionary connecting two distinct theoretical toolboxes, and an example of constructively incorporating physical interpretability in applications of deep learning in physics.
At low energies, the microscopic characteristics and changes of physical systems as viewed at different distance scales are described by universal scale invariant properties investigated by the Renormalization Group (RG) apparatus, an efficient tool used to deal with scaling problems in effective field theories. We employ an information-theoretic approach in a deep learning setup by introducing an artificial neural network algorithm to map and identify new physical degrees of freedom. Using deep learning methods mapped to a genuine field theory, we develop a mechanism capable to identify relevant degrees of freedom and induce scale invariance without prior knowledge about a system. We show that deep learning algorithms that use an RG-like scheme to learn relevant features from data could help to understand the nature of the holographic entanglement entropy and the holographic principle in context of the AdS/CFT correspondence.
Identifying the relevant degrees of freedom in a complex physical system is a key stage in developing effective theories in and out of equilibrium. The celebrated renormalization group provides a framework for this, but its practical execution in unfamiliar systems is fraught with ad hoc choices, whereas machine learning approaches, though promising, lack formal interpretability. Here we present an algorithm employing state-of-the-art results in machine-learning-based estimation of information-theoretic quantities, overcoming these challenges, and use this advance to develop a new paradigm in identifying the most relevant operators describing properties of the system. We demonstrate this on an interacting model, where the emergent degrees of freedom are qualitatively different from the microscopic constituents. Our results push the boundary of formally interpretable applications of machine learning, conceptually paving the way toward automated theory building.
Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.
Within the past decade, machine learning algorithms have been proposed as a po-tential solution to a variety of research problems which emerge within physics. As this cross-fertilization matures, one is able to investigate if the efficiency of machine learning algorithms can be increased by interpreting them physically and if there exist fundamental connections that can be established between the two research fields. In this thesis, we pursue research directions intimately related to the above questions.First, we investigate the practical implications of interpreting machine learning functions as statistical-mechanical observables. Through this perspective, we explore if we can extend the classification capabilities of machine learning algorithms and if we are able to include neural networks within Hamiltonians to induce phase transitions in systems. A related direction concerns the use of machine learning to construct inverse renormalization group transformations to arbitrarily increase the size of a system. These techniques are then utilized to study the infinite volume limit of discrete spin systems and of quantum field theories, in order to investigate if machine learning is a powerful tool to study phase transitions.In another research direction we explore the derivation of machine learning algo-rithms from quantum field theories. We investigate if the φ4 scalar field theory satisfies the Hammersley-Clifford theorem and if it can be recast as a Markov random field. We then explore if φ4 neural networks can be derived that generalize a certain class of standard neural network architectures, and we present relevant numerical appli-cations. Finally, we discuss how this research direction opens up the opportunity to investigate machine learning within quantum field theory and how it solidifies a rig-orous connection between the research fields of machine learning, probability theory, statistical mechanics, lattice and constructive quantum field theory.
Recently, a novel real-space renormalization group (RG) algorithm was introduced. By maximizing an information-theoretic quantity, the real-space mutual information, the algorithm identifies the relevant low-energy degrees of freedom. Motivated by this insight, we investigate the information-theoretic properties of coarse-graining procedures for both translationally invariant and disordered systems. We prove that a perfect real-space mutual information coarse graining does not increase the range of interactions in the renormalized Hamiltonian, and, for disordered systems, it suppresses the generation of correlations in the renormalized disorder distribution, being in this sense optimal. We empirically verify decay of those measures of complexity as a function of information retained by the RG, on the examples of arbitrary coarse grainings of the clean and random Ising chain. The results establish a direct and quantifiable connection between properties of RG viewed as a compression scheme and those of physical objects, i.e., Hamiltonians and disorder distributions. We also study the effect of constraints on the number and type of coarse-grained degrees of freedom on a generic RG procedure.
Recently, a novel real-space renormalization group (RG) algorithm was introduced.By maximizing an information-theoretic quantity, the real-space mutual information, the algorithm identifies the relevant lowenergy degrees of freedom.Motivated by this insight, we investigate the information-theoretic properties of coarse-graining procedures for both translationally invariant and disordered systems.We prove that a perfect real-space mutual information coarse graining does not increase the range of interactions in the renormalized Hamiltonian, and, for disordered systems, it suppresses the generation of correlations in the renormalized disorder distribution, being in this sense optimal.We empirically verify decay of those measures of complexity as a function of information retained by the RG, on the examples of arbitrary coarse grainings of the clean and random Ising chain.The results establish a direct and quantifiable connection between properties of RG viewed as a compression scheme and those of physical objects, i.e., Hamiltonians and disorder distributions.We also study the effect of constraints on the number and type of coarse-grained degrees of freedom on a generic RG procedure.