Knowledge of Fisher information is not part of the educational background of most physicists. Why should a physicist bother to learn about this concept? Surely the (related) concept of entropy is sufficient to describe the degree of disorder of a given phenomenon. These important questions may be answered as follows. (a) The point made about entropy is true, but does not go far enough. Why not seek a measure of disorder whose variation derives the phenomenon? The concept of entropy cannot do this, for reasons discussed in Sec. 1.3. Fisher information will turn out to be the appropriate measure of disorder for this purpose. (b) Why should a physicist bother to learn this concept? Aside from the partial answer in (a): (i) Fisher information is a simple and intuitive concept. As theories go, it is quite elementary. To understand it does not require mathematics beyond differential equations. Even no prior knowledge of statistics is needed: this is easy enough to learn ‘on the fly’. The derivation of the defining property of Fisher information, in Sec. 1.2.3, is readily understood. (ii) The subject has very little specialized jargon or notation. The beginner does not need a glossary of terms and symbols to aid in its understanding. (iii) Most importantly, once understood, the concept gives strong payoff – one might call it ‘phenomen-all’ – in scope of application. It's simply worth learning.
Knowledge of Fisher information is not part of the educational background of most physicists. Why should a physicist bother to learn about this concept? Surely the (related) concept of entropy is sufficient to describe the degree of disorder of a given phenomenon. These important questions may be answered as follows. (a) The point made about entropy is true, but does not go far enough. Why not seek a measure of disorder whose variation derives the phenomenon? The concept of entropy cannot do this, for reasons discussed in Sec. 1.3. Fisher information will turn out to be the appropriate measure of disorder for this purpose. (b) Why should a physicist bother to learn this concept? Aside from the partial answer in (a): (i) Fisher information is a simple and intuitive concept. As theories go, it is quite elementary. To understand it does not require mathematics beyond differential equations. Even no prior knowledge of statistics is needed: this is easy enough to learn ‘on the fly’. The derivation of the defining property of Fisher information, in Sec. 1.2.3, is readily understood. (ii) The subject has very little specialized jargon or notation. The beginner does not need a glossary of terms and symbols to aid in its understanding. (iii) Most importantly, once understood, the concept gives strong payoff – one might call it ‘phenomen-all’ – in scope of application. It's simply worth learning.
This book defines and develops a unifying principle of physics, that of 'extreme physical information'. The information in question is, perhaps surprisingly, not Shannon or Boltzmann entropy but, rather, Fisher information, a simple concept little known to physicists. Both statistical and physical properties of Fisher information are developed. This information is shown to be a physical measure of disorder, sharing with entropy the property of monotonic change with time. The information concept is applied 'phenomenally' to derive most known physics, from statistical mechanics and thermodynamics to quantum mechanics, the Einstein field equations, and quantum gravity. Many new physical relations and concepts are developed, including new definitions of disorder, time and temperature. The information principle is based upon a new theory of measurement, one which incorporates the observer into the phenomenon that he/she observes. The 'request' for data creates the law that, ultimately, gives rise to the data. The observer creates his or her local reality.
This book defines and develops a unifying principle of physics, that of 'extreme physical information'. The information in question is, perhaps surprisingly, not Shannon or Boltzmann entropy but, rather, Fisher information, a simple concept little known to physicists. Both statistical and physical properties of Fisher information are developed. This information is shown to be a physical measure of disorder, sharing with entropy the property of monotonic change with time. The information concept is applied 'phenomenally' to derive most known physics, from statistical mechanics and thermodynamics to quantum mechanics, the Einstein field equations, and quantum gravity. Many new physical relations and concepts are developed, including new definitions of disorder, time and temperature. The information principle is based upon a new theory of measurement, one which incorporates the observer into the phenomenon that he/she observes. The 'request' for data creates the law that, ultimately, gives rise to the data. The observer creates his or her local reality.
Consider an isolated statistical system specified by a coordinate x and its probability density p(x). A functional of p(x) called ``Fisher information'' can be used to measure the degree of disorder of the system due to the spread in p(x). Fisher information may be minimized, subject to a physical constraint, to attain a temporal equilibrium solution p(x). When the constraint is linear in the mean kinetic energy of the system, the equilibrium solution p(x) often obeys the correct differential equation for the system. In this way, the Schr\"odinger (energy) wave equation, Klein-Gordon equation, Helmholtz wave equation, diffusion equation, Boltzmann law, and Maxwell-Boltzmann law may be derived from one classical principle of disorder. The convergence rate for Fisher information is about that for alternative use of maximum entropy (in problems where both have the same equilibrium solution). This suggests that Fisher information defines an arrow of time. The arrow points in the direction of decreasing accuracy for the determination of the mean, or ideal, value of a parameter.
I. Introduction.Statistical mechanics is the science of explaining, predicting and understanding the gross, macroscopic attributes of matter (which may be taken to mean mechanical systems with essentially an infinite number of degrees of freedom) in terms of the elementary dynamical laws governing its atomic constituents.The problems that arise are sufficiently complex and intriguing, but at the same time sufficiently well posed, that the subject is nowadays as much a part of mathematics as of physics.The fields of information theory and ergodic theory had their genesis in statistical mechanical modes of thought and are now well established in the mathematics literature; there will be more to come.Ludwig Boltzmann, who died in 1906, was one of the principal founders of statistical mechanics, and his monument in Vienna contains the following eloquent testimonial to his scientific creativity:(1) S = k log W.Surely, this hypothesis of Boltzmann [1] is one of the most important and daring in statistical mechanics, for it relates S, the macroscopic entropy of a system, to W, the number of microscopic states of the system which have the same, given macroscopic properties.The number k is a universal constant, called Boltzmann's constant, and, for our purposes, we can consider it to be 1.In these lectures we shall explore some of the abstract properties of entropy, after first giving a precise formulation of it, and will include some recent results (with M. B. Ruskai) which extend formerly known facts about the strong subadditivity of entropy from the domain of classical mechanics to the quantum-mechanical domain.The presentation here will be sketchy and the reader is referred to the original papers [3], [4], [5] for more details.An expanded version of an invited address delivered before the M.I.T.
What is information? A short answer could be: Anything that changes our minds and states of thinking! Information is virtual. It can be carried by speech, an image, scratches on a stone, patterns in a photon field, the connections of neurons in our brains or even by the wavefunction of an electron. Information does not depend on the actual means of transmission nor does its identity change when changing the carrier. Information is physical. It needs to be sustained by a physical substrate and rearranging that substrate to contain the wanted information requires to spend energy or work. Without a physical world, information can not be stored, processed, or transmitted. Our world is permeated with different representations of information. Actually Information about the physical laws governing our Universe can be found everywhere and in everything. Could it then be that the real fundamental elements of this world are tiny bits of information? Could we get it from bit?1 A number of researcher investigate this possibility, and several of their attempts to recover physical laws from the principles governing information are reported about in this special issue on physics of information. The articles by Gregg Jaeger,2 by Ariel Caticha,3 and by John Skilling and Kevin Knuth4 address the reconstruction of quantum mechanics from information theory. Furthermore, Kevin Knuth and James Walsh5 show how to recover space-time from particles sending each others signals in an initially geometry-free setting. A central element of most works on information is the concept of a probability. Probabilities can be defined via various routes. Most familiar might be their definitions by the frequentists school as relative frequencies of events (in the limit of infinite amounts of such). A more general definition of probabilities is the Bayesians' one, which regards probabilities as quantifiers of statements, expressing how much those should be expected to be true. Although this definition looks subjective on the first glance, it rests on a solid mathematical theorem by Richard Cox6 that any extension of binary logic to a continuum of single valued truthfulness quantifier that respects a minimum of requirements must be isomorphic to probability theory. And this definition turns out to embrace the frequentists' definition in case that infinite numbers of comparable events are actually available and can be counted. The majority of authors of this special issues seem to favor the Bayesian perspective, which is probably a result of our own, the guest editors', preference and not necessarily a reflection of the prevailing position of contemporary scientists. The different views on probabilities of frequentists and Bayesians are addressed by the article of Allen Caldwell.7 He builds a bridge between these antagonistic camps by explaining frequentists' constructions in a Bayesian language. In particular, his argumentation why the omnipresent p-value in statistical and experimental literature might often capture a relevant signal of model misfit is reassuring, given that a large fraction of our medical system is based on decisions backed up by p-values. A Bayesian might add that it would be even more reassuring, if a system rigorously build on logic and its continuous extension would be used for decisions affecting live-or-death questions. The laws governing probabilities themselves seem to reflect that of more elementary mathematical structures. Kevin Knuth shows how these emerge from lattice theory, which deals with relations in partially ordered sets like that of collections of possibilities.8 The sum rule of probability theory is a direct consequence of the underlying lattice structure. Interestingly, a number of other theories are resting also on such lattices and therefore exhibit their own versions of sum rules: measures of sets, mutual information, the Euler characteristics of topological spaces, as well as quantum mechanical interactions that are quantified by pairs of numbers, the real and imaginary parts of complex probability amplitudes (see also4). The theory of information has very practical application in physics and elsewhere. Manfred Opper discusses how the solutions of stochastic differential equations can be inferred via variational methods, which have their origin in thermodynamics.9 Torsten Enßlin introduces information field theory (IFT), the information theory for fields, as a means to infer the configuration of a physical field from incomplete and noisy data.10 The mathematical apparatus for this is that of quantum field theory (QFT), just with ordinary probabilities instead of complex probability amplitudes as in the latter. Could it be that the difference between IFT and QFT is just be the same as between classical information theory and quantum mechanics, the transition from real valued to complex valued quantifier for statements? Anyhow, any practical inference machinery, be it for dynamical systems or for fields, requires sophisticated numerical methods. Michael Betancourt reviews Markov Chain Monte Carlo Methods to calculate expectation values over high dimensional probabilities.11 Theo Steininger and coworkers present the NIFTy3 software package for numerical information field theory calculations, which can handle very high dimensional field inference problems.12 Finally, in an article focused on the subtleties of the Monty Hall problem and mostly aiming at information theoretical entertainment, Torsten Enßlin and Margret Westercamp investigate the possibility that human intuition might be more rational than contemporary psychology thinks.13 A number of possible topics for an issue on physics of information are not addressed here. We would have loved to have articles on Landauer's principle, which states that erasing information has to cost energy, on QBism, a Bayesian version of quantum mechanics, on artificial and biological neural networks, as well as any other aspect of the information-soaked biological world. What has a reader of this special issue to be equipped with in order to follow the lines of thoughts? In addition to the usual physicist's education, the minimal additional equipment are the basics of probability theory and the concept of entropy. Most of the other concepts used in the articles rest on those and are introduced there. Let A, B, and C be propositions (or possibilities). The probability P ( A | B ) of A being true assuming that B is true is a number between 0 and 1, with P ( A | B ) = 1 expressing certainty of A under B ( B ⇒ A ) and P ( A | B ) = 0 expressing certainty of not A being the case under B ( B ⇒ ¬ A ).
The astute reader might have formed the impression that quantum in formation science is a rather qualitative discipline because we have not, as yet, explained how to quantify quantum information. There are three good reasons for leaving this important question until the final chapter. Firstly, quantum information theory is technically demanding and to treat it at an earlier stage might have suggested that our subject was more complicated than it is. Secondly, there is the fact that many of the ideas in the field, such as teleportation and quantum circuits, are unfamiliar and it was important to present these as simply as possible. Finally, and most importantly, the theory of quantum information is not yet fully developed. It has not yet reached, in particular, the level of completeness of its classical counterpart. For this reason we can answer only some of the many questions we would like a quantum theory of information to address. Having said this, we can say that however, there are beautiful and useful mathematical results and it seems certain that these will continue to form an important part of the theory as it develops. We noted in the introduction to Chapter 1 that ‘quantum mechanics is a probabilistic theory and so it was inevitable that a quantum information theory would be developed’. A presentation of at least the beginnings of a quantitative theory is the objective of this final chapter. The entropy or information derived from a given probability distribution is, as we have seen, a convenient measure of the uncertainty associated with the distribution. If many of the probabilities are large, so that many of the possible events are comparably likely, then the entropy will be large. If one probability is close to unity, however, then the entropy will be small. It is convenient to introduce entropy in quantum mechanics as a measure of the uncertainty, or lack of knowledge, of the form of the state vector. If we know that our system is in a particular pure state then the associated uncertainty or entropy should be zero. For mixed states, however, it will take a non-zero value.