The science of information theory begins with the observation that there is a fundamental link between probabilities and information. As early as the mid eighteenth century, Bayes recognized that probabilities depend on what we know; if we acquire additional information then this modifies the probability. For example, the probability that it is raining when I leave for work in the morning is about 0.2, but if I look out of the window ten minutes before leaving and see that it is raining then this additional information adjusts the probability to in excess of 0.9. Information is a function of probabilities: it is the entropy associated with the probability distribution. This conclusion grew out of investigations into the physical nature of entropy by Boltzmann and his followers. The full power of entropy as the quantity of information was revealed by Shannon in his mathematical theory of communication. This work laid the foundations for the development of information and communications theory by proving two powerful theorems which limit our ability to communicate information. The link between probability and information has far wider application than just communications. Indeed, we can expect the ideas of information theory to be applicable to any statistical or probabilistic problem. Quantum mechanics is a probabilistic theory and so it was inevitable that a quantum information theory would be developed. In quantum theory, probabilities are secondary quantities calculated by taking the squared modulus of probability amplitudes and this gives rise to interference effects. Consider, for example, the famous two-slit experiment depicted in Fig. 1.1. A single particle launched at the slits can arrive at a point P on the screen by passing either through slit 1 or through slit 2. In classical statistical mechanics this leads to a probability . . . P = P1 + P2, (1.1) . . . where P1 and P2 are, respectively, the probabilities that the particle passed through slit 1 or slit 2 and went on to arrive at the point P.
The science of information theory begins with the observation that there is a fundamental link between probabilities and information. As early as the mid eighteenth century, Bayes recognized that probabilities depend on what we know; if we acquire additional information then this modifies the probability. For example, the probability that it is raining when I leave for work in the morning is about 0.2, but if I look out of the window ten minutes before leaving and see that it is raining then this additional information adjusts the probability to in excess of 0.9. Information is a function of probabilities: it is the entropy associated with the probability distribution. This conclusion grew out of investigations into the physical nature of entropy by Boltzmann and his followers. The full power of entropy as the quantity of information was revealed by Shannon in his mathematical theory of communication. This work laid the foundations for the development of information and communications theory by proving two powerful theorems which limit our ability to communicate information. The link between probability and information has far wider application than just communications. Indeed, we can expect the ideas of information theory to be applicable to any statistical or probabilistic problem. Quantum mechanics is a probabilistic theory and so it was inevitable that a quantum information theory would be developed. In quantum theory, probabilities are secondary quantities calculated by taking the squared modulus of probability amplitudes and this gives rise to interference effects. Consider, for example, the famous two-slit experiment depicted in Fig. 1.1. A single particle launched at the slits can arrive at a point P on the screen by passing either through slit 1 or through slit 2. In classical statistical mechanics this leads to a probability . . . P = P1 + P2, (1.1) . . . where P1 and P2 are, respectively, the probabilities that the particle passed through slit 1 or slit 2 and went on to arrive at the point P.
The astute reader might have formed the impression that quantum in formation science is a rather qualitative discipline because we have not, as yet, explained how to quantify quantum information. There are three good reasons for leaving this important question until the final chapter. Firstly, quantum information theory is technically demanding and to treat it at an earlier stage might have suggested that our subject was more complicated than it is. Secondly, there is the fact that many of the ideas in the field, such as teleportation and quantum circuits, are unfamiliar and it was important to present these as simply as possible. Finally, and most importantly, the theory of quantum information is not yet fully developed. It has not yet reached, in particular, the level of completeness of its classical counterpart. For this reason we can answer only some of the many questions we would like a quantum theory of information to address. Having said this, we can say that however, there are beautiful and useful mathematical results and it seems certain that these will continue to form an important part of the theory as it develops. We noted in the introduction to Chapter 1 that ‘quantum mechanics is a probabilistic theory and so it was inevitable that a quantum information theory would be developed’. A presentation of at least the beginnings of a quantitative theory is the objective of this final chapter. The entropy or information derived from a given probability distribution is, as we have seen, a convenient measure of the uncertainty associated with the distribution. If many of the probabilities are large, so that many of the possible events are comparably likely, then the entropy will be large. If one probability is close to unity, however, then the entropy will be small. It is convenient to introduce entropy in quantum mechanics as a measure of the uncertainty, or lack of knowledge, of the form of the state vector. If we know that our system is in a particular pure state then the associated uncertainty or entropy should be zero. For mixed states, however, it will take a non-zero value.
Abstract. At this point in time, two major areas of physics, statistical mechanics and quantum mechanics, rest on the foundations of probability and entropy. The last century saw several significant fundamental advances in our understanding of the process of inference, which make it clear that these are inferential theories. That is, rather than being a description of the behavior of the universe, these theories describe how observers can make optimal predictions about the universe. In such a picture, information plays a critical role. What is more is that little clues, such as the fact that black holes have entropy, continue to suggest that information is fundamental to physics in general. In the last decade, our fundamental understanding of probability theory has led to a Bayesian revolution. In addition, we have come to recognize that the foundations go far deeper and that Cox’s approach of generalizing a Boolean algebra to a probability calculus is the first specific example of the more fundamental idea of assigning valuations to partially-ordered sets. By considering this as a natural way to introduce quantification to the more fundamental notion of ordering, one obtains an entirely new way of deriving physical laws. I will introduce this new way of thinking by demonstrating how one can quantify partially-ordered sets and, in the process, derive physical laws. The implication is that physical law does not reflect the order in the universe, instead it is derived from the order imposed by our description of the universe. Information physics, which is based on understanding the ways in which we both quantify and process information about the world around us, is a fundamentally new approach to science.
At this point in time, two major areas of physics, statistical mechanics and quantum mechanics, rest on the foundations of probability and entropy. The last century saw several significant fundamental advances in our understanding of the process of inference, which make it clear that these are inferential theories. That is, rather than being a description of the behavior of the universe, these theories describe how observers can make optimal predictions about the universe. In such a picture, information plays a critical role. What is more is that little clues, such as the fact that black holes have entropy, continue to suggest that information is fundamental to physics in general. In the last decade, our fundamental understanding of probability theory has led to a Bayesian revolution. In addition, we have come to recognize that the foundations go far deeper and that Cox's approach of generalizing a Boolean algebra to a probability calculus is the first specific example of the more fundamental idea of assigning valuations to partially-ordered sets. By considering this as a natural way to introduce quantification to the more fundamental notion of ordering, one obtains an entirely new way of deriving physical laws. I will introduce this new way of thinking by demonstrating how one can quantify partially-ordered sets and, in the process, derive physical laws. The implication is that physical law does not reflect the order in the universe, instead it is derived from the order imposed by our description of the universe. Information physics, which is based on understanding the ways in which we both quantify and process information about the world around us, is a fundamentally new approach to science.
We have seen that there is an intimate relationship between probability and information. The values we assign to probabilities depend on the information available, and information is a function of probabilities. This connection makes it inevitable that information will be an important concept in any statistical theory, including thermodynamics and, of course, quantum physics. The probabilistic interpretation of quantum theory has probability amplitudes rather than probabilities as the fundamental quantities. This feature, together with the associated superposition principle, is responsible for intrinsically quantum phenomena and gives quantum information theory its distinctive flavour. We shall see that the quantum rules for dynamical evolution and measurement, together with the existence of entangled states, have important implications for quantum information. They also make it possible to perform tasks which are either impractical or impossible within the classical domain. In describing these we shall make extensive use of simple but fundamental ideas in quantum theory. This chapter introduces the mathematical description of quantum physics and the concepts which will be employed in our study of quantum information.
Given a coin, one has two complementary pieces of information and a corresponding probability of .5 and .5. Similar considerations apply to a die. As a result, one may get the impression that there exists a unique probability distribution in a probabilistic system and that this single probability scheme is linked to the constrained maximization of Shannon’s entropy or Fisher’s information. We argue however, that there may be different probability schemes in the same problem associated with different pieces of information which are relevant to the physics which occurs in the system. Thus, we argue that it is the physics which selects the probability distribution to be examined. For example, one may loosely think in terms of a detailed probability scheme versus a kind-of coarse grained one in the same system. For example, for a quantum free particle, P(x)=1 i.e there is the same probability for a particle to be at any x point. This, however, is based on velocity, i.e. the relationship between dx and dt. There is another piece of information in the system, namely mo which together with v yields momentum (either relativistic or nonrelativistic) .This need not be measured using a clock (which velocity requires), but by the impact of an impulse on a target. Thus, nonrelativistically p=m1v1=m2v2 give the same impulse, but have different velocities. Quantum mechanics shows that a free particle is governed by exp(ipx) which contains two unnormalized probability distributions cos(px) and sin(px). These are key for impulse interactions (i.e. the physics of interactions such as with a 2-slit apparatus etc), yet exp(ipx)/sqrt(L) maps into the “coarse-grained” P(x)=exp(ipx)exp(-ipx) / L = constant. P(x) is one probability scheme in the problem and exp(ipx) another (for a free particle). Each answers some physical questions, but one needs to know which probability to use (i.e. what information is relevant) to a particular situation. P(x) is relevant to motion in space not involving interactions. exp(ipx) is relevant to interactions (even with a magnetic vector potential causing no magnetic field) or with a 2-slit apparatus. Furthermore, in a quantum bound state, one has W*(x)W(x) = P(x) (W(x)=wavefunction =Sum over p a(p)exp(ipx)) and a*(p)a(p).=P(p). Each probability scheme corresponds to the analysis of a different kind of measurement, but various schemes may also be linked. Thus one may have to start with one to create the others. For example, exp(ipx) is the base for creating W(x)=Sum over p a(p)exp(ipx), which ultimately yields P(x)=W*(x)W(x). As a result, we argue that if probability exists in a system, it is not necessarily the case that there is a unique probability distribution. There may be various probability schemes linked with different information and different measurements/interactions. Thus, physics leads one to focus on a specific probability scheme. As a result, we argue that one must be careful when discussing pure statistical quantities and their extremization, because these quantities, like Shannon’s entropy and Fisher’ s information, contain a classical probability distribution, but they do not indicate which distribution to use in the case that a system has more than one. . As a result, the analysis of a system requires knowledge of the physics (interactions) occurring.We argue against thinking in terms of physics following directly from statistical considerations. The constrained extremization of one statistical quantity may yield a probability distribution related to one piece of information, but not to another. Furthermore, Shannon’s entropy and Fisher’s information pertain to classical probability 0<=P(x)<1 and Integral P(x)dx =1. The wavefunction, exp(ipx)/sqrt(L), is comprised of two probability distributions and ADD and MULTIPLIES statistically (i.e. OR and AND ) situations, but is involved in relative average calculations because it is not normalized. It is also different because it allows for positive and negative values. These are properties which do not follow from statistics using a classical P(x).
Information theory, introduced by Claude Shannon in 1948, provides a mathematical model, predominantly probabilistic, for addressing the challenges associated with measuring the amount of information in a message and efficiently and reliably storing the message (compression and protection). This theory has always been intricately connected to communication theory. Information theory not only relies on communication theory but also draws from fields such as statistics, statistical physics, and inference theory. In this chapter, a concise overview is presented, covering the mathematical foundations of classical and quantum information, their manipulation and associated qualifications. The fundamental concepts of classical information theory, including Shannon entropy, joint entropy, conditional entropy, and mutual information, are introduced. Additionally, basic concepts related to quantum information and quantum correlation measurements are presented.