Nihil est in intellectu quod non prius fuerit in sensu. The Physics of Information This book describes the limits for the communication of information with waves. How many ideas can we communicate by writing on a sheet of paper? How well can we hear a concert? How many details can we distinguish in an image? How much data can we get from our internet connection? These are all questions related to the transport of information by waves. Our sensing ability to capture the differences between distinct waveforms dictates the limits of the amount of information that is delivered by a propagating wave. The problem of quantifying this precisely requires a mathematical description and a physical understanding of both the propagation and the communication processes. We focus on the propagation of electromagnetic waves as described by Maxwell's theory of electromagnetism, and on communication as described by Shannon's theory of information. Although our treatment is mostly based on classical field theory, we also consider limiting regimes where the classical theory must give way to discrete quantum formulations. The old question of whether information is physics or mathematics resounds here. Information is certainly described mathematically, but we argue that it also has a definite physical structure. The central theme of this book is that Shannon's information-theoretic limits are natural. They are revealed by observing physical quantities at certain asymptotic scales where finite dimensionality emerges and observational uncertainties are averaged out. These limits are also rigorous, and obey the mathematical rules that govern the model of reality on which the physical theories are based. Shannon's Laws Originally introduced by Claude Elwood Shannon in 1948, and continuing up to its latest developments in multi-user communication networks, information theory describes in the language of mathematics the limits for the communication of information. These limits are operational, of real engineering significance, and independent of the semantic aspects associated with the communication process.
Nihil est in intellectu quod non prius fuerit in sensu. The Physics of Information This book describes the limits for the communication of information with waves. How many ideas can we communicate by writing on a sheet of paper? How well can we hear a concert? How many details can we distinguish in an image? How much data can we get from our internet connection? These are all questions related to the transport of information by waves. Our sensing ability to capture the differences between distinct waveforms dictates the limits of the amount of information that is delivered by a propagating wave. The problem of quantifying this precisely requires a mathematical description and a physical understanding of both the propagation and the communication processes. We focus on the propagation of electromagnetic waves as described by Maxwell's theory of electromagnetism, and on communication as described by Shannon's theory of information. Although our treatment is mostly based on classical field theory, we also consider limiting regimes where the classical theory must give way to discrete quantum formulations. The old question of whether information is physics or mathematics resounds here. Information is certainly described mathematically, but we argue that it also has a definite physical structure. The central theme of this book is that Shannon's information-theoretic limits are natural. They are revealed by observing physical quantities at certain asymptotic scales where finite dimensionality emerges and observational uncertainties are averaged out. These limits are also rigorous, and obey the mathematical rules that govern the model of reality on which the physical theories are based. Shannon's Laws Originally introduced by Claude Elwood Shannon in 1948, and continuing up to its latest developments in multi-user communication networks, information theory describes in the language of mathematics the limits for the communication of information. These limits are operational, of real engineering significance, and independent of the semantic aspects associated with the communication process.
What is information? A short answer could be: Anything that changes our minds and states of thinking! Information is virtual. It can be carried by speech, an image, scratches on a stone, patterns in a photon field, the connections of neurons in our brains or even by the wavefunction of an electron. Information does not depend on the actual means of transmission nor does its identity change when changing the carrier. Information is physical. It needs to be sustained by a physical substrate and rearranging that substrate to contain the wanted information requires to spend energy or work. Without a physical world, information can not be stored, processed, or transmitted. Our world is permeated with different representations of information. Actually Information about the physical laws governing our Universe can be found everywhere and in everything. Could it then be that the real fundamental elements of this world are tiny bits of information? Could we get it from bit?1 A number of researcher investigate this possibility, and several of their attempts to recover physical laws from the principles governing information are reported about in this special issue on physics of information. The articles by Gregg Jaeger,2 by Ariel Caticha,3 and by John Skilling and Kevin Knuth4 address the reconstruction of quantum mechanics from information theory. Furthermore, Kevin Knuth and James Walsh5 show how to recover space-time from particles sending each others signals in an initially geometry-free setting. A central element of most works on information is the concept of a probability. Probabilities can be defined via various routes. Most familiar might be their definitions by the frequentists school as relative frequencies of events (in the limit of infinite amounts of such). A more general definition of probabilities is the Bayesians' one, which regards probabilities as quantifiers of statements, expressing how much those should be expected to be true. Although this definition looks subjective on the first glance, it rests on a solid mathematical theorem by Richard Cox6 that any extension of binary logic to a continuum of single valued truthfulness quantifier that respects a minimum of requirements must be isomorphic to probability theory. And this definition turns out to embrace the frequentists' definition in case that infinite numbers of comparable events are actually available and can be counted. The majority of authors of this special issues seem to favor the Bayesian perspective, which is probably a result of our own, the guest editors', preference and not necessarily a reflection of the prevailing position of contemporary scientists. The different views on probabilities of frequentists and Bayesians are addressed by the article of Allen Caldwell.7 He builds a bridge between these antagonistic camps by explaining frequentists' constructions in a Bayesian language. In particular, his argumentation why the omnipresent p-value in statistical and experimental literature might often capture a relevant signal of model misfit is reassuring, given that a large fraction of our medical system is based on decisions backed up by p-values. A Bayesian might add that it would be even more reassuring, if a system rigorously build on logic and its continuous extension would be used for decisions affecting live-or-death questions. The laws governing probabilities themselves seem to reflect that of more elementary mathematical structures. Kevin Knuth shows how these emerge from lattice theory, which deals with relations in partially ordered sets like that of collections of possibilities.8 The sum rule of probability theory is a direct consequence of the underlying lattice structure. Interestingly, a number of other theories are resting also on such lattices and therefore exhibit their own versions of sum rules: measures of sets, mutual information, the Euler characteristics of topological spaces, as well as quantum mechanical interactions that are quantified by pairs of numbers, the real and imaginary parts of complex probability amplitudes (see also4). The theory of information has very practical application in physics and elsewhere. Manfred Opper discusses how the solutions of stochastic differential equations can be inferred via variational methods, which have their origin in thermodynamics.9 Torsten Enßlin introduces information field theory (IFT), the information theory for fields, as a means to infer the configuration of a physical field from incomplete and noisy data.10 The mathematical apparatus for this is that of quantum field theory (QFT), just with ordinary probabilities instead of complex probability amplitudes as in the latter. Could it be that the difference between IFT and QFT is just be the same as between classical information theory and quantum mechanics, the transition from real valued to complex valued quantifier for statements? Anyhow, any practical inference machinery, be it for dynamical systems or for fields, requires sophisticated numerical methods. Michael Betancourt reviews Markov Chain Monte Carlo Methods to calculate expectation values over high dimensional probabilities.11 Theo Steininger and coworkers present the NIFTy3 software package for numerical information field theory calculations, which can handle very high dimensional field inference problems.12 Finally, in an article focused on the subtleties of the Monty Hall problem and mostly aiming at information theoretical entertainment, Torsten Enßlin and Margret Westercamp investigate the possibility that human intuition might be more rational than contemporary psychology thinks.13 A number of possible topics for an issue on physics of information are not addressed here. We would have loved to have articles on Landauer's principle, which states that erasing information has to cost energy, on QBism, a Bayesian version of quantum mechanics, on artificial and biological neural networks, as well as any other aspect of the information-soaked biological world. What has a reader of this special issue to be equipped with in order to follow the lines of thoughts? In addition to the usual physicist's education, the minimal additional equipment are the basics of probability theory and the concept of entropy. Most of the other concepts used in the articles rest on those and are introduced there. Let A, B, and C be propositions (or possibilities). The probability P ( A | B ) of A being true assuming that B is true is a number between 0 and 1, with P ( A | B ) = 1 expressing certainty of A under B ( B ⇒ A ) and P ( A | B ) = 0 expressing certainty of not A being the case under B ( B ⇒ ¬ A ).
The main view promoted by this book is that underlying many different aspects of reality is some form of information processing. The theory of information started rather innocently, as the result of a very specific question that Shannon considered, which was how to maximize the capacity of communication between two users. Shannon showed that all we need is to associate a probability to an event, and defined a metric that allowed you to quantify the information content of that event. Interestingly, because of its simplicity and intuitiveness, Shannon’s views have been successfully applied to many other problems. We can view biological information through Shannon’s theory as a communication in time (where the objective of natural selection is to propagate the gene pool into the future). But it is not only that communications and biology are trying to optimize information. In physics, systems arrange themselves so that entropy is maximized, and this entropy is quantified in the same way as Shannon’s information. We encounter the same form of information in other phenomena. Financial speculation is also governed by the same concept of entropy, and optimizing your profit is the same problem as optimizing your channel capacity. In social theory, society is governed by its interconnectedness or correlation and this correlation itself is quantified by Shannon’s entropy. Underlying all these phenomena was the classical Boolean logic where events had clear outcomes, either yes or no, on or off, and so on. In our most accurate description of reality, given by quantum theory, we know that bits of information are an approximation to a much more precise concept of qubits. Qubits, unlike bits, can exist in a multitude of states, any combination of yes and no, on and off. Shannon’s information theory has been extended to account for quantum theory and the resulting framework, quantum information theory, has already shown a number of advantages. The greater power of quantum information theory is manifested in more secure cryptographic protocols, a completely new order of computing, quantum teleportation, and a number of other applications that were simply not possible according to Shannon’s view.
We cannot overstate the importance of Shannon's contribution to modern science. His introduction of the field of information theory and his solutions to its two main theorems demonstrate that his ideas on communication were far beyond the other prevailing ideas in this domain around 1948. In this chapter, our aim is to discuss Shannon's two main contributions in a descriptive fashion. The goal of this high-level discussion is to build up the intuition for the problem domain of information theory and to understand the main concepts before we delve into the analogous quantum information-theoretic ideas. We avoid going into deep technical detail in this chapter, leaving such details for later chapters where we formally prove both classical and quantum Shannon-theoretic coding theorems. We do use some mathematics from probability theory, namely, the law of large numbers. We will be delving into the technical details of this chapter's material in later chapters (specifically, Chapters 10, 12, and 13). Once you have reached later chapters that develop some more technical details, it might be helpful to turn back to this chapter to get an overall flavor for the motivation of the development. Data Compression We first discuss the problem of data compression. Those who are familiar with the Internet have used several popular data formats such as JPEG, MPEG, ZIP, GIF, etc. All of these file formats have corresponding algorithms for compressing the output of an information source.
We cannot overstate the importance of Shannon's contribution to modern science. His introduction of the field of information theory and his solutions to its two main theorems demonstrate that his ideas on communication were far beyond the other prevailing ideas in this domain around 1948. In this chapter, our aim is to discuss Shannon's two main contributions in a descriptive fashion. The goal of this high-level discussion is to build up the intuition for the problem domain of information theory and to understand the main concepts before we delve into the analogous quantum information-theoretic ideas. We avoid going into deep technical detail in this chapter, leaving such details for later chapters where we formally prove both classical and quantum Shannontheoretic coding theorems. We do use some mathematics from probability theory (namely, the law of large numbers). We will be delving into the technical details of this chapter's material in later chapters (specifically, Chapters 10, 13, and 14). Once you have reached later chapters that develop some more technical details, it might be helpful to turn back to this chapter to get an overall flavor for the motivation of the development. Data Compression We first discuss the problem of data compression. Those who are familiar with the Internet have used several popular data formats such as JPEG, MPEG, ZIP, GIF, etc. All of these file formats have corresponding algorithms for compressing the output of an information source. A first glance at the compression problem might lead one to believe that it is not possible to compress the output of the information source to an arbitrarily small size, and Shannon proved that this is the case. This result is the content of Shannon's first noiseless coding theorem. An Example of Data Compression We begin with a simple example that illustrates the concept of an information source. We then develop a scheme for coding this source so that it requires fewer bits to represent its output faithfully. Suppose that Alice is a sender and Bob is a receiver. Suppose further that a noiseless bit channel connects Alice to Bob—a noiseless bit channel is one that transmits information perfectly from sender to receiver, e.g., Bob receives “0” if Alice transmits “0” and Bob receives “1” if Alice transmits “1.”
In these first few chapters, our aim is to establish a firm grounding so that we can address some fundamental questions regarding information transmission over quantum channels. This area of study has become known as “quantum Shannon theory” in the broader quantum information community, in order to distinguish this topic from other areas of study in quantum information science. In this text, we will use the terms “quantum Shannon theory” and “quantum information theory” somewhat interchangeably. We will begin by briefly overviewing several fundamental aspects of the quantum theory. Our study of the quantum theory, in this chapter and future ones, will be at an abstract level, without giving preference to any particular physical system such as a spin- 1/2 particle or a photon. This approach will be more beneficial for the purposes of our study, but, here and there, we will make some reference to actual physical systems to ground us in reality. You may be wondering, what is quantum Shannon theory and why do we name this area of study as such? In short, quantum Shannon theory is the study of the ultimate capability of noisy physical systems, governed by the laws of quantum mechanics, to preserve information and correlations. Quantum information theorists have chosen the name quantum Shannon theory to honor Claude Shannon, who single-handedly founded the field of classical information theory with a groundbreaking paper (Shannon, 1948). In particular, the name refers to the asymptotic theory of quantum information, which is the main topic of study in this book. Information theorists since Shannon have dubbed him the “Einstein of the information age.”1 The name quantum Shannon theory is fit to capture this area of study because we often use quantum versions of Shannon's ideas to prove some of the main theorems in quantum Shannon theory. We prefer the name “quantum Shannon theory” over such names as “quantum information science” or just “quantum information.” These other names are too broad, encompassing subjects as diverse as quantum computation, quantum algorithms, quantum complexity theory, quantum communication complexity, entanglement theory, quantum key distribution, quantum error correction, and even the experimental implementation of quantum protocols.
We may be on the brink of a quantum transfor -mation driven by our increasing ability to realize, detect, control, and manipulate high-purity quantum states. For decades the quantum world has crept into our lives through modern information technology, the most visible side of science that permeates almost every aspect of our daily conduct. Yet quantumness in its full glory remains a promise for the future, as existing mainstream technologies are essentially based on properties of mixed quantum states realized in, e.g., metal–oxide–semiconductor heterostructures. The scientific backbone of current technologies combines quantum theory with information theory and the theory of computation, everything being sprinkled with a healthy dose of mathematics and brought into reality by remarkable engineering achievements. This multidisciplinary approach has enabled us to understand key physical phenomena and material properties, to communicate and process information efficiently, and, ultimately, to build functional devices that provide us with unprecedented convenience and mastery over nature, possibly with the price of being in danger to become the slaves of our computers and gadgets. However, our technological success is built on a combination of essentially distinct and largely independent scientific fields. Harnessing high-purity quantum states, on the other hand, sets these previously separated “continents of knowledge” on a collision path. The main goal of the last three chapters of this book is to provide a bird's eye view of the “collision zone” and prepare the reader to explore, enjoy, and perhaps exploit the newly raising mountains. We start with a brief introduction to basic concepts in quantum information, which also includes a summary of the operational formulation of quantum mechanics (the “natural” language of quantum information theory) and a synopsis of classical information theory ( Chapter 10 ). For a thorough treatment of concepts, problems, and techniques in quantum information theory the reader is referred to the books by Nielsen and Chuang [ 389 ] and Wilde [ 548 ]. The book by Watrous [ 530 ] focuses on the mathematical theory of quantum information. An undergraduate level introduction to this subject can be found, for example, in Schumacher and Westmoreland [ 458 ]. Some of the philosophical implications of the relationship between quantum mechanics and information theory are discussed in Timpson [ 505 ]. Our survey continues with Chapter 11 , where we discuss a few key aspects pertaining to quantum computation, which can be viewed as the quintessential quantum technology involving the controlled realization and manipulation of high-purity quantum states within a large Hilbert space. Finally, Chapter 12 focuses on topological quantum computation, a promising approach to overcoming the fundamental challenge associated with quantum decoherence based on harnessing the underlying robustness of topological quantum states. This also marks our final return to the realm of topological quantum matter.