Abstract. We propose that entropy is a universal co-homological class in a theory associated to a family of observable quantities and a family of probability distributions. Three cases are presented: 1) classical probabilities and random variables; 2) quantum probabilities and observable operators; 3) dynamic probabilities and observation trees. This gives rise to a new kind of topology for information processes. We discuss briefly its application to complex data, in particular to the structures of information flows in biological systems. This short note summarizes results obtained during the last years by the authors. The proofs are not included, but the definitions and theorems are stated with precision.
Abstract. We propose that entropy is a universal co-homological class in a theory associated to a family of observable quantities and a family of probability distributions. Three cases are presented: 1) classical probabilities and random variables; 2) quantum probabilities and observable operators; 3) dynamic probabilities and observation trees. This gives rise to a new kind of topology for information processes. We discuss briefly its application to complex data, in particular to the structures of information flows in biological systems. This short note summarizes results obtained during the last years by the authors. The proofs are not included, but the definitions and theorems are stated with precision.
We propose that entropy is a universal co-homological class in a theory associated to a family of observable quantities and a family of probability distributions. Three cases are presented: (1) classical probabilities and random variables; (2) quantum probabilities and observable operators; (3) dynamic probabilities and observation trees. This gives rise to a new kind of topology for information processes, that accounts for the main information functions: entropy, mutual-informations at all orders, and Kullback–Leibler divergence and generalizes them in several ways. The article is divided into two parts, that can be read independently. In the first part, the introduction, we provide an overview of the results, some open questions, future results and lines of research, and discuss briefly the application to complex data. In the second part we give the complete definitions and proofs of the theorems A, C and E in the introduction, which show why entropy is the first homological invariant of a structure of information in four contexts: static classical or quantum probability, dynamics of classical or quantum strategies of observation of a finite system.
Baudot and Bennequin introduced a cohomological construction adapted to information theory, called \emph{information cohomology}; it characterizes Shannon entropy as the only nontrivial cohomology class in degree one, for suitable coefficients. This text develops the relation between information cohomology and topos theory. To this end, we redefine information structures, as categories of random variables related by a notion of extension or refinement; classical and quantum probability spaces appear as models (or representations) of them. These generalized information structures form a category with finite products and coproducts. Information cohomology is a derived functor in the abelian category of modules over an algebra of random variables, a subcategory of the presheaf topos associated with the information structure. The cohomology is invariant under isomorphisms of generalized structures. We prove that the relatively free bar construction gives a projective object for the computation of this invariant. We provide detailed computations of H0 and H1 when the coefficients are functionals of classical probabilities, including the degenerate cases. In the process, we also establish the homological nature of Tsallis entropies, as the only cocycles in degree one for certain one-parameter family of modules.
In this manuscript, we study discrete-time dynamics of systems that arise in physics and information theory, and the measure of disorder in these systems known as dy- namical entropy. The study of dynamics in classical systems is done from two distinct viewpoints: random walks and dynamical systems. Random walks are probabilistic in nature and are described by stochastic processes. On the other hand, dynami- cal systems are described algebraically and deterministic in nature. The measure of disorder from either viewpoint is known as dynamical entropy. Entropy is an essential notion in physics and information theory. Motivated by the study of disorder for the positions and velocities of gas molecules, the notion of entropy was first introduced mathematically by Boltzmann near the end of the 19th Century and gives rise to the second law of thermodynamics. Almost eighty years after Boltzmann, Shannon became the father of the new field of information theory when he produced his groundbreaking works where he used entropy as a measure of information transfer between two sources. In the last two years of the 1950’s, Kolmogorov and Sinai extended the notions of Boltzmann to a dynamical entropy. The Kolmogorov-Sinai dynamical entropy gives a measure for the disorder of a system of particles (e.g. gas molecules) averaged over time, quantifying the uncertainty in the dynamics of a system. The advent of quantum mechanics and its pervasiveness in nature has required the development of non-commutative generalizations of dynamics and dynamical entropy to the quantum regime. Many of each have been proposed. In particular, we recall the definitions of quantum random walks, dynamical systems and Markov chains. We motivate each generalization by relating to its classical counterpart. Quantum dynamical entropy (QDE) is a generalization of the Kolmogorov-Sinai dynamical entropy to quantum mechanics. There have been numerous definitions for QDE beginning with that of Connes, Narnhofer and Thirring in 1987. We focus on the semi- classical approach given by Słomczyński and Życzkowski in 1994 and the quantum Markov chain approach which started with Accardi, Ohya and Watanabe in 1997. Linearity of a dynamical entropy means that the dynamical entropy of the n-fold composition of a dynamical map with itself is equal to n times the dynamical entropy of the map for every positive integer n. We show that the quantum dynamical entropy introduced by Słomczyński and Życzkowski is nonlinear in the time interval between successive measurements of a quantum dynamical system. This is in contrast to Kolmogorov-Sinai dynamical entropy for classical dynamical systems, which is linear in time. We also compute the exact values of quantum dynamical entropy for the Hadamard walk with varying Lüders-von Neumann instruments and partitions. In 1948, Shannon proved the Source Coding Theorem which gives upper and lower bounds on the minimal expected codeword length in terms of the entropy of a ran- dom variable. This theorem can be leveraged to give the minimal expected average codeword length for a string of symbols in terms of entropy rate, which can be in- terpreted as the dynamical entropy of a stochastic process. In 1994, Schumacher defined indeterminate-length quantum codes and proved a Quantum Source Coding Theorem. We introduce the notion of stochastic ensembles of pure states and give a novel representation in terms of quantum Markov chains. Moreover, this representa- tion allows us to extend the Quantum Source Coding Theorem, giving the minimal expected average codeword length of an indeterminate-length quantum code in terms of quantum dynamical entropy. This manuscript includes joint work with Dr. George Androulakis (Mathematics, University of South Carolina).
We introduce the category of information structures, whose objects are suitable diagrams of measurable sets that encode the possible outputs of a given family of observables and their mutual relationships of refinement; they serve as mathematical models of contextuality in classical and quantum settings.Each information structure can be regarded as a ringed site with trivial topology; the structure ring is generated by the observables themselves and its multiplication corresponds to joint measurement.We extend Baudot and Bennequin's definition of information cohomology to this setting, as a derived functor in the category of modules over the structure ring, and show explicitly that the bar construction gives a projective resolution in that category, recovering in this way the cochain complexes previously considered in the literature.Finally, we study the particular case of a one-parameter family of coefficients made of functions of probability distributions.The only 1-cocycles are Shannon entropy or Tsallis α-entropy, depending on the value of the parameter.
This paper introduces a novel definition of non-classical information entropy, rooted in the principles of statistical physics, particularly quantum mechanics. The core premise is that traditional information entropy, derived from classical probability theory, is insufficient to capture the essence of information within non-classical systems exhibiting phenomena such as superposition and entanglement. We propose a definition based on the concept of accessible states and their associated probabilities, extending the notion of entropy beyond the realm of purely deterministic systems. The paper details the mathematical formulation of this non-classical information entropy, denoted as *Hnc*, and presents a method for its calculation. The method relies on identifying the set of accessible states for a given system and then applying a generalized form of Shannon's entropy formula, modified to account for the inherent uncertainties associated with quantum mechanics. The resulting *Hnc* provides a quantitative measure of the information content within non-classical systems, offering a potential framework for studying information processing and dynamics in quantum environments. The key contribution lies in bridging the gap between information theory and non-classical statistical physics, offering a new lens through which to understand information in quantum systems.
Entropies are fundamental measures of uncertainty with central importance in information theory and statistics and applications across all the quantitative sciences. Under a natural set of operational axioms, the most general form of entropy is captured by the family of Rényi entropies, parameterized by a real number α. Conditional entropy extends the notion of entropy by quantifying uncertainty from the viewpoint of an observer with access to potentially correlated side information. However, despite their significance and the emergence of various useful definitions, a complete characterization of measures of conditional entropy that satisfy a natural set of operational axioms has remained elusive. In this work, we provide a complete characterization of conditional entropy, defined through a set of axioms that are essential for any operationally meaningful definition: additivity for independent random variables, invariance under relabeling, and monotonicity under bistochastic maps conditioned on the side information. We prove that the most general form of conditional entropy is captured by a family of conditional entropies that are averages of Rényi entropies of the conditioned distribution and parameterized by a probability measure on the positive reals.