Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

Europhysics Letters (EPL)

Entropy, cross-entropy, relative entropy: Deformation theory (a)

J. Zhang, Hiroshi Matsuzoe

Published Apr 1, 2021• 5 citations
Publisher

Abstract

Abstract Attempts at generalizing Shannon entropy and Kullback-Leibler divergence (relative entropy) led to a plenthora of deformation models in theoretical physics, including q -model, κ -model, etc. Naudts and Zhang ( Inf. Geom. , 1 (2018) 79) established that these models can be unified under two notions: deformed ϕ -exponential family ( Naudts, J. , J. Inequal. Pure Appl. Math. , 5 (2004) 102) and conjugate -embedding ( Zhang J. , Neural Comput. , 16 (2004) 159) of probability functions. Conjugate -embedding has a gauge freedom which, upon its fixing, subsumes the U -model of Eguchi ( Sugaku Expositions , 19 (2006) 197) proposed in a statistical machine learning context. The generalization by -entropy, -cross-entropy, -divergence, when applied to the ϕ -exponential family, yields either a Hessian structure or a conformal Hessian structure under different gauge selections —this “splitting” is the hallmark when deforming the exponential family with its dually flat (Hessian) geometry. This letter provides a unified information geometric perspective of deformation of the exponential model, with calculations for Tsallis q -model.

DOI
10.1209/0295-5075/134/18001

Related papers

The Homological Nature of Entropy

Pierre Baudot, Daniel Bennequin

We propose that entropy is a universal co-homological class in a theory associated to a family of observable quantities and a family of probability distributions. Three cases are presented: (1) classical probabilities and random variables; (2) quantum probabilities and observable operators; (3) dynamic probabilities and observation trees. This gives rise to a new kind of topology for information processes, that accounts for the main information functions: entropy, mutual-informations at all orders, and Kullback–Leibler divergence and generalizes them in several ways. The article is divided into two parts, that can be read independently. In the first part, the introduction, we provide an overview of the results, some open questions, future results and lines of research, and discuss briefly the application to complex data. In the second part we give the complete definitions and proofs of the theorems A, C and E in the introduction, which show why entropy is the first homological invariant of a structure of information in four contexts: static classical or quantum probability, dynamics of classical or quantum strategies of observation of a finite system.

Entropy•May 13, 2015•70 citations

$r$-deformed $α$-$z$-Rényi relative entropy

Srikrishna Maity, Shigeru Furuichi, Supriyo Dutta

In this article, we consider the $r$-logarithm for defining three-parameter family of Rényi relative entropies that are generalization of the $α$-$z$-Rényi relative entropies. All the members of $r$-deformed $α$-$z$-Rényi relative entropies satisfy the necessary axioms to be a divergence. We expose the range of parameters $α$, $z$ and $r$ for which the data processing inequality holds. We also establish that $r$-deformed $α$-$z$-Rényi relative entropy is an upper bound of the Tsallis relative entropy. Now, we have two upper bounds of the Tsallis relative entropy, which are $r$-deformed $α$-$z$-Rényi relative entropy and the other one, which is discussed in literature. We investigate the order relationship between these two upper bounds of the Tsallis relative entropy. We observe that our new upper bound is more tighter when applicable to the density operators.

arXiv:2607.01805•Jul 2, 2026•0 citations

A generalized quantum relative entropy

Luiza Helena Félix de Andrade, Rui F. Vigelis, Charles C. Cavalcante

We propose a generalization of the quantum relative entropy by considering the geodesic on a manifold formed by all the invertible density matrices $ \mathcal{P} $. This geodesic is defined from a deformed exponential function $ \varphi $ which allows to work with a wider class of families of probability distributions. Such choice allows important flexibility in the statistical model. We show and discuss some properties of this proposed generalized quantum relative entropy.

Advances in Mathematics of Communications•Jan 1, 2020•2 citations

Classical and quantum relative entropy

Patrícia Szokol

The concept of entropy rst arose in the study of thermodynamics and statistical me- chanics. In information theory the so-called Shannon entropy plays a signi cant role, which measures how much information we gain, on average, when we learn the value of a random variable. An alternative interpretation is that the entropy of the random variable quanti es the amount of uncertainty about the random variable before we learn its value. The concept was introduced by Claude E. Shannon in his 1948 paper "A Mathematical Theory of Communication". In the rst part of the present work we consider this fundamental measure of information and show some of its properties, such as non-negativity. The relative entropy or the Kullback-Leibler divergence is a very useful measure of the distance between two prob- ability distributions. After introducing its de nition and other notions (joint entropy, conditional entropy or mutual information) we can prove additional assertions about the Shannon entropy and present statements about the relations between the above men- tioned quantities. In quantum information theory, there is a quantity, namely von Neumann entropy, which corresponds to Shannon entropy and originates from it. In the section Von Neu- mann entropy we will introduce this quantity which is a continuous functional on the states, or on their mathematical representatives, the density operators, and it ful ls in- equalities such as subadditivity and strong subaddititvity. Just as with the Shannon entropy, it is extremely useful to de ne a quantum version of the relative entropy, which is one of the most important numerical quantities appearing in quantum information theory. It is used as a measure of distinguishability between quantum states. Among others its properties are jointly convexity, monotonicity which are speci cally convenient. Transformations on structures which preserve operations, quantities or relations among elements appear in many areas of mathematics. These are the so-called preservers which show up also in physics and even in chemistry where they are usually called symmetries. Therefore, the study of preserver transformations is an important area of research. The third section of this work is based on a paper, in which Lajos Molnár described the structure every bijective map on the space of all density operators on a given Hilbert space, which preserves the relative entropy. This result is closely related to Wigner's famous theorem on the form of quantum mechanical symmetry transformations. The above theorem of Molnár can be extended into two directions. On the one hand, the result holds also in the case when the transformation under consideration acts on the space of all positive semide nite operators not only on density operators (i.e., we omit the condition of unit trace). On the other hand the statement remains valid even if we omit the strongly restrictive condition that the transformation is bijective. The second result is a joint work with Prof. Lajos Molnár which appeared in the journal Linear Algebra and Its Applications.

•May 6, 2010•0 citations

Properties of classical and quantum Jensen-Shannon divergence

Jop Briët, Peter Harremöes D

Jensen-Shannon divergence (JD) is a symmetrized and smoothed version of the most important divergence measure of information theory, Kullback divergence. As opposed to Kullback divergence it determines in a very direct way a metric; indeed, it is the square of a metric. We consider a family of divergence measures (${\text{JD}}_{\ensuremath{\alpha}}$ for $\ensuremath{\alpha}>0$), the Jensen divergences of order $\ensuremath{\alpha}$, which generalize JD as ${\text{JD}}_{1}=\text{JD}$. Using a result of Schoenberg, we prove that ${\text{JD}}_{\ensuremath{\alpha}}$ is the square of a metric for $\ensuremath{\alpha}∊(0,2]$, and that the resulting metric space of probability distributions can be isometrically embedded in a real Hilbert space. Quantum Jensen-Shannon divergence (QJD) is a symmetrized and smoothed version of quantum relative entropy and can be extended to a family of quantum Jensen divergences of order $\ensuremath{\alpha}$ $({\text{QJD}}_{\ensuremath{\alpha}})$. We strengthen results by Lamberti and co-workers by proving that for qubits and pure states, ${\text{QJD}}_{\ensuremath{\alpha}}^{1/2}$ is a metric space which can be isometrically embedded in a real Hilbert space when $\ensuremath{\alpha}∊(0,2]$. In analogy with Burbea and Rao's generalization of JD, we also define general QJD by associating a Jensen-type quantity to any weighted family of states. Appropriate interpretations of quantities introduced are discussed and bounds are derived in terms of the total variation and trace distance.

arXiv:0806.4472•Physical Review A•May 12, 2009•207 citations

Geometry from divergence functions and complex structures

Florio M. Ciaglia, Fabio Di Cosmo, Armando Figueroa +2 more

Motivated by the geometrical structures of quantum mechanics, we introduce an almost complex structure [Formula: see text] on the product [Formula: see text] of any parallelizable statistical manifold [Formula: see text]. Then, we use [Formula: see text] to extract a pre-symplectic form and a metric-like tensor on [Formula: see text] from a divergence function. These tensors may be pulled back to [Formula: see text], and we compute them in the case of an N-dimensional symplex with respect to the Kullback–Leibler relative entropy, and in the case of (a suitable unfolding space of) the manifold of faithful density operators with respect to the von Neumann–Umegaki relative entropy.

arXiv:2002.02891•International Journal of Quantum Information•Feb 1, 2020•0 citations
Abstract Attempts at generalizing Shannon entropy and Kullback-Leibler divergence (relative entropy) led to a plenthora of deformation models in theoretical physics, including q -model, κ -model, etc. Naudts and Zhang ( Inf. Geom. , 1 (2018) 79) established that these models can be unified under two notions: deformed ϕ -exponential family ( Naudts, J. , J. Inequal. Pure Appl. Math. , 5 (2004) 102) and conjugate -embedding ( Zhang J. , Neural Comput. , 16 (2004) 159) of probability functions. Conjugate -embedding has a gauge freedom which, upon its fixing, subsumes the U -model of Eguchi ( Sugaku Expositions , 19 (2006) 197) proposed in a statistical machine learning context. The generalization by -entropy, -cross-entropy, -divergence, when applied to the ϕ -exponential family, yields either a Hessian structure or a conformal Hessian structure under different gauge selections —this “splitting” is the hallmark when deforming the exponential family with its dually flat (Hessian) geometry. This letter provides a unified information geometric perspective of deformation of the exponential model, with calculations for Tsallis q -model.
We propose that entropy is a universal co-homological class in a theory associated to a family of observable quantities and a family of probability distributions. Three cases are presented: (1) classical probabilities and random variables; (2) quantum probabilities and observable operators; (3) dynamic probabilities and observation trees. This gives rise to a new kind of topology for information processes, that accounts for the main information functions: entropy, mutual-informations at all orders, and Kullback–Leibler divergence and generalizes them in several ways. The article is divided into two parts, that can be read independently. In the first part, the introduction, we provide an overview of the results, some open questions, future results and lines of research, and discuss briefly the application to complex data. In the second part we give the complete definitions and proofs of the theorems A, C and E in the introduction, which show why entropy is the first homological invariant of a structure of information in four contexts: static classical or quantum probability, dynamics of classical or quantum strategies of observation of a finite system.
rrr-deformed ααα-zzz-Rényi relative entropy
In this article, we consider the rrr-logarithm for defining three-parameter family of Rényi relative entropies that are generalization of the ααα-zzz-Rényi relative entropies. All the members of rrr-deformed ααα-zzz-Rényi relative entropies satisfy the necessary axioms to be a divergence. We expose the range of parameters ααα, zzz and rrr for which the data processing inequality holds. We also establish that rrr-deformed ααα-zzz-Rényi relative entropy is an upper bound of the Tsallis relative entropy. Now, we have two upper bounds of the Tsallis relative entropy, which are rrr-deformed ααα-zzz-Rényi relative entropy and the other one, which is discussed in literature. We investigate the order relationship between these two upper bounds of the Tsallis relative entropy. We observe that our new upper bound is more tighter when applicable to the density operators.
The concept of entropy rst arose in the study of thermodynamics and statistical me- chanics. In information theory the so-called Shannon entropy plays a signi cant role, which measures how much information we gain, on average, when we learn the value of a random variable. An alternative interpretation is that the entropy of the random variable quanti es the amount of uncertainty about the random variable before we learn its value. The concept was introduced by Claude E. Shannon in his 1948 paper "A Mathematical Theory of Communication". In the rst part of the present work we consider this fundamental measure of information and show some of its properties, such as non-negativity. The relative entropy or the Kullback-Leibler divergence is a very useful measure of the distance between two prob- ability distributions. After introducing its de nition and other notions (joint entropy, conditional entropy or mutual information) we can prove additional assertions about the Shannon entropy and present statements about the relations between the above men- tioned quantities. In quantum information theory, there is a quantity, namely von Neumann entropy, which corresponds to Shannon entropy and originates from it. In the section Von Neu- mann entropy we will introduce this quantity which is a continuous functional on the states, or on their mathematical representatives, the density operators, and it ful ls in- equalities such as subadditivity and strong subaddititvity. Just as with the Shannon entropy, it is extremely useful to de ne a quantum version of the relative entropy, which is one of the most important numerical quantities appearing in quantum information theory. It is used as a measure of distinguishability between quantum states. Among others its properties are jointly convexity, monotonicity which are speci cally convenient. Transformations on structures which preserve operations, quantities or relations among elements appear in many areas of mathematics. These are the so-called preservers which show up also in physics and even in chemistry where they are usually called symmetries. Therefore, the study of preserver transformations is an important area of research. The third section of this work is based on a paper, in which Lajos Molnár described the structure every bijective map on the space of all density operators on a given Hilbert space, which preserves the relative entropy. This result is closely related to Wigner's famous theorem on the form of quantum mechanical symmetry transformations. The above theorem of Molnár can be extended into two directions. On the one hand, the result holds also in the case when the transformation under consideration acts on the space of all positive semide nite operators not only on density operators (i.e., we omit the condition of unit trace). On the other hand the statement remains valid even if we omit the strongly restrictive condition that the transformation is bijective. The second result is a joint work with Prof. Lajos Molnár which appeared in the journal Linear Algebra and Its Applications.
Jensen-Shannon divergence (JD) is a symmetrized and smoothed version of the most important divergence measure of information theory, Kullback divergence. As opposed to Kullback divergence it determines in a very direct way a metric; indeed, it is the square of a metric. We consider a family of divergence measures (${\text{JD}}_{\ensuremath{\alpha}}$ for $\ensuremath{\alpha}>0$), the Jensen divergences of order $\ensuremath{\alpha}$, which generalize JD as JD1=JD{\text{JD}}_{1}=\text{JD}JD1​=JD. Using a result of Schoenberg, we prove that ${\text{JD}}_{\ensuremath{\alpha}}$ is the square of a metric for $\ensuremath{\alpha}∊(0,2]$, and that the resulting metric space of probability distributions can be isometrically embedded in a real Hilbert space. Quantum Jensen-Shannon divergence (QJD) is a symmetrized and smoothed version of quantum relative entropy and can be extended to a family of quantum Jensen divergences of order $\ensuremath{\alpha}$ $({\text{QJD}}_{\ensuremath{\alpha}})$. We strengthen results by Lamberti and co-workers by proving that for qubits and pure states, ${\text{QJD}}_{\ensuremath{\alpha}}^{1/2}$ is a metric space which can be isometrically embedded in a real Hilbert space when $\ensuremath{\alpha}∊(0,2]$. In analogy with Burbea and Rao's generalization of JD, we also define general QJD by associating a Jensen-type quantity to any weighted family of states. Appropriate interpretations of quantities introduced are discussed and bounds are derived in terms of the total variation and trace distance.
Motivated by the geometrical structures of quantum mechanics, we introduce an almost complex structure [Formula: see text] on the product [Formula: see text] of any parallelizable statistical manifold [Formula: see text]. Then, we use [Formula: see text] to extract a pre-symplectic form and a metric-like tensor on [Formula: see text] from a divergence function. These tensors may be pulled back to [Formula: see text], and we compute them in the case of an N-dimensional symplex with respect to the Kullback–Leibler relative entropy, and in the case of (a suitable unfolding space of) the manifold of faithful density operators with respect to the von Neumann–Umegaki relative entropy.