Entropy and Information Sources

Berry Schoenmakers, Jilles Tjoelker, Pim Tuyls +1 more

We investigate the recently introduced notion of smooth Renyi entropy for the case of ergodic information sources, thereby generalizing previous work which concentrated mainly on i.i.d. information sources. We will actually consider ergodic quantum information sources, of which ergodic classical information sources are a special case. We prove that the average smooth Renyi entropy rate will approach the entropy rate of a stationary, ergodic source, which is equal to the Shannon entropy rate for a classical source and the von Neumann entropy rate for a quantum source.

David Johnston

Berry Schoenmakers, Jilles Tjoelker, Pim Tuyls +1 more

We prove that the average smooth Renyi entropy rate will approach the entropy rate of a stationary, ergodic information source, which is equal to the Shannon entropy rate for a classical information source and the von Neumann entropy rate for a quantum information source.

Jilles Tjoelker

We investigate therecently introduced notion of smoothRenyi entropy forthecaseofergodic information sources, thereby generalizing previous workwhichconcentrated mainly oni.i.d. information sources. We willactually consider ergodic quantuminformation sources, ofwhichergodic classical infor- mationsources area special case. We provethattheaverage smoothRenyientropy ratewill approach theentropy rateofa stationary, ergodic source, whichisequal totheShannon entropy rateforaclassical source andthevonNeumannentropy rate foraquantumsource. I.INTRODUCTION Theelegant notion ofsmooth Renyi entropy wasintroduced recently byRennerandWolfin(6)forclassical information sources, andthenatural extension toquantuminformation sources wasdefined byRenner andKonigin(5). Inthese two papers andfurther workbyRennerandWolf(7), (4), many properties ofsmooth Renyi entropyandsmooth min-entropy andsmooth max-entropy inparticular havebeenstudied in detail. A central property ofsmooth Renyi entropy proved inthese worksisthat formemoryless (i.i.d.) information sources, the average smooth Renyi entropy ratewill approach theentropy rate ofthesource, whichisequal totheShannon entropy fora classical source andthevonNeumannentropy foraquantum source. Whereas, ingeneral, theaverage (conventional) Renyi entropy rateofamemoryless source doesnotconverge tothe source's entropy rate. Inthis paper weextend thestudy ofsmooth Renyi entropy tothemoregeneral class ofstationary, ergodic sources rather thanmemoryless sources. We willprovethatforboththe classical andthequantum casethat theaverage smooth Renyi entropy ratewill approach theShannon andthevonNeumann entropy rate, respectively. We will dosobyfirst treating the classical caseandthenreducing thequantumcasetothe classical onewithout losing generality. Ingeneral, smooth Renyi entropy oforder a > 1,anda oc(min-entropy) inparticular, isofcryptographic relevance (e.g., forrandomness-extraction), andsmoothRenyientropy oforder a < 1,anda = 0(max-entropy) inparticular, are relevant todatacompression (minimum encoding length). In these contexts, theimportance ofsmooth Renyi entropy isthat its rate isbasically equal totheShannon/von Neumannentropy rate forani.i.d. source (andforergodic sources aswell, aswe showinthis paper). Thisisnotthecaseforconventional Renyi entropy. Moregenerally, asshowninthepapers byRenner et al.mentioned above, smooth Renyientropy behaves muchas Shannon/von Neumannentropy does. Inthis paperwefocus ontheunconditional case, whereas muchoftheabovementioned workbyRenner etal.treats the moregeneral conditional case. We leave theextension tothe conditional caseforfuture work.However, we doconsider twonotions ofe-closeness, onebasedontrace distance (also knownasvariational orstatistical distance) andonebased on non-normalized density matrices (orprobability distributions), wherethelatter ismoresuitable tohandle theconditional case.' Thus, webelieve that ourresults canbeextended tothe conditional caseaswell.

Eduardo Jose Bayro-Corrochano