A demon that remembers: an agential approach towards quantum thermodynamics of temporal correlations

This thesis develops a decision-theoretic framework for extracting thermodynamic work from temporal correlations in quantum systems. We model a classical agent -- lacking quantum memory -- performing adaptive work extraction through continuous inference and decision-making under uncertainty. By introducing $ρ^*$ -ideal protocols, we demonstrate that exploiting memory effects allows adaptive strategies to surpass non-adaptive bounds. We formalize this via the Time-Ordered Free Energy (TOFE), a novel upper bound for causal, adaptive operations that reveals a thermodynamic gap linked to adaptive ordered discord. Additionally, we tackle work extraction from unknown sources using reinforcement learning. By adapting multi-armed bandit algorithms, we show an agent can simultaneously learn an unknown i.i.d. quantum state and extract work, achieving polylogarithmic cumulative dissipation that significantly outperforms standard tomography. Overall, this work lays the foundation for predictive and learning-based quantum thermodynamics.This thesis develops a decision-theoretic framework for extracting thermodynamic work from temporal correlations in quantum systems. We model a classical agent -- lacking quantum memory -- performing adaptive work extraction through continuous inference and decision-making under uncertainty. By introducing $ρ^*$-ideal protocols, we demonstrate that exploiting memory effects allows adaptive strategies to surpass non-adaptive bounds. We formalize this via the Time-Ordered Free Energy (TOFE), a novel upper bound for causal, adaptive operations that reveals a thermodynamic gap linked to adaptive ordered discord. Additionally, we tackle work extraction from unknown sources using reinforcement learning. By adapting multi-armed bandit algorithms, we show an agent can simultaneously learn an unknown i.i.d. quantum state and extract work, achieving polylogarithmic cumulative dissipation that significantly outperforms standard tomography. Overall, this work lays the foundation for predictive and learning-based quantum thermodynamics.

quant-pharXiv:2604.04462Apr 7, 20260 citations

Reinforcement learning for quantum processes with memory

Josep Lumbreras, Ruo Cheng Huang, Yanglin Hu +2 more

In reinforcement learning, an agent interacts sequentially with an environment to maximize a reward, receiving only partial, probabilistic feedback. This creates a fundamental exploration-exploitation trade-off: the agent must explore to learn the hidden dynamics while exploiting this knowledge to maximize its target objective. While extensively studied classically, applying this framework to quantum systems requires dealing with hidden quantum states that evolve via unknown dynamics. We formalize this problem via a framework where the environment maintains a hidden quantum memory evolving via unknown quantum channels, and the agent intervenes sequentially using quantum instruments. For this setting, we adapt an optimistic maximum-likelihood estimation algorithm. We extend the analysis to continuous action spaces, allowing us to model general positive operator-valued measures (POVMs). By controlling the propagation of estimation errors through quantum channels and instruments, we prove that the cumulative regret of our strategy scales as $\widetilde{\mathcal{O}}(\sqrt{K})$ over $K$ episodes. Furthermore, via a reduction to the multi-armed quantum bandit problem, we establish information-theoretic lower bounds demonstrating that this sublinear scaling is strictly optimal up to polylogarithmic factors. As a physical application, we consider state-agnostic work extraction. When extracting free energy from a sequence of non-i.i.d. quantum states correlated by a hidden memory, any lack of knowledge about the source leads to thermodynamic dissipation. In our setting, the mathematical regret exactly quantifies this cumulative dissipation. Using our adaptive algorithm, the agent uses past energy outcomes to improve its extraction protocol on the fly, achieving sublinear cumulative dissipation, and, consequently, an asymptotically zero dissipation rate.In reinforcement learning, an agent interacts sequentially with an environment to maximize a reward, receiving only partial, probabilistic feedback. This creates a fundamental exploration-exploitation trade-off: the agent must explore to learn the hidden dynamics while exploiting this knowledge to maximize its target objective. While extensively studied classically, applying this framework to quantum systems requires dealing with hidden quantum states that evolve via unknown dynamics. We formalize this problem via a framework where the environment maintains a hidden quantum memory evolving via unknown quantum channels, and the agent intervenes sequentially using quantum instruments. For this setting, we adapt an optimistic maximum-likelihood estimation algorithm. We extend the analysis to continuous action spaces, allowing us to model general positive operator-valued measures (POVMs). By controlling the propagation of estimation errors through quantum channels and instruments, we prove that the cumulative regret of our strategy scales as $\widetilde{\mathcal{O}}(\sqrt{K})$ over $K$ episodes. Furthermore, via a reduction to the multi-armed quantum bandit problem, we establish information-theoretic lower bounds demonstrating that this sublinear scaling is strictly optimal up to polylogarithmic factors. As a physical application, we consider state-agnostic work extraction. When extracting free energy from a sequence of non-i.i.d. quantum states correlated by a hidden memory, any lack of knowledge about the source leads to thermodynamic dissipation. In our setting, the mathematical regret exactly quantifies this cumulative dissipation. Using our adaptive algorithm, the agent uses past energy outcomes to improve its extraction protocol on the fly, achieving sublinear cumulative dissipation, and, consequently, an asymptotically zero dissipation rate.

quant-pharXiv:2603.25138Mar 27, 20260 citations

Time-ordered free energy in correlated quantum systems: An agentic approach

Ruo Cheng Huang, Isha Singh Le Xue, Yuxuan Qu +3 more

How much work can an agent extract from a temporal sequence of quantum states when it can only operate online under causal constraints---deciding which energy extraction method to use with knowledge of what it has observed before? Here, we study this problem in the context of quantum state sequences that are potentially non-Markovian---generated by some underlying hidden Markov machine that the agent cannot observe. Using techniques from dynamic programming and computational mechanics, we present a method to identify the provably optimal agent strategy, with time complexity that scales linearly with sequence length. This motivates us to introduce the maximum work such agents can extract---time-ordered free energy(TOFE)---as a fundamental measure of free energy available in a temporally correlated quantum system subject to causal considerations.How much work can an agent extract from a temporal sequence of quantum states when it can only operate online under causal constraints---deciding which energy extraction method to use with knowledge of what it has observed before? Here, we study this problem in the context of quantum state sequences that are potentially non-Markovian---generated by some underlying hidden Markov machine that the agent cannot observe. Using techniques from dynamic programming and computational mechanics, we present a method to identify the provably optimal agent strategy, with time complexity that scales linearly with sequence length. This motivates us to introduce the maximum work such agents can extract---time-ordered free energy(TOFE)---as a fundamental measure of free energy available in a temporally correlated quantum system subject to causal considerations.

quant-pharXiv:2608.12942Aug 14, 20260 citations

Learning to extract work optimally from unknown quantum states with exponentially reduced dissipation

Josep Lumbreras, Ruo Cheng Huang, Yanglin Hu +2 more

Abstract A pure qubit is a source of non-equilibrium free energy, but this energy cannot be fully extracted when the qubit state is unknown. Here we consider an agent tasked with harvesting work from a sequence of $N$ identically prepared pure qubits, without quantum memory to store them for joint processing. The agent must balance two competing objectives: extracting as much work as possible from each qubit given its current knowledge, while learning more about the state to improve extraction in subsequent rounds. We show that this learning cost can be reduced exponentially relative to existing approaches. By leveraging the exploration-exploitation trade-off from reinforcement learning, we construct fully adaptive work-extraction protocols whose cumulative dissipation scales logarithmically in $N$ , compared to the square-root scaling of existing strategies. Our results identify a concrete setting in which the thermodynamic cost of learning an unknown quantum state can be sharply reduced, and reveal a precise connection between reinforcement learning and quantum thermodynamics.Abstract A pure qubit is a source of non-equilibrium free energy, but this energy cannot be fully extracted when the qubit state is unknown. Here we consider an agent tasked with harvesting work from a sequence of $N$ identically prepared pure qubits, without quantum memory to store them for joint processing. The agent must balance two competing objectives: extracting as much work as possible from each qubit given its current knowledge, while learning more about the state to improve extraction in subsequent rounds. We show that this learning cost can be reduced exponentially relative to existing approaches. By leveraging the exploration-exploitation trade-off from reinforcement learning, we construct fully adaptive work-extraction protocols whose cumulative dissipation scales logarithmically in $N$, compared to the square-root scaling of existing strategies. Our results identify a concrete setting in which the thermodynamic cost of learning an unknown quantum state can be sharply reduced, and reveal a precise connection between reinforcement learning and quantum thermodynamics.

May 11, 20260 citations

Bandits roaming Hilbert space

Josep Lumbreras

This thesis studies the exploration and exploitation trade-off in online learning of properties of quantum states using multi-armed bandits. Given streaming access to an unknown quantum state, in each round we select an observable from a set of actions to maximize its expectation value. Using past information, we refine actions to minimize regret; the cumulative gap between current reward and the maximum possible. We derive information-theoretic lower bounds and optimal strategies with matching upper bounds, showing regret typically scales as the square root of rounds. As an application, we reframe quantum state tomography to both learn the state efficiently and minimize measurement disturbance. For pure states and continuous actions, we achieve polylogarithmic regret using a sample-optimal algorithm based on a weighted online least squares estimator. The algorithm relies on the optimistic principle and controls the eigenvalues of the design matrix. We also apply our framework to quantum recommender systems and thermodynamic work extraction from unknown states. In this last setting, our results demonstrate an exponential advantage in work dissipation over tomography-based protocols.This thesis studies the exploration and exploitation trade-off in online learning of properties of quantum states using multi-armed bandits. Given streaming access to an unknown quantum state, in each round we select an observable from a set of actions to maximize its expectation value. Using past information, we refine actions to minimize regret; the cumulative gap between current reward and the maximum possible. We derive information-theoretic lower bounds and optimal strategies with matching upper bounds, showing regret typically scales as the square root of rounds. As an application, we reframe quantum state tomography to both learn the state efficiently and minimize measurement disturbance. For pure states and continuous actions, we achieve polylogarithmic regret using a sample-optimal algorithm based on a weighted online least squares estimator. The algorithm relies on the optimistic principle and controls the eigenvalues of the design matrix. We also apply our framework to quantum recommender systems and thermodynamic work extraction from unknown states. In this last setting, our results demonstrate an exponential advantage in work dissipation over tomography-based protocols.

arXiv:2509.24569Sep 29, 20250 citations

Quantum work extraction from partial information and with finite resources

Giacomo Guarnieri, Diego Maragnano, Giulia Gamba +2 more

Information can be converted into work, but in quantum mechanics information about a state is not freely available: it must be inferred statistically from measurements on a finite number of copies. We study work extraction in this finite-resource setting by introducing a partial-information and finite-resources (PIFR) quantum Maxwell's demon. Given $N$ identical copies of a state with known Hamiltonian, the demon measures $M$ copies to estimate the state and the corresponding ergotropic unitary, which is then applied to the remaining $N-M$ copies. This protocol induces a trade-off between information acquisition, reconstruction accuracy, and thermodynamic yield, making the total extracted work normalized to the ideal ergotropic benchmark the relevant figure of merit. As our central result, we derive a universal closed-form trade-off bound that places this ergotropic efficiency between a Carnot-type ceiling $1-M/N$ and a floor controlled by a reconstruction precision rooted in finite-sample quantum estimation theory; optimizing the copy allocation yields $M^*\propto N^{2/3}$ and an $N^{-1/3}$ approach to the ideal limit, set by a conservative, worst-case reconstruction precision. By considering standard quantum state tomography, we numerically verify the presence of an optimal resource distribution, which also depends on the purity of the state under consideration. Our results identify finite-copy work extraction as a genuinely task-dependent inference problem, in which estimation strategies should be judged by thermodynamic performance rather than reconstruction fidelity alone.Information can be converted into work, but in quantum mechanics information about a state is not freely available: it must be inferred statistically from measurements on a finite number of copies. We study work extraction in this finite-resource setting by introducing a partial-information and finite-resources (PIFR) quantum Maxwell's demon. Given $N$ identical copies of a state with known Hamiltonian, the demon measures $M$ copies to estimate the state and the corresponding ergotropic unitary, which is then applied to the remaining $N-M$ copies. This protocol induces a trade-off between information acquisition, reconstruction accuracy, and thermodynamic yield, making the total extracted work normalized to the ideal ergotropic benchmark the relevant figure of merit. As our central result, we derive a universal closed-form trade-off bound that places this ergotropic efficiency between a Carnot-type ceiling $1-M/N$ and a floor controlled by a reconstruction precision rooted in finite-sample quantum estimation theory; optimizing the copy allocation yields $M^*\propto N^{2/3}$ and an $N^{-1/3}$ approach to the ideal limit, set by a conservative, worst-case reconstruction precision. By considering standard quantum state tomography, we numerically verify the presence of an optimal resource distribution, which also depends on the purity of the state under consideration. Our results identify finite-copy work extraction as a genuinely task-dependent inference problem, in which estimation strategies should be judged by thermodynamic performance rather than reconstruction fidelity alone.

quant-pharXiv:2608.29394Sep 1, 20260 citations

ρ^*

-ideal protocols, we demonstrate that exploiting memory effects allows adaptive strategies to surpass non-adaptive bounds. We formalize this via the Time-Ordered Free Energy (TOFE), a novel upper bound for causal, adaptive operations that reveals a thermodynamic gap linked to adaptive ordered discord. Additionally, we tackle work extraction from unknown sources using reinforcement learning. By adapting multi-armed bandit algorithms, we show an agent can simultaneously learn an unknown i.i.d. quantum state and extract work, achieving polylogarithmic cumulative dissipation that significantly outperforms standard tomography. Overall, this work lays the foundation for predictive and learning-based quantum thermodynamics.

\widetilde{\mathcal{O}}(\sqrt{K})

over

K

episodes. Furthermore, via a reduction to the multi-armed quantum bandit problem, we establish information-theoretic lower bounds demonstrating that this sublinear scaling is strictly optimal up to polylogarithmic factors. As a physical application, we consider state-agnostic work extraction. When extracting free energy from a sequence of non-i.i.d. quantum states correlated by a hidden memory, any lack of knowledge about the source leads to thermodynamic dissipation. In our setting, the mathematical regret exactly quantifies this cumulative dissipation. Using our adaptive algorithm, the agent uses past energy outcomes to improve its extraction protocol on the fly, achieving sublinear cumulative dissipation, and, consequently, an asymptotically zero dissipation rate.

N

identically prepared pure qubits, without quantum memory to store them for joint processing. The agent must balance two competing objectives: extracting as much work as possible from each qubit given its current knowledge, while learning more about the state to improve extraction in subsequent rounds. We show that this learning cost can be reduced exponentially relative to existing approaches. By leveraging the exploration-exploitation trade-off from reinforcement learning, we construct fully adaptive work-extraction protocols whose cumulative dissipation scales logarithmically in

N

, compared to the square-root scaling of existing strategies. Our results identify a concrete setting in which the thermodynamic cost of learning an unknown quantum state can be sharply reduced, and reveal a precise connection between reinforcement learning and quantum thermodynamics.

N

identical copies of a state with known Hamiltonian, the demon measures

M

copies to estimate the state and the corresponding ergotropic unitary, which is then applied to the remaining

N-M

copies. This protocol induces a trade-off between information acquisition, reconstruction accuracy, and thermodynamic yield, making the total extracted work normalized to the ideal ergotropic benchmark the relevant figure of merit. As our central result, we derive a universal closed-form trade-off bound that places this ergotropic efficiency between a Carnot-type ceiling

1-M/N

and a floor controlled by a reconstruction precision rooted in finite-sample quantum estimation theory; optimizing the copy allocation yields

M^*\propto N^{2/3}

and an

N^{-1/3}

approach to the ideal limit, set by a conservative, worst-case reconstruction precision. By considering standard quantum state tomography, we numerically verify the presence of an optimal resource distribution, which also depends on the purity of the state under consideration. Our results identify finite-copy work extraction as a genuinely task-dependent inference problem, in which estimation strategies should be judged by thermodynamic performance rather than reconstruction fidelity alone.