We consider a general class of models, where a reinforcement learning (RL) agent learns from cyclic interactions with an external environment via classical signals. Perceptual inputs are encoded as quantum states, which are subsequently transformed by a quantum channel representing the agent's memory, while the outcomes of measurements performed at the channel's output determine the agent's actions. The learning takes place via stepwise modifications of the channel properties. They are described by an update rule that is inspired by the projective simulation (PS) model and equipped with a glow mechanism that allows for a backpropagation of policy changes, analogous to the eligibility traces in RL and edge glow in PS. In this way, the model combines features of PS with the ability for generalization, offered by its physical embodiment as a quantum system. We apply the agent to various setups of an invasion game and a grid world, which serve as elementary model tasks allowing a direct comparison with a basic classical PS agent.
We consider a general class of models, where a reinforcement learning (RL) agent learns from cyclic interactions with an external environment via classical signals. Perceptual inputs are encoded as quantum states, which are subsequently transformed by a quantum channel representing the agent's memory, while the outcomes of measurements performed at the channel's output determine the agent's actions. The learning takes place via stepwise modifications of the channel properties. They are described by an update rule that is inspired by the projective simulation (PS) model and equipped with a glow mechanism that allows for a backpropagation of policy changes, analogous to the eligibility traces in RL and edge glow in PS. In this way, the model combines features of PS with the ability for generalization, offered by its physical embodiment as a quantum system. We apply the agent to various setups of an invasion game and a grid world, which serve as elementary model tasks allowing a direct comparison with a basic classical PS agent.
In recent years, the interest in leveraging quantum effects for enhancing machine learning tasks has significantly increased. Many algorithms speeding up supervised and unsupervised learning were established. The first framework in which ways to exploit quantum resources specifically for the broader context of reinforcement learning were found is projective simulation. Projective simulation presents an agent-based reinforcement learning approach designed in a manner which may support quantum walk-based speedups. Although classical variants of projective simulation have been benchmarked against common reinforcement learning algorithms, very few formal theoretical analyses have been provided for its performance in standard learning scenarios. In this paper, we provide a detailed formal discussion of the properties of this model. Specifically, we prove that one version of the projective simulation model, understood as a reinforcement learning approach, converges to optimal behavior in a large class of Markov decision processes. This proof shows that a physically inspired approach to reinforcement learning can guarantee to converge.
Quantum Reinforcement Learning (QRL) is an emerging field with many novel approaches suitable for noisy intermediate-scale quantum (NISQ) devices being proposed recently. Projective Simulation (PS) is a classical machine learning method based on random walks in a Directed Acyclic Graph (DAG), and is mainly used to train agents in a Reinforcement Learning setup. This graph corresponds to the Episodic Compositional Memory (ECM) of the agent. For the quantum version of PS we use quantum walks of single photons in quantum optical circuits, which go through tunable components (beamsplitters and phase shifters). We call this specific model Quantum Optical Projective Simulation (QOPS). In this work, we propose a general framework to translate any 2-layer ECM to a quantum optical circuit and an algorithm for solving quantum strategic games using QOPS. Moreover, we introduce QOPS with an entangled state as input. The application that we are using to evaluate the performance of the framework is the Prisoner's Dilemma. We executed the implementation using Altair's noisy simulator, designed to emulate a single photon-based quantum processor by Quandela. The algorithm provided promising results even in the presence of noise, which makes it a strong candidate for runs on single-photon-based quantum processors.
Reinforcement learning is a machine learning paradigm where an agent learns how to optimize its behavior solely through its interaction with the environment. It has been extensively studied and successfully applied to complex problems of many different domains in the past decades, i.e., robotics, games, scheduling. However, the performance of these algorithms becomes limited as the complexity and dimension of the state-action space increases. Recent advances in quantum computing and quantum information have sparked interest in possible applications to machine learning. By taking advantage of quantum mechanics, it is possible to efficiently process immense quantities of information and improve computational speed. In this work, we combined quantum computing with reinforcement learning and studied its application to a board game to assess the benefits that it can introduce, namely its impact on the learning efficiency of an agent. From the results, we concluded that the proposed quantum exploration policy improved the convergence rate of the agent and promoted a more efficient exploration of the state space.
We propose a novel and interdisciplinary conceptual framework that bridges Feynman’s path integral formulation of quantum mechanics with reinforcement learning (RL) in artificial intelligence. In the path integral approach, a quantum system does not follow a single predetermined trajectory but instead explores all possible paths simultaneously, assigning each a complex amplitude weighted by eiS/h, where S represents the classical action. Constructive and destructive interference across these paths acts as a natural filter, amplifying trajectories of stationary action and suppressing suboptimal ones, thereby leaving paths of least action as the observable outcomes. We argue this process is strongly analogous to a quantum agent evaluating an entire policy space in superposition, where interference effectively encodes a reward mechanism that eliminates non-optimal policies. This perspective not only deepens our understanding of light’s propagation in complex refractive media but also inspires the design of quantum-inspired Reinforcement learning architectures capable of leveraging the intrinsic parallelism of quantum mechanics. Furthermore, the advent of quantum computing, with its inherent properties of superposition, entanglement, and quantum interference, provides a tangible pathway for implementing such algorithms in practice. To illustrate this paradigm, we propose a toy model wherein policies are encoded as quantum states, rewards are mapped to phase shifts, and measurement collapses the superposed state into an optimal policy. The implications of this framework extend beyond algorithmic innovation, offering insights into the possibility that nature itself operates as a quantum learning system, with physical laws emerging from a process akin to reinforcement learning.
In reinforcement learning, an agent interacts sequentially with an environment to maximize a reward, receiving only partial, probabilistic feedback. This creates a fundamental exploration-exploitation trade-off: the agent must explore to learn the hidden dynamics while exploiting this knowledge to maximize its target objective. While extensively studied classically, applying this framework to quantum systems requires dealing with hidden quantum states that evolve via unknown dynamics. We formalize this problem via a framework where the environment maintains a hidden quantum memory evolving via unknown quantum channels, and the agent intervenes sequentially using quantum instruments. For this setting, we adapt an optimistic maximum-likelihood estimation algorithm. We extend the analysis to continuous action spaces, allowing us to model general positive operator-valued measures (POVMs). By controlling the propagation of estimation errors through quantum channels and instruments, we prove that the cumulative regret of our strategy scales as O(K) over K episodes. Furthermore, via a reduction to the multi-armed quantum bandit problem, we establish information-theoretic lower bounds demonstrating that this sublinear scaling is strictly optimal up to polylogarithmic factors. As a physical application, we consider state-agnostic work extraction. When extracting free energy from a sequence of non-i.i.d. quantum states correlated by a hidden memory, any lack of knowledge about the source leads to thermodynamic dissipation. In our setting, the mathematical regret exactly quantifies this cumulative dissipation. Using our adaptive algorithm, the agent uses past energy outcomes to improve its extraction protocol on the fly, achieving sublinear cumulative dissipation, and, consequently, an asymptotically zero dissipation rate.
This paper investigates the application of quantum-inspired algorithms to the field of reinforcement learning (RL). Traditional RL methods often struggle with exploration challenges and can be computationally expensive, particularly in complex environments. We propose a novel RL algorithm that leverages concepts from quantum mechanics, specifically superposition and entanglement, to address these limitations. The core idea is to represent states, actions, and rewards using quantum-inspired representations, allowing the agent to explore the environment more efficiently and potentially converge to optimal policies faster. This work presents a detailed formulation of the algorithm, including the mathematical framework and key considerations for implementation. The potential benefits of this approach, including improved exploration and reduced computational complexity, are discussed. The algorithm is designed to iteratively refine its policy based on simulated quantum measurements, aiming to minimize the expected return. The exploration strategy is guided by the inherent probabilistic nature of quantum states, leading to a more diverse and potentially faster learning process. Further research is warranted to fully explore the capabilities and limitations of this approach across various RL benchmark problems.