Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

Synthese Library/Synthese library

Projective Simulation

Hans J. Briegel, Thomas Müller

Published Jan 1, 2025• 0 citations
Publisher

Abstract

Abstract Projective Simulation (PS) is a model for agency that incorporates aspects of reinforcement learning, an indeterministic basic dynamics inspired by physical hopping processes as studied in quantum optics, and an overall orientation towards agency as a continuous process of interaction with the environment as described by phenomenology. We explain the formal structure of PS in detail and compare it to the more standard paradigm of reinforcement learning. PS stands out mostly due to its specific memory structure (Episodic and Compositional Memory, ECM) and its openness, which makes it apt as an agency model rather than just a learning model: PS does not start from a fixed environment model to be “solved”. The focus is, rather, on the internal dynamics of deliberation, which is resolved as a stochastic process within the ECM, as well as on learning and adaptation over a long time, allowing for significant and even disruptive changes in the environment to be incorporated. After discussing the model, we showcase several applications of PS, prominently in basic research in quantum mechanics and especially in quantum information. This strengthens our claim about the intimate connection between agency and quantum physics that we already defended in Chaps. 3 and 4 . We also consider quantum versions of PS, which promise to show quantum advantages in machine learning while retaining a maximum of transparency. These applications motivate a separate discussion of transparency and traceability in the context of explainable (quantum) AI.

DOI
10.1007/978-3-031-98119-7_5

Related papers

On the convergence of projective-simulation–based reinforcement learning in Markov decision processes

W. L. Boyajian, J. Clausen, L. M. Trenkwalder +2 more

In recent years, the interest in leveraging quantum effects for enhancing machine learning tasks has significantly increased. Many algorithms speeding up supervised and unsupervised learning were established. The first framework in which ways to exploit quantum resources specifically for the broader context of reinforcement learning were found is projective simulation. Projective simulation presents an agent-based reinforcement learning approach designed in a manner which may support quantum walk-based speedups. Although classical variants of projective simulation have been benchmarked against common reinforcement learning algorithms, very few formal theoretical analyses have been provided for its performance in standard learning scenarios. In this paper, we provide a detailed formal discussion of the properties of this model. Specifically, we prove that one version of the projective simulation model, understood as a reinforcement learning approach, converges to optimal behavior in a large class of Markov decision processes. This proof shows that a physically inspired approach to reinforcement learning can guarantee to converge.

arXiv:1910.11914•Quantum Machine Intelligence•Nov 5, 2020•18 citations

Benchmarking Projective Simulation in Navigation Problems

Alexey A. Melnikov, Adi Makmal, Hans J. Briegel

Projective simulation is a model for intelligent agents with a deliberation capacity that is based on episodic memory. The model has been shown to provide a flexible framework for constructing reinforcement-learning agents, and it allows for quantum mechanical generalization, which leads to a speedup in deliberation time. Projective simulation agents have been applied successfully in the context of complex skill learning in robotics and the design of the state-of-the-art quantum experiments. In this paper, we study the performance of projective simulation in two benchmarking problems in navigation, namely, the grid world and the mountain car problem. The performance of projective simulation is compared to standard tabular reinforcement learning approaches, Q-learning, and SARSA. Our comparison demonstrates that the performance of projective simulation and standard learning approaches are qualitatively and quantitatively similar, while it is much easier to choose optimal model parameters in the case of projective simulation with reduced computational effort of one to two orders of magnitude. Our results show that the projective simulation model stands out for its simplicity in terms of the number of model parameters, which makes it simple to set up the learning agent in unknown task environments.

arXiv:1804.08607•IEEE Access•Jan 1, 2018•23 citations

Interpreting Quantum Learning Models via Stochastic Processes

Johannes Fankhauser, Lukas J. Fiderer, Hans J. Briegel

Quantum machine learning models define probabilistic input--output maps through coherent quantum evolution and measurement. While such models can exhibit computational advantages, their internal functioning and decision making generally resists interpretation in terms of stochastic trajectories through intermediate configurations. In contrast to classical (Markovian) stochastic processes, quantum dynamics generically violates the Chapman--Kolmogorov divisibility condition, preventing a decomposition into probabilistically meaningful intermediate transitions. We develop a probabilistic framework for representing quantum learning models as stochastic processes over configuration spaces where the dynamics are modeled as linear maps on probability distributions. Starting from a fixed POVM, arbitrary quantum channels induce transition kernels on the associated probability representation. For informationally complete POVMs, and in particular SIC-POVMs, these kernels are Markovian but generally quasi-stochastic, with non-classicality appearing as negativity. By contrast, projective spaces admit positive stochastic kernels but generally require non-Markovian dynamics due to the failure of Chapman--Kolmogorov divisibility. This yields a trade-off between negativity and dependence on past configurations, i.e. quantum dynamics can be represented either by Markovian quasi-stochastic maps or by positive stochastic processes with higher Markov order. We discuss how such representations of quantum dynamics can be interpreted as stochastic walks through a memory space in the spirit of Projective Simulation, a model of learning and agency in which decisions arise from random walks over an episodic memory network. We further outline how finite-order stochastic kernels can approximate such quantum deliberation processes and show in what regimes the classical machine learning model is recovered.

quant-pharXiv:2607.17327•Jul 21, 2026•0 citations

Entanglement-enhanced Quantum Reinforcement Learning: an Application using Single-Photons

Joaquim M. Gaspar, Alexandre Bergerault, Vassilis Apostolou +1 more

Quantum Reinforcement Learning (QRL) is an emerging field with many novel approaches suitable for noisy intermediate-scale quantum (NISQ) devices being proposed recently. Projective Simulation (PS) is a classical machine learning method based on random walks in a Directed Acyclic Graph (DAG), and is mainly used to train agents in a Reinforcement Learning setup. This graph corresponds to the Episodic Compositional Memory (ECM) of the agent. For the quantum version of PS we use quantum walks of single photons in quantum optical circuits, which go through tunable components (beamsplitters and phase shifters). We call this specific model Quantum Optical Projective Simulation (QOPS). In this work, we propose a general framework to translate any 2-layer ECM to a quantum optical circuit and an algorithm for solving quantum strategic games using QOPS. Moreover, we introduce QOPS with an entangled state as input. The application that we are using to evaluate the performance of the framework is the Prisoner's Dilemma. We executed the implementation using Altair's noisy simulator, designed to emulate a single photon-based quantum processor by Quandela. The algorithm provided promising results even in the presence of noise, which makes it a strong candidate for runs on single-photon-based quantum processors.

IEEE International Conference on Quantum Computing and Engineering (QCE)•Sep 15, 2024•4 citations

Demonstration of quantum projective simulation on a single-photon-based quantum computer

Giacomo Franceschetto, Arno Ricou

Variational quantum algorithms show potential in effectively operating on noisy intermediate-scale quantum devices. A variational approach to reinforcement learning has been recently proposed, incorporating linear-optical interferometers and a classical learning model known as projective simulation (PS). PS is a decision-making tool for reinforcement learning and can be classically represented as a random walk on a graph that describes the agent's memory. In its optical quantum version, this approach utilizes quantum walks of single photons on a mesh of tunable beamsplitters and phase shifters to select actions. In this work, we present the implementation of this algorithm on Ascella, a single-photon-based quantum computer from Quandela. The focus is drawn on solving a test bed task to showcase the potential of the quantum agent with respect to the classical agent. Published by the American Physical Society 2024

arXiv:2404.12729•Physical Review A•Dec 17, 2024•3 citations

Quantum machine learning with glow for episodic tasks and decision games

Jens Clausen, Hans J. Briegel

We consider a general class of models, where a reinforcement learning (RL) agent learns from cyclic interactions with an external environment via classical signals. Perceptual inputs are encoded as quantum states, which are subsequently transformed by a quantum channel representing the agent's memory, while the outcomes of measurements performed at the channel's output determine the agent's actions. The learning takes place via stepwise modifications of the channel properties. They are described by an update rule that is inspired by the projective simulation (PS) model and equipped with a glow mechanism that allows for a backpropagation of policy changes, analogous to the eligibility traces in RL and edge glow in PS. In this way, the model combines features of PS with the ability for generalization, offered by its physical embodiment as a quantum system. We apply the agent to various setups of an invasion game and a grid world, which serve as elementary model tasks allowing a direct comparison with a basic classical PS agent.

arXiv:1601.07358•Physical Review A•Feb 5, 2018•14 citations
Abstract Projective Simulation (PS) is a model for agency that incorporates aspects of reinforcement learning, an indeterministic basic dynamics inspired by physical hopping processes as studied in quantum optics, and an overall orientation towards agency as a continuous process of interaction with the environment as described by phenomenology. We explain the formal structure of PS in detail and compare it to the more standard paradigm of reinforcement learning. PS stands out mostly due to its specific memory structure (Episodic and Compositional Memory, ECM) and its openness, which makes it apt as an agency model rather than just a learning model: PS does not start from a fixed environment model to be “solved”. The focus is, rather, on the internal dynamics of deliberation, which is resolved as a stochastic process within the ECM, as well as on learning and adaptation over a long time, allowing for significant and even disruptive changes in the environment to be incorporated. After discussing the model, we showcase several applications of PS, prominently in basic research in quantum mechanics and especially in quantum information. This strengthens our claim about the intimate connection between agency and quantum physics that we already defended in Chaps. 3 and 4 . We also consider quantum versions of PS, which promise to show quantum advantages in machine learning while retaining a maximum of transparency. These applications motivate a separate discussion of transparency and traceability in the context of explainable (quantum) AI.
In recent years, the interest in leveraging quantum effects for enhancing machine learning tasks has significantly increased. Many algorithms speeding up supervised and unsupervised learning were established. The first framework in which ways to exploit quantum resources specifically for the broader context of reinforcement learning were found is projective simulation. Projective simulation presents an agent-based reinforcement learning approach designed in a manner which may support quantum walk-based speedups. Although classical variants of projective simulation have been benchmarked against common reinforcement learning algorithms, very few formal theoretical analyses have been provided for its performance in standard learning scenarios. In this paper, we provide a detailed formal discussion of the properties of this model. Specifically, we prove that one version of the projective simulation model, understood as a reinforcement learning approach, converges to optimal behavior in a large class of Markov decision processes. This proof shows that a physically inspired approach to reinforcement learning can guarantee to converge.
Projective simulation is a model for intelligent agents with a deliberation capacity that is based on episodic memory. The model has been shown to provide a flexible framework for constructing reinforcement-learning agents, and it allows for quantum mechanical generalization, which leads to a speedup in deliberation time. Projective simulation agents have been applied successfully in the context of complex skill learning in robotics and the design of the state-of-the-art quantum experiments. In this paper, we study the performance of projective simulation in two benchmarking problems in navigation, namely, the grid world and the mountain car problem. The performance of projective simulation is compared to standard tabular reinforcement learning approaches, Q-learning, and SARSA. Our comparison demonstrates that the performance of projective simulation and standard learning approaches are qualitatively and quantitatively similar, while it is much easier to choose optimal model parameters in the case of projective simulation with reduced computational effort of one to two orders of magnitude. Our results show that the projective simulation model stands out for its simplicity in terms of the number of model parameters, which makes it simple to set up the learning agent in unknown task environments.
Quantum machine learning models define probabilistic input--output maps through coherent quantum evolution and measurement. While such models can exhibit computational advantages, their internal functioning and decision making generally resists interpretation in terms of stochastic trajectories through intermediate configurations. In contrast to classical (Markovian) stochastic processes, quantum dynamics generically violates the Chapman--Kolmogorov divisibility condition, preventing a decomposition into probabilistically meaningful intermediate transitions. We develop a probabilistic framework for representing quantum learning models as stochastic processes over configuration spaces where the dynamics are modeled as linear maps on probability distributions. Starting from a fixed POVM, arbitrary quantum channels induce transition kernels on the associated probability representation. For informationally complete POVMs, and in particular SIC-POVMs, these kernels are Markovian but generally quasi-stochastic, with non-classicality appearing as negativity. By contrast, projective spaces admit positive stochastic kernels but generally require non-Markovian dynamics due to the failure of Chapman--Kolmogorov divisibility. This yields a trade-off between negativity and dependence on past configurations, i.e. quantum dynamics can be represented either by Markovian quasi-stochastic maps or by positive stochastic processes with higher Markov order. We discuss how such representations of quantum dynamics can be interpreted as stochastic walks through a memory space in the spirit of Projective Simulation, a model of learning and agency in which decisions arise from random walks over an episodic memory network. We further outline how finite-order stochastic kernels can approximate such quantum deliberation processes and show in what regimes the classical machine learning model is recovered.
Quantum Reinforcement Learning (QRL) is an emerging field with many novel approaches suitable for noisy intermediate-scale quantum (NISQ) devices being proposed recently. Projective Simulation (PS) is a classical machine learning method based on random walks in a Directed Acyclic Graph (DAG), and is mainly used to train agents in a Reinforcement Learning setup. This graph corresponds to the Episodic Compositional Memory (ECM) of the agent. For the quantum version of PS we use quantum walks of single photons in quantum optical circuits, which go through tunable components (beamsplitters and phase shifters). We call this specific model Quantum Optical Projective Simulation (QOPS). In this work, we propose a general framework to translate any 2-layer ECM to a quantum optical circuit and an algorithm for solving quantum strategic games using QOPS. Moreover, we introduce QOPS with an entangled state as input. The application that we are using to evaluate the performance of the framework is the Prisoner's Dilemma. We executed the implementation using Altair's noisy simulator, designed to emulate a single photon-based quantum processor by Quandela. The algorithm provided promising results even in the presence of noise, which makes it a strong candidate for runs on single-photon-based quantum processors.
Variational quantum algorithms show potential in effectively operating on noisy intermediate-scale quantum devices. A variational approach to reinforcement learning has been recently proposed, incorporating linear-optical interferometers and a classical learning model known as projective simulation (PS). PS is a decision-making tool for reinforcement learning and can be classically represented as a random walk on a graph that describes the agent's memory. In its optical quantum version, this approach utilizes quantum walks of single photons on a mesh of tunable beamsplitters and phase shifters to select actions. In this work, we present the implementation of this algorithm on Ascella, a single-photon-based quantum computer from Quandela. The focus is drawn on solving a test bed task to showcase the potential of the quantum agent with respect to the classical agent. Published by the American Physical Society 2024
We consider a general class of models, where a reinforcement learning (RL) agent learns from cyclic interactions with an external environment via classical signals. Perceptual inputs are encoded as quantum states, which are subsequently transformed by a quantum channel representing the agent's memory, while the outcomes of measurements performed at the channel's output determine the agent's actions. The learning takes place via stepwise modifications of the channel properties. They are described by an update rule that is inspired by the projective simulation (PS) model and equipped with a glow mechanism that allows for a backpropagation of policy changes, analogous to the eligibility traces in RL and edge glow in PS. In this way, the model combines features of PS with the ability for generalization, offered by its physical embodiment as a quantum system. We apply the agent to various setups of an invasion game and a grid world, which serve as elementary model tasks allowing a direct comparison with a basic classical PS agent.