Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:1504.02247 · Scientific Reports

Projective simulation with generalization

Alexey A. Melnikov, Adi Makmal, Vedran Dunjko, Hans J. Briegel

Published Oct 25, 2017• 45 citations
arXivHTMLPDF

Abstract

The ability to generalize is an important feature of any intelligent agent. Not only because it may allow the agent to cope with large amounts of data, but also because in some environments, an agent with no generalization capabilities cannot learn. In this work we outline several criteria for generalization, and present a dynamic and autonomous machinery that enables projective simulation agents to meaningfully generalize. Projective simulation, a novel, physical approach to artificial intelligence, was recently shown to perform well in standard reinforcement learning problems, with applications in advanced robotics as well as quantum experiments. Both the basic projective simulation model and the presented generalization machinery are based on very simple principles. This allows us to provide a full analytical analysis of the agent's performance and to illustrate the benefit the agent gains by generalizing. Specifically, we show that already in basic (but extreme) environments, learning without generalization may be impossible, and demonstrate how the presented generalization machinery enables the projective simulation agent to learn.

DOI
10.1038/s41598-017-14740-y

Related papers

Benchmarking Projective Simulation in Navigation Problems

Alexey A. Melnikov, Adi Makmal, Hans J. Briegel

Projective simulation is a model for intelligent agents with a deliberation capacity that is based on episodic memory. The model has been shown to provide a flexible framework for constructing reinforcement-learning agents, and it allows for quantum mechanical generalization, which leads to a speedup in deliberation time. Projective simulation agents have been applied successfully in the context of complex skill learning in robotics and the design of the state-of-the-art quantum experiments. In this paper, we study the performance of projective simulation in two benchmarking problems in navigation, namely, the grid world and the mountain car problem. The performance of projective simulation is compared to standard tabular reinforcement learning approaches, Q-learning, and SARSA. Our comparison demonstrates that the performance of projective simulation and standard learning approaches are qualitatively and quantitatively similar, while it is much easier to choose optimal model parameters in the case of projective simulation with reduced computational effort of one to two orders of magnitude. Our results show that the projective simulation model stands out for its simplicity in terms of the number of model parameters, which makes it simple to set up the learning agent in unknown task environments.

arXiv:1804.08607•IEEE Access•Jan 1, 2018•23 citations

On the convergence of projective-simulation–based reinforcement learning in Markov decision processes

W. L. Boyajian, J. Clausen, L. M. Trenkwalder +2 more

In recent years, the interest in leveraging quantum effects for enhancing machine learning tasks has significantly increased. Many algorithms speeding up supervised and unsupervised learning were established. The first framework in which ways to exploit quantum resources specifically for the broader context of reinforcement learning were found is projective simulation. Projective simulation presents an agent-based reinforcement learning approach designed in a manner which may support quantum walk-based speedups. Although classical variants of projective simulation have been benchmarked against common reinforcement learning algorithms, very few formal theoretical analyses have been provided for its performance in standard learning scenarios. In this paper, we provide a detailed formal discussion of the properties of this model. Specifically, we prove that one version of the projective simulation model, understood as a reinforcement learning approach, converges to optimal behavior in a large class of Markov decision processes. This proof shows that a physically inspired approach to reinforcement learning can guarantee to converge.

arXiv:1910.11914•Quantum Machine Intelligence•Nov 5, 2020•18 citations

Projective Simulation

Hans J. Briegel, Thomas Müller

Abstract Projective Simulation (PS) is a model for agency that incorporates aspects of reinforcement learning, an indeterministic basic dynamics inspired by physical hopping processes as studied in quantum optics, and an overall orientation towards agency as a continuous process of interaction with the environment as described by phenomenology. We explain the formal structure of PS in detail and compare it to the more standard paradigm of reinforcement learning. PS stands out mostly due to its specific memory structure (Episodic and Compositional Memory, ECM) and its openness, which makes it apt as an agency model rather than just a learning model: PS does not start from a fixed environment model to be “solved”. The focus is, rather, on the internal dynamics of deliberation, which is resolved as a stochastic process within the ECM, as well as on learning and adaptation over a long time, allowing for significant and even disruptive changes in the environment to be incorporated. After discussing the model, we showcase several applications of PS, prominently in basic research in quantum mechanics and especially in quantum information. This strengthens our claim about the intimate connection between agency and quantum physics that we already defended in Chaps. 3 and 4 . We also consider quantum versions of PS, which promise to show quantum advantages in machine learning while retaining a maximum of transparency. These applications motivate a separate discussion of transparency and traceability in the context of explainable (quantum) AI.

Synthese Library/Synthese library•Jan 1, 2025•0 citations

Quantum Projective Simulation with Hamiltonian Evolution: A study in reinforcement learning

Amara Katabarwa, Nima Karimatari

Projective Simulation was introduced as a novel approach to Artificial Intelligence. It involves a deliberation procedure that consists of a random walk on a graph of clips and allows for the learning agent to project itself into the future before committing to an action. Here we study and analyze a quantum mechanical version in which the random walk is performed by two kinds of Hamiltonians. The first kind is implemented by naively embedding the classical model in a quantum model by turning the clips into qubits. The other allows for storing clips in superpositions of qubits allowing for a potentially purely quantum mechanical learning procedure in which the perception of the environment is purely quantum mechanical but the action is classical. We lastly introduce the concept of interacting projective agents for both the classical and quantum mechanical case.

arXiv:1708.00123•Aug 1, 2017•0 citations

Advances in quantum reinforcement learning

Vedran Dunjko, Jacob M. Taylor, Hans J. Briegel

In recent times, there has been much interest in quantum enhancements of machine learning, specifically in the context of data mining and analysis. Reinforcement learning, an interactive form of learning, is, in turn, vital in artificial intelligence-type applications. Also in this case, quantum mechanics was shown to be useful, in certain instances. Here, we elucidate these results, and show that quantum enhancements can be achieved in a new setting: the setting of learning models which learn how to improve themselves - that is, those that metalearn. While not all learning models meta-learn, all non-trivial models have the potential of being "lifted", enhanced, to meta-learning models. Our results show that also such models can be quantum-enhanced to make even better learners. In parallel, we address one of the bottlenecks of current quantum reinforcement learning approaches: the need for so-called oracularized variants of task environments. Here we elaborate on a method which realizes these variants, with minimal changes in the setting, and with no corruption of the operative specification of the environments. This result may be important in near-term experimental demonstrations of quantum reinforcement learning.

arXiv:1811.08676•Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics•Oct 1, 2017•81 citations

Multi-Excitation Projective Simulation with a Many-Body Physics-Inspired Inductive Bias

Philip A. LeMaitre, Marius Krumm, Hans J. Briegel

With the impressive progress of deep learning, applications relying on machine learning are increasingly being integrated into daily life. However, most deep learning models have an opaque, oracle-like nature making it difficult to interpret and understand their decisions. This problem led to the development of the field known as eXplainable Artificial Intelligence (XAI). One method in this field known as Projective Simulation (PS) models a chain-of-thought as a random walk of a particle on a graph with vertices that have concepts attached to them. While this description has various benefits, including the possibility of quantization, it cannot be naturally used to model thoughts that combine several concepts simultaneously. To overcome this limitation, we introduce Multi-Excitation Projective Simulation (mePS), a generalization that considers a chain-of-thought to be a random walk of several particles on a hypergraph. A definition for a dynamic hypergraph is put forward to describe the agent's training history along with applications to AI and hypergraph visualization. An inductive bias inspired by the remarkably successful few-body interaction models used in quantum many-body physics is formalized for our classical mePS framework and employed to tackle the exponential complexity associated with naive implementations of hypergraphs. We prove that our inductive bias reduces the complexity from exponential to polynomial, with the exponent representing the cutoff on how many particles can interact. We numerically apply our method to two toy environments and a more complex scenario modelling the diagnosis of a broken computer. These environments demonstrate the resource savings provided by an appropriate choice of inductive bias, as well as showcasing aspects of interpretability. A quantum model for mePS is also briefly outlined and some future directions for it are discussed.

International Joint Conference on Artificial Intelligence•Sep 1, 2026•0 citations
The ability to generalize is an important feature of any intelligent agent. Not only because it may allow the agent to cope with large amounts of data, but also because in some environments, an agent with no generalization capabilities cannot learn. In this work we outline several criteria for generalization, and present a dynamic and autonomous machinery that enables projective simulation agents to meaningfully generalize. Projective simulation, a novel, physical approach to artificial intelligence, was recently shown to perform well in standard reinforcement learning problems, with applications in advanced robotics as well as quantum experiments. Both the basic projective simulation model and the presented generalization machinery are based on very simple principles. This allows us to provide a full analytical analysis of the agent's performance and to illustrate the benefit the agent gains by generalizing. Specifically, we show that already in basic (but extreme) environments, learning without generalization may be impossible, and demonstrate how the presented generalization machinery enables the projective simulation agent to learn.
Projective simulation is a model for intelligent agents with a deliberation capacity that is based on episodic memory. The model has been shown to provide a flexible framework for constructing reinforcement-learning agents, and it allows for quantum mechanical generalization, which leads to a speedup in deliberation time. Projective simulation agents have been applied successfully in the context of complex skill learning in robotics and the design of the state-of-the-art quantum experiments. In this paper, we study the performance of projective simulation in two benchmarking problems in navigation, namely, the grid world and the mountain car problem. The performance of projective simulation is compared to standard tabular reinforcement learning approaches, Q-learning, and SARSA. Our comparison demonstrates that the performance of projective simulation and standard learning approaches are qualitatively and quantitatively similar, while it is much easier to choose optimal model parameters in the case of projective simulation with reduced computational effort of one to two orders of magnitude. Our results show that the projective simulation model stands out for its simplicity in terms of the number of model parameters, which makes it simple to set up the learning agent in unknown task environments.
In recent years, the interest in leveraging quantum effects for enhancing machine learning tasks has significantly increased. Many algorithms speeding up supervised and unsupervised learning were established. The first framework in which ways to exploit quantum resources specifically for the broader context of reinforcement learning were found is projective simulation. Projective simulation presents an agent-based reinforcement learning approach designed in a manner which may support quantum walk-based speedups. Although classical variants of projective simulation have been benchmarked against common reinforcement learning algorithms, very few formal theoretical analyses have been provided for its performance in standard learning scenarios. In this paper, we provide a detailed formal discussion of the properties of this model. Specifically, we prove that one version of the projective simulation model, understood as a reinforcement learning approach, converges to optimal behavior in a large class of Markov decision processes. This proof shows that a physically inspired approach to reinforcement learning can guarantee to converge.
Abstract Projective Simulation (PS) is a model for agency that incorporates aspects of reinforcement learning, an indeterministic basic dynamics inspired by physical hopping processes as studied in quantum optics, and an overall orientation towards agency as a continuous process of interaction with the environment as described by phenomenology. We explain the formal structure of PS in detail and compare it to the more standard paradigm of reinforcement learning. PS stands out mostly due to its specific memory structure (Episodic and Compositional Memory, ECM) and its openness, which makes it apt as an agency model rather than just a learning model: PS does not start from a fixed environment model to be “solved”. The focus is, rather, on the internal dynamics of deliberation, which is resolved as a stochastic process within the ECM, as well as on learning and adaptation over a long time, allowing for significant and even disruptive changes in the environment to be incorporated. After discussing the model, we showcase several applications of PS, prominently in basic research in quantum mechanics and especially in quantum information. This strengthens our claim about the intimate connection between agency and quantum physics that we already defended in Chaps. 3 and 4 . We also consider quantum versions of PS, which promise to show quantum advantages in machine learning while retaining a maximum of transparency. These applications motivate a separate discussion of transparency and traceability in the context of explainable (quantum) AI.
Projective Simulation was introduced as a novel approach to Artificial Intelligence. It involves a deliberation procedure that consists of a random walk on a graph of clips and allows for the learning agent to project itself into the future before committing to an action. Here we study and analyze a quantum mechanical version in which the random walk is performed by two kinds of Hamiltonians. The first kind is implemented by naively embedding the classical model in a quantum model by turning the clips into qubits. The other allows for storing clips in superpositions of qubits allowing for a potentially purely quantum mechanical learning procedure in which the perception of the environment is purely quantum mechanical but the action is classical. We lastly introduce the concept of interacting projective agents for both the classical and quantum mechanical case.
In recent times, there has been much interest in quantum enhancements of machine learning, specifically in the context of data mining and analysis. Reinforcement learning, an interactive form of learning, is, in turn, vital in artificial intelligence-type applications. Also in this case, quantum mechanics was shown to be useful, in certain instances. Here, we elucidate these results, and show that quantum enhancements can be achieved in a new setting: the setting of learning models which learn how to improve themselves - that is, those that metalearn. While not all learning models meta-learn, all non-trivial models have the potential of being "lifted", enhanced, to meta-learning models. Our results show that also such models can be quantum-enhanced to make even better learners. In parallel, we address one of the bottlenecks of current quantum reinforcement learning approaches: the need for so-called oracularized variants of task environments. Here we elaborate on a method which realizes these variants, with minimal changes in the setting, and with no corruption of the operative specification of the environments. This result may be important in near-term experimental demonstrations of quantum reinforcement learning.
With the impressive progress of deep learning, applications relying on machine learning are increasingly being integrated into daily life. However, most deep learning models have an opaque, oracle-like nature making it difficult to interpret and understand their decisions. This problem led to the development of the field known as eXplainable Artificial Intelligence (XAI). One method in this field known as Projective Simulation (PS) models a chain-of-thought as a random walk of a particle on a graph with vertices that have concepts attached to them. While this description has various benefits, including the possibility of quantization, it cannot be naturally used to model thoughts that combine several concepts simultaneously. To overcome this limitation, we introduce Multi-Excitation Projective Simulation (mePS), a generalization that considers a chain-of-thought to be a random walk of several particles on a hypergraph. A definition for a dynamic hypergraph is put forward to describe the agent's training history along with applications to AI and hypergraph visualization. An inductive bias inspired by the remarkably successful few-body interaction models used in quantum many-body physics is formalized for our classical mePS framework and employed to tackle the exponential complexity associated with naive implementations of hypergraphs. We prove that our inductive bias reduces the complexity from exponential to polynomial, with the exponent representing the cutoff on how many particles can interact. We numerically apply our method to two toy environments and a more complex scenario modelling the diagnosis of a broken computer. These environments demonstrate the resource savings provided by an appropriate choice of inductive bias, as well as showcasing aspects of interpretability. A quantum model for mePS is also briefly outlined and some future directions for it are discussed.