Reinforcement learning algorithms are a powerful tool to manage the interaction between a system and its environment. Here we present an approach to apply these algorithms within modern-day photonic technologies. Numerical tests, performed on typical learning tasks, show that the architecture is robust against experimental noise, which can even be beneficial for the learning process. The proposed architecture, based on single-photon evolution on a tree structure of tunable beamsplitters, is simple, easy to implement and an integration in quantum optics applications appears to be within the reach of near-term technology.
Reinforcement learning algorithms are a powerful tool to manage the interaction between a system and its environment. Here we present an approach to apply these algorithms within modern-day photonic technologies. Numerical tests, performed on typical learning tasks, show that the architecture is robust against experimental noise, which can even be beneficial for the learning process. The proposed architecture, based on single-photon evolution on a tree structure of tunable beamsplitters, is simple, easy to implement and an integration in quantum optics applications appears to be within the reach of near-term technology.
Abstract The last decade has seen an unprecedented growth in artificial intelligence and photonic technologies, both of which drive the limits of modern-day computing devices. In line with these recent developments, this work brings together the state of the art of both fields within the framework of reinforcement learning. We present the blueprint for a photonic implementation of an active learning machine incorporating contemporary algorithms such as SARSA, Q-learning, and projective simulation. We numerically investigate its performance within typical reinforcement learning environments, showing that realistic levels of experimental noise can be tolerated or even be beneficial for the learning process. Remarkably, the architecture itself enables mechanisms of abstraction and generalization, two features which are often considered key ingredients for artificial intelligence. The proposed architecture, based on single-photon evolution on a mesh of tunable beamsplitters, is simple, scalable, and a first integration in quantum optical experiments appears to be within the reach of near-term technology.
Many challenges arising in Quantum Technology can be successfully addressed using a set of machine learning algorithms collectively known as reinforcement learning (RL), based on adaptive decision-making through interaction with the quantum device. After a concise and intuitive introduction to RL aimed at a broad physics readership, we discuss the key ideas and core concepts in reinforcement learning with a particular focus on quantum systems. We then survey recent progress in RL in all relevant areas. We discuss state preparation in few- and many-body quantum systems, the design and optimization of high-fidelity quantum gates, and the automated construction of quantum circuits, including applications to variational quantum eigensolvers and architecture search. We further highlight the interactive capabilities of RL agents, emphasizing recent progress in quantum feedback control and quantum error correction, and briefly discuss quantum reinforcement learning as well as applications to quantum metrology. The review concludes with a discussion of open challenges -- such as scalability, interpretability, and integration with experimental platforms -- and outlines promising directions for future research. Throughout, we highlight experimental implementations that exemplify the increasing role of reinforcement learning in shaping the development of quantum technologies.
As the field of artificial intelligence is pushed forward, the question arises of how fast autonomous machines can learn. Within artificial intelligence, an important paradigm is reinforcement learning, where agents - learning entities capable of decision making - interact with the world they are placed in, called an environment. Thanks to these interactions, agents receive feedback from the environment and thus progressively adjust their behaviour to accomplish a given goal. An important question in reinforcement learning is how fast agents can learn to fulfill their tasks. To answer this question we consider a novel reinforcement learning framework where quantum mechanics is used. In particular, we quantize the agent and the environment and grant them the possibility to also interact quantum-mechanically, that is, by using a quantum channel for their communication. We demonstrate that this feature enables a speed-up in the agent's learning process, and we further show that combining this scenario with classical communication enables the evaluation of such an improvement. This learning protocol is implemented on an integrated re-programmable photonic platform interfaced with photons at telecommunication wavelengths. Thanks to the full tunability of the device, this platform proves the best candidate for the implementation of learning protocols, where a continuous update of the learning process is required.
Quantum Reinforcement Learning (QRL) is an emerging field with many novel approaches suitable for noisy intermediate-scale quantum (NISQ) devices being proposed recently. Projective Simulation (PS) is a classical machine learning method based on random walks in a Directed Acyclic Graph (DAG), and is mainly used to train agents in a Reinforcement Learning setup. This graph corresponds to the Episodic Compositional Memory (ECM) of the agent. For the quantum version of PS we use quantum walks of single photons in quantum optical circuits, which go through tunable components (beamsplitters and phase shifters). We call this specific model Quantum Optical Projective Simulation (QOPS). In this work, we propose a general framework to translate any 2-layer ECM to a quantum optical circuit and an algorithm for solving quantum strategic games using QOPS. Moreover, we introduce QOPS with an entangled state as input. The application that we are using to evaluate the performance of the framework is the Prisoner's Dilemma. We executed the implementation using Altair's noisy simulator, designed to emulate a single photon-based quantum processor by Quandela. The algorithm provided promising results even in the presence of noise, which makes it a strong candidate for runs on single-photon-based quantum processors.
The advantage of quantum computers over classical computers fuels the recent trend of developing machine learning algorithms on quantum computers, which can potentially lead to breakthroughs and new learning models in this area. The aim of our study is to explore deep quantum reinforcement learning (RL) on photonic quantum computers, which can process information stored in the quantum states of light. These quantum computers can naturally represent continuous variables, making them an ideal platform to create quantum versions of neural networks. Using quantum photonic circuits, we implement Q learning and actor-critic algorithms with multilayer quantum neural networks and test them in the grid world environment. Our experiments show that 1) these quantum algorithms can solve the RL problem and 2) compared to one layer, using three layer quantum networks improves the learning of both algorithms in terms of rewards collected. In summary, our findings suggest that having more layers in deep quantum RL can enhance the learning outcome.
Recently, extensive studies on photonic reinforcement learning to accelerate the process of calculation by exploiting the physical nature of light have been conducted. Previous studies utilized quantum interference of photons to achieve collective decision-making without choice conflicts when solving the competitive multi-armed bandit problem, a fundamental example of reinforcement learning. However, the bandit problem deals with a static environment where the agent's action does not influence the reward probabilities. This study aims to extend the conventional approach to a more general multi-agent reinforcement learning targeting the grid world problem. Unlike the conventional approach, the proposed scheme deals with a dynamic environment where the reward changes because of agents' actions. A successful photonic reinforcement learning scheme requires both a photonic system that contributes to the quality of learning and a suitable algorithm. This study proposes a novel learning algorithm, discontinuous bandit Q-learning, in view of a potential photonic implementation. Here, state-action pairs in the environment are regarded as slot machines in the context of the bandit problem and an updated amount of Q-value is regarded as the reward of the bandit problem. We perform numerical simulations to validate the effectiveness of the bandit algorithm. In addition, we propose a multi-agent architecture in which agents are indirectly connected through quantum interference of light and quantum principles ensure the conflict-free property of state-action pair selections among agents. We demonstrate that multi-agent reinforcement learning can be accelerated owing to conflict avoidance among multiple agents.