Decision-making enables artificial intelligence to dynamically adjust and acquire knowledge from experiences, distinguishing it from traditional computing intelligence based on predetermined and rigid logic rules. However, the hardness of decision-making within the Turing framework increases exponentially with the number of decisions and decision-agents, thereby limiting the speed and scaling for artificial intelligence to process intensity-heavy tasks. Here, by introducing the quantum advantages of both photons and atoms, we report a photon-atom hybrid decision framework, whose decision-making exploration is accomplished through time-correlated atomic excitation within a quantum memory material. We develop a pseudophotonic blockade effect within memory materials to ensure that decision-making conflicts are hardly generated. With exploring a two-agent N -armed bandit in a concurrent manner, an N 2 times acceleration of decision-making exploration compared to nonconcurrent methods is demonstrated. Furthermore, the experimentally characterized performance in preference satisfaction and conflict avoidance shows an advanced capacity for distributed decision-making on a significant scale. Our research advances scalable distributed frameworks to address future reinforcement learning challenges.
Decision-making enables artificial intelligence to dynamically adjust and acquire knowledge from experiences, distinguishing it from traditional computing intelligence based on predetermined and rigid logic rules. However, the hardness of decision-making within the Turing framework increases exponentially with the number of decisions and decision-agents, thereby limiting the speed and scaling for artificial intelligence to process intensity-heavy tasks. Here, by introducing the quantum advantages of both photons and atoms, we report a photon-atom hybrid decision framework, whose decision-making exploration is accomplished through time-correlated atomic excitation within a quantum memory material. We develop a pseudophotonic blockade effect within memory materials to ensure that decision-making conflicts are hardly generated. With exploring a two-agent N -armed bandit in a concurrent manner, an N 2 times acceleration of decision-making exploration compared to nonconcurrent methods is demonstrated. Furthermore, the experimentally characterized performance in preference satisfaction and conflict avoidance shows an advanced capacity for distributed decision-making on a significant scale. Our research advances scalable distributed frameworks to address future reinforcement learning challenges.
Understanding and using natural processes for intelligent functionalities, referred to as natural intelligence, has recently attracted interest from a variety of fields, including post-silicon computing for artificial intelligence and decision making in the behavioural sciences. In a past study, we successfully used the wave-particle duality of single photons to solve the two-armed bandit problem, which constitutes the foundation of reinforcement learning and decision making. In this study, we propose and confirm a hierarchical architecture for single-photon-based reinforcement learning and decision making that verifies the scalability of the principle. Specifically, the four-armed bandit problem is solved given zero prior knowledge in a two-layer hierarchical architecture, where polarization is autonomously adapted in order to effect adequate decision making using single-photon measurements. In the hierarchical structure, the notion of layer-dependent decisions emerges. The optimal solutions in the coarse layer and in the fine layer, however, conflict with each other in some contradictive problems. We show that while what we call a tournament strategy resolves such contradictions, the probabilistic nature of single photons allows for the direct location of the optimal solution even for contradictive problems, hence manifesting the exploration ability of single photons. This study provides insights into photon intelligence in hierarchical architectures for future artificial intelligence as well as the potential of natural processes for intelligent functionalities.
Recently, extensive studies on photonic reinforcement learning to accelerate the process of calculation by exploiting the physical nature of light have been conducted. Previous studies utilized quantum interference of photons to achieve collective decision-making without choice conflicts when solving the competitive multi-armed bandit problem, a fundamental example in reinforcement learning. However, the bandit problem deals with a static environment where the agent’s actions do not influence the reward probabilities. This study aims to extend the conventional approach to a more general type of parallel reinforcement learning targeting the grid world problem. Unlike the conventional approach, the proposed scheme deals with a dynamic environment where the reward changes because of the agent’s actions. A successful photonic reinforcement learning scheme requires both a photonic system that contributes to the quality of learning and a suitable algorithm. This study proposes a novel learning algorithm, a modified bandit Q-learning method, in view of a potential photonic implementation. Here, state–action pairs in the environment are regarded as slot machines in the context of the bandit problem and a change in Q -value is regarded as the reward of the bandit problem. We perform numerical simulations to validate the effectiveness of the bandit algorithm. In addition, we propose a parallel architecture in which multiple agents are indirectly connected through quantum interference of light and quantum principles ensure the conflict-free property of state–action pair selections among agents. We demonstrate that parallel reinforcement learning can be accelerated owing to conflict avoidance among multiple agents.
Recently, extensive studies on photonic reinforcement learning to accelerate the process of calculation by exploiting the physical nature of light have been conducted. Previous studies utilized quantum interference of photons to achieve collective decision-making without choice conflicts when solving the competitive multi-armed bandit problem, a fundamental example of reinforcement learning. However, the bandit problem deals with a static environment where the agent's action does not influence the reward probabilities. This study aims to extend the conventional approach to a more general multi-agent reinforcement learning targeting the grid world problem. Unlike the conventional approach, the proposed scheme deals with a dynamic environment where the reward changes because of agents' actions. A successful photonic reinforcement learning scheme requires both a photonic system that contributes to the quality of learning and a suitable algorithm. This study proposes a novel learning algorithm, discontinuous bandit Q-learning, in view of a potential photonic implementation. Here, state-action pairs in the environment are regarded as slot machines in the context of the bandit problem and an updated amount of Q-value is regarded as the reward of the bandit problem. We perform numerical simulations to validate the effectiveness of the bandit algorithm. In addition, we propose a multi-agent architecture in which agents are indirectly connected through quantum interference of light and quantum principles ensure the conflict-free property of state-action pair selections among agents. We demonstrate that multi-agent reinforcement learning can be accelerated owing to conflict avoidance among multiple agents.
Understanding and using photonic processes for intelligent functionalities, referred to as photonic intelligence, has recently attracted interest from a variety of fields, including postsilicon computing for artificial intelligence and decision making in the behavioral sciences. In a past study, we successfully used the wave–particle duality of single photons to solve the two-armed bandit problem, which constitutes one of the important foundations of decision making and reinforcement learning. In this paper, we propose and confirm a hierarchical architecture for single-photon-based decision making that verifies the scalability of the principle. Specifically, the four-armed bandit problem is solved given zero prior knowledge in a two-layer hierarchical architecture, where the polarization of single photons is autonomously adapted in order to effect adequate decision making. In the hierarchical structure, the notion of layer-dependent decisions emerges. The optimal solutions in the coarse layer and the fine layer, however, conflict with each other in some contradictory problems. We show that while what we call a tournament strategy resolves such contradictions, the probabilistic nature of single photons allows for the direct location of the optimal solution, even for contradictory problems, hence manifesting the exploration capability of single photons. This study provides insights into photon intelligence in hierarchical architectures for artificial intelligence as well as a novel aspect of photonic processes for intelligent functionalities.
In the realm of artificial intelligence, deep rein-forcement learning (RL) agents struggle with generalizability and require substantial computational resources, unlike humans who easily adapt and generalize across tasks. To address these challenges, we introduce Quantum Rainbow, a hybrid algorithm that leverages the neural mechanisms of human decision-making and the efficiency of quantum computing. Quantum Rainbow combines variational quantum circuits with the Rainbow Deep Q-Network (DQN) model to create a novel approach in rein-forcement learning that integrates quantum principles into deep learning paradigms. We evaluate our model using behavioral experiments through the Iowa Gambling Task and 4-Armed Bandit Task. Our investigations reveal a significant relationship between the architecture of quantum circuits and the performance of quantum RL agents. Specifically, using causal discovery methods, we demonstrate the critical role of quantum entanglement in enhancing model performance. These findings not only show promising results but also pave the way for future explorations into optimizing quantum circuit architectures for reinforcement learning applications. This study underscores the potential of quantum-enhanced algorithms to achieve “quantum advantage” by addressing fundamental limitations in conventional deep RL methods.
In recent cross-disciplinary studies involving both optics and computing, single-photon-based decision-making has been demonstrated by utilizing the wave-particle duality of light to solve multi-armed bandit problems. Furthermore, entangled-photon-based decision-making has managed to solve a competitive multi-armed bandit problem in such a way that conflicts of decisions among players are avoided while ensuring equality. However, as these studies are based on the polarization of light, the number of available choices is limited to two, corresponding to two orthogonal polarization states. Here we propose a scalable principle to solve competitive decision-making situations by using the orbital angular momentum as the tunable degree of freedom of photons, which theoretically allows an unlimited number of arms. Moreover, by extending the Hong-Ou-Mandel effect to more than two states, we theoretically establish an experimental configuration able to generate entangled photon states with orbital angular momentum and conditions that provide conflict-free selections at every turn. We numerically examine total rewards regarding three-armed bandit problems, for which the proposed strategy accomplishes almost the theoretical maximum, which is greater than a conventional mixed strategy intending to realize Nash equilibrium. This is thanks to the entanglement property that achieves no-conflict selections, even in the exploring phase to find the best arms.