Collaboration is a key challenge in distributed multi-agent reinforcement learning (MARL) environments. Learning frameworks for these decentralized systems must weigh the benefits of explicit player coordination against the communication overhead and computational cost of sharing local observations and environmental data. Quantum computing has sparked a potential synergy between quantum entanglement and cooperation in multi-agent environments, which could enable more efficient distributed collaboration with minimal information sharing. This relationship is largely unexplored, however, as current state-of-the-art quantum MARL (QMARL) implementations rely on classical information sharing rather than entanglement over a quantum channel as a coordination medium. In contrast, in this paper, a novel framework dubbed entangled QMARL (eQMARL) is proposed. The proposed eQMARL is a distributed actor-critic framework that facilitates cooperation over a quantum channel and eliminates local observation sharing via a quantum entangled split critic. Introducing a quantum critic uniquely spread across the agents allows coupling of local observation encoders through entangled input qubits over a quantum channel, which requires no explicit sharing of local observations and reduces classical communication overhead. Further, agent policies are tuned through joint observation-value function estimation via joint quantum measurements, thereby reducing the centralized computational burden. Experimental results show that eQMARL with $Ψ^{+}$ entanglement converges to a cooperative strategy up to $17.8\%$ faster and with a higher overall score compared to split classical and fully centralized classical and quantum baselines. The results also show that eQMARL achieves this performance with a constant factor of $25$-times fewer centralized parameters compared to the split classical baseline.
Collaboration is a key challenge in distributed multi-agent reinforcement learning (MARL) environments. Learning frameworks for these decentralized systems must weigh the benefits of explicit player coordination against the communication overhead and computational cost of sharing local observations and environmental data. Quantum computing has sparked a potential synergy between quantum entanglement and cooperation in multi-agent environments, which could enable more efficient distributed collaboration with minimal information sharing. This relationship is largely unexplored, however, as current state-of-the-art quantum MARL (QMARL) implementations rely on classical information sharing rather than entanglement over a quantum channel as a coordination medium. In contrast, in this paper, a novel framework dubbed entangled QMARL (eQMARL) is proposed. The proposed eQMARL is a distributed actor-critic framework that facilitates cooperation over a quantum channel and eliminates local observation sharing via a quantum entangled split critic. Introducing a quantum critic uniquely spread across the agents allows coupling of local observation encoders through entangled input qubits over a quantum channel, which requires no explicit sharing of local observations and reduces classical communication overhead. Further, agent policies are tuned through joint observation-value function estimation via joint quantum measurements, thereby reducing the centralized computational burden. Experimental results show that eQMARL with Ψ+ entanglement converges to a cooperative strategy up to 17.8% faster and with a higher overall score compared to split classical and fully centralized classical and quantum baselines. The results also show that eQMARL achieves this performance with a constant factor of 25-times fewer centralized parameters compared to the split classical baseline.
We present an empirical evaluation of quantum entanglement in agent coordination within quantum multi agent reinforcement learning (QMARL). While QMARL has attracted growing interest recently, most prior work evaluates quantum policies without provable baselines, making it impossible to rigorously distinguish quantum advantage from algorithmic coincidence. We address this directly by evaluating a decentralized QMARL framework with variational quantum circuit (VQC) actors with shared entangled states. In the CHSH game, which has a mathematically proven classical performance ceiling of 0.75 win rate, we show that entangled QMARL agents approach the Tsirelson limit of 0.854, providing clear evidence of their quantum advantage. We show that unentangled quantum circuits match the classical baseline, confirming that entanglement and not the quantum circuit itself is the active coordination mechanism. We also explore the effect of specific entanglement structures, as some Bell states enable coordination gains while others actively harm performance. On cooperative navigation (CoopNav), QMARL without entanglement achieves ∼2× improvement in success rate over classical MAA2C ($\sim$0.85 versus $\sim$0.40), with the hybrid configuration, quantum actor paired with a classical centralised critic, outperforming both fully classical and fully quantum solutions. We present our experimental analysis and discuss future work.
This paper presents a comprehensive survey of Quantum Multi-Agent Reinforcement Learning (QMARL), a nascent field at the intersection of quantum computing and multi-agent systems. The survey begins by introducing the fundamentals of quantum computing, highlighting its potential to revolutionize computational capabilities. We then delve into the principles of multi-agent reinforcement learning (MARL), examining how quantum computing can enhance learning efficiency and decision-making processes in complex environments. The core of the survey focuses on the current state of QMARL, reviewing existing literature, methodologies, and case studies that demonstrate the integration of quantum algorithms with MARL frameworks. The paper also addresses the unique challenges and opportunities presented by quantum technologies in multi-agent systems, such as quantum entanglement and superposition, and their implications for agent coordination and learning dynamics. Additionally, the survey explores the practical applications of QMARL in various domains, including cybersecurity, finance, and robotics, underscoring its transformative potential. The paper concludes by identifying key research gaps and proposing future directions for the development of QMARL. This includes the need for scalable quantum algorithms, the exploration of quantum-resistant strategies in adversarial settings, and the integration of quantum principles in agent communication and collaboration. Overall, this survey serves as a foundational guide for researchers and practitioners interested in the emerging field of QMARL, offering insights into its current achievements and future possibilities.
This paper presents a comprehensive survey of Quantum Multi-Agent Reinforcement Learning (QMARL), a nascent field at the intersection of quantum computing and multi-agent systems. The survey begins by introducing the fundamentals of quantum computing, highlighting its potential to revolutionize computational capabilities. We then delve into the principles of multi-agent reinforcement learning (MARL), examining how quantum computing can enhance learning efficiency and decision-making processes in complex environments. The core of the survey focuses on the current state of QMARL, reviewing existing literature, methodologies, and case studies that demonstrate the integration of quantum algorithms with MARL frameworks. The paper also addresses the unique challenges and opportunities presented by quantum technologies in multi-agent systems, such as quantum entanglement and superposition, and their implications for agent coordination and learning dynamics. Additionally, the survey explores the practical applications of QMARL in various domains, including cybersecurity, finance, and robotics, underscoring its transformative potential. The paper concludes by identifying key research gaps and proposing future directions for the development of QMARL. This includes the need for scalable quantum algorithms, the exploration of quantum-resistant strategies in adversarial settings, and the integration of quantum principles in agent communication and collaboration. Overall, this survey serves as a foundational guide for researchers and practitioners interested in the emerging field of QMARL, offering insights into its current achievements and future possibilities.
The inability to communicate poses a major challenge to coordination in multi-agent reinforcement learning (MARL). Prior work has explored correlating local policies via shared randomness, sometimes in the form of a correlation device, as a mechanism to assist in decentralized decision-making. In contrast, this work introduces the first framework for training MARL agents to exploit shared quantum entanglement as a coordination resource, which permits a larger class of communication-free correlated policies than shared randomness alone. This is motivated by well-known results in quantum physics which posit that, for certain single-round cooperative games with no communication, shared quantum entanglement enables strategies that outperform those that only use shared randomness. In such cases, we say that there is quantum advantage. Our framework is based on a novel differentiable policy parameterization that enables optimization over quantum measurements, together with a novel policy architecture that decomposes joint policies into a quantum coordinator and decentralized local actors. To illustrate the effectiveness of our proposed method, we first show that we can learn, purely from experience, strategies that attain quantum advantage in single-round games that are treated as black box oracles. We then demonstrate how our machinery can learn policies with quantum advantage in an illustrative multi-agent sequential decision-making problem formulated as a decentralized partially observable Markov decision process (Dec-POMDP).
This paper surveys Quantum Multi-Agent Reinforcement Learning (QMARL), an emerging fusion of quantum computing and multi-agent systems. It begins by introducing the transformative potential of quantum computing in computational capabilities. Examining the principles of multi-agent reinforcement learning (MARL), the paper explores how quantum computing enhances learning efficiency and decision-making. Focusing on the current state of QMARL, it reviews literature, methodologies, and case studies showcasing the integration of quantum algorithms with MARL frameworks. The survey addresses challenges and opportunities arising from quantum technologies in multi-agent systems, such as entanglement and superposition, exploring their implications for agent coordination and learning dynamics. Practical applications in domains like cybersecurity and finance underscore QMARL’s transformative potential. Concluding, the paper identifies research gaps and proposes future directions, emphasizing the need for scalable quantum algorithms, exploring quantum-resistant strategies in adversarial settings, and integrating quantum principles in agent communication and collaboration. This concise survey serves as a foundational guide for researchers and practitioners, offering insights into the current state and future possibilities of QMARL.
For Industry 4.0 Revolution, cooperative autonomous mobility systems are widely used based on multiagent reinforcement learning (MARL). However, the MARLbased algorithms suffer from huge parameter utilization and convergence difficulties with many agents. To tackle these problems, a quantum MARL (QMARL) algorithm based on the concept of actor-critic network is proposed, which is beneficial in terms of scalability, to deal with the limitations in the noisy intermediatescale quantum (NISQ) era. Additionally, our QMARL is also beneficial in terms of efficient parameter utilization and fast convergence due to quantum supremacy. Note that the reward in our QMARL is defined as task precision over computation time in multiple agents, thus, multi-agent cooperation can be realized. For further improvement, an additional technique for scalability is proposed, which is called projection value measure (PVM). Based on PVM, our proposed QMARL can achieve the highest reward, by reducing the action dimension into a logarithmicscale. Finally, we can conclude that our proposed QMARL with PVM outperforms the other algorithms in terms of efficient parameter utilization, fast convergence, and scalability.