The Multi-Arm bandit(MAB) problem is a classical problem in the field of reinforcement learning with only one state. The Grid problem is a multi-state problem for reinforcement learning. In this work, we focus on how to combine the classical value function method to quantum computation, and we propose three novel quantum reinforcement learning(QRL) algorithms for the MAB problem and one novel QRL algorithm, which is combined with the quantum random walk and Grover algorithm, for the Grid problem. From the experiments, the learning process is speed-up by combining the value function with quantum computation.
Quantum computing has the potential to solve complex problems faster and more efficiently than classical computing. It can achieve speedups by leveraging quantum phenomena like superposition, entanglement, and tunneling. Quantum walks (QWs) form the foundation for many quantum algorithms. Unlike classical random walks, QWs exhibit quantum interference, leading to unique behaviors such as linear spreading and localization. These properties make QWs valuable for various applications, including universal computation, time series prediction, encryption, and quantum hash functions. One emerging application of QWs is decision-making. Previous research has used QWs to model human decision processes and solve multiarmed bandit problems. This paper extends QWs to collective decision-making, focusing on minimizing decision conflicts, cases where multiple agents choose the same option, leading to inefficiencies like traffic congestion or overloaded servers. Prior research using quantum interference has addressed two-player conflict avoidance but struggled with three-player scenarios. This paper proposes a method using QWs to entirely eliminate decision conflicts in three-player cases, demonstrating its effectiveness in collective decision-making.
Multi-arm bandit (MAB) and stochastic linear bandit (SLB) are important models in reinforcement learning, and it is well-known that classical algorithms for bandits with time horizon T suffer from the regret of at least the square root of T. In this paper, we study MAB and SLB with quantum reward oracles and propose quantum algorithms for both models with the order of the polylog T regrets, exponentially improving the dependence in terms of T. To the best of our knowledge, this is the first provable quantum speedup for regrets of bandit problems and in general exploitation in reinforcement learning. Compared to previous literature on quantum exploration algorithms for MAB and reinforcement learning, our quantum input model is simpler and only assumes quantum oracles for each individual arm.
Quantum multi-armed bandits (MAB) and stochastic linear bandits (SLB) have recently attracted significant attention, as their quantum counterparts can achieve quadratic speedups over classical MAB and SLB. However, most existing quantum MAB algorithms assume ideal quantum Monte Carlo (QMC) procedures on noise-free circuits, overlooking the impact of noise in current noisy intermediate-scale quantum (NISQ) devices. In this paper, we study a noise-robust QMC algorithm that improves estimation accuracy when querying quantum reward oracles. Building on this estimator, we propose noise-robust QMAB and QSLB algorithms that enhance performance in noisy environments while preserving the advantage over classical methods. Experiments show that our noise-robust approach improves QMAB estimation accuracy and reduces regret under several quantum noise models.
We study quantum multi-armed bandits (QMAB) and quantum linear bandits (QLB), where the learner queries each arm or action through a quantum reward oracle or its inverse. Prior work gives algorithms over horizon
T with regret
O(KlogT) for QMAB with
K arms and
O(d2polylogT) for
d-dimensional QLB. This leaves open the optimal dependence on
K and
T and whether the dependence on
d can be further improved. In this work, we prove the first tight minimax regret bound of
Θ(Klog(1+T/K)) for QMAB and the first lower bound of
Ω(dlog(1+T/d)) for finite-action QLB, ruling out regret independent of
T. Our lower bounds rely on a high-confidence single-arm quantum testing lower bound for distinguishing a fixed reward mean from an interval of alternatives. A bandit-to-testing reduction then lifts it to the QMAB lower bound, while a linear embedding gives the finite-action QLB lower bound. The matching QMAB upper bound is obtained using a tail bound for the Quantum Monte Carlo (QMC) estimator. For finite-action QLB, we propose a phased elimination algorithm that combines a low-bias low-variance quantum mean estimator with a small-support
G-optimal design through a query allocation matched to the design weights. When the action set has size
poly(d), its regret is nearly linear in
d and matches our lower bound up to polylogarithmic factors.