Gradient descent methods have long been the de facto standard for training deep neural networks. Millions of training samples are fed into models with billions of parameters, which are slowly updated over hundreds of epochs. Recently, it's been shown that large, randomly initialized neural networks contain subnetworks that perform as well as fully trained models. This insight offers a promising avenue for training future neural networks by simply pruning weights from large, random models. However, this problem is combinatorically hard and classical algorithms are not efficient at finding the best subnetwork. In this paper, we explore how quantum algorithms could be formulated and applied to this neuron selection problem. We introduce several methods for local quantum neuron selection that reduce the entanglement complexity that large scale neuron selection would require, making this problem more tractable for current quantum hardware.Gradient descent methods have long been the de facto standard for training deep neural networks. Millions of training samples are fed into models with billions of parameters, which are slowly updated over hundreds of epochs. Recently, it's been shown that large, randomly initialized neural networks contain subnetworks that perform as well as fully trained models. This insight offers a promising avenue for training future neural networks by simply pruning weights from large, random models. However, this problem is combinatorically hard and classical algorithms are not efficient at finding the best subnetwork. In this paper, we explore how quantum algorithms could be formulated and applied to this neuron selection problem. We introduce several methods for local quantum neuron selection that reduce the entanglement complexity that large scale neuron selection would require, making this problem more tractable for current quantum hardware.

arXiv:2302.05984Feb 12, 20230 citations

Quantum-Inspired Optimization of Neural Network Training Dynamics

Jincheng Zhang

Training deep neural networks remains a computationally intensive and often challenging process. Traditional optimization methods, such as stochastic gradient descent (SGD) and its variants, frequently struggle to escape local minima and converge to optimal solutions, leading to suboptimal network performance. This paper proposes a novel approach to neural network training that leverages principles inspired by quantum mechanics. Specifically, we investigate the application of quantum-inspired optimization techniques, including simulated annealing and quantum walks, to dynamically adjust the training process. The core idea is to mimic the probabilistic exploration inherent in quantum systems to effectively navigate the complex loss landscape of neural networks. We formulate the training process as a quantum-inspired optimization problem, where the network's weights are treated as quantum bits (Qubits) and the optimization algorithm seeks to find the lowest energy state, analogous to finding the minimum loss. The proposed method demonstrates the potential for accelerating convergence and improving the overall accuracy of neural networks compared to conventional optimization strategies. Our analysis focuses on the key aspects of the approach, including the mathematical formulation of the quantum-inspired optimization problem and the selection of appropriate parameters to govern the exploration and exploitation phases. We present a theoretical framework for understanding the dynamics of this approach and discuss potential avenues for further research, including hybrid quantum-classical algorithms and the integration of quantum computing hardware.Training deep neural networks remains a computationally intensive and often challenging process. Traditional optimization methods, such as stochastic gradient descent (SGD) and its variants, frequently struggle to escape local minima and converge to optimal solutions, leading to suboptimal network performance. This paper proposes a novel approach to neural network training that leverages principles inspired by quantum mechanics. Specifically, we investigate the application of quantum-inspired optimization techniques, including simulated annealing and quantum walks, to dynamically adjust the training process. The core idea is to mimic the probabilistic exploration inherent in quantum systems to effectively navigate the complex loss landscape of neural networks. We formulate the training process as a quantum-inspired optimization problem, where the network's weights are treated as quantum bits (Qubits) and the optimization algorithm seeks to find the lowest energy state, analogous to finding the minimum loss. The proposed method demonstrates the potential for accelerating convergence and improving the overall accuracy of neural networks compared to conventional optimization strategies. Our analysis focuses on the key aspects of the approach, including the mathematical formulation of the quantum-inspired optimization problem and the selection of appropriate parameters to govern the exploration and exploitation phases. We present a theoretical framework for understanding the dynamics of this approach and discuss potential avenues for further research, including hybrid quantum-classical algorithms and the integration of quantum computing hardware.

Sep 5, 20260 citations

A quantum algorithm for training wide and deep classical neural networks

Alexander Zlokapa, Hartmut Neven, Seth Lloyd

Given the success of deep learning in classical machine learning, quantum algorithms for traditional neural network architectures may provide one of the most promising settings for quantum machine learning. Considering a fully-connected feedforward neural network, we show that conditions amenable to classical trainability via gradient descent coincide with those necessary for efficiently solving quantum linear systems. We propose a quantum algorithm to approximately train a wide and deep neural network up to $O(1/n)$ error for a training set of size $n$ by performing sparse matrix inversion in $O(\log n)$ time. To achieve an end-to-end exponential speedup over gradient descent, the data distribution must permit efficient state preparation and readout. We numerically demonstrate that the MNIST image dataset satisfies such conditions; moreover, the quantum algorithm matches the accuracy of the fully-connected network. Beyond the proven architecture, we provide empirical evidence for $O(\log n)$ training of a convolutional neural network with pooling.Given the success of deep learning in classical machine learning, quantum algorithms for traditional neural network architectures may provide one of the most promising settings for quantum machine learning. Considering a fully-connected feedforward neural network, we show that conditions amenable to classical trainability via gradient descent coincide with those necessary for efficiently solving quantum linear systems. We propose a quantum algorithm to approximately train a wide and deep neural network up to $O(1/n)$ error for a training set of size $n$ by performing sparse matrix inversion in $O(\log n)$ time. To achieve an end-to-end exponential speedup over gradient descent, the data distribution must permit efficient state preparation and readout. We numerically demonstrate that the MNIST image dataset satisfies such conditions; moreover, the quantum algorithm matches the accuracy of the fully-connected network. Beyond the proven architecture, we provide empirical evidence for $O(\log n)$ training of a convolutional neural network with pooling.

arXiv:2107.09200Jul 19, 202110 citations

Quantum algorithms for feedforward neural networks

Jonathan Allcock, Chang‐Yu Hsieh, Iordanis Kerenidis +1 more

Quantum machine learning has the potential for broad industrial applications, and the development of quantum algorithms for improving the performance of neural networks is of particular interest given the central role they play in machine learning today. In this paper we present quantum algorithms for training and evaluating feedforward neural networks based on the canonical classical feedforward and backpropagation algorithms. Our algorithms rely on an efficient quantum subroutine for approximating the inner products between vectors in a robust way, and on implicitly storing large intermediate values in quantum random access memory for fast retrieval at later stages. The running times of our algorithms can be quadratically faster in the size of the network than their standard classical counterparts since they depend linearly on the number of neurons in the network, as opposed to the number of connections between neurons as in the classical case. This makes our algorithms suited for large-scale, highly-connected networks where the number of edges in the network dominates the classical algorithmic running time. Furthermore, networks trained by our quantum algorithm may have an intrinsic resilience to overfitting, as the algorithm naturally mimics the effects of classical techniques such as drop-out used to regularize networks. Our algorithms can also be used as the basis for new quantum-inspired classical algorithms which have the same dependence on the network dimensions as their quantum counterparts, but with quadratic overhead in other parameters that makes them relatively impractical.Quantum machine learning has the potential for broad industrial applications, and the development of quantum algorithms for improving the performance of neural networks is of particular interest given the central role they play in machine learning today. In this paper we present quantum algorithms for training and evaluating feedforward neural networks based on the canonical classical feedforward and backpropagation algorithms. Our algorithms rely on an efficient quantum subroutine for approximating the inner products between vectors in a robust way, and on implicitly storing large intermediate values in quantum random access memory for fast retrieval at later stages. The running times of our algorithms can be quadratically faster in the size of the network than their standard classical counterparts since they depend linearly on the number of neurons in the network, as opposed to the number of connections between neurons as in the classical case. This makes our algorithms suited for large-scale, highly-connected networks where the number of edges in the network dominates the classical algorithmic running time. Furthermore, networks trained by our quantum algorithm may have an intrinsic resilience to overfitting, as the algorithm naturally mimics the effects of classical techniques such as drop-out used to regularize networks. Our algorithms can also be used as the basis for new quantum-inspired classical algorithms which have the same dependence on the network dimensions as their quantum counterparts, but with quadratic overhead in other parameters that makes them relatively impractical.

Dec 15, 20190 citations

Training deep quantum neural networks

Kerstin Beer, Dmytro Bondarenko, Terry Farrelly +4 more

Neural networks enjoy widespread success in both research and industry and, with the advent of quantum technology, it is a crucial challenge to design quantum neural networks for fully quantum learning tasks. Here we propose a truly quantum analogue of classical neurons, which form quantum feedforward neural networks capable of universal quantum computation. We describe the efficient training of these networks using the fidelity as a cost function, providing both classical and efficient quantum implementations. Our method allows for fast optimisation with reduced memory requirements: the number of qudits required scales with only the width, allowing deep-network optimisation. We benchmark our proposal for the quantum task of learning an unknown unitary and find remarkable generalisation behaviour and a striking robustness to noisy training data.Neural networks enjoy widespread success in both research and industry and, with the advent of quantum technology, it is a crucial challenge to design quantum neural networks for fully quantum learning tasks. Here we propose a truly quantum analogue of classical neurons, which form quantum feedforward neural networks capable of universal quantum computation. We describe the efficient training of these networks using the fidelity as a cost function, providing both classical and efficient quantum implementations. Our method allows for fast optimisation with reduced memory requirements: the number of qudits required scales with only the width, allowing deep-network optimisation. We benchmark our proposal for the quantum task of learning an unknown unitary and find remarkable generalisation behaviour and a striking robustness to noisy training data.

arXiv:1902.10445Nature CommunicationsFeb 10, 2020726 citations

Next-Generation Optimization in Quantum and Hybrid Neural Frameworks

Sara Ali, Polla Fattah, Sanar Fawzi

The problem of the neural network optimization is local minima and flat areas. According to quantum-inspired methods, quantum mechanics, superposition, entanglement and interference are inspirational in order to possibly surpass these constraints in a more efficient manner. Variational quantum algorithms use parameterized quantum circuits parameterized quantum circuits (ansatz) to optimize weights. Quantum kernel methods also take advantage of quantum embeddings in order to transform features better. Hybrid quantum-classical systems incorporate quantum-inspired processing with already known reliability of optimization. There are, however, serious weaknesses that limit implementation. There are exponentially decreasing gradients due to the barren plateau phenomenon that makes it hard to train at scale. The modern quantum machines have a 0.1-1% error and a microsecond coherence time. The overheads in state preparation, the cost of measurement, circuit depth, and poor qubit connectivity are still impediments. The quantum-inspired classical algorithms, which include the tensor networks and optimization heuristics and run-on standard computers, are the most effective in the near future. These come with small advances in certain areas without involving quantum hardware. Although both quantum kernel methods and variational approaches show theoretical potential, the advantages are still small (relative to the advantages of optimized classical methods). Recent evaluations show that quantum benefits are more constrained than originally claimed them to be. Real quantum applications benefits in neural network optimization may only happen 10-20 years ahead, notwithstanding failure resistant quantum hardware and algorithm developments.The problem of the neural network optimization is local minima and flat areas. According to quantum-inspired methods, quantum mechanics, superposition, entanglement and interference are inspirational in order to possibly surpass these constraints in a more efficient manner. Variational quantum algorithms use parameterized quantum circuits parameterized quantum circuits (ansatz) to optimize weights. Quantum kernel methods also take advantage of quantum embeddings in order to transform features better. Hybrid quantum-classical systems incorporate quantum-inspired processing with already known reliability of optimization. There are, however, serious weaknesses that limit implementation. There are exponentially decreasing gradients due to the barren plateau phenomenon that makes it hard to train at scale. The modern quantum machines have a 0.1-1% error and a microsecond coherence time. The overheads in state preparation, the cost of measurement, circuit depth, and poor qubit connectivity are still impediments. The quantum-inspired classical algorithms, which include the tensor networks and optimization heuristics and run-on standard computers, are the most effective in the near future. These come with small advances in certain areas without involving quantum hardware. Although both quantum kernel methods and variational approaches show theoretical potential, the advantages are still small (relative to the advantages of optimized classical methods). Recent evaluations show that quantum benefits are more constrained than originally claimed them to be. Real quantum applications benefits in neural network optimization may only happen 10-20 years ahead, notwithstanding failure resistant quantum hardware and algorithm developments.

CRC Press eBooksMay 20, 20260 citations

O(1/n)

error for a training set of size

n

by performing sparse matrix inversion in

O(\log n)

time. To achieve an end-to-end exponential speedup over gradient descent, the data distribution must permit efficient state preparation and readout. We numerically demonstrate that the MNIST image dataset satisfies such conditions; moreover, the quantum algorithm matches the accuracy of the fully-connected network. Beyond the proven architecture, we provide empirical evidence for

O(\log n)

training of a convolutional neural network with pooling.