Self-Attention Mechanisms (SAMs) enhance model performance by extracting key information but are inefficient when dealing with long sequences. To this end, a photonic Gaussian Kernel Enhanced Transformer (PGKET) is proposed, based on the Photonic Gaussian Kernel Self-Attention Mechanism (PGKSAM). The PGKSAM calculates the Photonic Gaussian Kernel Self-Attention Score (PGKSAS) using photon interferometry and superposition to process multiple inputs in parallel. Experimental results show that PGKET outperforms some state-of-the-art transformers in multi-classification tasks on MedMNIST v2 and CIFAR-10, and is expected to improve performance in complex tasks and accelerate the convergence of Photonic Computing (PC) and machine learning.
Self-Attention Mechanisms (SAMs) enhance model performance by extracting key information but are inefficient when dealing with long sequences. To this end, a photonic Gaussian Kernel Enhanced Transformer (PGKET) is proposed, based on the Photonic Gaussian Kernel Self-Attention Mechanism (PGKSAM). The PGKSAM calculates the Photonic Gaussian Kernel Self-Attention Score (PGKSAS) using photon interferometry and superposition to process multiple inputs in parallel. Experimental results show that PGKET outperforms some state-of-the-art transformers in multi-classification tasks on MedMNIST v2 and CIFAR-10, and is expected to improve performance in complex tasks and accelerate the convergence of Photonic Computing (PC) and machine learning.
The Self-Attention Mechanism (SAM) excels at distilling important information from the interior of data to improve the computational efficiency of models. Nevertheless, many Quantum Machine Learning (QML) models lack the ability to distinguish the intrinsic connections of information like SAM, which limits their effectiveness on massive high-dimensional quantum data. To tackle the above issue, a Quantum Kernel Self-Attention Mechanism (QKSAM) is introduced to combine the data representation merit of Quantum Kernel Methods (QKM) with the efficient information extraction capability of SAM. Further, a Quantum Kernel Self-Attention Network (QKSAN) framework is proposed based on QKSAM, which ingeniously incorporates the Deferred Measurement Principle (DMP) and conditional measurement techniques to release half of quantum resources by mid-circuit measurement, thereby bolstering both feasibility and adaptability. Simultaneously, the Quantum Kernel Self-Attention Score (QKSAS) with an exponentially large characterization space is spawned to accommodate more information and determine the measurement conditions. Eventually, four QKSAN sub-models are deployed on PennyLane and IBM Qiskit platforms to perform binary classification on MNIST and Fashion MNIST, where the QKSAS tests and correlation assessments between noise immunity and learning ability are executed on the best-performing sub-model. The paramount experimental finding is that the QKSAN subclasses possess the potential learning advantage of acquiring impressive accuracies exceeding 98.05% with far fewer parameters than classical machine learning models. Predictably, QKSAN lays the foundation for future quantum computers to perform machine learning on massive amounts of data while driving advances in areas such as quantum computer vision.
Quantum computing, with its unique principles such as superposition and entanglement, promises to transform various domains, including artificial intelligence. This study investigates the fusion of quantum computing with contemporary AI technologies through quantum transfer learning. We introduce the Complex Quantum Kernel Self-Attention Network (CQKSAN), a novel model that leverages quantum circuits and self-attention mechanisms to boost text classification performance. By applying quantum transfer learning, CQKSAN harnesses the strengths of pre-trained classical models and enhances them with quantum capabilities. Experimental results on multiple binary classification datasets demonstrate that CQKSAN achieves a more efficient learning process. Specifically, it improves training accuracy and validation accuracy by up to 24.31% and 28.35%, respectively, and increases the Matthews Correlation Coefficient by 113.10%. Moreover, CQKSAN consistently outperforms the BERT baseline in both training and validation accuracy across various datasets, showcasing its strong learning capability and generalization performance in handling tasks with diverse linguistic structures and complex semantics. Our findings underscore the potential of hybrid quantum-classical approaches, while also identifying the current limitations and paving the way for future research in fully quantum-based learning systems.
Recently, machine learning has had remarkable impact in scientific to everyday-life applications. However, complex tasks often require the consumption of unfeasible amounts of energy and computational power. Quantum computation may lower such requirements, although it is unclear whether enhancements are reachable with current technologies. Here we demonstrate a kernel method on a photonic integrated processor to perform a binary classification task. We show that our protocol outperforms state-of-the-art kernel methods such as gaussian and neural tangent kernels by exploiting quantum interference, and provides further improvements in accuracy by offering single-photon coherence. Our scheme does not require entangling gates and can modify the system dimension through additional modes and injected photons. This result gives access to more efficient algorithms and to formulating tasks where quantum effects improve standard methods.
The recent exploding growth in size of state-of-the-art machine learning models highlights a well-known issue where exponential parameter growth, which has grown to trillions as in the case of the Generative Pre-trained Transformer (GPT), leads to training time and memory requirements which limit their advancement in the near term. The predominant models use the so-called transformer network and have a large field of applicability, including predicting text and images, classification, and even predicting solutions to the dynamics of physical systems. Here we present a variational quantum circuit architecture named Self-Attention Sequential Quantum Transformer Channel (SASQuaTCh), which builds networks of qubits that perform analogous operations of the transformer network, namely the keystone self-attention operation, and leads to an exponential improvement in parameter complexity and run-time complexity over its classical counterpart. Our approach leverages recent insights from kernel-based operator learning in the context of predicting spatiotemporal systems to represent deep layers of a vision transformer network using simple gate operations and a set of multi-dimensional quantum Fourier transforms. To validate our approach, we consider image classification tasks in simulation and with hardware, where with only 9 qubits and a handful of parameters we are able to simultaneously embed and classify a grayscale image of handwritten digits with high accuracy.
Quantum self-attention is a promising approach to replace traditional self-attention mechanisms in classical deep learning, which often require a massive number of GPUs for training. Although several early studies have proposed quantum self-attention architectures, these models often struggle with classification tasks in vision transformers. In this study, we address this limitation by enhancing quantum self-attention through integration with circuit classifier design and, crucially, by optimizing the number of strong entangling layers. Evaluation results show that incorporating an optimal number of entangling layers substantially improves classification performance, outperforming models that rely solely on directly measured quantum self-attention. We further analyze how varying the number of quantum classifier layers affects classification accuracy during training. Experiments on downscaled MNIST and MC datasets reveal that the nature of the input data has minimal impact on the final outcomes. Notably, testing accuracy reaches 56% when using 10–16 strong entangling layers, highlighting the critical role of entanglement in model performance. For SOTA comparison, we replicate existing quantum self-attention methods and demonstrate that our model consistently achieves 10-23% better performance results. Additionally, we compare different optimizers and find that Nesterov Momentum delivers the most effective convergence for our architecture.
The success of the self-attention mechanism in classical machine learning models has inspired the development of quantum analogs aimed at reducing the computational overhead. Self-attention integrates learnable query and key matrices to calculate attention scores between all pairs of tokens in a sequence. These scores are then multiplied by a learnable value matrix to obtain the output self-attention matrix, enabling the model to effectively capture long-range dependencies within the input sequence. Here, we propose a hybrid quantum-classical self-attention mechanism as part of a transformer decoder, the architecture underlying large language models (LLMs). To demonstrate its utility in chemistry, we train this model on the QM9 dataset for conditional generation, using SMILES strings as input, each labeled with a set of physicochemical properties that serve as conditions during inference. Our theoretical analysis shows that the time complexity of the query-key dot product is reduced from O ( n 2 d ) in a classical model to O ( n 2 log d ) in our quantum model, where n and d represent the sequence length and the embedding dimension, respectively. We perform simulations using NVIDIA’s CUDA-Q platform, which is designed for efficient GPU scalability. This work provides a promising avenue for quantum-enhanced natural language processing (NLP).