Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2606.11673

Higher-Order Token Interactions via Quantum Attention

Jian Xu, Chao Li, Delu Zeng, John Paisley, Qibin Zhao

quant-phcs.LGSubmitted Jun 10, 2026• Announced Jun 11, 2026• 0 citations
v1 · Jun 10, 2026v2 · Jun 18, 2026
arXivHTMLPDF

Abstract

Standard dot-product self-attention computes, in a single layer, only pairwise (order-2) interactions between tokens; representing a generic order-$k$ interaction is known to require either super-quadratic resources in one layer or composition across depth. We introduce \textbf{Quantum Higher-Order Attention (QHA)}, a shallow, hardware-realizable quantum attention head that, via data re-uploading and an all-to-all non-Clifford entangler, synthesizes order-$k$ token interactions inside the circuit and exposes them through a local single-qubit read-out. We prove (i) an expressivity separation: any single standard self-attention layer with embedding dimension $m$, $H$ heads and $p$-bit precision satisfying $mHp=o(N/\log\log N)$ cannot represent the order-$k$ correlation family that one QHA head represents with circuit depth $O(\log k)$ ($O(k)$ two-qubit gates); and (ii) a trainability guarantee for its local-design instantiation: with a local read-out and $O(\log n)$ depth the gradient variance is $Ω(1/\mathrm{poly}(n))$ (no barren plateau), which we confirm empirically -- while being explicit that the more expressive all-to-all instantiation we benchmark is trained empirically and shows exponentially decaying gradients. Empirically, at a $6.5\times$ smaller parameter budget, QHA generalizes hidden-subset parity of every order $k\le6$ from disjoint inputs, whereas the larger classical attention head collapses past order~2; consistent with theory, the size of the advantage tracks the target's Fourier degree - largest for parity and shrinking when low-order structure is present. As an application, QHA serves as a compact high-order interaction detector across three domains - genetic epistasis, learning-parity-with-noise, and graph triangle detection - reaching the noise ceiling at the smallest parameter budget where field-standard linear methods fail.

Related papers

Quantum Adaptive Self-Attention for quantum Transformer models

Chi‐Sheng Chen, En-Jui Kuo

Abstract A recurring weakness in quantum machine learning (QML) is that reported ‘quantum advantages’ are seldom tested against a capacity-matched classical control, leaving it unclear whether a gain comes from the quantum substrate or from the architectural change that accompanies it. Our primary contribution is methodological: a protocol for attributing such gains honestly—a capacity-matched classical bottleneck of identical parameter budget, transparent reporting of where quantum does not help, and validation on real quantum hardware—which we develop and apply through a concrete case study. That case study is Quantum Adaptive Self-Attention (QASA), a hybrid Transformer that replaces the value projection of a single encoder layer with a 36-parameter parameterized quantum circuit (PQC), keeping all other layers classical. Across nine synthetic benchmarks and the real-world ETTh1 dataset, QASA improves on a full-capacity classical Transformer for chaotic and trend-dominated signals. To ask whether this is a genuinely quantum effect, we introduce a control rarely applied in QML—a capacity-matched classical bottleneck with the same parameter budget—and find that it matches the PQC on the error metrics. The gain is therefore attributable to the low-rank value-projection bottleneck (an architectural parsimony principle), not to quantumness; adding further quantum layers only degrades performance and trainability. We accordingly position the quantum layer not as a source of accuracy advantage but as a competitive instantiation of this principle: its low-rank compression onto the signal’s intrinsic dimensionality is matched by a classical bottleneck, so the gain is architectural rather than quantum. The quantum layer’s distinguishing features are physical—a high circuit entanglement (Meyer–Wallach Q = 0.981 with only 27 CNOTs, which we report as a circuit-level property) and, most concretely, noisy intermediate-scale quantum deployability, which we verify by executing the trained model on a real IBM Quantum processor (one-step prediction within 7.5 % of noiseless simulation on the quantum-favored task and 25 % , with overlapping error bars, on a classical-favored control; no error mitigation). Relative to other quantum sequence models (quantum long short-term memory (QLSTM), QnnFormer), QASA is moreover the most resource-efficient of the three: it reaches competitive accuracy with the fewest quantum parameters (36 vs 90–128) and the strongest entanglement-per-gate ( Q = 0.981 with 27 CNOTs vs QLSTM’s 56), and retains trainable gradients that deeper circuits lose. We argue that capacity-matched baselines and honest reporting of where quantum does not help are prerequisites for credible quantum-machine-learning claims.

arXiv:2504.05336•Quantum Science and Technology•Jul 15, 2026•2 citations

Quantum-Enhanced Attention Mechanism in NLP: A Hybrid Classical-Quantum Approach

S. M. Yousuf Iqbal Tomal, Abdullah Al Shafin, Bhattacharjee, Debojit +2 more

Recent advances in quantum computing have opened new pathways for enhancing deep learning architectures, particularly in domains characterized by high-dimensional and context-rich data such as natural language processing (NLP). In this work, we present a hybrid classical-quantum Transformer model that integrates a quantum-enhanced attention mechanism into the standard classical architecture. By embedding token representations into a quantum Hilbert space via parameterized variational circuits and exploiting entanglement-aware kernel similarities, the model captures complex semantic relationships beyond the reach of conventional dot-product attention. We demonstrate the effectiveness of this approach across diverse NLP benchmarks, showing improvements in both efficiency and representational capacity. The results section reveal that the quantum attention layer yields globally coherent attention maps and more separable latent features, while requiring comparatively fewer parameters than classical counterparts. These findings highlight the potential of quantum-classical hybrid models to serve as a powerful and resource-efficient alternative to existing attention mechanisms in NLP.

arXiv:2501.15630•Jan 26, 2025•3 citations

Do Quantum Transformers Help? A Systematic VQC Architecture Comparison on Tabular Benchmarks

Chi-Sheng Chen, En-Jui Kuo

Variational quantum circuits (VQCs) are a leading approach to quantum machine learning on near-term devices, yet it remains unclear which circuit architecture yields the best accuracy-parameter trade-off on classical tabular data. We present a systematic empirical comparison of four VQC families -- multi-layer fully-connected (FC-VQC), residual (ResNet-VQC), hybrid quantum-classical transformer (QT), and fully quantum transformer (FQT) -- across five regression and classification benchmarks. Our key findings are: \textbf{(i)}~FC-VQCs achieve 90-96\% of the $R^2$ of attention-based VQCs while using 40-50\% fewer parameters, and consistently outperform equal-capacity MLPs (mean $R^2{=}0.829$ vs.\ MLP$_{720}$'s $0.753$ on Boston Housing, 3-seed average); \textbf{(ii)}~FC-VQC's Type~4 inter-block connectivity provides partial cross-token mixing that approximates the role of attention -- explicit quantum self-attention yields only marginal gains on most datasets while significantly increasing parameter count; \textbf{(iii)}~expressibility saturates at circuit depth~${\approx}\,3$, explaining why shallow VQCs already cover the Hilbert space effectively; \textbf{(iv)}~LayerNorm on the fully quantum transformer improves classification accuracy, suggesting normalization is important when all operations are quantum; \textbf{(v)}~in our noise study on Boston Housing, FQT degrades gracefully under depolarizing noise while QT collapses. All results are validated across three random seeds. These findings provide practical architectural guidance for deploying VQCs on near-term quantum hardware.

quant-pharXiv:2604.23931•Apr 28, 2026•0 citations

QAMA: Scalable Quantum Annealing Multi-Head Attention Operator for Deep Learning

Du, Peng, Jinjing Shi, Wenxuan Wang +3 more

Attention mechanisms underpin modern deep learning, while the quadratic time and space complexity limit scalability for long sequences. To address this, Quantum Annealing Multi-Head Attention (QAMA) is proposed, a novel drop-in operator that reformulates attention as an energy-based Hamiltonian optimization problem. In this framework, token interactions are encoded into binary quadratic terms, and quantum annealing is employed to search for low-energy configurations that correspond to effective attention patterns. Unlike classical sparse or approximate attention methods that rely on hand-crafted heuristics, QAMA allows sparsity structures to emerge naturally from the optimization process. Theoretically, computational complexity is analysed through single-spin flip dynamics, providing time to solution runtime bounds that depend on the spectral properties of the annealing Hamiltonian. Empirically, evaluation on both natural language and vision benchmarks shows that, across tasks, accuracy deviates by at most 2.7 points from standard multi-head attention, while requiring only linear qubits in sequence length. Visualizations further reveal that the Hamiltonian penalty terms induce meaningful and interpretable sparsity across heads. Finally, deployment on a coherent Ising machine validates the feasibility of running QAMA on real quantum hardware, showing tangible inference-time reductions compared with classical implementations. These results highlight QAMA as a pioneering and scalable step toward integrating quantum optimization devices into deep neural architectures, providing a seamlessly integrable and hardware-compatible alternative to conventional attention mechanisms. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

arXiv:2504.11083•Apr 15, 2025•0 citations

Quantum Interference Transformer (QIT): Emergent Sequence Intelligence from Amplitude Dynamics

Mihail Alexandru Stancescu

This working paper introduces the Quantum Interference Transformer (QIT), a sequence model in which token relationships are resolved by constructive and destructive interference in a parameterised quantum circuit, rather than by classical dot-product attention. We present QIT-0, a minimal PennyLane/PyTorch prototype. Tokens are encoded as rotation angles, cross-token correlations are seeded by a ring CNOT ladder, a learned strongly-entangling unitary acts as the attention operator, and Pauli-Z expectation values are decoded by a linear layer. On a 4-bit parity memorisation benchmark (16 inputs; 5 random seeds), QIT-0 (78 parameters) reaches 100% accuracy in 6.2 ± 1.7 epochs. Classical baselines with more parameters converge more slowly or fail within the epoch budget. Additional experiments include ablations, a first-token negative control, and related structured tasks. This is an independent technical report, not a peer-reviewed publication. We do not claim general quantum advantage for machine learning. Code: https://github.com/MihailStancescu/qit

•May 16, 2026•0 citations

Quantum Transformer BSDE Solver via Multi-Layer Fully-Connected Variational Quantum Circuits

Howard Su, Huan-Hsin Tseng, Chi-Sheng Chen +1 more

Solving high-dimensional parabolic partial differential equations (PDEs) is important in engineering, physics, and stochastic control. Deep BSDE methods reformulate semilinear PDEs as backward stochastic differential equations and admit a model-based reinforcement learning interpretation, where trajectories are generated from known stochastic dynamics while a trainable model learns the gradient-related control process. We propose a Quantum Transformer BSDE solver based on Multi-Layer Fully-Connected Variational Quantum Circuits (FC-VQC). The method treats the normalized state trajectory as time--coordinate tokens and applies causal self-attention to learn interactions in the adapted BSDE gradient process. All trainable model parameters are contained within the FC-VQC embedding, projection, feed-forward, and decoder modules, while attention and structural operations remain classical and parameter-free. Experiments on three d=36 PDE benchmarks show that QTransformer consistently improves over the non-attentive FC-VQC baseline and outperforms the classical Transformer at compact hidden widths, while the wider classical Transformer achieves the best overall accuracy. These results demonstrate that combining causal attention with FC-VQC provides an effective quantum architecture for high-dimensional BSDE trajectory learning.

quant-pharXiv:2607.25162•Jul 29, 2026•0 citations
Standard dot-product self-attention computes, in a single layer, only pairwise (order-2) interactions between tokens; representing a generic order-kkk interaction is known to require either super-quadratic resources in one layer or composition across depth. We introduce \textbf{Quantum Higher-Order Attention (QHA)}, a shallow, hardware-realizable quantum attention head that, via data re-uploading and an all-to-all non-Clifford entangler, synthesizes order-kkk token interactions inside the circuit and exposes them through a local single-qubit read-out. We prove (i) an expressivity separation: any single standard self-attention layer with embedding dimension mmm, HHH heads and ppp-bit precision satisfying mHp=o(N/log⁡log⁡N)mHp=o(N/\log\log N)mHp=o(N/loglogN) cannot represent the order-kkk correlation family that one QHA head represents with circuit depth O(log⁡k)O(\log k)O(logk) (O(k)O(k)O(k) two-qubit gates); and (ii) a trainability guarantee for its local-design instantiation: with a local read-out and O(log⁡n)O(\log n)O(logn) depth the gradient variance is Ω(1/poly(n))Ω(1/\mathrm{poly}(n))Ω(1/poly(n)) (no barren plateau), which we confirm empirically -- while being explicit that the more expressive all-to-all instantiation we benchmark is trained empirically and shows exponentially decaying gradients. Empirically, at a 6.5×6.5\times6.5× smaller parameter budget, QHA generalizes hidden-subset parity of every order k≤6k\le6k≤6 from disjoint inputs, whereas the larger classical attention head collapses past order~2; consistent with theory, the size of the advantage tracks the target's Fourier degree - largest for parity and shrinking when low-order structure is present. As an application, QHA serves as a compact high-order interaction detector across three domains - genetic epistasis, learning-parity-with-noise, and graph triangle detection - reaching the noise ceiling at the smallest parameter budget where field-standard linear methods fail.
Abstract A recurring weakness in quantum machine learning (QML) is that reported ‘quantum advantages’ are seldom tested against a capacity-matched classical control, leaving it unclear whether a gain comes from the quantum substrate or from the architectural change that accompanies it. Our primary contribution is methodological: a protocol for attributing such gains honestly—a capacity-matched classical bottleneck of identical parameter budget, transparent reporting of where quantum does not help, and validation on real quantum hardware—which we develop and apply through a concrete case study. That case study is Quantum Adaptive Self-Attention (QASA), a hybrid Transformer that replaces the value projection of a single encoder layer with a 36-parameter parameterized quantum circuit (PQC), keeping all other layers classical. Across nine synthetic benchmarks and the real-world ETTh1 dataset, QASA improves on a full-capacity classical Transformer for chaotic and trend-dominated signals. To ask whether this is a genuinely quantum effect, we introduce a control rarely applied in QML—a capacity-matched classical bottleneck with the same parameter budget—and find that it matches the PQC on the error metrics. The gain is therefore attributable to the low-rank value-projection bottleneck (an architectural parsimony principle), not to quantumness; adding further quantum layers only degrades performance and trainability. We accordingly position the quantum layer not as a source of accuracy advantage but as a competitive instantiation of this principle: its low-rank compression onto the signal’s intrinsic dimensionality is matched by a classical bottleneck, so the gain is architectural rather than quantum. The quantum layer’s distinguishing features are physical—a high circuit entanglement (Meyer–Wallach Q = 0.981 with only 27 CNOTs, which we report as a circuit-level property) and, most concretely, noisy intermediate-scale quantum deployability, which we verify by executing the trained model on a real IBM Quantum processor (one-step prediction within 7.5 % of noiseless simulation on the quantum-favored task and 25 % , with overlapping error bars, on a classical-favored control; no error mitigation). Relative to other quantum sequence models (quantum long short-term memory (QLSTM), QnnFormer), QASA is moreover the most resource-efficient of the three: it reaches competitive accuracy with the fewest quantum parameters (36 vs 90–128) and the strongest entanglement-per-gate ( Q = 0.981 with 27 CNOTs vs QLSTM’s 56), and retains trainable gradients that deeper circuits lose. We argue that capacity-matched baselines and honest reporting of where quantum does not help are prerequisites for credible quantum-machine-learning claims.
Recent advances in quantum computing have opened new pathways for enhancing deep learning architectures, particularly in domains characterized by high-dimensional and context-rich data such as natural language processing (NLP). In this work, we present a hybrid classical-quantum Transformer model that integrates a quantum-enhanced attention mechanism into the standard classical architecture. By embedding token representations into a quantum Hilbert space via parameterized variational circuits and exploiting entanglement-aware kernel similarities, the model captures complex semantic relationships beyond the reach of conventional dot-product attention. We demonstrate the effectiveness of this approach across diverse NLP benchmarks, showing improvements in both efficiency and representational capacity. The results section reveal that the quantum attention layer yields globally coherent attention maps and more separable latent features, while requiring comparatively fewer parameters than classical counterparts. These findings highlight the potential of quantum-classical hybrid models to serve as a powerful and resource-efficient alternative to existing attention mechanisms in NLP.
Variational quantum circuits (VQCs) are a leading approach to quantum machine learning on near-term devices, yet it remains unclear which circuit architecture yields the best accuracy-parameter trade-off on classical tabular data. We present a systematic empirical comparison of four VQC families -- multi-layer fully-connected (FC-VQC), residual (ResNet-VQC), hybrid quantum-classical transformer (QT), and fully quantum transformer (FQT) -- across five regression and classification benchmarks. Our key findings are: \textbf{(i)}~FC-VQCs achieve 90-96\% of the R2R^2R2 of attention-based VQCs while using 40-50\% fewer parameters, and consistently outperform equal-capacity MLPs (mean R2=0.829R^2{=}0.829R2=0.829 vs.\ MLP720_{720}720​'s 0.7530.7530.753 on Boston Housing, 3-seed average); \textbf{(ii)}~FC-VQC's Type~4 inter-block connectivity provides partial cross-token mixing that approximates the role of attention -- explicit quantum self-attention yields only marginal gains on most datasets while significantly increasing parameter count; \textbf{(iii)}~expressibility saturates at circuit depth~≈ 3{\approx}\,3≈3, explaining why shallow VQCs already cover the Hilbert space effectively; \textbf{(iv)}~LayerNorm on the fully quantum transformer improves classification accuracy, suggesting normalization is important when all operations are quantum; \textbf{(v)}~in our noise study on Boston Housing, FQT degrades gracefully under depolarizing noise while QT collapses. All results are validated across three random seeds. These findings provide practical architectural guidance for deploying VQCs on near-term quantum hardware.
Attention mechanisms underpin modern deep learning, while the quadratic time and space complexity limit scalability for long sequences. To address this, Quantum Annealing Multi-Head Attention (QAMA) is proposed, a novel drop-in operator that reformulates attention as an energy-based Hamiltonian optimization problem. In this framework, token interactions are encoded into binary quadratic terms, and quantum annealing is employed to search for low-energy configurations that correspond to effective attention patterns. Unlike classical sparse or approximate attention methods that rely on hand-crafted heuristics, QAMA allows sparsity structures to emerge naturally from the optimization process. Theoretically, computational complexity is analysed through single-spin flip dynamics, providing time to solution runtime bounds that depend on the spectral properties of the annealing Hamiltonian. Empirically, evaluation on both natural language and vision benchmarks shows that, across tasks, accuracy deviates by at most 2.7 points from standard multi-head attention, while requiring only linear qubits in sequence length. Visualizations further reveal that the Hamiltonian penalty terms induce meaningful and interpretable sparsity across heads. Finally, deployment on a coherent Ising machine validates the feasibility of running QAMA on real quantum hardware, showing tangible inference-time reductions compared with classical implementations. These results highlight QAMA as a pioneering and scalable step toward integrating quantum optimization devices into deep neural architectures, providing a seamlessly integrable and hardware-compatible alternative to conventional attention mechanisms. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.
This working paper introduces the Quantum Interference Transformer (QIT), a sequence model in which token relationships are resolved by constructive and destructive interference in a parameterised quantum circuit, rather than by classical dot-product attention. We present QIT-0, a minimal PennyLane/PyTorch prototype. Tokens are encoded as rotation angles, cross-token correlations are seeded by a ring CNOT ladder, a learned strongly-entangling unitary acts as the attention operator, and Pauli-Z expectation values are decoded by a linear layer. On a 4-bit parity memorisation benchmark (16 inputs; 5 random seeds), QIT-0 (78 parameters) reaches 100% accuracy in 6.2 ± 1.7 epochs. Classical baselines with more parameters converge more slowly or fail within the epoch budget. Additional experiments include ablations, a first-token negative control, and related structured tasks. This is an independent technical report, not a peer-reviewed publication. We do not claim general quantum advantage for machine learning. Code: https://github.com/MihailStancescu/qit
Solving high-dimensional parabolic partial differential equations (PDEs) is important in engineering, physics, and stochastic control. Deep BSDE methods reformulate semilinear PDEs as backward stochastic differential equations and admit a model-based reinforcement learning interpretation, where trajectories are generated from known stochastic dynamics while a trainable model learns the gradient-related control process. We propose a Quantum Transformer BSDE solver based on Multi-Layer Fully-Connected Variational Quantum Circuits (FC-VQC). The method treats the normalized state trajectory as time--coordinate tokens and applies causal self-attention to learn interactions in the adapted BSDE gradient process. All trainable model parameters are contained within the FC-VQC embedding, projection, feed-forward, and decoder modules, while attention and structural operations remain classical and parameter-free. Experiments on three d=36 PDE benchmarks show that QTransformer consistently improves over the non-attentive FC-VQC baseline and outperforms the classical Transformer at compact hidden widths, while the wider classical Transformer achieves the best overall accuracy. These results demonstrate that combining causal attention with FC-VQC provides an effective quantum architecture for high-dimensional BSDE trajectory learning.