Knowledge distillation (KD) is a widely adopted technique for compressing large models into smaller, more efficient student models that can be deployed on devices with limited computational resources. Among various KD methods, Relational Knowledge Distillation (RKD) improves student performance by aligning relational structures in the feature space, such as pairwise distances and angles. In this work, we propose Quantum Relational Knowledge Distillation (QRKD), which extends RKD by incorporating quantum relational information. Specifically, we map classical features into a Hilbert space, interpret them as quantum states, and compute quantum kernel values to capture richer inter-sample relationships. These quantum-informed relations are then used to guide the distillation process. We evaluate QRKD on both vision and language tasks, including CNNs on MNIST and CIFAR-10, and GPT-2 on WikiText-2, Penn Treebank, and IMDB. Across all benchmarks, QRKD consistently improves student model performance compared to classical RKD. Importantly, both teacher and student models remain classical and deployable on standard hardware, with quantum computation required only during training. This work presents the first demonstration of quantum-enhanced knowledge distillation in a fully classical deployment setting.
Knowledge distillation (KD) is a widely adopted technique for compressing large models into smaller, more efficient student models that can be deployed on devices with limited computational resources. Among various KD methods, Relational Knowledge Distillation (RKD) improves student performance by aligning relational structures in the feature space, such as pairwise distances and angles. In this work, we propose Quantum Relational Knowledge Distillation (QRKD), which extends RKD by incorporating quantum relational information. Specifically, we map classical features into a Hilbert space, interpret them as quantum states, and compute quantum kernel values to capture richer inter-sample relationships. These quantum-informed relations are then used to guide the distillation process. We evaluate QRKD on both vision and language tasks, including CNNs on MNIST and CIFAR-10, and GPT-2 on WikiText-2, Penn Treebank, and IMDB. Across all benchmarks, QRKD consistently improves student model performance compared to classical RKD. Importantly, both teacher and student models remain classical and deployable on standard hardware, with quantum computation required only during training. This work presents the first demonstration of quantum-enhanced knowledge distillation in a fully classical deployment setting.
Photonic quantum processors naturally produce intrinsically stochastic measurement outcomes, offering a hardware-native source of structured randomness that can be exploited during machine-learning training. Here we introduce Photonic Quantum-Enhanced Knowledge Distillation (PQKD), a hybrid quantum photonic--classical framework in which a programmable photonic circuit generates a compact conditioning signal that constrains and guides a parameter-efficient student network during distillation from a high-capacity teacher. PQKD replaces fully trainable convolutional kernels with dictionary convolutions: each layer learns only a small set of shared spatial basis filters, while sample-dependent channel-mixing weights are derived from shot-limited photonic features and mapped through a fixed linear transform. Training alternates between standard gradient-based optimisation of the student and sampling-robust, gradient-free updates of photonic parameters, avoiding differentiation through photonic hardware. Across MNIST, Fashion-MNIST and CIFAR-10, PQKD traces a controllable compression--accuracy frontier, remaining close to teacher performance on simpler benchmarks under aggressive convolutional compression. Performance degrades predictably with finite sampling, consistent with shot-noise scaling, and exponential moving-average feature smoothing suppresses high-frequency shot-noise fluctuations, extending the practical operating regime at moderate shot budgets.
Abstract Quantum computing has emerged as a promising paradigm for addressing computational tasks intractable for classical systems, leveraging quantum mechanical principles such as superposition and entanglement to efficiently explore high-dimensional solution spaces. In recent years, hybrid quantum-classical approaches have gained increasing attention as a means to exploit the representational power of quantum systems while preserving the practicality of classical machine learning frameworks. This paper presents a study on quantum-enhanced model compression via feature-based knowledge distillation, in which a large classical neural network (the teacher) transfers knowledge to a substantially smaller hybrid student model. The proposed hybrid quantum-classical student architecture incorporates a parameterized quantum circuit (PQC) implemented in Qiskit, employing a linear $$\varvec{R_y}$$ angle-encoding feature map followed by an EfficientSU2 variational ansatz, with features bounded through a $$\varvec{\tanh (\cdot )\times \frac{\pi }{2}}$$ scaling to ensure stable gradient propagation. Gradient computation is performed via Reverse-Mode (adjoint) differentiation through the StatevectorEstimator primitive, which provides exact, shot-noise-free gradients at the cost of a single forward–backward pass per optimization step. The hybrid student is compared against a parameter-matched purely classical student under strict 1:1 parameter parity across a six-task binary benchmark spanning MNIST and Fashion-MNIST, as well as a four-class multi-class benchmark, totaling eleven classification experiments. A hardware validation experiment is additionally conducted on the IQM Garnet 20-qubit quantum processor using the Parameter-Shift Rule. The results indicate that the hybrid student outperforms its classical counterpart in five of six binary tasks (83%) and in the two hardest multi-class configurations, with advantages most pronounced in high-confusion regimes where compact classical classifiers approach the limits of their expressive capacity. On real quantum hardware, accuracy decreases from 99.04% in noiseless simulation to 80.50%, reflecting the impact of gate noise, readout errors, and transpilation overhead. While the hybrid model demonstrates improved parameter efficiency and expressive capacity, it incurs higher computational overhead and exhibits greater epoch-to-epoch training variability relative to classical baselines. We discuss practical limitations related to noise, scalability, and current hardware constraints, and identify directions for future work including hardware-efficient gradient estimation and advanced error mitigation protocols. Overall, our findings suggest that variational quantum circuits can serve as compact yet expressive components within neural compression schemes, with their representational advantage most effectively realized in binary and high-ambiguity classification scenarios.
Knowledge distillation is a widely explored technique in classical machine learning, in which a smaller or more efficient model is trained to mimic the behavior of a larger, more complex model. In this study, we extend the concept of knowledge distillation from classical architectures to quantum architectures, with the goal of improving the training of quantum models while potentially reducing the number of parameters compared to their classical counterparts. Given the inherent challenges in training quantum neural networks, leveraging knowledge from well-established classical models could provide valuable insights and advantages, particularly in terms of model efficiency and performance. In this work we explore the potential benefits of this approach, evaluating a hybrid quantum model against a non-hybrid quantum baseline. While the proposed study is still in the preliminary stage, it aims to set the scene for further investigation into the most appropriate architecture for classical-to-quantum knowledge distillation in order to enhance the development and optimization of quantum neural networks more generally.
As foundational tools in natural language processing, Large Language Models (LLMs) have immense parameter scales, which makes deployment and inference increasingly prohibitive, especially in resource-constrained devices. Therefore, knowledge distillation for LLMs, i.e., compressing the LLM to a smaller model, is meaningful. With strong parameter representation capacity, quantum computing is regarded as a promising solution. Here, we propose a Quantum knowledge Distillation model for LLMs (QD-LLM) that leverages variational quantum circuits to learn from LLMs. In classical simulation, QD-LLM outperforms several mainstream distillation methods on multiple text classification tasks in terms of both accuracy and efficiency using only 11 qubits. The results reveal an interesting phenomenon that the simulation of quantum student models may be regarded as a new class of quantum-inspired classical algorithms. Remarkably, we deploy the obtained circuits on the Baihua superconducting quantum processor via the Quafu platform to assess practical feasibility. The model maintains stable inference performance despite hardware constraints such as decoherence and finite sampling. In summary, QD-LLM marks a foundational step in connecting quantum computing with LLMs, demonstrating the feasibility of quantum-native approaches that aim to compress and deploy models of increasingly larger scales. The code of this article has been open-sourced at https://github.com/Lilingxiao-bupt/QD-LLM.
Point cloud upsampling is crucial for high-fidelity 3D reconstruction in real-time applications such as autonomous systems. Existing methods based on CNNs or Transformers face three limitations: (1) prohibitive computational complexity hindering real-time deployment, (2) insufficient modeling of multi-scale geometric dependencies in sparse data, (3) sensitivity to noise and outliers. To address these challenges, we propose QCTKD-PU, a framework integrating Quantum Convolutional Transformers (QCT) and Knowledge Distillation (KD) for Point cloud Upsampling. The QCT leverages quantum superposition and self-attention to encode high-dimensional features, enabling efficient multi-scale point interaction learning. Simultaneously, KD transfers knowledge from a teacher model to a lightweight student network, reducing computational costs while maintaining accuracy. Experiments on benchmark datasets demonstrate superior performance in geometric accuracy and noise robustness compared to state-of-the-art methods. This work pioneers the synergy of quantum computing and lightweight learning for resource-constrained 3D vision tasks, while the student model achieves real-time and compact deployment, offering a practical solution for collaborative edge systems.
While hybrid quantum–classical architectures and knowledge distillation have each been explored independently in medical imaging and model compression, no prior work has unified these domains within a single framework. This study addresses this gap by investigating the integration of quantum–classical neural networks with knowledge distillation for medical image analysis, with a focus on circuit behavior, parameter efficiency, and diagnostic performance. The framework (QuantumMedKD) employs classical CNN architectures (ResNet-50, EfficientNet-B0, Xception) as teacher models, transferring learned representations to parameter-efficient quantum student networks via parameterized quantum circuits across qubit configurations (3-8), demonstrated through pneumonia detection from pediatric chest radiographs. Experimental validation reveals remarkable parameter efficiency, requiring only 24-36 trainable parameters compared to millions for classical counterparts, achieving compression ratios exceeding 1 0 5 while maintaining competitive diagnostic performance. Optimal configurations achieve 84.00% accuracy (EfficientNet-B0, 6-qubit) and 73.33% (Xception, 4-qubit), with knowledge distillation providing statistically significant improvements ( p -vallues < 0.001, Cohen’s d > 2.0). Medical evaluation confirms diagnostic capability with sensitivity 81.4%, specificity 73.9%, and AUC 0.89, establishing quantum–classical knowledge transfer viability for resource-constrained healthcare deployment. QuantumMedKD reveals essential principles and proof-of-concept for quantum-enhanced healthcare AI systems within and beyond the noisy intermediate-scale quantum (NISQ) era for high efficiency and deployability, paving the way for future advancements in the field.