Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.
Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.
Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar quantization encodes weights independently, while explicit-codebook vector quantization introduces costly search during training. We propose E8-QAT, a 2-bit quantization-aware training framework that uses the E8 lattice as an implicit eight-dimensional codebook. It jointly quantizes weight groups through arithmetic projection without storing or searching an explicit codebook during the training forward pass. A 2-bit core encoding and sparse extension preserve the selected lattice points exactly, adding less than 0.1\% to core index storage. On Qwen3.6-35B-A3B, E8-QAT matches the performance of the full-precision model in terms of the average score across the four tasks. E8-QAT improves over the state-of-the-art baseline by 2.03\% on IFEval and by 1.51\% on LiveCodeBench, under the same data, schedule, and bit width. The 2-bit representation reduces peak memory by 80\% and supports a 4× larger batch. At each representation's largest feasible batch, E8-QAT reaches 1.97× the decoding throughput of BF16.
Low-density parity-check (LDPC) codes have been successfully commercialized in communication systems due to their strong error correction capabilities and simple decoding process. However, the error-floor phenomenon of LDPC codes, in which the error rate stops decreasing rapidly at a certain level, presents challenges for achieving extremely low error rates and deploying LDPC codes in scenarios demanding ultra-high reliability. In this work, we propose training methods for neural min-sum (NMS) decoders to eliminate the error-floor effect. First, by leveraging the boosting learning technique of ensemble networks, we divide the decoding network into two neural decoders and train the post decoder to be specialized for uncorrected words that the first decoder fails to correct. Secondly, to address the vanishing gradient issue in training, we introduce a block-wise training schedule that locally trains a block of weights while retraining the preceding block. Lastly, we show that assigning different weights to unsatisfied check nodes effectively lowers the error-floor with a minimal number of weights. By applying these training methods to standard LDPC codes, we achieve the best error-floor performance compared to other decoding methods. The proposed NMS decoder, optimized solely through novel training methods without additional modules, can be integrated into existing LDPC decoders without incurring extra hardware costs. The source code is available at https://github.com/ghy1228/LDPC_Error_Floor .
Post-training quantization (PTQ) is a primary approach for deploying large language models without fine-tuning, and the quantized performance is often strongly affected by the calibration in PTQ. By contrast, in vision-language models (VLMs), substantial differences between visual and text tokens in their activation distributions and sensitivities to quantization error pose significant challenges for effective calibration during PTQ. In this work, we rethink what PTQ calibration should align with in VLMs and propose the Token-level Importance-aware Layer-wise Quantization framework (TLQ). Guided by gradient information, we design a token-level importance integration mechanism for quantization error, and use it to construct a token-level calibration set, enabling a more fine-grained calibration strategy. Furthermore, TLQ introduces a multi-GPU, quantization-exposed layer-wise calibration scheme. This scheme keeps the layer-wise calibration procedure consistent with the true quantized inference path and distributes the complex layer-wise calibration workload across multiple RTX3090 GPUs, thereby reducing reliance on the large memory of A100 GPUs. TLQ is evaluated across two models, three model scales, and two quantization settings, consistently achieving performance improvements across all settings, indicating its strong quantization stability. The code will be released publicly.
RÉSUMÉ: Cette thèse présente la conception et l'évaluation d'un réseau neuronal adaptatif quantique (QEANN-QGD), qui intègre des paradigmes d'apprentissage quantique et classique pour traiter efficacement du données complexes. Le modèle exploite la descente de gradient quan- tique (QGD) et intègre une Fusion de caractéristiques hybride (HFI), où les caractéristiques améliorées par le quantum sont combinées avec des caractéristiques extraites à l'aide de méthodes classiques. Cette architecture utilise les forces des deux paradigmes, garantissant une performance d'apprentissage évolutive et adaptative. Notre approche été évaluée sur des ensembles de données de référence, ycompris MNIST et CIFAR-10, enutilisant diversestech- niques d'optimisation telles que SGD et Adam comme points de référence. Pour l'ensemble de données MNIST on a atteint une précision de 99,26% avec QEANN-QGD, comparé à 99,84% avec SGD et 99,97% avec Adam, tout en maintenant une faible valeur de perte de 0,0281. Pour CIFAR-10, le modèle atteint une précision de 82,57%, restant compétitif a plu- sieurs modèles classiques, avec une valeur finale de perte de 0,2791. Les couches contribuent considérablement à capturer des motifs non triviaux, reflétés dans les courbes de précision et la convergence progressive de l'énergie de l'état fondamental au cours de l'optimisation. De plus, des techniques de modélisation et de réduction du bruit ont été appliquées pour atténuer les erreurs quantiques et assurer la fiabilité des calculs quantiques. En utilisant les circuits quantiques variationnels et les techniques de décalage de paramètres ont permis une optimisation efficace des caractéristiques quantiques. Les résultats montrent que notre entraînement basé sur la QGD offreune robustesse et une capacité d'extraction de caractéris- tiques comparables aux méthodes classiques conventionnelles. Cela démontre la capacité des systèmes hybrides quantiques-classiques à s'adapter à des ensembles de données diversifiés avec des améliorations en termes de précision, rappel, score F1 et ROC-AUC. Cependant, l'architecture hybride introduit des défis tels qu'une surcharge computationnelle accrue et des besoins en qubits, qui sont discutés en détail. Les travaux futurs se concentreront sur le développement de circuits quantiques plus efficaces pour réduire l'utilisation des qubits et explorer des algorithmes quantiques alternatifs pour l'optimisation afin d'améliorer encore les performances du modèle. Cette étude fait progresser l'apprentissage machine quantique en proposant un réseau neuronal hybride et évolutif qui équilibre avec succès les composant quantiques et classiques, avec des applications potentielles dans des domaines nécessitant une extraction de caractéristiques améliorées et des capacités d'apprentissage adaptatives. ABSTRACT: This thesis presents the design and verification of a Quantum-Enhanced Adaptive Neural Network with quantum gradient descent (QEANN-QGD), which integrates quantum and classical learning paradigms to address complex data patterns effectively. The model lever- ages Quantum Gradient Descent(QGD) and incorporates Hybrid Feature Integration (HFI), where quantum-enhanced features are combined with classical features extracted through traditional methods. This architecture exploits the strengths of both paradigms, ensuring scalable and adaptive learning performance. The proposed approach is verified on bench- mark datasets, including MNIST and CIFAR-10, using various optimization techniquessuch as SGD and Adam as baselines. For the MNIST dataset, we achieve an accuracy of 99.26% with QEANN-QGD, compared to 99.84% with SGD and 99.97% with Adam, while main- taining a low loss value of 0.0281. For CIFAR-10, the model has 82.57% accuracy, remaining competitive with several classical models, with a final loss value of 0.2791. The quantum layers significantly contributes to capturing non-trivial patterns, reflected in the gradual convergence of the ground-state energy during optimization. In addition, noise reduction techniques are applied to mitigate quantum errors and ensure the reliability of the quantum computations. Using variational quantum circuits and parameter-shift techniques enable ef- fective optimization of quantum features. The results show that our QGD-based training provides comparable robustnessand feature extraction capabilities to conventional classical methods. But the classical counterpart showed better accuracy and precision when compared to hyprid model. The hybrid architecture introduces challenges such as increased computa- tional overhead and qubit requirements, which are discussed in detail. Future work will focus on developing more efficient quantum circuit designs to reduce gthe number of qubit used and exploring alternative quantum algorithms for optimization to further enhance model per- formance. This study advances quantum machine learning by proposing a scalable, hybrid neural network that successfully balances quantum and classical components, with potential applications in domains requiring enhanced feature extraction and adaptive learning capa- bilities.
The burgeoning complexity and scale of 3D geometry models pose significant challenges for deployment on resource-constrained platforms. While Post-Training Quantization (PTQ) enables efficient inference without retraining, conventional methods, primarily optimized for 2D Vision Transformers, fail to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead. To address these challenges, we propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline specifically engineered for 3D geometric learning. Our contribution is threefold: (1) To overcome the data-scale bottleneck in 3D datasets, we develop a progressive coarse-to-fine calibration construction strategy that constructs a highly compact subset to achieve both statistical purity and geometric representativeness. (2) We reformulate the quantization interval search as an optimization problem and introduce a ternary-search-based solver, reducing the computational complexity from O(N) to O(logN) for accelerated deployment. (3) To mitigate quantization error accumulation, we propose TRE-Guided Module-wise Compensation, which utilizes a Tail Relative Error (TRE) metric to adaptively identify and rectify distortions in modules sensitive to long-tailed activation outliers. Extensive experiments on the VGGT and Pi3 benchmarks demonstrate that TAPTQ consistently outperforms state-of-the-art PTQ methods in accuracy while significantly reducing calibration time. The code will be released soon.