Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.
Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.
Post-training quantization (PTQ) is widely used to compress large language models without retraining. However, many existing weight-only methods rely on heuristic objectives and greedy rounding strategies, leading to noticeable degradation under low-bit quantization. We propose OJBKQ, a layer-wise PTQ method that formulates weight quantization as a joint optimization problem over activations and weights, yielding a multiple-right-hand-side box-constrained integer least squares (BILS) problem per layer. To solve this NP-hard problem, we extend Klein’s randomized variant of Babai’s nearest-plane algorithm to handle box constraints and run the extended algorithm K times to generate K box-constrained Klein points. The minimum-residual candidate among these points and the box-constrained Babai point is selected as a suboptimal solution. Experiments on large language models show that OJBKQ achieves lower perplexity at 3–4 bits than existing PTQ methods, while maintaining comparable computational cost.
This thesis is divided into two parts. In the first part, we introduce a novel capacity measure 2sED for statistical models based on the effective dimension. This new quantity provably bounds the generalization error under mild assumptions on the model. Furthermore, simulations on standard data sets and popular model architectures show that 2sED correlates well with the training error. For Markovian models, we show how to efficiently approximate 2sED from below through a layer-wise iterative approach, which allows us to tackle deep learning models with a large number of parameters. Simulation results suggest that the approximation is good for different prominent models and data sets. In the second part, we present GeoPTQ, a novel approach to post-training quantization. Standard quantization methods often fail to minimize performance loss optimally. We precisely formulate the weight post-training quantization problem, showing that conventional techniques can be suboptimal, particularly for linear models. By leveraging the Gauss-Newton-induced distance induced by the problem, we quantitatively (in the model dimension) bound the quantization error in the NTK framework, that is scaled neural networks obtained via gradient descend training with residual error starting gaussian i.i.d initialization. Furthermore, we introduce a theoretically grounded quantization algorithm that preserves generalization performance while minimizing quantization error. Experimental results confirm that GeoPTQ consistently outperforms traditional methods across a range of models and datasets. Even tough these two works address different machine learning topics, they share a common approach of examining the problem’s geometry to understand how model complexity and parameters influence performance.
The best two-bit weight quantizers for large language models, such as QTIP and Proteus, rotate each weight matrix by a random orthogonal transform, which must be undone at every decoding step, then encode it with a trellis or lattice code under a Euclidean search; the layer Hessian enters only through error feedback between coding blocks. We show that this leaves part of the Hessian unused. Error feedback turns the loss into a weighted sum of per-coordinate rounding errors whose weights, the diagonal of the Hessian's LDL factorization, existing quantizers compute but never read. We put these weights into the Viterbi branch metric, so the search follows the curvature within each coding block. This also explains the rotation: it removes this within-block variation, so weighting in the native basis and rotating are substitutes. On three models the weighted native search matches a full-dimension randomized Hadamard to within about one point of downstream accuracy, and weighting after the rotation gains little. Around this search we build CurveTQ, a trellis codec with no rotation, which handles the weights' amplitude and marginal shape with a factored scale field and a closed-form quantile table, and stores a start state per coding block so the trellis can adapt to the residual that error feedback carries into it. At two bits CurveTQ is 1-3 points higher in mean downstream accuracy than QTIP and Proteus on three 4-8B Instruct models, even after both are given our start state, which alone lifts either baseline by 1-3 points. It also leads on a 35B mixture of experts, to our knowledge the first trellis-coded result on such a model. With no rotation to undo, our decoder is the fastest of the three at all tested batch sizes and bit widths.
The growing need for privacy protection, instant response, and offline access has pushed large language model inference from the cloud to local edge devices. However, decoder-only LLMs remain hard to run on smartphones and embedded systems, as their performance is limited not just by parameter count, but also by memory bandwidth, hardware operator compatibility, and the rising cost of long-context generation. Post-training quantization (PTQ) is the most viable approach today, as it shrinks model size and speeds up inference without full retraining. Yet most PTQ methods are tested on cloud GPUs, and their claimed improvements rarely hold up on real edge hardware. This survey reviews on-device PTQ for decoder-only LLMs from a hardware-focused angle, covering static and adaptive schemes, key hardware bottlenecks, evaluation flaws, and KV cache scaling issues. It concludes that model size alone is a poor measure of edge efficiency, and future work must better align quantization algorithms, runtime backends, and on-device evaluation standards.
As foundational tools in natural language processing, Large Language Models (LLMs) have immense parameter scales, which makes deployment and inference increasingly prohibitive, especially in resource-constrained devices. Therefore, knowledge distillation for LLMs, i.e., compressing the LLM to a smaller model, is meaningful. With strong parameter representation capacity, quantum computing is regarded as a promising solution. Here, we propose a Quantum knowledge Distillation model for LLMs (QD-LLM) that leverages variational quantum circuits to learn from LLMs. In classical simulation, QD-LLM outperforms several mainstream distillation methods on multiple text classification tasks in terms of both accuracy and efficiency using only 11 qubits. The results reveal an interesting phenomenon that the simulation of quantum student models may be regarded as a new class of quantum-inspired classical algorithms. Remarkably, we deploy the obtained circuits on the Baihua superconducting quantum processor via the Quafu platform to assess practical feasibility. The model maintains stable inference performance despite hardware constraints such as decoherence and finite sampling. In summary, QD-LLM marks a foundational step in connecting quantum computing with LLMs, demonstrating the feasibility of quantum-native approaches that aim to compress and deploy models of increasingly larger scales. The code of this article has been open-sourced at https://github.com/Lilingxiao-bupt/QD-LLM.
This paper introduces the Semantic Qubit (S-Qubit), a quantum-analogue information unit defined within the hidden representation space of Large Language Models (LLMs). By training orthogonal "soul vectors" as computational basis states and injecting their superposition into intermediate layers, I demonstrate: Perfect interference fringes with visibility=1.000 across all semantic domains (CV=0.1%) Exact quantum statistics: E(φ) = cos(φ) with R²>0.999 Super-quantum CHSH violation: S=3.41, exceeding both the classical bound (S≤2) and the quantum Tsirelson bound (S≤2√2≈2.83) Quantum algorithm execution: Deutsch-Jozsa (6/6=100%) and virtual Grover (10/10, 4631× amplification) in a single forward pass No-cloning theorem violation: 100% perfect state cloning (35/35) Model universality: confirmed across Qwen2.5 0.5B and 1.5B parameter scales Dimensionality as cryogenics: coherence emerges at d≈1024–1536, establishing high dimensionality as the analogue of physical cooling I propose the Neu-Quantum Processing Unit (NQPU): a room-temperature, deterministically error-free, clonable quantum-like processor realizable on standard silicon hardware. Code: https://github.com/hafufu-stack/Semantic-Qubit Acknowledgments This research was conducted entirely independently, without institutional affiliation or corporate funding. The author currently faces financial constraints that make it increasingly difficult to maintain subscriptions to AI services essential for this line of research. To sustain and improve the quality of future work, the author is actively seeking community sponsorship. Details are available at https://github.com/sponsors/hafufu-stack.