The growing need for privacy protection, instant response, and offline access has pushed large language model inference from the cloud to local edge devices. However, decoder-only LLMs remain hard to run on smartphones and embedded systems, as their performance is limited not just by parameter count, but also by memory bandwidth, hardware operator compatibility, and the rising cost of long-context generation. Post-training quantization (PTQ) is the most viable approach today, as it shrinks model size and speeds up inference without full retraining. Yet most PTQ methods are tested on cloud GPUs, and their claimed improvements rarely hold up on real edge hardware. This survey reviews on-device PTQ for decoder-only LLMs from a hardware-focused angle, covering static and adaptive schemes, key hardware bottlenecks, evaluation flaws, and KV cache scaling issues. It concludes that model size alone is a poor measure of edge efficiency, and future work must better align quantization algorithms, runtime backends, and on-device evaluation standards.
The growing need for privacy protection, instant response, and offline access has pushed large language model inference from the cloud to local edge devices. However, decoder-only LLMs remain hard to run on smartphones and embedded systems, as their performance is limited not just by parameter count, but also by memory bandwidth, hardware operator compatibility, and the rising cost of long-context generation. Post-training quantization (PTQ) is the most viable approach today, as it shrinks model size and speeds up inference without full retraining. Yet most PTQ methods are tested on cloud GPUs, and their claimed improvements rarely hold up on real edge hardware. This survey reviews on-device PTQ for decoder-only LLMs from a hardware-focused angle, covering static and adaptive schemes, key hardware bottlenecks, evaluation flaws, and KV cache scaling issues. It concludes that model size alone is a poor measure of edge efficiency, and future work must better align quantization algorithms, runtime backends, and on-device evaluation standards.
Post-training quantization (PTQ) is widely used to compress large language models without retraining. However, many existing weight-only methods rely on heuristic objectives and greedy rounding strategies, leading to noticeable degradation under low-bit quantization. We propose OJBKQ, a layer-wise PTQ method that formulates weight quantization as a joint optimization problem over activations and weights, yielding a multiple-right-hand-side box-constrained integer least squares (BILS) problem per layer. To solve this NP-hard problem, we extend Klein’s randomized variant of Babai’s nearest-plane algorithm to handle box constraints and run the extended algorithm K times to generate K box-constrained Klein points. The minimum-residual candidate among these points and the box-constrained Babai point is selected as a suboptimal solution. Experiments on large language models show that OJBKQ achieves lower perplexity at 3–4 bits than existing PTQ methods, while maintaining comparable computational cost.
Large Language Models (LLMs) such as GPT and LLaMA have transformed artificial intelligence, but their rapid growth has brought major challenges in computation, energy use, and scalability. Quantum computing (QC) offers a fundamentally different way to process information, with the potential to accelerate key operations in training, inference, optimization, and representation learning. This survey provides a structured and comprehensive review of how QC can support the future development of LLMs. It examines quantum algorithms, hybrid quantum-classical methods, quantum neural networks, quantum embeddings, and security applications. It also evaluates their feasibility in both the current noisy intermediate-scale quantum (NISQ) era and the future fault-tolerant era. The survey highlights current progress, identifies major technical barriers, and outlines practical research directions needed to build scalable, efficient, and secure quantum-enhanced language models.
Post-training quantization (PTQ) is a primary approach for deploying large language models without fine-tuning, and the quantized performance is often strongly affected by the calibration in PTQ. By contrast, in vision-language models (VLMs), substantial differences between visual and text tokens in their activation distributions and sensitivities to quantization error pose significant challenges for effective calibration during PTQ. In this work, we rethink what PTQ calibration should align with in VLMs and propose the Token-level Importance-aware Layer-wise Quantization framework (TLQ). Guided by gradient information, we design a token-level importance integration mechanism for quantization error, and use it to construct a token-level calibration set, enabling a more fine-grained calibration strategy. Furthermore, TLQ introduces a multi-GPU, quantization-exposed layer-wise calibration scheme. This scheme keeps the layer-wise calibration procedure consistent with the true quantized inference path and distributes the complex layer-wise calibration workload across multiple RTX3090 GPUs, thereby reducing reliance on the large memory of A100 GPUs. TLQ is evaluated across two models, three model scales, and two quantization settings, consistently achieving performance improvements across all settings, indicating its strong quantization stability. The code will be released publicly.
Large Language Models (LLMs) face deployment challenges due to high computational costs, and while Post-Training Quantization (PTQ) offers a solution, existing rotation-based methods struggle at very low bit-widths like 2-bit.We introduce a novel, training-free approach to construct an improved rotation matrix, addressing the limitations of current methods.The key contributions include leveraging the Walsh-Hadamard transform with sequency ordering, which clusters similar frequency components to reduce quantization error compared to standard Hadamard matrices, significantly improving performance.Furthermore, we propose a Grouped Sequencyarranged Rotation (GSR) using block-diagonal matrices with smaller Walsh blocks, effectively isolating outlier impacts and achieving performance comparable to optimization-based methods without requiring any training.Our method demonstrates robust performance on reasoning tasks and Perplexity (PPL) score on WikiText-2.Our method also enhances results even when applied over existing learned rotation techniques.
Edge artificial intelligence (Edge AI) requires real-time inference under stringent constraints on computation, memory, energy, and connectivity. Although training can be offloaded to servers, efficient, high-capacity inference and rapid on-device adaptation remain central challenges. Cloud-based inference offers substantial computational power but depends on connectivity, latency, privacy, and reliability conditions that edge deployments cannot always guarantee. Classical model-compression methods—including quantization, pruning, distillation, and neural architecture search—have extended the feasibility of on-device inference, yet they leave largely unchanged the fundamental cost of the linear-algebraic, sampling, and optimization primitives that dominate modern deep learning. Quantum computing has therefore been proposed as a complementary accelerator for selected AI workloads, with theoretical advantages in linear systems, singular value decomposition, sampling, kernel evaluation, and optimization. This review surveys the emerging field of quantum-accelerated AI for edge systems under a hybrid architectural premise: edge devices remain classical, while quantum processors operate as remote, cloud, MEC, or near-edge accelerators. We synthesize advances across quantum learning models, hybrid optimization methods, hardware and deployment architectures, and quantum-inspired approaches suitable for constrained devices. We also assess the practical barriers that currently separate asymptotic quantum advantage from deployable edge intelligence, including data loading, measurement overhead, noise, latency, and benchmarking gaps. Finally, we outline a staged research roadmap from near-term hybrid workflows to fault-tolerant and integrated quantum-edge architectures.
Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains an intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models. Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.