PEQuant: 거대 언어 모델의 양자화 오류를 최소화하기 위한 실용적이고 경험적인 방법
최동훈
Abstract
Post-training quantization (PTQ) is simpler and faster to implement and consumes fewer resources than quantization-aware training (QAT), which requires retraining the entire model. Existing PTQ methods adjust the quantization granularity, modify the model architecture, or apply equivalent transformations to reduce quantization errors. However, these PTQ approaches often increase computational overhead, complicate implementation, and require resource-consuming efforts to find the optimal quantization configuration, diminishing PTQs advantages over QAT. To address these challenges, we propose PEQuant, a set of effective yet simple techniques that exploit model characteristics and quantization features to quantize large language models (LLMs). Our method uses Per-tensor Static Uniform Symmetric Round-to-Nearest Quantization with minimal quantization errors, eliminating the need for intricate quantization schemes. PEQuant comprises three main techniques. First, we identify the optimal sweet spot for clipping activation outliers for each model. Second, based on our observation that certain layers exhibit significant outliers, we apply selective quantization to layers sensitive to quantization while using W8A8 (weight INT8, activation INT8) for the remaining layers, substantially reducing quantization errors. Finally, we introduce a calibration layer at the models output, trained via knowledge distillation, which further mitigates quantization errors. We implement PEQuant with NVIDIA CUTLASS and evaluate it on various LLMs, including OPT, LLaMA2, and LLaMA3, across perplexity and zero-shot downstream tasks. We also compare PEQuant with two state-of-the-art PTQ methods, LLM.int8() and SmoothQuant. The experimental results indicate that PEQuant is practical and efficient in saving memory spaces, increasing inference speed, and reducing quantization errors.