Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2602.01602

Spectral-Aligned Pruning for Universal Error-Correcting Code Transformers

Sanghyeon Cho, Taewoo Park, Seong‐Joon Park, Dae-Young Yun, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim

Published Feb 2, 2026• 0 citations
arXivHTMLPDF

Abstract

Universal channel decoders based on transformers-such as the Foundation Error Correction Code Transformer (FECCT)-achieve competitive decoding performance across diverse code families with a single shared backbone, optionally followed by code-specific finetuning. However, the high computational complexity and large parameter footprint of FECCT present substantial obstacles to practical deployment. To address these challenges, we investigate structured pruning for FECCT and propose Spectral-Aligned Pruning (SAP), a structure-aware framework that enables cross-code reuse of structured pruning masks by leveraging the spectrum of the corresponding bipartite graph. SAP is grounded in classical graph analysis of codes: the two algebraically largest adjacency eigenvalues provide compact spectral proxies for degree scale, expansion ratio, and minimum-distance lower bounds. These quantities are directly relevant to decoding performance: degree scale reflects how densely codeword bits and parity checks are connected; expansion ratio influences how information propagates across the bipartite graph; and minimum distance characterizes codeword separation. Based on this connection, SAP uses these two leading eigenvalues as a lightweight code signature for pruning-mask retrieval. Empirically, this two-dimensional signature yields stable library selection equivalent to higher-dimensional spectral signatures in our evaluation. After pruning, SAP uses parameter-efficient low-rank adaptation (LoRA) to recover decoding performance while storing only a small number of additional adapter parameters per code. Experiments across diverse codes show that SAP achieves decoding performance comparable to dedicated per-code pruning, while enabling substantial reductions in computational cost and model memory footprint through kernel-level structured pruning. Code is available at https://github.com/iil-postech/sap.

DOI
10.48550/arxiv.2602.01602

Related papers

Unified Error Correction Code Transformer With Low Complexity

Yongli Yan, Jieao Zhu, Tianyue Zheng +3 more

Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer (ECCT) and its enhanced version, the foundation error correction code Transformer (FECCT), have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity ofO(N2) due to the self-attention mechanism in the Transformer limits scalability, whereNrepresents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length and code rate across different code types, while a redesigned low-rank unified attention module, with computational complexity ofO(N), is shared across various heads in the Transformer. Additionally, a sparse mask, derived from the parity-check matrix’s sparsity, is introduced to enhance the decoder’s ability to capture inherent constraints between information and parity-check bits, improving decoding accuracy and further reducing computational complexity by 86%. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.

arXiv:2410.03364•IEEE Internet of Things Journal•Jan 28, 2026•4 citations

Spectral-Guided Mask Generation for Error Correction Code Transformers

Linh Nguyen, Quoc Bao Phan, Tuy Tan Nguyen

Neural decoders have recently advanced error correction performance, with transformer-based architectures in the error correction code transformer (ECCT) framework achieving state-of-the-art results among neural decoders. However, the multiple-masks ECCT suffers from suboptimal mask selection, which impacts decoding performance. In this work, we propose spectral-guided ECCT, leveraging the Laplacian matrix and its eigenvectors from the variable-variable graph to identify weakly connected node clusters for optimized mask generation. By embedding spectral properties into the decoding process, our method constructs more effective masks that improve information flow across variable and check nodes. Experimental results demonstrate consistent gains over existing ECCT-based models, achieving superior decoding accuracy and reliability while significantly improving training efficiency. This work highlights the potential of spectral graph theory to address structural challenges in neural decoders and provides a principled direction for future error correction design.

IEEE Transactions on Machine Learning in Communications and Networking•Jan 1, 2026•0 citations

Multiple-Masks Error Correction Code Transformer for Short Block Codes

Seong‐Joon Park, Hee-Youl Kwak, Sang‐Hyo Kim +3 more

With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.

IEEE Journal on Selected Areas in Communications•Apr 10, 2025•13 citations

A Low-Complexity Unified Error Correction Transformer

Yongli Yan, Jieao Zhu, Tianyue Zheng +3 more

Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer and its enhanced version, the foundation error correction code Transformer, have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity of $\mathcal{O}\left( {{N^2}} \right)$ due to the self-attention mechanism in the Transformer limits scalability, where N represents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length across different code types, while a redesigned low-rank unified attention module, with computational complexity of $\mathcal{O}\left( N \right)$, is shared across various heads in the Transformer. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.

International Conference on Communications•May 24, 2026•0 citations

Accelerating Error Correction Code Transformers

Matan Levy, Yoni Choukroun, Lior Wolf

Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.

arXiv:2410.05911•Oct 8, 2024•7 citations

Denoising Diffusion Decoding with U-Shaped Error Correction Code Transformers

Elaheh Bassak, Elvino Silveira Sousa

Recent neural decoders substantially improve error correction for short block codes but remain computationally intensive and lack adaptive refinement. This work introduces enhanced U-shaped transformer architectures with two key innovations: (i) double-masked attention from complementary parity-check matrices for improved structural diversity, and (ii) cross-attention message-passing that bridges neural and classical decoding principles. To enable adaptive iterative refinement, we extend these single-shot U-shaped decoders into the denoising diffusion paradigm, where channel corruption is modeled as a forward diffusion process and iteratively reversed based on continuous soft syndrome conditioning. Our key finding demonstrates that U-shaped backbones provide disproportionately larger performance gains under diffusion compared to shallow or deep alternatives, nearly matching deep architectures while utilizing half the parameters. Extensive experiments on LDPC(49,24) and Polar(64,32) codes show that the proposed diffusion-enhanced U-shaped transformers achieve state-of-the-art bit-error-rate performance while achieving superior computational efficiency.

IEEE International Conference on Communications workshops•May 24, 2026•0 citations
Universal channel decoders based on transformers-such as the Foundation Error Correction Code Transformer (FECCT)-achieve competitive decoding performance across diverse code families with a single shared backbone, optionally followed by code-specific finetuning. However, the high computational complexity and large parameter footprint of FECCT present substantial obstacles to practical deployment. To address these challenges, we investigate structured pruning for FECCT and propose Spectral-Aligned Pruning (SAP), a structure-aware framework that enables cross-code reuse of structured pruning masks by leveraging the spectrum of the corresponding bipartite graph. SAP is grounded in classical graph analysis of codes: the two algebraically largest adjacency eigenvalues provide compact spectral proxies for degree scale, expansion ratio, and minimum-distance lower bounds. These quantities are directly relevant to decoding performance: degree scale reflects how densely codeword bits and parity checks are connected; expansion ratio influences how information propagates across the bipartite graph; and minimum distance characterizes codeword separation. Based on this connection, SAP uses these two leading eigenvalues as a lightweight code signature for pruning-mask retrieval. Empirically, this two-dimensional signature yields stable library selection equivalent to higher-dimensional spectral signatures in our evaluation. After pruning, SAP uses parameter-efficient low-rank adaptation (LoRA) to recover decoding performance while storing only a small number of additional adapter parameters per code. Experiments across diverse codes show that SAP achieves decoding performance comparable to dedicated per-code pruning, while enabling substantial reductions in computational cost and model memory footprint through kernel-level structured pruning. Code is available at https://github.com/iil-postech/sap.
Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer (ECCT) and its enhanced version, the foundation error correction code Transformer (FECCT), have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity ofO(N2) due to the self-attention mechanism in the Transformer limits scalability, whereNrepresents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length and code rate across different code types, while a redesigned low-rank unified attention module, with computational complexity ofO(N), is shared across various heads in the Transformer. Additionally, a sparse mask, derived from the parity-check matrix’s sparsity, is introduced to enhance the decoder’s ability to capture inherent constraints between information and parity-check bits, improving decoding accuracy and further reducing computational complexity by 86%. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.
Neural decoders have recently advanced error correction performance, with transformer-based architectures in the error correction code transformer (ECCT) framework achieving state-of-the-art results among neural decoders. However, the multiple-masks ECCT suffers from suboptimal mask selection, which impacts decoding performance. In this work, we propose spectral-guided ECCT, leveraging the Laplacian matrix and its eigenvectors from the variable-variable graph to identify weakly connected node clusters for optimized mask generation. By embedding spectral properties into the decoding process, our method constructs more effective masks that improve information flow across variable and check nodes. Experimental results demonstrate consistent gains over existing ECCT-based models, achieving superior decoding accuracy and reliability while significantly improving training efficiency. This work highlights the potential of spectral graph theory to address structural challenges in neural decoders and provides a principled direction for future error correction design.
With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.
Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer and its enhanced version, the foundation error correction code Transformer, have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity of O(N2)\mathcal{O}\left( {{N^2}} \right)O(N2) due to the self-attention mechanism in the Transformer limits scalability, where N represents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length across different code types, while a redesigned low-rank unified attention module, with computational complexity of O(N)\mathcal{O}\left( N \right)O(N), is shared across various heads in the Transformer. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.
Recent neural decoders substantially improve error correction for short block codes but remain computationally intensive and lack adaptive refinement. This work introduces enhanced U-shaped transformer architectures with two key innovations: (i) double-masked attention from complementary parity-check matrices for improved structural diversity, and (ii) cross-attention message-passing that bridges neural and classical decoding principles. To enable adaptive iterative refinement, we extend these single-shot U-shaped decoders into the denoising diffusion paradigm, where channel corruption is modeled as a forward diffusion process and iteratively reversed based on continuous soft syndrome conditioning. Our key finding demonstrates that U-shaped backbones provide disproportionately larger performance gains under diffusion compared to shallow or deep alternatives, nearly matching deep architectures while utilizing half the parameters. Extensive experiments on LDPC(49,24) and Polar(64,32) codes show that the proposed diffusion-enhanced U-shaped transformers achieve state-of-the-art bit-error-rate performance while achieving superior computational efficiency.