Error correction codes (ECCs) are essential for reliable communication over noisy channels. While Transformer-based decoders surpass traditional methods on short and medium-length codes, they insufficiently exploit the structural information of Tanner graphs. In this paper, we propose a novel Transformer decoder with Tanner Graph-Based Positional Encoding (TGPE) which introduces a dual-level positional encoding mechanism to effectively integrate graph topology into the attention process. The intra-type encoding utilizes a distance matrix to embed shortest-path relationships among nodes of the same type, thereby enhancing the modeling of intra-group dependencies. Meanwhile, the inter-type encoding extends the differential attention mechanism by combining mask-based edge connectivity—derived from the parity-check matrix—with a distance-modulated attention term that captures distance-aware dependencies between variable and check nodes while suppressing background attention noise. This dual encoding enables the decoder to capture both global and local, indirect and direct topological dependencies. Extensive experiments on standard ECC benchmarks demonstrate that TGPE consistently outperforms previous methods including BP, ECCT and CrossMPT. Moreover, interpretability analysis confirms that the learned distance encodings reflect theoretical coding principles, with smaller graph distances corresponding to stronger correlations, highlighting TGPE’s robustness and theoretical soundness.
Error correction codes (ECCs) are essential for reliable communication over noisy channels. While Transformer-based decoders surpass traditional methods on short and medium-length codes, they insufficiently exploit the structural information of Tanner graphs. In this paper, we propose a novel Transformer decoder with Tanner Graph-Based Positional Encoding (TGPE) which introduces a dual-level positional encoding mechanism to effectively integrate graph topology into the attention process. The intra-type encoding utilizes a distance matrix to embed shortest-path relationships among nodes of the same type, thereby enhancing the modeling of intra-group dependencies. Meanwhile, the inter-type encoding extends the differential attention mechanism by combining mask-based edge connectivity—derived from the parity-check matrix—with a distance-modulated attention term that captures distance-aware dependencies between variable and check nodes while suppressing background attention noise. This dual encoding enables the decoder to capture both global and local, indirect and direct topological dependencies. Extensive experiments on standard ECC benchmarks demonstrate that TGPE consistently outperforms previous methods including BP, ECCT and CrossMPT. Moreover, interpretability analysis confirms that the learned distance encodings reflect theoretical coding principles, with smaller graph distances corresponding to stronger correlations, highlighting TGPE’s robustness and theoretical soundness.
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.
Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.
Transformers have recently emerged as effective neural decoders, capable of capturing complex interdependencies and delivering strong performance. Cyclic codes, including BCH codes, Reed-Solomon codes, and Reed-Muller codes, are fundamental in practical channel coding applications. In this work, we propose a novel approach for Transformer-based neural decoders specifically designed for cyclic codes. By leveraging the relational properties among the parity-check matrix, Tanner graph, and mask matrix, we integrate these with the inherent algebraic structure of cyclic codes. We then identify a cyclic shift reuse property that can be effectively applied to both parameter matrices and self-attention matrices. Extensive simulations on BCH codes demonstrate that our method reduces the total number of parameters by up to 64.4% compared to prior Transformer-based decoders while achieving comparable decoding performance. Additionally, our method decreases the computational cost of self-attention by up to 10.6%. Finally, through experiments across various dimensions, we explore the effect of embedding dimensions on performance and provide insights into the relationship between embedding dimensions and code positions.
The Error Correction Code Transformer (ECCT) is a seminal state-of-the-art model-free neural decoder for linear block codes, leveraging parity-check-matrix-aware masked self-attention to exploit algebraic code structure. A well-known limitation of deep Transformer architectures trained with Pre-Layer Normalization (PreNorm), however, is the uncontrolled growth of hidden-state magnitudes with depth, which progressively dilutes the contribution of individual layers, a phenomenon known as PreNorm dilution, thus effectively limiting the benefit of increased architectural depth. In this paper, we propose EC-CRT, a novel Transformer-based neural decoder that integrates Attention Residuals (AttnRes) into the ECCT framework to address this limitation. AttnRes replaces fixed, additive residual connections with learned, input-dependent softmax attention over all preceding layer outputs, enabling each decoding layer to selectively aggregate the most informative earlier representations and directly mitigating PreNorm dilution. Extensive experiments on LDPC(49,24), Polar(64,32), BCH(31,16), and BCH(63,51) codes under a deep configuration and a shallow configuration demonstrate that ECCRT consistently and substantially outperforms ECCT in the deep setting across all codes and SNR values, with gains that widen at higher SNRs, while delivering comparable performance in the shallow setting. The improvement is achieved with negligible additional parameters, only one learned pseudo-query vector per Transformer layer. To the best of our knowledge, this is the first work to demonstrate the benefit of Attention Residuals outside the large language model regime, confirming that this technique generalizes across model scales and application domains.
With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.
Neural decoders have recently advanced error correction performance, with transformer-based architectures in the error correction code transformer (ECCT) framework achieving state-of-the-art results among neural decoders. However, the multiple-masks ECCT suffers from suboptimal mask selection, which impacts decoding performance. In this work, we propose spectral-guided ECCT, leveraging the Laplacian matrix and its eigenvectors from the variable-variable graph to identify weakly connected node clusters for optimized mask generation. By embedding spectral properties into the decoding process, our method constructs more effective masks that improve information flow across variable and check nodes. Experimental results demonstrate consistent gains over existing ECCT-based models, achieving superior decoding accuracy and reliability while significantly improving training efficiency. This work highlights the potential of spectral graph theory to address structural challenges in neural decoders and provides a principled direction for future error correction design.