Transformers have recently emerged as effective neural decoders, capable of capturing complex interdependencies and delivering strong performance. Cyclic codes, including BCH codes, Reed-Solomon codes, and Reed-Muller codes, are fundamental in practical channel coding applications. In this work, we propose a novel approach for Transformer-based neural decoders specifically designed for cyclic codes. By leveraging the relational properties among the parity-check matrix, Tanner graph, and mask matrix, we integrate these with the inherent algebraic structure of cyclic codes. We then identify a cyclic shift reuse property that can be effectively applied to both parameter matrices and self-attention matrices. Extensive simulations on BCH codes demonstrate that our method reduces the total number of parameters by up to 64.4% compared to prior Transformer-based decoders while achieving comparable decoding performance. Additionally, our method decreases the computational cost of self-attention by up to 10.6%. Finally, through experiments across various dimensions, we explore the effect of embedding dimensions on performance and provide insights into the relationship between embedding dimensions and code positions.
Transformers have recently emerged as effective neural decoders, capable of capturing complex interdependencies and delivering strong performance. Cyclic codes, including BCH codes, Reed-Solomon codes, and Reed-Muller codes, are fundamental in practical channel coding applications. In this work, we propose a novel approach for Transformer-based neural decoders specifically designed for cyclic codes. By leveraging the relational properties among the parity-check matrix, Tanner graph, and mask matrix, we integrate these with the inherent algebraic structure of cyclic codes. We then identify a cyclic shift reuse property that can be effectively applied to both parameter matrices and self-attention matrices. Extensive simulations on BCH codes demonstrate that our method reduces the total number of parameters by up to 64.4% compared to prior Transformer-based decoders while achieving comparable decoding performance. Additionally, our method decreases the computational cost of self-attention by up to 10.6%. Finally, through experiments across various dimensions, we explore the effect of embedding dimensions on performance and provide insights into the relationship between embedding dimensions and code positions.
With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.
Neural decoders were introduced as a generalization of the classic Belief Propagation (BP) decoding algorithms, where the Trellis graph in the BP algorithm is viewed as a neural network, and the weights in the Trellis graph are optimized by training the neural network. In this work, we propose a novel neural decoder for cyclic codes by exploiting their cyclically invariant property. More precisely, we impose a shift invariant structure on the weights of our neural decoder so that any cyclic shift of inputs results in the same cyclic shift of outputs. Extensive simulations with BCH codes and punctured Reed-Muller (RM) codes show that our new decoder consistently outperforms previous neural decoders when decoding cyclic codes. Finally, we propose a list decoding procedure that can significantly reduce the decoding error probability for BCH codes and punctured RM codes. For certain high-rate codes, the gap between our list decoder and the Maximum Likelihood decoder is less than 0.1dB. Code available at https://github.com/cyclicallyneuraldecoder/CyclicallyEquivariantNeuralDecoders
Effective iterative decoding of short BCH codes faces two primary challenges: identifying an appropriate parity-check matrix and accelerating decoder convergence. To address these issues, we propose a systematic scheme to derive an optimized parity-check matrix through a heuristic approach. This involves a series of binary sum and row shift operations, resulting in a low-density, quasi-regular column weight distribution with a reduced number of shortest cycles in the underlying redundant Tanner graph. For the revised normalized min-sum decoder, we concurrently integrate three types of random permutations into the alternated messages across iterations, leading to significantly faster convergence compared to existing methods. Furthermore, by utilizing the iterative trajectories of failed normalized min-sum decoding, we enhance the reliability measurement of codeword bits with the assistance of a neural network model from prior work, which accommodates more failures for the post-processing of ordered statistics decoding. Additionally, we report the types of undetected errors for the design of iterative decoders for short BCH codes, which potentially challenge efforts to approach the maximum likelihood limit. Extensive simulations demonstrate that the proposed hybrid framework achieves an attractive balance between performance, latency, and complexity.
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.
Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer (ECCT) and its enhanced version, the foundation error correction code Transformer (FECCT), have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity ofO(N2) due to the self-attention mechanism in the Transformer limits scalability, whereNrepresents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length and code rate across different code types, while a redesigned low-rank unified attention module, with computational complexity ofO(N), is shared across various heads in the Transformer. Additionally, a sparse mask, derived from the parity-check matrix’s sparsity, is introduced to enhance the decoder’s ability to capture inherent constraints between information and parity-check bits, improving decoding accuracy and further reducing computational complexity by 86%. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.
Recently, deep learning has demonstrated improvements over classical decoding algorithms in various families of error-correcting codes with short to moderate block lengths. Neural decoders like neural belief propagation (NBP), constructed based on the Tanner graphs of linear codes, have been extensively studied, but their application to longer codes is limited due to increased network complexity with longer block lengths. This complexity leads to higher computational costs and deployment challenges, such as GPU memory usage. To address this, we propose a novel revolving framework for NBP-like decoders tailored to cyclic and QC-LDPC codes, a commonly used class of error correction codes. Our approach leverages the cyclic structure and section-wise cyclic structure inherent in cyclic and QC-LDPC codes respectively, significantly simplifying the complexity of the network. Experimental results demonstrate the effectiveness of our method across various cyclic and QC-LDPC codes. Especially, compared to the traditional decoding scheme based on the section-wise cyclic structure, our proposed decoder shows substantial improvements on 5G LDPC codes, exhibits better performance than the traditional min-sum decoder, and approaches the sum-product decoder without a noticeable error floor within the investigated noise levels.