Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2507.01038

Cross-Attention Message-Passing Transformers for Code-Agnostic Decoding in 6G Networks

Seong‐Joon Park, Kwak, Hee-Youl, Sang‐Hyo Kim, Yongjune Kim, Jong‐Seon No

Published Jun 22, 2025• 0 citations
arXivHTMLPDF

Abstract

Channel coding for 6G networks is expected to support a wide range of requirements arising from heterogeneous communication scenarios. These demands challenge traditional code-specific decoders, which lack the flexibility and scalability required for next-generation systems. To tackle this problem, we propose an AI-native foundation model for unified and code-agnostic decoding based on the transformer architecture. We first introduce a cross-attention message-passing transformer (CrossMPT). CrossMPT employs two masked cross-attention blocks that iteratively update two distinct input representations-magnitude and syndrome vectors-allowing the model to effectively learn the decoding problem. Notably, our CrossMPT has achieved state-of-the-art decoding performance among single neural decoders. Building on this, we develop foundation CrossMPT (FCrossMPT) by making the architecture invariant to code length, rate, and class, allowing a single trained model to decode a broad range of codes without retraining. To further enhance decoding performance, particularly for short blocklength codes, we propose CrossMPT ensemble decoder (CrossED), an ensemble decoder composed of multiple parallel CrossMPT blocks employing different parity-check matrices. This architecture can also serve as a foundation model, showing strong generalization across diverse code types. Overall, the proposed AI-native code-agnostic decoder offers flexibility, scalability, and high performance, presenting a promising direction to channel coding for 6G networks.

DOI
10.48550/arxiv.2507.01038

Related papers

CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes

Seong‐Joon Park, Kwak, Hee-Youl, Sang‐Hyo Kim +2 more

Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.

arXiv:2405.01033•May 2, 2024•1 citation

E2E DC-CrossMPT: Cross-Attention Message-Passing Transformer for Joint Design and Decoding of Linear Block Codes

Yeji Cho, Junghyun Kim

In this paper, we propose a novel deep learning-based framework for the joint design and decoding of linear block codes, the end-to-end deep coding cross-attention message-passing Transformer (E2E DC-CrossMPT). To improve linear block code design and decoding, we redesign the conventional error correction code (ECC) decoder, CrossMPT, to fit within an end-to-end framework. The redesigned decoder separately utilizes magnitude and syndrome vectors obtained from the received signals as inputs. It further employs one-hot encoding based syndrome embedding and incorporates a parity-check matrix into the output layers. Experimental results demonstrate that, across various code lengths and code rates, E2E DC-CrossMPT consistently outperforms both traditional decoding algorithms and a conventional end-to-end deep coding model in terms of decoding performance. Moreover, the codes designed by E2E DC-CrossMPT achieve superior error-correction capability compared with both traditional linear block codes and those designed by the conventional end-to-end deep coding model, while requiring lower computational complexity.

Electronics•Jun 25, 2026•1 citation

CrossMPM: Cross-Selective Scan Message-Passing Mamba for Short-Length LDPC Codes

Yoonju Cho, Junghyun Kim

Error-correcting codes (ECCs) are fundamental to reliable communication over noisy channels. Transformer-based neural decoders achieve strong ECC decoding performance, but attention operations cause computational complexity to grow substantially with code length. We propose Cross-Selective Scan Message-Passing Mamba (CrossMPM), a low-complexity Mamba-based neural decoder, to address this issue. CrossMPM replaces cross-attention-based information exchange between magnitude and syndrome vectors in Cross-Attention Message-Passing Transformer (CrossMPT) with Tanner-graph projection and Cross-Selective Scan State-Space Model (Cross-Selective Scan SSM). Tanner-graph projection aggregates connected node representations according to the parity-check matrix to incorporate code structure, while Cross-Selective Scan SSM dynamically generates selection parameters from the aggregated representations to control state updates and outputs. CrossMPM thus models local parity constraints and long-range dependencies without attention operations and reduces computational complexity. Simulation results show that CrossMPM achieves competitive decoding performance with lower computational complexity than CrossMPT and Hybrid Mamba-Transformer Decoder (ECCM), and this performance is maintained at longer code lengths. These results show that CrossMPM provides a favorable trade-off between error correction performance and computational efficiency.

Electronics•Oct 28, 2026•0 citations

Unified Error Correction Code Transformer With Low Complexity

Yongli Yan, Jieao Zhu, Tianyue Zheng +3 more

Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer (ECCT) and its enhanced version, the foundation error correction code Transformer (FECCT), have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity ofO(N2) due to the self-attention mechanism in the Transformer limits scalability, whereNrepresents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length and code rate across different code types, while a redesigned low-rank unified attention module, with computational complexity ofO(N), is shared across various heads in the Transformer. Additionally, a sparse mask, derived from the parity-check matrix’s sparsity, is introduced to enhance the decoder’s ability to capture inherent constraints between information and parity-check bits, improving decoding accuracy and further reducing computational complexity by 86%. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.

arXiv:2410.03364•IEEE Internet of Things Journal•Jan 28, 2026•0 citations

On the Design and Performance of Machine Learning Based Error Correcting Decoders

Yuncheng Yuan, Péter Scheepers, Lydia Tasiou +3 more

This paper analyzes the design and competitiveness of four neural network (NN) architectures recently proposed as decoders for forward error correction (FEC) codes. We first consider the so-called single-label neural network (SLNN) and the multi-label neural network (MLNN) decoders which have been reported to achieve near maximum likelihood (ML) performance. Here, we show analytically that SLNN and MLNN decoders can always achieve ML performance, regardless of the code dimensions-although at the cost of computational complexityand no training is in fact required. We then turn our attention to two transformer-based decoders: the error correction code transformer (ECCT) and the cross-attention message passing transformer (CrossMPT). We compare their performance against traditional decoders, and show that ordered statistics decoding outperforms these transformer-based decoders. The results in this paper cast serious doubts on the application of NN-based FEC decoders in the short and medium blocklength regime.

arXiv:2410.15899•International ITG Conference on Systems, Communications and Coding (SCC)•Mar 10, 2025•2 citations

Multiple-Masks Error Correction Code Transformer for Short Block Codes

Seong‐Joon Park, Hee-Youl Kwak, Sang‐Hyo Kim +3 more

With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.

IEEE Journal on Selected Areas in Communications•Apr 10, 2025•8 citations
Channel coding for 6G networks is expected to support a wide range of requirements arising from heterogeneous communication scenarios. These demands challenge traditional code-specific decoders, which lack the flexibility and scalability required for next-generation systems. To tackle this problem, we propose an AI-native foundation model for unified and code-agnostic decoding based on the transformer architecture. We first introduce a cross-attention message-passing transformer (CrossMPT). CrossMPT employs two masked cross-attention blocks that iteratively update two distinct input representations-magnitude and syndrome vectors-allowing the model to effectively learn the decoding problem. Notably, our CrossMPT has achieved state-of-the-art decoding performance among single neural decoders. Building on this, we develop foundation CrossMPT (FCrossMPT) by making the architecture invariant to code length, rate, and class, allowing a single trained model to decode a broad range of codes without retraining. To further enhance decoding performance, particularly for short blocklength codes, we propose CrossMPT ensemble decoder (CrossED), an ensemble decoder composed of multiple parallel CrossMPT blocks employing different parity-check matrices. This architecture can also serve as a foundation model, showing strong generalization across diverse code types. Overall, the proposed AI-native code-agnostic decoder offers flexibility, scalability, and high performance, presenting a promising direction to channel coding for 6G networks.
Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.
In this paper, we propose a novel deep learning-based framework for the joint design and decoding of linear block codes, the end-to-end deep coding cross-attention message-passing Transformer (E2E DC-CrossMPT). To improve linear block code design and decoding, we redesign the conventional error correction code (ECC) decoder, CrossMPT, to fit within an end-to-end framework. The redesigned decoder separately utilizes magnitude and syndrome vectors obtained from the received signals as inputs. It further employs one-hot encoding based syndrome embedding and incorporates a parity-check matrix into the output layers. Experimental results demonstrate that, across various code lengths and code rates, E2E DC-CrossMPT consistently outperforms both traditional decoding algorithms and a conventional end-to-end deep coding model in terms of decoding performance. Moreover, the codes designed by E2E DC-CrossMPT achieve superior error-correction capability compared with both traditional linear block codes and those designed by the conventional end-to-end deep coding model, while requiring lower computational complexity.
Error-correcting codes (ECCs) are fundamental to reliable communication over noisy channels. Transformer-based neural decoders achieve strong ECC decoding performance, but attention operations cause computational complexity to grow substantially with code length. We propose Cross-Selective Scan Message-Passing Mamba (CrossMPM), a low-complexity Mamba-based neural decoder, to address this issue. CrossMPM replaces cross-attention-based information exchange between magnitude and syndrome vectors in Cross-Attention Message-Passing Transformer (CrossMPT) with Tanner-graph projection and Cross-Selective Scan State-Space Model (Cross-Selective Scan SSM). Tanner-graph projection aggregates connected node representations according to the parity-check matrix to incorporate code structure, while Cross-Selective Scan SSM dynamically generates selection parameters from the aggregated representations to control state updates and outputs. CrossMPM thus models local parity constraints and long-range dependencies without attention operations and reduces computational complexity. Simulation results show that CrossMPM achieves competitive decoding performance with lower computational complexity than CrossMPT and Hybrid Mamba-Transformer Decoder (ECCM), and this performance is maintained at longer code lengths. These results show that CrossMPM provides a favorable trade-off between error correction performance and computational efficiency.
Channel coding is vital for reliable sixth-generation (6G) data transmission, employing diverse error correction codes for various application scenarios. Traditional decoders require dedicated hardware for each code, leading to high hardware costs. Recently, artificial intelligence (AI)-driven approaches, such as the error correction code Transformer (ECCT) and its enhanced version, the foundation error correction code Transformer (FECCT), have been proposed to reduce the hardware cost by leveraging the Transformer to decode multiple codes. However, their excessively high computational complexity ofO(N2) due to the self-attention mechanism in the Transformer limits scalability, whereNrepresents the sequence length. To reduce computational complexity, we propose a unified Transformer-based decoder that handles multiple linear block codes within a single framework. Specifically, a standardized unit is employed to align code length and code rate across different code types, while a redesigned low-rank unified attention module, with computational complexity ofO(N), is shared across various heads in the Transformer. Additionally, a sparse mask, derived from the parity-check matrix’s sparsity, is introduced to enhance the decoder’s ability to capture inherent constraints between information and parity-check bits, improving decoding accuracy and further reducing computational complexity by 86%. Extensive experimental results demonstrate that the proposed unified Transformer-based decoder outperforms existing methods and provides a high-performance, low-complexity solution for next-generation wireless communication systems.
This paper analyzes the design and competitiveness of four neural network (NN) architectures recently proposed as decoders for forward error correction (FEC) codes. We first consider the so-called single-label neural network (SLNN) and the multi-label neural network (MLNN) decoders which have been reported to achieve near maximum likelihood (ML) performance. Here, we show analytically that SLNN and MLNN decoders can always achieve ML performance, regardless of the code dimensions-although at the cost of computational complexityand no training is in fact required. We then turn our attention to two transformer-based decoders: the error correction code transformer (ECCT) and the cross-attention message passing transformer (CrossMPT). We compare their performance against traditional decoders, and show that ordered statistics decoding outperforms these transformer-based decoders. The results in this paper cast serious doubts on the application of NN-based FEC decoders in the short and medium blocklength regime.
With the broadening applications of deep learning, neural decoders have emerged as a key research focus, specifically aimed at improving the decoding performance of conventional decoding algorithms. In particular, error correction code transformer (ECCT), which utilizes the transformer architecture, has achieved state-of-the-art performance among neural network-based decoders. We present three technical contributions to significantly enhance the performance of ECCT. First, we propose a novel transformer architecture of ECCT, termed themultiple-masks ECCT (MM ECCT). We employ multiple masked self-attention blocks with different mask matrices in a parallel manner to learn diverse relationships among the codeword bits. Second, we discover that constructing mask matrices based on systematic parity check matrices (PCMs) can make the attention mapssparse, which not only enhances the decoding performance but also reduces computational complexity. Finally, we propose using complementary mask matrices derived from cyclic permutations of the systematic PCM. These complementary mask matrices are specifically designed to enhance the decoding of cyclic codes. Our extensive simulation results show that the proposed MM ECCT architecture with carefully designed mask matrices outperforms the original ECCT by a large margin, achieving state-of-the-art decoding performance among neural decoders. The source code is available at https://github.com/iil-postech/mm-ecct.