Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2305.04347 · IEEE International Symposium on Information Theory (ISIT)

Interpreting Training Aspects of Deep-Learned Error-Correcting Codes

Natasha Devroye, Abhijeet Mulgund, Raj Shekhar, György Turán, Miloš Žefran, Yingyao Zhou

Published Jun 25, 2023• 3 citations
arXivHTMLPDF

Abstract

As new deep-learned error-correcting codes continue to be introduced, it is important to develop tools to interpret the designed codes and understand the training process. Prior work focusing on the deep-learned TurboAE has both interpreted the learned encoders post-hoc by mapping these onto nearby "interpretable" encoders, and experimentally evaluated the performance of these interpretable encoders with various decoders. Here we look at developing tools for interpreting the training process for deep-learned error-correcting codes, focusing on: 1) using the Goldreich-Levin algorithm to quickly interpret the learned encoder; 2) using Fourier coefficients as a tool for understanding the training dynamics and the loss landscape; 3) reformulating the training loss, the binary cross entropy, by relating it to encoder and decoder parameters, and the bit error rate (BER); 4) using these insights to formulate and study a new training procedure. All tools are demonstrated on TurboAE, but are applicable to other deep-learned forward error correcting codes (without feedback).

DOI
10.1109/isit54713.2023.10206783

Related papers

ProductAE: Toward Deep Learning Driven Error-Correction Codes of Large Dimensions

Mohammad Vahid Jamali, Hamid Saber, Homayoon Hatami +1 more

While decades of theoretical research have led to the invention of several classes of error-correction codes, the design of such codes is an extremely challenging task, mostly driven by human ingenuity. Recent studies demonstrate that such designs can be effectively automated and accelerated via tools from machine learning (ML), thus enabling ML-driven classes of error-correction codes with promising performance gains compared to classical designs. A fundamental challenge, however, is that it is prohibitively complex, if not impossible, to design and train fully ML-driven encoder and decoder pairs for large code dimensions. In this paper, we propose Product Autoencoder (ProductAE) -- a computationally-efficient family of deep learning driven (encoder, decoder) pairs -- aimed at enabling the training of relatively large codes (both encoder and decoder) with a manageable training complexity. We build upon ideas from classical product codes and propose constructing large neural codes using smaller code components. ProductAE boils down the complex problem of training the encoder and decoder for a large code dimension $k$ and blocklength $n$ to less-complex sub-problems of training encoders and decoders for smaller dimensions and blocklengths. Our training results show successful training of ProductAEs of dimensions as large as $k = 300$ bits with meaningful performance gains compared to state-of-the-art classical and neural designs. Moreover, we demonstrate excellent robustness and adaptivity of ProductAEs to channel models different than the ones used for training.

arXiv:2303.16424•Mar 29, 2023•3 citations

Higher-order Interpretations of Deepcode, a Learned Feedback Code

Yaoyu Zhou, Natasha Devroye, Gy. Turán +1 more

We present an interpretation of Deepcode, a learned feedback code that showcases higher-order error correction relative to an earlier interpretable model [1]. By interpretation, we mean succinct analytical encoder and decoder expressions (albeit with learned parameters) in which the role of feedback in achieving error correction is easy to understand. By higher-order, we mean that longer sequences of large noise values are acted upon by the encoder (which has access to these through the feedback) and used in error correction at the decoder in a two-stage decoding process.11This work was supported by NSF under awards 1900911, 2217023, and 2240532, and by the AI National Laboratory Program (RRF-2.3.1-21-2022-00004). The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NSF.

arXiv:2408.11907•Allerton Conference on Communication, Control, and Computing•Sep 24, 2024•3 citations

On deep learning-based channel decoding

Tobias Gruber, Sebastian Cammerer, Jakob Hoydis +1 more

We revisit the idea of using deep neural networks for one-shot decoding of random and structured codes, such as polar codes. Although it is possible to achieve maximum a posteriori (MAP) bit error rate (BER) performance for both code families and for short codeword lengths, we observe that (i) structured codes are easier to learn and (ii) the neural network is able to generalize to codewords that it has never seen during training for structured, but not for random codes. These results provide some evidence that neural networks can learn a form of decoding algorithm, rather than only a simple classifier. We introduce the metric normalized validation error (NVE) in order to further investigate the potential and limitations of deep learning-based decoding with respect to performance and complexity.

Conference on Information Sciences and Systems•Mar 1, 2017•574 citations

On Deep Learning-Based Channel Decoding

Tobias Gruber, Sebastian Cammerer, Jakob Hoydis +1 more

We revisit the idea of using deep neural networks for one-shot decoding of random and structured codes, such as polar codes. Although it is possible to achieve maximum a posteriori (MAP) bit error rate (BER) performance for both code families and for short codeword lengths, we observe that (i) structured codes are easier to learn and (ii) the neural network is able to generalize to codewords that it has never seen during training for structured, but not for random codes. These results provide some evidence that neural networks can learn a form of decoding algorithm, rather than only a simple classifier. We introduce the metric normalized validation error (NVE) in order to further investigate the potential and limitations of deep learning-based decoding with respect to performance and complexity.

arXiv:1701.07738•Jan 26, 2017•42 citations

Interpreting KO Codes

R. Shekhar, Natasha Devroye, Gy. Turán +1 more

The KO (Kronecker Operation) code is a recent deep-learned error-correcting code using a neural network architecture to generalize a Reed-Muller code with Dumer decoding. Analyzing the encoder modules and using ablation techniques, we give interpretations of the KO encoder which significantly reduce the number of parameters. We also discuss interpretability aspects of the KO decoder. The interpretation opens up possibilities to give an explicit representation of KO codes, which could be useful for more efficient learning of KO codes and explaining the learning mechanism underlying the empirical observations made about its performance in previous work.

IEEE International Symposium on Information Theory (ISIT)•Jun 22, 2025•0 citations

Deep Learning Methods for Channel Decoding: A Brief Tutorial

Kai Niu, Jincheng Dai, Kailin Tan +1 more

The recent development of deep learning methods demonstrates a new insight to optimize the decoding of linear codes. In this paper, we survey the typical neural network decoding methods, including data-driven and model-driven schemes. We investigate the design principle, algorithm mechanism, parameter assignment, and training process of these neural decoders for high-density parity check (HDPC), low-density parity-check (LDPC), and polar codes. Finally, we summarize the advantages of neural network decoding and point out some research directions in the future.

IEEE/CIC International Conference on Communications in China (ICCC)•Jul 28, 2021•8 citations
As new deep-learned error-correcting codes continue to be introduced, it is important to develop tools to interpret the designed codes and understand the training process. Prior work focusing on the deep-learned TurboAE has both interpreted the learned encoders post-hoc by mapping these onto nearby "interpretable" encoders, and experimentally evaluated the performance of these interpretable encoders with various decoders. Here we look at developing tools for interpreting the training process for deep-learned error-correcting codes, focusing on: 1) using the Goldreich-Levin algorithm to quickly interpret the learned encoder; 2) using Fourier coefficients as a tool for understanding the training dynamics and the loss landscape; 3) reformulating the training loss, the binary cross entropy, by relating it to encoder and decoder parameters, and the bit error rate (BER); 4) using these insights to formulate and study a new training procedure. All tools are demonstrated on TurboAE, but are applicable to other deep-learned forward error correcting codes (without feedback).
While decades of theoretical research have led to the invention of several classes of error-correction codes, the design of such codes is an extremely challenging task, mostly driven by human ingenuity. Recent studies demonstrate that such designs can be effectively automated and accelerated via tools from machine learning (ML), thus enabling ML-driven classes of error-correction codes with promising performance gains compared to classical designs. A fundamental challenge, however, is that it is prohibitively complex, if not impossible, to design and train fully ML-driven encoder and decoder pairs for large code dimensions. In this paper, we propose Product Autoencoder (ProductAE) -- a computationally-efficient family of deep learning driven (encoder, decoder) pairs -- aimed at enabling the training of relatively large codes (both encoder and decoder) with a manageable training complexity. We build upon ideas from classical product codes and propose constructing large neural codes using smaller code components. ProductAE boils down the complex problem of training the encoder and decoder for a large code dimension kkk and blocklength nnn to less-complex sub-problems of training encoders and decoders for smaller dimensions and blocklengths. Our training results show successful training of ProductAEs of dimensions as large as k=300k = 300k=300 bits with meaningful performance gains compared to state-of-the-art classical and neural designs. Moreover, we demonstrate excellent robustness and adaptivity of ProductAEs to channel models different than the ones used for training.
We present an interpretation of Deepcode, a learned feedback code that showcases higher-order error correction relative to an earlier interpretable model [1]. By interpretation, we mean succinct analytical encoder and decoder expressions (albeit with learned parameters) in which the role of feedback in achieving error correction is easy to understand. By higher-order, we mean that longer sequences of large noise values are acted upon by the encoder (which has access to these through the feedback) and used in error correction at the decoder in a two-stage decoding process.11This work was supported by NSF under awards 1900911, 2217023, and 2240532, and by the AI National Laboratory Program (RRF-2.3.1-21-2022-00004). The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NSF.
We revisit the idea of using deep neural networks for one-shot decoding of random and structured codes, such as polar codes. Although it is possible to achieve maximum a posteriori (MAP) bit error rate (BER) performance for both code families and for short codeword lengths, we observe that (i) structured codes are easier to learn and (ii) the neural network is able to generalize to codewords that it has never seen during training for structured, but not for random codes. These results provide some evidence that neural networks can learn a form of decoding algorithm, rather than only a simple classifier. We introduce the metric normalized validation error (NVE) in order to further investigate the potential and limitations of deep learning-based decoding with respect to performance and complexity.
We revisit the idea of using deep neural networks for one-shot decoding of random and structured codes, such as polar codes. Although it is possible to achieve maximum a posteriori (MAP) bit error rate (BER) performance for both code families and for short codeword lengths, we observe that (i) structured codes are easier to learn and (ii) the neural network is able to generalize to codewords that it has never seen during training for structured, but not for random codes. These results provide some evidence that neural networks can learn a form of decoding algorithm, rather than only a simple classifier. We introduce the metric normalized validation error (NVE) in order to further investigate the potential and limitations of deep learning-based decoding with respect to performance and complexity.
The KO (Kronecker Operation) code is a recent deep-learned error-correcting code using a neural network architecture to generalize a Reed-Muller code with Dumer decoding. Analyzing the encoder modules and using ablation techniques, we give interpretations of the KO encoder which significantly reduce the number of parameters. We also discuss interpretability aspects of the KO decoder. The interpretation opens up possibilities to give an explicit representation of KO codes, which could be useful for more efficient learning of KO codes and explaining the learning mechanism underlying the empirical observations made about its performance in previous work.
The recent development of deep learning methods demonstrates a new insight to optimize the decoding of linear codes. In this paper, we survey the typical neural network decoding methods, including data-driven and model-driven schemes. We investigate the design principle, algorithm mechanism, parameter assignment, and training process of these neural decoders for high-density parity check (HDPC), low-density parity-check (LDPC), and polar codes. Finally, we summarize the advantages of neural network decoding and point out some research directions in the future.