Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2002.05185

A Tensor Network Approach to Finite Markov Decision Processes

Edward Gillman, Rose, Dominic C., Juan P. Garrahan

Published Feb 12, 2020• 5 citations
arXivHTMLPDF

Abstract

Tensor network (TN) techniques - often used in the context of quantum many-body physics - have shown promise as a tool for tackling machine learning (ML) problems. The application of TNs to ML, however, has mostly focused on supervised and unsupervised learning. Yet, with their direct connection to hidden Markov chains, TNs are also naturally suited to Markov decision processes (MDPs) which provide the foundation for reinforcement learning (RL). Here we introduce a general TN formulation of finite, episodic and discrete MDPs. We show how this formulation allows us to exploit algorithms developed for TNs for policy optimisation, the key aim of RL. As an application we consider the issue - formulated as an RL problem - of finding a stochastic evolution that satisfies specific dynamical conditions, using the simple example of random walk excursions as an illustration.

DOI
10.48550/arxiv.2002.05185

Related papers

A Tensor Network Implementation of Multi Agent Reinforcement Learning

Sunny Howard

Recently it has been shown that tensor networks (TNs) have the ability to represent the expected return of a single-agent finite Markov decision process (FMDP). The TN represents a distribution model, where all possible trajectories are considered. When extending these ideas to a multi-agent setting, distribution models suffer from the curse of dimensionality: the exponential relation between the number of possible trajectories and the number of agents. The key advantage of using TNs in this setting is that there exists a large number of established optimisation and decomposition techniques that are specific to TNs, that one can apply to ensure the most efficient representation is found. In this report, these methods are used to form a TN that represents the expected return of a multi-agent reinforcement learning (MARL) task. This model is then applied to a 2 agent random walker example, where it was shown that the policy is correctly optimised using a DMRG technique. Finally, I demonstrate the use of an exact decomposition technique, reducing the number of elements in the tensors by 97.5%, without experiencing any loss of information.

arXiv:2401.03896•Jan 8, 2024•1 citation

Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations

Edward Gillman, Dominic C. Rose, Juan P. Garrahan

We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.

arXiv:2209.14089•Physical Review Letters•May 7, 2024•9 citations

Discrete generative diffusion models without stochastic differential equations: A tensor network approach

Luke Causer, Grant M. Rotskoff, Juan P. Garrahan

Diffusion models (DMs) are a class of generative machine learning methods that sample a target distribution by transforming samples of a trivial (often Gaussian) distribution using a learned stochastic differential equation. In standard DMs, this is done by learning a "score function" that reverses the effect of adding diffusive noise to the distribution of interest. Here we consider the generalisation of DMs to lattice systems with discrete degrees of freedom, and where noise is added via Markov chain jump dynamics. We show how to use tensor networks (TNs) to efficiently define and sample such "discrete diffusion models" (DDMs) without explicitly having to solve a stochastic differential equation. We show the following: (i) by parametrising the data and evolution operators as TNs, the denoising dynamics can be represented exactly; (ii) the auto-regressive nature of TNs allows to generate samples efficiently and without bias; (iii) for sampling Boltzmann-like distributions, TNs allow to construct an efficient learning scheme that integrates well with Monte Carlo. We illustrate this approach to study the equilibrium of two models with non-trivial thermodynamics, the d=1 constrained Fredkin chain and the d=2 Ising model.

arXiv:2407.11133•Physical review. E•Feb 7, 2025•2 citations

Tensor Networks for Interpretable and Efficient Quantum-Inspired Machine Learning

Shi-Ju Ran, Gang Su

It is a critical challenge to simultaneously achieve high interpretability and high efficiency with the current schemes of deep machine learning (ML). The tensor network (TN), a well-established mathematical tool originating from quantum mechanics, has shown its unique advantages in developing efficient “white-box” ML schemes. Here, we provide a brief review of the inspiring progress in TN-based ML. On the one hand, the interpretability of TN ML can be accommodated by a solid theoretical foundation based on quantum information and many-body physics. On the other hand, high efficiency can be obtained from powerful TN representations and the advanced computational techniques developed in quantum many-body physics. Keeping pace with the rapid development of quantum computers, TNs are expected to produce novel schemes runnable on quantum hardware in the direction of “quantum artificial intelligence” in the near future.

arXiv:2311.11258•Intelligent Computing•Jan 1, 2023•24 citations

Tensor Network States: Optimizations and Applications in Quantum Many-Body Physics and Machine Learning

Justin A. Reyes

Tensor network states are ubiquitous in the investigation of quantum many-body (QMB) physics. Their advantage over other state representations is evident from their reduction in the computational complexity required to obtain various quantities of interest, namely observables. Additionally, they provide a natural platform for investigating entanglement properties within a system. In this dissertation, we develop various novel algorithms and optimizations to tensor networks for the investigation of QMB systems, including classical and quantum circuits. Specifically, we study optimizations for the two-dimensional Ising model in a transverse field, we create an algorithm for the $k$-SAT problem, and we study the entanglement properties of random unitary circuits. In addition to these applications, we reinterpret renormalization group principles from QMB physics in the context of machine learning to develop a novel algorithm for the tasks of classification and regression, and then utilize machine learning architectures for the time evolution of operators in QMB systems.

Journal of International Crisis and Risk Communication Research•Jan 1, 2020•0 citations

Planning with tensor networks based on active inference

Samuel T. Wauthier, Tim Verbelen, Bart Dhoedt +1 more

Abstract Tensor networks (TNs) have seen an increase in applications in recent years. While they were originally developed to model many-body quantum systems, their usage has expanded into the field of machine learning. This work adds to the growing range of applications by focusing on planning by combining the generative modeling capabilities of matrix product states and the action selection algorithm provided by active inference. Their ability to deal with the curse of dimensionality, to represent probability distributions, and to dynamically discover hidden variables make matrix product states specifically an interesting choice to use as the generative model in active inference, which relies on ‘beliefs’ about hidden states within an environment. We evaluate our method on the T-maze and Frozen Lake environments, and show that the TN-based agent acts Bayes optimally as expected under active inference.

Machine Learning Science and Technology•Aug 29, 2024•0 citations
Tensor network (TN) techniques - often used in the context of quantum many-body physics - have shown promise as a tool for tackling machine learning (ML) problems. The application of TNs to ML, however, has mostly focused on supervised and unsupervised learning. Yet, with their direct connection to hidden Markov chains, TNs are also naturally suited to Markov decision processes (MDPs) which provide the foundation for reinforcement learning (RL). Here we introduce a general TN formulation of finite, episodic and discrete MDPs. We show how this formulation allows us to exploit algorithms developed for TNs for policy optimisation, the key aim of RL. As an application we consider the issue - formulated as an RL problem - of finding a stochastic evolution that satisfies specific dynamical conditions, using the simple example of random walk excursions as an illustration.
Recently it has been shown that tensor networks (TNs) have the ability to represent the expected return of a single-agent finite Markov decision process (FMDP). The TN represents a distribution model, where all possible trajectories are considered. When extending these ideas to a multi-agent setting, distribution models suffer from the curse of dimensionality: the exponential relation between the number of possible trajectories and the number of agents. The key advantage of using TNs in this setting is that there exists a large number of established optimisation and decomposition techniques that are specific to TNs, that one can apply to ensure the most efficient representation is found. In this report, these methods are used to form a TN that represents the expected return of a multi-agent reinforcement learning (MARL) task. This model is then applied to a 2 agent random walker example, where it was shown that the policy is correctly optimised using a DMRG technique. Finally, I demonstrate the use of an exact decomposition technique, reducing the number of elements in the tensors by 97.5%, without experiencing any loss of information.
We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.
Diffusion models (DMs) are a class of generative machine learning methods that sample a target distribution by transforming samples of a trivial (often Gaussian) distribution using a learned stochastic differential equation. In standard DMs, this is done by learning a "score function" that reverses the effect of adding diffusive noise to the distribution of interest. Here we consider the generalisation of DMs to lattice systems with discrete degrees of freedom, and where noise is added via Markov chain jump dynamics. We show how to use tensor networks (TNs) to efficiently define and sample such "discrete diffusion models" (DDMs) without explicitly having to solve a stochastic differential equation. We show the following: (i) by parametrising the data and evolution operators as TNs, the denoising dynamics can be represented exactly; (ii) the auto-regressive nature of TNs allows to generate samples efficiently and without bias; (iii) for sampling Boltzmann-like distributions, TNs allow to construct an efficient learning scheme that integrates well with Monte Carlo. We illustrate this approach to study the equilibrium of two models with non-trivial thermodynamics, the d=1 constrained Fredkin chain and the d=2 Ising model.
It is a critical challenge to simultaneously achieve high interpretability and high efficiency with the current schemes of deep machine learning (ML). The tensor network (TN), a well-established mathematical tool originating from quantum mechanics, has shown its unique advantages in developing efficient “white-box” ML schemes. Here, we provide a brief review of the inspiring progress in TN-based ML. On the one hand, the interpretability of TN ML can be accommodated by a solid theoretical foundation based on quantum information and many-body physics. On the other hand, high efficiency can be obtained from powerful TN representations and the advanced computational techniques developed in quantum many-body physics. Keeping pace with the rapid development of quantum computers, TNs are expected to produce novel schemes runnable on quantum hardware in the direction of “quantum artificial intelligence” in the near future.
Tensor network states are ubiquitous in the investigation of quantum many-body (QMB) physics. Their advantage over other state representations is evident from their reduction in the computational complexity required to obtain various quantities of interest, namely observables. Additionally, they provide a natural platform for investigating entanglement properties within a system. In this dissertation, we develop various novel algorithms and optimizations to tensor networks for the investigation of QMB systems, including classical and quantum circuits. Specifically, we study optimizations for the two-dimensional Ising model in a transverse field, we create an algorithm for the kkk-SAT problem, and we study the entanglement properties of random unitary circuits. In addition to these applications, we reinterpret renormalization group principles from QMB physics in the context of machine learning to develop a novel algorithm for the tasks of classification and regression, and then utilize machine learning architectures for the time evolution of operators in QMB systems.
Abstract Tensor networks (TNs) have seen an increase in applications in recent years. While they were originally developed to model many-body quantum systems, their usage has expanded into the field of machine learning. This work adds to the growing range of applications by focusing on planning by combining the generative modeling capabilities of matrix product states and the action selection algorithm provided by active inference. Their ability to deal with the curse of dimensionality, to represent probability distributions, and to dynamically discover hidden variables make matrix product states specifically an interesting choice to use as the generative model in active inference, which relies on ‘beliefs’ about hidden states within an environment. We evaluate our method on the T-maze and Frozen Lake environments, and show that the TN-based agent acts Bayes optimally as expected under active inference.