Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Log inSign up
Challenges
Datasets
Workspaces
Papers
Discussions
Leaderboard
Blog
Job Board
Q3AS

© 2026 Aqora Quantum S.A.S.

TermsPrivacyLegal Notice

arXiv:2311.16074 · International Symposium on Lattice Field Theory (LATTICE)

A solution for infinite variance problem of fermionic observables

Hyunwoo Oh, Andrei Alexandru, Paulo F. Bedaque, Andrea Carosso

Published Dec 27, 2023• 0 citations
arXivHTMLPDF

Abstract

Fermionic Monte Carlo calculations with continuous auxiliary fields often encounter infinite variance problem from fermionic observables. This issue renders the estimation of observables unreliable, even with an infinite number of samples. In this work, we show that the infinite variance problem stems from the fermionic determinant. Also, we propose an approach to address this problem by employing a reweighting method that utilizes the distribution from an extra time-slice. Two strategies to compute the reweighting factor are explored: one involves truncating and analytically calculating the reweighting factor, while the other employs a secondary Monte Carlo estimation. With Hubbard model as a testbed, we demonstrate that utilizing the sub-Monte Carlo estimation, coupled with an unbiased estimator, offers a solution that effectively mitigates the infinite variance problem at a minimal additional cost.

DOI
10.22323/1.453.0021

Related papers

Infinite variance problem in fermion models

Andrei Alexandru, Paulo F. Bedaque, Andrea Carosso +1 more

Monte Carlo calculations of fermionic systems with continuous auxiliary fields frequently suffer from a diverging variance for fermionic observables. If the simulations have an infinite variance problem, one cannot estimate observables reliably even with an arbitrarily large number of samples. In this paper, we explore a method to solve this problem using sampling based on the distribution of a system with an extra time slice. The necessary reweighting factor is computed both perturbatively and through a secondary Monte Carlo. We show that the Monte Carlo reweighting coupled to the use of an unbiased estimator of the reweighting factor leads to a method that eliminates the infinite variance problem at a very small extra cost. We compute the double occupancy in the Hubbard model at half-filling to demonstrate the method and compare the results to well-established results obtained by other methods.

arXiv:2211.06419•Physical review. D/Physical review. D.•May 3, 2023•7 citations

Addressing the infinite variance problem in fermionic Monte Carlo simulations: Retrospective error remediation and the exact bridge link method

Zhou‐Quan Wan, Shiwei Zhang

We revisit the infinite variance problem in fermionic Monte Carlo simulations, which is widely encountered in areas ranging from condensed matter to nuclear and high-energy physics. The different algorithms, which we broadly refer to as determinantal quantum Monte Carlo (DQMC), are applied in many situations and differ in details, but they share a foundation in field theory, and often involve fermion determinants whose symmetry properties make the algorithm sign-problem-free. We show that the infinite variance problem arises as the observables computed in DQMC tend to form heavy-tailed distributions. To remedy this issue retrospectively, we introduce a tail-aware error estimation method to correct the otherwise unreliable estimates of confidence intervals. Furthermore, we demonstrate how to perform DQMC calculations that eliminate the infinite variance problem for a broad class of observables. Our approach is an exact bridge link method, which involves a simple and efficient modification to the standard DQMC algorithm. The method introduces no systematic bias and is straightforward to implement with minimal computational overhead. Our results establish a practical and robust solution to the infinite variance problem, with broad implications for improving the reliability of a variety of fundamental fermion simulations.

arXiv:2507.08937•Physical review. E•Nov 24, 2025•2 citations

Infinite variance in fermion quantum Monte Carlo calculations

Hao Shi, Shiwei Zhang

For important classes of many-fermion problems, quantum Monte Carlo (QMC) methods allow exact calculations of ground-state and finite-temperature properties without the sign problem. The list spans condensed matter, nuclear physics, and high-energy physics, including the half-filled repulsive Hubbard model, the spin-balanced atomic Fermi gas, and lattice quantum chromodynamics calculations at zero density with Wilson Fermions, and is growing rapidly as a number of problems have been discovered recently to be free of the sign problem. In these situations, QMC calculations are relied on to provide definitive answers. Their results are instrumental to our ability to understand and compute properties in fundamental models important to multiple subareas in quantum physics. It is shown, however, that the most commonly employed algorithms in such situations have an infinite variance problem. A diverging variance causes the estimated Monte Carlo statistical error bar to be incorrect, which can render the results of the calculation unreliable or meaningless. We discuss how to identify the infinite variance problem. An approach is then proposed to solve the problem. The solution does not require major modifications to standard algorithms, adding a "bridge link" to the imaginary-time path integral. The general idea is applicable to a variety of situations where the infinite variance problem may be present. Illustrative results are presented for the ground state of the Hubbard model at half-filling.

arXiv:1511.04084•Physical review. E•Mar 7, 2016•59 citations

A method to estimate observables with infinite variance in fermionic systems

Çağın Yunus, William Detmold

Numerical estimation of fermionic observables often requires introduction of auxiliary bosonic fieldsthat have no direct physical relevance through a Hubbard-Stratonovich transformation. The varianceof some fermionic observables (for example, appropriately constructed local four-fermion operatorsin 2d Gross-Neveu model) when they are estimated using such auxiliary fields may not correspondto any physical observable and in particular may diverge. In such cases it is not clear how onecan reliably estimate the corresponding observable. An important example may be found in thecontext of exceptional configurations in quenched QCD. We demonstrate simpler examples in toymodels by constructing particular four-fermion operators. We propose two methods to overcome thisissue. The first method is a discrete Hubbard-Stratonovich transformation that does not suffer fromsuch divergences for these classes of observables. While this method works in principle, it is notpractically feasible. The second method we propose is a reweighting method.

International Symposium on Lattice Field Theory (LATTICE)•May 16, 2022•1 citation

Efficient calculation of unbiased expectation values in diffusion quantum Monte Carlo

Manolo C. Per, Ian K. Snook, Salvy P. Russo

Despite the proven utility of quantum Monte Carlo methods in addressing the quantum many-body problem, many important observables are difficult to calculate due to the presence of large, and sometimes divergent, statistical errors. The present state of the art allows the construction of renormalized estimators which result in finite variances, but which invariably include some systematic bias. We present a simple method for calculating unbiased expectation values of local operators in the diffusion quantum Monte Carlo method which is applicable to both bare and renormalized estimators, allowing the accurate calculation of important properties such as forces.

Physical Review B•Nov 29, 2012•12 citations

Mitigating spikes in fermion Monte Carlo methods by reshuffling measurements

Maksim Ulybyshev, Fakher F. Assaad

We propose a method to mitigate heavy-tailed distributions in fermion quantum Monte Carlo simulations originating from zeros of the fermion determinant. In this case, the second moment of the observables might not be well defined, and we show that by merely changing the synchronization between local updates and computation of observables, one can reduce the prefactor of the heavy-tailed distribution, thus substantially suppressing statistical fluctuations of observables. We also show that the average, or the first moment, is well defined and hence is independent on our measuring scheme. The method is especially suitable for local observables similar to, e.g., double occupancy, where the resulting speedup can reach two orders of magnitude. For observables containing spatial correlators, the speedup is more moderate but still ranges between five and ten. Our results are independent on the nature of the auxiliary field, discrete or continuous, and pave the way to improve measurement strategies for hybrid Monte Carlo simulations.

arXiv:2112.05260•Physical review. E•Aug 30, 2022•6 citations
Fermionic Monte Carlo calculations with continuous auxiliary fields often encounter infinite variance problem from fermionic observables. This issue renders the estimation of observables unreliable, even with an infinite number of samples. In this work, we show that the infinite variance problem stems from the fermionic determinant. Also, we propose an approach to address this problem by employing a reweighting method that utilizes the distribution from an extra time-slice. Two strategies to compute the reweighting factor are explored: one involves truncating and analytically calculating the reweighting factor, while the other employs a secondary Monte Carlo estimation. With Hubbard model as a testbed, we demonstrate that utilizing the sub-Monte Carlo estimation, coupled with an unbiased estimator, offers a solution that effectively mitigates the infinite variance problem at a minimal additional cost.
Monte Carlo calculations of fermionic systems with continuous auxiliary fields frequently suffer from a diverging variance for fermionic observables. If the simulations have an infinite variance problem, one cannot estimate observables reliably even with an arbitrarily large number of samples. In this paper, we explore a method to solve this problem using sampling based on the distribution of a system with an extra time slice. The necessary reweighting factor is computed both perturbatively and through a secondary Monte Carlo. We show that the Monte Carlo reweighting coupled to the use of an unbiased estimator of the reweighting factor leads to a method that eliminates the infinite variance problem at a very small extra cost. We compute the double occupancy in the Hubbard model at half-filling to demonstrate the method and compare the results to well-established results obtained by other methods.
We revisit the infinite variance problem in fermionic Monte Carlo simulations, which is widely encountered in areas ranging from condensed matter to nuclear and high-energy physics. The different algorithms, which we broadly refer to as determinantal quantum Monte Carlo (DQMC), are applied in many situations and differ in details, but they share a foundation in field theory, and often involve fermion determinants whose symmetry properties make the algorithm sign-problem-free. We show that the infinite variance problem arises as the observables computed in DQMC tend to form heavy-tailed distributions. To remedy this issue retrospectively, we introduce a tail-aware error estimation method to correct the otherwise unreliable estimates of confidence intervals. Furthermore, we demonstrate how to perform DQMC calculations that eliminate the infinite variance problem for a broad class of observables. Our approach is an exact bridge link method, which involves a simple and efficient modification to the standard DQMC algorithm. The method introduces no systematic bias and is straightforward to implement with minimal computational overhead. Our results establish a practical and robust solution to the infinite variance problem, with broad implications for improving the reliability of a variety of fundamental fermion simulations.
For important classes of many-fermion problems, quantum Monte Carlo (QMC) methods allow exact calculations of ground-state and finite-temperature properties without the sign problem. The list spans condensed matter, nuclear physics, and high-energy physics, including the half-filled repulsive Hubbard model, the spin-balanced atomic Fermi gas, and lattice quantum chromodynamics calculations at zero density with Wilson Fermions, and is growing rapidly as a number of problems have been discovered recently to be free of the sign problem. In these situations, QMC calculations are relied on to provide definitive answers. Their results are instrumental to our ability to understand and compute properties in fundamental models important to multiple subareas in quantum physics. It is shown, however, that the most commonly employed algorithms in such situations have an infinite variance problem. A diverging variance causes the estimated Monte Carlo statistical error bar to be incorrect, which can render the results of the calculation unreliable or meaningless. We discuss how to identify the infinite variance problem. An approach is then proposed to solve the problem. The solution does not require major modifications to standard algorithms, adding a "bridge link" to the imaginary-time path integral. The general idea is applicable to a variety of situations where the infinite variance problem may be present. Illustrative results are presented for the ground state of the Hubbard model at half-filling.
Numerical estimation of fermionic observables often requires introduction of auxiliary bosonic fieldsthat have no direct physical relevance through a Hubbard-Stratonovich transformation. The varianceof some fermionic observables (for example, appropriately constructed local four-fermion operatorsin 2d Gross-Neveu model) when they are estimated using such auxiliary fields may not correspondto any physical observable and in particular may diverge. In such cases it is not clear how onecan reliably estimate the corresponding observable. An important example may be found in thecontext of exceptional configurations in quenched QCD. We demonstrate simpler examples in toymodels by constructing particular four-fermion operators. We propose two methods to overcome thisissue. The first method is a discrete Hubbard-Stratonovich transformation that does not suffer fromsuch divergences for these classes of observables. While this method works in principle, it is notpractically feasible. The second method we propose is a reweighting method.
Despite the proven utility of quantum Monte Carlo methods in addressing the quantum many-body problem, many important observables are difficult to calculate due to the presence of large, and sometimes divergent, statistical errors. The present state of the art allows the construction of renormalized estimators which result in finite variances, but which invariably include some systematic bias. We present a simple method for calculating unbiased expectation values of local operators in the diffusion quantum Monte Carlo method which is applicable to both bare and renormalized estimators, allowing the accurate calculation of important properties such as forces.
We propose a method to mitigate heavy-tailed distributions in fermion quantum Monte Carlo simulations originating from zeros of the fermion determinant. In this case, the second moment of the observables might not be well defined, and we show that by merely changing the synchronization between local updates and computation of observables, one can reduce the prefactor of the heavy-tailed distribution, thus substantially suppressing statistical fluctuations of observables. We also show that the average, or the first moment, is well defined and hence is independent on our measuring scheme. The method is especially suitable for local observables similar to, e.g., double occupancy, where the resulting speedup can reach two orders of magnitude. For observables containing spatial correlators, the speedup is more moderate but still ranges between five and ten. Our results are independent on the nature of the auxiliary field, discrete or continuous, and pave the way to improve measurement strategies for hybrid Monte Carlo simulations.