Quantum and thermal phase transitions in a bosonic atom-molecule mixture in a two-dimensional optical lattice

Abstract

We study the ground state and the thermal phase diagram of a two-species Bose-Hubbard model, with $\text{U}(1)\ifmmode\times\else\texttimes\fi{}{\mathbb{Z}}_{2}$ symmetry, describing atoms and molecules on a two-dimensional optical lattice interacting via a Feshbach resonance. Using quantum Monte Carlo simulations and mean-field theory, we show that the conversion between the two species, coherently coupling the atomic and molecular states, has a crucial impact on the Mott-superfluid transition and stabilizes an insulating phase with a gap controlled by the conversion term---the Feshbach insulator---instead of a standard Mott-insulating phase. Depending on the detuning between atoms and molecules, this model exhibits three phases: the Feshbach insulator, a molecular condensate coexisting with noncondensed atoms, and a mixed atomic-molecular condensate. Employing finite-size scaling analysis, we observe three-dimensional (3D) $XY$ (3D Ising) transition when $\text{U}(1)$ (${\mathbb{Z}}_{2}$) symmetry is broken, whereas the transition is first order when both $\text{U}(1)$ and ${\mathbb{Z}}_{2}$ symmetries are spontaneously broken. The finite-temperature phase diagram is also discussed. The thermal disappearance of the molecular superfluid leads to a Berezinskii-Kosterlitz-Thouless transition with unusual universal jump in the superfluid density. The loss of the quasi-long-range coherence of the mixed atomic and molecular superfluid is more subtle since only atoms exhibit conventional Berezinskii-Kosterlitz-Thouless criticality. We also observe a signal compatible with a classical first-order transition between the mixed superfluid and the normal Bose liquid at low temperature.