We theoretically investigate the thermodynamic stability of a normal-state Bose-Fermi mixture, with a tunable Bose-Fermi pairing interaction $\ensuremath{-}{U}_{\mathrm{BF}}<0$ associated with a heteronuclear Feshbach resonance, as well as a weak repulsive Bose-Bose interaction ${U}_{\mathrm{BB}}\ensuremath{\ge}0$. Including strong heteropairing fluctuations associated with the former interaction within the self-consistent
T-matrix approximation, as well as the latter within the mean-field level, we calculate the compressibility matrix, to assess the stability of this system against density fluctuations. In the weak- and the intermediate-coupling regimes with respect to $\ensuremath{-}{U}_{\mathrm{BF}}$, we show that an effective attractive interaction between bosons mediated by density fluctuations in the Fermi component makes the system unstable below a certain temperature
Tclp (leading to density collapse). When
UBB=0,Tclp is always higher than the Bose-Einstein condensation (BEC) temperature
Tc. When
UBB>0, the density collapse is suppressed, and the BEC transition becomes possible. It is also suppressed by the formation of tightly bound Bose-Fermi molecules when the heteropairing interaction $\ensuremath{-}{U}_{\mathrm{BF}}$ is strong; however, since the system may be viewed as a molecular Fermi gas in this case, the BEC transition does not also occur. Since quantum gases involving Bose atoms are known to be sensitive to interparticle correlations, our results would be useful for the study of many-body properties of a Bose-Fermi mixture in a stable manner, without facing the unwanted density collapse.
We investigate a three-site ring system with a small number of quantum degenerate bosons and fermions. By means of the exact diagonalization of the Bose-Fermi-Hubbard Hamiltonian, we show that the symmetry of the ground state configuration is a function of both the boson-boson and the inter-species interaction in the system. The phase diagram of the system, constructed by computing the exact two-body spatial correlations, reveals nontrivial insulating phases that exist even in the strong bosonic tunneling limit and for incommensurate filling of bosons. These insulating phases are due to the inter-species interactions in the system and are not necessarily accompanied by the suppression of the particle number fluctuations.