Recent repoet on the formation of two-dimensional Bose-Einstein condensates (BECs) of spinning g-wave molecules is surprise. Here we study quantum phase transition in the quasi-2D atomic Bose gas with a g-wave Feshbach resonance, and show that there are two phase transitions in this system: from a phase with only a atomic Bose-Einstein condensate to a phase with coexistence of atomic and molecular condensation and from the coexistence phase to a phase with only a molecular Bose-Einstein condensate. We show that the g-wave resonance Feshbach has two features in the phase transitions, a phase with only a atomic Bose-Einstein condensate is no longer forbidden and resonance effects appear only at levels beyond the mean-field. We determine the T=0 beyond mean-field phase diagram of the gas as a function of magnetic field and molecular condensation density. We also detemine the ratio of atomic to molecular condensation in the coexistence phase, which can be tested in experiments.
In this paper, properties of a homogeneous Bose gas with a Feshbach resonance are studied in the dilute region at zero temperature. The stationary state contains condensations of atoms and molecules. The ratio of the molecule density to the atom density is $\ensuremath{\pi}{\mathrm{na}}^{3}.$ There are two types of excitations, molecular excitations and atomic excitations. Atomic excitations are gapless, consistent with the traditional theory of a dilute Bose gas. The molecular excitation energy is finite in the long-wavelength limit as observed in recent experiments on
85Rb. In addition, the decay process of the condensate is studied. The coefficient of the three-body recombination rate is about 140 times larger than that of a Bose gas without a Feshbach resonance, in reasonably good agreement with the experiment on
23Na.