We report on measurements of the binding energies of several weakly bound vibrational states of the paramagnetic
174Yb6Li molecule in the electronic ground state using two-photon spectroscopy in an ultracold atomic mixture confined in an optical dipole trap. We theoretically analyze the experimental spectrum to obtain an accurate description of the long-range potential of the ground-state molecule. Based on the measured binding energies, we arrive at an improved value of the interspecies
s-wave scattering length
as0=30a0. Employing coherent two-photon spectroscopy we also observe the creation of ``dark'' atom-molecule superposition states in the heteronuclear Yb-Li system. This paper is an important step towards the efficient production of ultracold YbLi molecules via association from an ultracold atomic mixture.
We derive the relativistic formulas for the van der Waals coefficients of Yb--alkali-metal dimers that correlate to ground and excited separated-atom limits. We calculate
C6 and
C8 coefficients of particular experimental interest. We also derive a semiempirical formula that expresses the
C8 coefficient of heteronuclear
A+B dimers in terms of the
C6 and
C8 coefficients of homonuclear dimers and the static dipole and quadrupole polarizabilities of the atomic states
A and
B. We report results of calculation of the
C6 coefficients for the Yb-Rb
3P1o+5s2S1/2 and
1S0+5p2P1/2o dimers, and the
C8 coefficients for the Yb-Li
1S0+2s2S1/2 and Yb-Rb
1S0+5s2S1/2 dimers. Uncertainties are estimated for all predicted properties.