The translational and internal level cooling of atoms and molecules in ultracold gases results from a combination of elastic and inelastic collisional processes. While elastic collisions lead to rapid thermalization, exoergic inelastic collisions may lead to heating and trap loss. To date, most collisional studies have targeted low-lying levels of diatomic molecules. Here we investigate inelastic quenching and elastic scattering of rotationally excited linear (H
2, HD, CO,
O2, and CO
2) and nonlinear (H
2O and NH
3) molecules in ultracold collisions with He and report the corresponding complex scattering lengths. It has been found that the ratio of the imaginary component $\ensuremath{\beta}$ to the real component $\ensuremath{\alpha}$ of the scattering length generally increases with decreasing rotational constant for linear molecules. With the exception of CO, $\ensuremath{\beta}$ becomes significantly smaller than $\ensuremath{\alpha}$ as the energy gap for rotational transitions increases. In all cases, $\ensuremath{\beta}$ decreases with rotational energy gap for relatively large rotational excitation, allowing for convenient fits to an exponential energy gap formula. Excited rotational levels of
H2 and HD appear to be collisionally stable due to the very low values of $\ensuremath{\beta}/\ensuremath{\alpha}$. Rotationally excited
H2O also appears to be a viable candidate for He buffer gas cooling due to relatively small values of $\ensuremath{\beta}$.
We report experimental observations of the rotational structure in photoassociative spectroscopy of ultralong-range Rydberg molecules (ULRRMs) in an ultracold gas of
86Sr. ULRRM spectroscopy probes scattering wave functions at much larger internuclear separations than photoassociative spectroscopy of low-lying electronic states. At such separations photon recoil momentum can lead to the transfer of significant angular momentum to the Rydberg molecule, evidence of which is provided through the distribution of excited rotational states. The visibility of the rotational structure is enhanced because, for collisions between ground-state
86Sr atoms, the large, near-resonant s-wave scattering length approaches the size of the ULRRM. Similar enhancement is not seen for
84Sr, which has a much smaller scattering length. Results are interpreted with the aid of a theory that accounts for the recoil momentum associated with photoexcitation and the large
s-wave scattering length in the entrance channel. While the observed rotational splittings and the qualitative trends in the relative intensities of the spectral features are well described by theory, the effects of recoil momentum are more prominent in the measured data than in the theory.