Regular and irregular spin-mixing dynamics are studied in a spin-1 atomic Bose-Einstein condensate coupled with a molecular condensate. Based on the experimental photoassociation spectroscopy of
87Rb spin-1 atoms, a two-channel model is proposed, in which two zero-spin atoms can be associated with a molecular state via a laser field, and a pair of
+1- and $\ensuremath{-}1$-spin atoms can also be associated with the same molecular state via the same laser field. We find that the spin-mixing dynamics can be manipulated by the laser field and a magnetic field. Both regular and irregular oscillations can be observed under different parameters. Using the Fourier transform analysis, the spin-mixing dynamics may be changed from the regular oscillation with localized spectral distributions to irregular oscillation with a broad spectral distribution.
The form of the wavefunction psi for a semiclassical regular quantum state (associated with classical motion on an N-dimensional torus in the 2N-dimensional phase space) is very different from the form of psi for an irregular state (associated with stochastic classical motion on all or part of the (2N-1)-dimensional energy surface in phase space). For regular states the local average probability density Pi rises to large values on caustics at the boundaries of the classically allowed region in coordinate space, and psi exhibits strong anisotropic interference oscillations. For irregular states Pi falls to zero (or in two dimensions stays constant) on 'anticaustics' at the boundary of the classically allowed region, and psi appears to be a Gaussian random function exhibiting more moderate interference oscillations which for ergodic classical motion are statistically isotropic with the autocorrelation of psi given by a Bessel function.