We study steady-state continuous-variable entanglement in a three-mode optomechanical system consisting of an active optical cavity (gain) coupled to a passive optical cavity (loss) supporting a mechanical mode. For a driving laser which is blue-detuned, we show that coupling between optical and mechanical modes is enhanced in the unbroken-
PT-symmetry regime. We analyze the stability and this shows that steady-state solutions are more stable in the gain and loss systems. We use these stable solutions to generate distant entanglement between the mechanical mode and the optical field inside the gain cavity. It results in a giant enhancement of entanglement compared to what is generated in the single lossy cavity. This work offers the prospect of exploring quantum state engineering and quantum information in such systems. Furthermore, such entanglement opens up an interesting possibility to study spatially separated quantum objects.
In the field of optomechanics, one typically studies the interaction of a single mechanical resonator and a single optical cavity mode. In this thesis, we investigate collective quantum effects in optomechanical systems with two or more optical and mechanical modes. First, we study the generation and destruction of entanglement between two mechanical resonators effected by optical cavity fields. We also describe how the optomechanical interaction can be employed to mediate strong coherent coupling and entanglement between a micromechanical membrane and a single atom. Secondly, we show that a configuration with two optical and a single mechanical mode can lead to enhanced nonlinear interactions, which can be exploited for quantum non-demolition measurements of the phonon and photon numbers. Finally, we explore the quantum many-body dynamics of optomechanical arrays. In particular, we research into the transition from a disordered state due to quantum noise to a state with phasecoherent mechanical oscillations. As an intermediate step towards this study, we also investigate the nonlinear classical dynamics of single- and multi-mode optomechanical setups and compare theoretical predictions to experimental data.