We investigate the physics of a gas of ultracold atoms coupled to three single-mode optical cavities and transversely pumped with a laser. Recent work has demonstrated that, for two cavities, the
Z2 symmetries of each cavity can be combined into a global U(1) symmetry [J. L\'eonard et al., Nature (London) 543, 87 (2017)]. Here we show that when adding an extra cavity mode, the low-energy description of this system can additionally exhibit an SO(3) rotational symmetry which can be spontaneously broken. This leads to a superradiant phase transition in all the cavities simultaneously, and the appearance of Goldstone and amplitude modes in the excitation spectrum. We determine the phase diagram of the system, which shows the emergence and breaking of the continuous symmetries and displays first- and second-order phase transitions. We also obtain the excitation spectrum for each phase and discuss the atomic self-organized structures that emerge in the different superradiant phases. We argue that coupling the atoms equally to
n different modes will in general generate a global
SO(n) symmetry if the mode frequencies can be tuned to the same value.
Atoms in transversely pumped optical cavities ``self-organize'' by forming a density wave and emitting superradiantly into the cavity mode(s). For a single-mode cavity, the properties of this self-organization transition are well characterized both theoretically and experimentally. Here, we explore the self-organization of a Bose-Einstein condensate in the presence of two cavity modes---a system that recently was realized experimentally [L\'eonard et al., Nature (London) 543, 87 (2017)]. We argue that this system can exhibit a ``vestigially ordered'' phase in which neither cavity mode exhibits superradiance but the cavity modes are mutually phase locked by the atoms. We argue that this vestigially ordered phase should generically be present in multimode cavity geometries.