We study a quantum version of the three-state Potts model that includes as special cases the effective models of bosons and fermions on the square lattice in the Mott-insulating limit. It can be viewed as a model of quantum permutations with amplitudes ${J}_{\ensuremath{\parallel}}$ and ${J}_{\ensuremath{\perp}}$ for identical and different colors, respectively. For ${J}_{\ensuremath{\parallel}}={J}_{\ensuremath{\perp}}>0$ it is equivalent to the SU(3) Heisenberg model, which describes the Mott-insulating phase of 3-color fermions, while the parameter range ${J}_{\ensuremath{\perp}}<\mathrm{min}(0,\ensuremath{-}{J}_{\ensuremath{\parallel}})$ can be realized in the Mott insulating phase of 3-color bosonic atoms. Using linear flavor wave theory, infinite projected entangled-pair states (iPEPS), and continuous-time quantum Monte Carlo simulations, we construct the full
T=0 phase diagram, and we explore the $T\ensuremath{\ne}0$ properties for ${J}_{\ensuremath{\perp}}<0$. For dominant antiferromagnetic ${J}_{\ensuremath{\parallel}}$ interactions, a three-sublattice long-range ordered stripe state is selected out of the ground-state manifold of the antiferromagnetic Potts model by quantum fluctuations. Upon increasing $|{J}_{\ensuremath{\perp}}|$, this state is replaced by a uniform superfluid for ${J}_{\ensuremath{\perp}}<0$, and by an exotic three-sublattice superfluid followed by a two-sublattice superfluid for ${J}_{\ensuremath{\perp}}>0$. The transition out of the uniform superfluid (that can be realized with bosons) is shown to be a peculiar type of Kosterlitz-Thouless transition with three types of elementary vortices.