We present the theory of a three-dimensional fractionalized topological insulator in the form of a
U(1) spin liquid with gapped fermionic spinons in the bulk and topologically protected gapless spinon surface states. Starting from a spin-rotation-invariant spin-
1/2 model on a pyrochlore lattice with frustrated antiferromagnetic and ferromagnetic exchange interactions, we show that decomposition of the latter interactions, within slave-fermion representation of the spins, can naturally give rise to an emergent spin-orbit coupling for the spinons by spontaneously breaking the spin rotation symmetry. The time-reversal symmetry, however, is preserved. This stabilizes a fractionalized topological insulator which also has bulk bond spin-nematic order. We describe the low-energy properties of this state.
We propose an experimentally feasible setup with ultracold alkaline-earth-metal atoms to simulate the dynamics of U(1) lattice gauge theories in 2
+ 1 dimensions with a Chern-Simons term. To this end we consider the ground-state properties of spin-5/2 alkaline-earth-metal fermions in a honeycomb lattice. We use the Gutzwiller projected variational approach in the strongly repulsive regime in the case of filling 1/6. The ground state of the system is a chiral spin-liquid state with $2\ensuremath{\pi}/3$ flux per plaquette, which violates time-reversal invariance. We demonstrate that due to the breaking of time-reversal symmetry the system exhibits quantum Hall effect and chiral edge states. We relate the experimentally accessible spin fluctuations to the emerging gauge-field dynamics. We discuss also properties of the lowest energy competing orders.