We explore the nature of the quantum phase transition between a magnetically ordered state with collinear spin pattern and a gapless
Z2 spin liquid in the Heisenberg-Kitaev model. We construct a slave particle mean-field theory for the Heisenberg-Kitaev model in terms of complex fermionic spinons. It is shown that this theory, formulated in the appropriate basis, is capable of describing the Kitaev spin liquid as well as the transition between the gapless
Z2 spin liquid and the so-called stripy antiferromagnet. Within our mean-field theory, we find a discontinuous transition from the
Z2 spin liquid to the stripy antiferromagnet. We argue that subtle spinon confinement effects, associated with the instability of gapped
U(1) spin liquid in two spatial dimensions, play an important role at this transition. The possibility of an exotic continuous transition is briefly addressed.
The global phase diagram of a doped Kitaev-Heisenberg model is studied using an
SU(2) slave-boson mean-field method. Near the Kitaev limit,
p-wave superconducting states which break the time-reversal symmetry are stabilized as reported by You et al. [Phys. Rev. B 86, 085145 (2012)] irrespective of the sign of the Kitaev interaction. By further doping, a
d-wave superconducting state appears when the Kitaev interaction is antiferromagnetic, while another
p-wave superconducting state appears when the Kitaev interaction is ferromagnetic. This
p-wave superconducting state does not break the time-reversal symmetry as reported by Hyart et al. [Phys. Rev. B 85, 140510 (2012)], and such a superconducting state also appears when the antiferromagnetic Kitaev interaction and the ferromagnetic Heisenberg interaction compete. This work, thus, demonstrates the clear difference between the antiferromagnetic Kitaev model and the ferromagnetic Kitaev model when carriers are doped while these models are equivalent in the undoped limit, and how novel superconducting states emerge when the Kitaev interaction and the Heisenberg interaction compete.
Recent experiments show that the ground state of some layered materials with localized moments is in close proximity to the Kitaev spin liquid, calling for a proper model to describe the measurements. The Kitaev-Hubbard (KHu) model is the minimal model that captures the essential ingredients of these systems; it yields the Kitaev-Heisenberg spin model at the strong coupling limit and contains the charge fluctuations present in these materials as well. Despite its relevance, the phase diagram of the KHu model has not been rigorously revealed yet. In this work, we study the full phase diagram of the KHu model using the
Z2 slave-spin mean-field theory as well as the auxiliary field quantum Monte Carlo on rather large systems and at low temperatures. The Mott transition is signaled by a vanishing quasiparticle weight evaluated using the slave-spin construction. Moreover, we demonstrate that there are multiple magnetic phase transitions within the Mott phase including magnetically ordered phases and most notably a quantum spin liquid phase for $1.0\ensuremath{\lesssim}{t}^{\ensuremath{'}}/t\ensuremath{\lesssim}1.11$ at
U/t=5.