We study a spin model on honeycomb lattice with two types of Heisenberg exchange couplings,
J and
J~, where
J is for the conventional spin and and
J~ for rotated spin. When
J~=0, this is the conventional Heisenberg model. When J=0, the system is either in a stripy antiferromagnetic order(
J~0, antiferromagnetic for rotated spin). The competition between two ferromagnetic orders or two antiferromagnetic orders induces Kitaev's spin liquid phase characterized by the exactly solvable Kitaev model (
J=J~). Our model can be applied to layered Mott insulators A
2IrO
3 (A=Li, Na). For a monolayer of Li
2IrO
3, we show that it is possible to tune the controlling parameter into the Kitaev spin liquid regime by a link-dependent Rashba spin-orbital coupling.
Recent experiments show that the ground state of some layered materials with localized moments is in close proximity to the Kitaev spin liquid, calling for a proper model to describe the measurements. The Kitaev-Hubbard (KHu) model is the minimal model that captures the essential ingredients of these systems; it yields the Kitaev-Heisenberg spin model at the strong coupling limit and contains the charge fluctuations present in these materials as well. Despite its relevance, the phase diagram of the KHu model has not been rigorously revealed yet. In this work, we study the full phase diagram of the KHu model using the
Z2 slave-spin mean-field theory as well as the auxiliary field quantum Monte Carlo on rather large systems and at low temperatures. The Mott transition is signaled by a vanishing quasiparticle weight evaluated using the slave-spin construction. Moreover, we demonstrate that there are multiple magnetic phase transitions within the Mott phase including magnetically ordered phases and most notably a quantum spin liquid phase for $1.0\ensuremath{\lesssim}{t}^{\ensuremath{'}}/t\ensuremath{\lesssim}1.11$ at
U/t=5.
We study the extent of the spin liquid phases in the Kitaev-Heisenberg model using infinite projected entangled-pair states tensor network ansatz wave functions directly in the thermodynamic limit. To assess the accuracy of the ansatz wave functions, we perform benchmarks against exact results for the Kitaev model and find very good agreement for various observables. In the case of the Kitaev-Heisenberg model, we confirm the existence of six different phases: N\'eel, stripy, ferromagnetic, zigzag, and two spin liquid phases. We find finite extents for both spin liquid phases and discontinuous phase transitions connecting them to symmetry-broken phases.