We study the Kitaev--Heisenberg model on a triangular lattice by using the two-dimensional density-matrix renormalization group method. Calculating the ground-state energy and spin structure factors, we obtain a ground-state phase diagram of the Kitaev--Heisenberg model. As suggested by previous studies, we find a 120
∘ antiferromagnetic (AFM) phase, a
Z2-vortex crystal phase, a nematic phase, a dual
Z2-vortex crystal phase (the dual counterpart of the
Z2-vortex crystal phase), a
Z6 ferromagnetic phase, and a dual ferromagnetic phase (the dual counterpart of the $\mathbb{Z}_6 $ ferromagnetic phase). Spin correlations discontinuously change at phase boundaries because of first-order phase transitions. We also study the relation among the von Neumann entanglement entropy, entanglement spectrum, and phase transitions of the model. We find that the Schmidt gap closes at phase boundaries and thus the entanglement entropy clearly changes as well. This is different from the Kitaev--Heisenberg model on a honeycomb lattice, where the Schmidt gap and entanglement entropy are not necessarily a good measure of phase transitions.
The Kitaev-Heisenberg model defined on both honeycomb and triangular lattices has been studied intensively in recent years as a possible model to describe spin-orbital physics in iridium oxides. In the model, there are many phases characteristic for each lattice. However, there is no study of how the phases in the two lattices merge with each other when geometry changes from the honeycomb lattice to triangular lattice. We investigate the ground state of the Kitaev-Heisenberg model defined on the system connecting the honeycomb and triangular lattices, named a honeycomb-triangular lattice. We obtain a ground state phase diagram of this model with classical spins by using the Luttinger-Tisza method and classical Monte Carlo simulation. In addition to known phases in the honeycomb and triangular lattices, we find coexisting phases consisting of the known phases. From the exact diagonalization and density-matrix renormalization-group calculations for the model with quantum spins, we find phases similar to the classical results, implying a small effect of quantum fluctuation on the phase diagram.