Comprehensive study of the global phase diagram of the J−K−Γ model on a triangular lattice

Abstract

The celebrated Kitaev honeycomb model provides an analytically tractable example with an exact quantum spin liquid ground state. While in real materials, other types of interactions besides the Kitaev coupling ($K$) are present, such as Heisenberg ($J$) and symmetric off-diagonal ($\mathrm{\ensuremath{\Gamma}}$) terms, these interactions can also be generalized to a triangular lattice. Here, we carry out a comprehensive study of the $J\ensuremath{-}K\ensuremath{-}\mathrm{\ensuremath{\Gamma}}$ model on a triangular lattice covering the whole parameter region, using a combination of exact diagonalization, classical Monte Carlo, and analytic methods, with an emphasis on the effects of the $\mathrm{\ensuremath{\Gamma}}$ term. In the HK limit ($\mathrm{\ensuremath{\Gamma}}=0$), we find five quantum phases which are quite similar to their classical counterparts. Among them, stripe-A and dual N\'eel phases are robust against the introduction of the $\mathrm{\ensuremath{\Gamma}}$ term, in particular, stripe A extends to the region connecting $K=\ensuremath{-}1$ and $K=1$ for $\mathrm{\ensuremath{\Gamma}}<0$. Though the ${120}^{\ensuremath{\circ}}$ N\'eel phase also extends to a finite $\mathrm{\ensuremath{\Gamma}}$, its region is largely reduced compared to the previous classical result. Interestingly, the ferromagnetic (dubbed FM-A) and stripe-B phases are unstable in response to an infinitesimal $\mathrm{\ensuremath{\Gamma}}$ interaction. Moreover, we find five additional phases for $\mathrm{\ensuremath{\Gamma}}\ensuremath{\ne}0$ which are elaborated by both the quantum and classical numerical methods. Part of the parameter space previously identified as the ${120}^{\ensuremath{\circ}}$ N\'eel phase in the classical study is found to give way to the modulated stripe phase. Depending on the sign of the $\mathrm{\ensuremath{\Gamma}}$ term, the FM-A phase transits into FM-B ($\mathrm{\ensuremath{\Gamma}}>0$) and FM-C ($\mathrm{\ensuremath{\Gamma}}<0$) phases with different spin orientations. Similarly, the stripe-B phase transits into stripe-C ($\mathrm{\ensuremath{\Gamma}}>0$) and stripe-A ($\mathrm{\ensuremath{\Gamma}}<0$) phases. Around the positive $\mathrm{\ensuremath{\Gamma}}$ point, due to the interplay of the Heisenberg, Kitaev, and $\mathrm{\ensuremath{\Gamma}}$ interactions, we find a possible quantum spin liquid in a noticeable region with a continuum in spin excitations.