Recently there has been considerable excitement surrounding the promising realization of high-spin Kitaev material, such as the quasi-2D compound CrI
3 and CrGeTe
3. However, the stability of quantum spin liquids (QSL) against single ion anisotropy (SIA) in these materials and the global quantum phase diagram of the extended spin-3/2 Kitaev model with finite SIA remain unclear. In this study, we perform large-scale density matrix renormalization group (DMRG) to explore the quantum phase diagram of the generalized spin-3/2 Kitaev-Heisenberg (K-H) model accompanied with SIA
Ac. In the
Ac=0 limit, the spin-3/2 K-H model exhibits a quantum phase diagram similar to that of a spin-1/2 system, including two QSLs around antiferromagnetic and ferromagnetic Kitaev models. For models with finite
Ac, we map out the quantum phase diagram around two Kitaev points and observe distinct types of in-plane vortex orders developed from these two QSL phases. Interestingly, series of nearly degenerate vortex configurations are discovered in each vortex phases. Using linear spin-wave theory, we demonstrate that these vortex configurations can be understood as a consequence of the quantum correction on a continuous family of degenerate classical states.
We study an extended Kitaev-Heisenberg model including additional anisotropic couplings by using the two-dimensional density-matrix renormalization group method. Calculating the ground-state energy, entanglement entropy, and spin-spin correlation functions, we make a phase diagram of the extended Kitaev-Heisenberg model around the spin-liquid phase. We find a zigzag antiferromagnetic phase, a ferromagnetic phase, a ${120}^{\ensuremath{\circ}}$ antiferromagnetic phase, and two kinds of incommensurate phases around the Kitaev spin-liquid phase. Furthermore, we study the entanglement spectrum of the model, and we find that entanglement levels in the Kitaev spin-liquid phase are degenerate forming pairs, but those in the magnetically ordered phases are nondegenerate. The Schmidt gap defined as the energy difference between the lowest two levels changes at the phase boundary adjacent to the Kitaev spin-liquid phase. However, we find that phase boundaries between magnetically ordered phases do not necessarily agree with the change of the Schmidt gap.