The spin structure of an axial next-nearest-neighbor Ising (ANNNI) model in two dimensions (2D) is a renewed problem because different Monte Carlo (MC) simulation methods predicted different spin orderings. The usual equilibrium simulation predicts the occurrence of a floating incommensurate (IC) Kosterlitz-Thouless (KT) type phase, which never emerges in non-equilibrium relaxation (NER) simulations. In this paper, we first examine previously published results of both methods, and then investigate a higher transition temperature
Tc1 between the IC and paramagnetic phases. In the usual equilibrium simulation, we calculate the chain magnetization on larger lattices (up to $512\ifmmode\times\else\texttimes\fi{}512$ sites) and estimate ${T}_{c1}\ensuremath{\approx}1.16J$ with frustration ratio $\ensuremath{\kappa}(\ensuremath{\equiv}\ensuremath{-}{J}_{2}/{J}_{1})=0.6$. We examine the nature of the phase transition in terms of the Binder ratio
gL of spin overlap functions and the correlation-length ratio $\ensuremath{\xi}/L$. In the NER simulation, we observe the spin dynamics in equilibrium states by means of an autocorrelation function and also observe the chain magnetization relaxations from the ground and disordered states. These quantities exhibit an algebraic decay at $T\ensuremath{\lesssim}1.17J$. We conclude that the two-dimensional ANNNI model actually admits an IC phase transition of the KT type.