We investigate the XY model with ferromagnetic nearest and antiferromagnetic next-to-nearest neighbours couplings (ANNNXY model). We formulate the study of both phases of the model in terms of different Coulomb gas models. The non-frustrated ferromagnetic phase is thus transformed into a usual Coulomb gas with one species, whereas the frustrated phase is translated into a Coulomb gas with three species interacting with each other in an anisotropic lattice. We then generalize the Kosterlitz - Thouless renormalization group equations for both phases by treating the Ising and Kosterlitz - Thouless order parameters in an independent way. This enables us to discuss the nature of the transition (Ising and (or) Kosterlitz - Thouless) in the frustrated phase.