Abstract This paper investigates physical layer security analysis of cooperative non-orthogonal multiple access (NOMA) communication system. A virtual full-duplex (VFD) relaying scheme with an untrusted amplify-and-forward (AF) half-duplex (HD) relay and a trusted decode-and-forward (DF) HD relay is used in this system to improve the spectral efficiency. In order to prevent the untrusted relay from eavesdropping, a simple and practical cooperative jamming scheme is designed to confuse the untrusted relay. The exact expressions of effective secrecy throughput (EST) for NOMA users and approximate expression of EST for non-NOMA user are derived. All theoretical results are validated by numerical simulations which demonstrate that the proposed VFD-NOMA scheme is superior to existing HD-NOMA scheme in cooperative system and jamming plays an important role for obtaining acceptable EST. In addition, simulation results shows that the best secrecy performance highly depends on the system parameters such as transmit powers and jamming signal power.