Enhancements of Modulation-Layer Phase Encryption and Their Performance Under Physical Impairments in Coherent Optical Communication

Abstract

This study investigates a multi-stage digital phase-encryption scheme designed for secure coherent optical communication. While the proposed method is implemented in DSP and can be modeled as a software-level stream cipher, it is explicitly co-designed with and tightly coupled to the physical-layer modulation structure. Unlike conventional XOR-based bitwise schemes, the encryption is applied to the symbol phase domain. It is integrated into the transmitter's modulation process, enabling symbol-level security evaluated in conjunction with transmission physics. By applying K-level phase-shift secret keys toM-ary quadrature amplitude modulation (QAM) signals, this method increases the number of secret-key bits per symbol without altering the signal constellation in its baseline configuration. This approach enhances encryption strength while maintaining signal integrity and reducing quantization overhead in phase-only operation. Experimental evaluations on nonlinear optical transmission channels confirm that baseline digital phase encryption preserves Q-factor stability under self-phase modulation, cross-phase modulation, and four-wave mixing. The proposed scheme ensures error-free transmission at forward error correction thresholds, effectively preventing eavesdroppers from reconstructing symbols under worst-case observation conditions. Additionally, simulations analyzing generalized mutual information and quantization errors demonstrate that the encryption scheme can minimize signal-to-noise ratio penalties, even when employing high-order QAM formats with limited digital-to-analog and analog-to-digital conversion resolutions. As an optional extension, we also analyze pseudo-quantum-noise-based IQ spreading and explicitly quantify the associated trade-off between additional noise masking and increased sensitivity to quantization. These findings position multi-stage digital phase encryption as a practical and power-efficient modulation-layer encryption technique whose behavior is systematically characterized under realistic nonlinear channel and hardware constraints, rather than as an entirely new cryptographic primitive.