With the rapid evolution toward 6G networks, ensuring robust physical layer security (PLS) in highly dynamic and heterogeneous wireless environments has become a key challenge. Traditional security methods often struggle to adapt to time-varying channels, especially in the absence of perfect channel state information. Furthermore, the dynamic nature of node selection and power allocation in heterogeneous networks creates a complex hybrid action space operating across multiple timescales, significantly complicating the design of efficient and adaptive security strategies. To address this, this paper proposes a novel constrained hierarchical reinforcement learning (CHRL) framework for secure cooperative communications in next-generation wireless systems. The framework is designed to optimize secrecy performance within a hybrid action space comprising both discrete node selection and continuous power allocation, operating at different timescales. By hierarchically decoupling the joint optimization problem, the upper layer performs risk-aware node selection to maximize long-term secrecy capacity (SC) while guaranteeing a stable and secure link. At the lower layer, we develop a constrained MiniMax Multi-objective Deep Deterministic Policy Gradient (M3DDPG) algorithm that optimizes power allocation considering worst-case conditions. Lagrange multipliers are integrated to enforce a strictly positive SC constraint throughout transmission, effectively preventing security outages. Simulation results under time-varying Rayleigh fading channels demonstrate that the proposed CHRL framework outperforms existing HRL methods, achieving up to 17% improvement in SC while strictly maintaining security constraints. These results validate the effectiveness of the proposed approach for enhancing PLS in next-generation cooperative wireless networks.
With the rapid evolution toward 6G networks, ensuring robust physical layer security (PLS) in highly dynamic and heterogeneous wireless environments has become a key challenge. Traditional security methods often struggle to adapt to time-varying channels, especially in the absence of perfect channel state information. Furthermore, the dynamic nature of node selection and power allocation in heterogeneous networks creates a complex hybrid action space operating across multiple timescales, significantly complicating the design of efficient and adaptive security strategies. To address this, this paper proposes a novel constrained hierarchical reinforcement learning (CHRL) framework for secure cooperative communications in next-generation wireless systems. The framework is designed to optimize secrecy performance within a hybrid action space comprising both discrete node selection and continuous power allocation, operating at different timescales. By hierarchically decoupling the joint optimization problem, the upper layer performs risk-aware node selection to maximize long-term secrecy capacity (SC) while guaranteeing a stable and secure link. At the lower layer, we develop a constrained MiniMax Multi-objective Deep Deterministic Policy Gradient (M3DDPG) algorithm that optimizes power allocation considering worst-case conditions. Lagrange multipliers are integrated to enforce a strictly positive SC constraint throughout transmission, effectively preventing security outages. Simulation results under time-varying Rayleigh fading channels demonstrate that the proposed CHRL framework outperforms existing HRL methods, achieving up to 17% improvement in SC while strictly maintaining security constraints. These results validate the effectiveness of the proposed approach for enhancing PLS in next-generation cooperative wireless networks.
Stacked intelligent metasurfaces (SIMs) have recently emerged as a powerful wave-domain technology that enables multi-stage manipulation of electromagnetic signals through multilayer programmable architectures. While SIMs offer unprecedented degrees of freedom for enhancing physical-layer security, their extremely large number of meta-atoms leads to a high-dimensional and strongly coupled optimization space, making conventional design approaches inefficient and difficult to scale. Moreover, existing deep reinforcement learning (DRL) techniques suffer from slow convergence and performance degradation in dynamic wireless environments with imperfect knowledge of passive eavesdroppers. To address these challenges, we propose a hybrid quantum proximal policy optimization (QPPO) framework for SIM-assisted secure communications that jointly optimizes transmit power allocation and SIM phase shifts to maximize the average secrecy rate under power and quality-of-service constraints. Specifically, a parameterized quantum circuit is embedded into the actor network, forming a hybrid classical-quantum policy architecture that enhances policy representation capability and exploration efficiency in high-dimensional continuous action spaces. Extensive simulations demonstrate that the proposed Q-PPO scheme consistently outperforms DRL baselines, achieving approximately 15% higher secrecy rates and 30% faster convergence under imperfect eavesdropper channel state information. These results establish Q-PPO as a powerful optimization paradigm for SIM-enabled secure wireless networks.
This paper investigates a dynamic game framework for physical layer security (PLS)-aware wireless network with third-party collaborative users (TCUs). In the considered system, legitimate users (LUs) aim to transmit secret signals under the threat of eavesdroppers (EVs). There exists TCUs rationally selecting to form coalitions with LUs or EVs, strategically enhancing the secrecy transmission rates for LUs or increasing eavesdropping rates for EVs to obtain rewards. Due to the unpredictability of wireless systems (e.g., time-varying channel conditions, user mobility), coalitions among the three parties may be unfixed and dynamically changing. A hierarchical game integrating a matching subgame with a coalition formation subgame is formulated to model the interactions among LUs, EVs, and TCUs. In order to derive the optimal long-term strategies while solving the equilibrium of the hierarchical game, we first introduce a constrained Gale-Shapley algorithm which determines the optimal power, mode, and channel selections of TCUs with the known coalition partition, and then propose a coalition selection algorithm for reaching the stability of coalition partitions. Moreover, we develop a deep reinforcement learning (DRL)-based solution to concatenate the aforementioned algorithms, producing the long-term optimal strategies for LUs, EVs and TCUs over multiple time slots. Simulations evaluate the performance of the proposed scheme, and demonstrate its superiority over counterparts.
In this paper, a novel hierarchical game framework for physical layer security (PLS) aware wireless communications with dynamic trilateral coalitions is studied. In the considered system, legitimate users (LUs) aim to transmit secret data to associated base stations (BSs) via uplink communications under the threat of eavesdroppers (EVs), while there also exists jammers (JAs) which may choose to form coalitions with either LUs for increasing their secrecy transmission rates or EVs for increasing their eavesdropping rates in exchange for potential rewards. Different from the existing work, we explore such complicated while dynamic coalition relationships under uncertainties of wireless systems (e.g., time-varying channel conditions), and formulate a hierarchical game integrated with a dynamic trilateral coalition formation game to model strategic interactions among LUs, JAs and EVs. Particularly, we first analyze stability conditions of the trilateral coalitions and propose a hedonic coalition selection and formation algorithm for reaching the stable coalition partition in each time slot. On top of this, we propose a deep reinforcement learning (DRL) based solution, which can achieve the equilibrium with long-term performance guarantees for the hierarchical game running over multiple time slots with dynamic evolutions. Simulations evaluate the proposed solution and show its superiority over counterparts.
Security has become an increasingly significant and urgent issue in wireless networks. The growing computational capability of the eavesdropper (Eve) and more and more complicated secure key management in large scale and heterogeneous networks, pose more and more stringent requirement to the cryptographic protocols. As a new design paradigm, physical layer security has been recognised as a promising solution to enhance the security of wireless links relying on the channel properties and advanced signal processing, and can act as either an alternative or a complementary solution to the conventional cryptographic methods. Without complicated secure key generation and management, physical layer security is quite suitable for the large scale distributed networks. In this thesis, we aim to improve the secrecy performance in emerging wireless communication systems relying on different physical layer security techniques. Firstly, the security of an amplify-and-forward successive relaying network with multiple untrusted relay nodes is investigated, where the conventional detrimental inter-relay interference is exploited to jam the untrusted nodes without assistance of external helpers. Considering different complexity requirements, several relay selection schemes are proposed, and the closed-form expressions of the lower bound of the secrecy outage probability and the maximum secrecy diversity order are derived accordingly. It is shown that the maximum secrecy diversity order of N-1 can be achieved for an N relay nodes network. Secondly, the artificial noise (AN) aided secure transmission strategy for a multiple-input single-output multiple-Eve (MISOME) system with a secure user (Bob) and a normal user (NU) is investigated. The power allocations among Bob, NU and AN, as well as the wiretap code rates, are jointly optimized to maximize the effective secrecy throughput (EST), under the average throughput constraint of NU. An alternative optimization algorithm with guaranteed convergence is proposed to obtain the optimal parameters. It is shown that the EST increases with the transmitting power and the number of transmit antennas, and decreases with the throughput constraint of NU, and the EST can be improved through injecting AN and concurrent transmission of Bob and NU. Thirdly, physical layer security in a multi-antenna small-cell network is investigated, where the multi-antenna base stations (BSs), cellular users, and Eves are all randomly distributed according to independent Poisson point processes. Stochastic geometry is applied to analyze the connection and secrecy outage probabilities and the average achievable secrecy rate. The impact of different parameters, including power allocation, BS and Eve density, and the adaptive eavesdropping or jamming on the secrecy performance is analyzed. It is shown that the average secrecy rate is a quasi-concave function of the power allocation factor and monotonically decreases with the ratio of the Eve-BS density. Finally, we study the physical layer security in a large-scale heterogeneous network consisting of both sub-6 GHz massive multi-input multi-output (MIMO) macro cells and millimeter wave (mmWave) small cells. By considering pilot spoofing attacks from the Eves, the coverage and secrecy probabilities are derived using stochastic geometry and the conditions under which the millimeter wave tier outperforms the sub-6 GHz counterpart are discussed in terms of both coverage and secrecy.It is shown that the mmWave small cell can provide better coverage performance in the high transmission code-rate region, and the secrecy performance of the mmWave system outperforms the sub-6 GHz counterpart in the low redundant rate region, which reveals the advantage of using mmWave for secure communication.
Physical layer security (PLS) is an emerging area for information security against eavesdroppers (Eve). Information security of any system can also be improved by using friendly jammers that produce interference signals to Eve. Traditional security techniques are limited by the processing power of the wireless nodes, whereas PLS can achieve communication secrecy without requiring computationally expensive cryptographic operations. The relay networks have emerged as a promising technology to enhance the performance of the wireless systems. This paper proposes a joint resource optimization framework for the PLS of dual-hop decode and forward (DF) relay network with and without cooperative jamming. In particular, the proposed framework consists of a base station (BS), multiple users, DF relays, multiple subcarriers, and an Eve. Our objective is to maximize the sum secrecy rate (SR) through optimal power loading over different subcarriers at BS and relay nodes, and efficient subcarrier assignment. We formulate a mixed binary integer programing problem for secrecy optimization and adopt Lagrangian dual method to achieve the efficient solutions. We also provide three benchmark frameworks, i.e., joint power optimization with random subcarrier assignment, equal power allocation with efficient subcarrier assignment and equal power with random subcarrier assignment to guage the performance of our joint resource optimization framework. Simulation results unveil that the proposed joint resource optimization framework under cooperative jamming and without jamming performs significantly better than the benchmark frameworks.
With the breakthrough advancements in deep neural networks (DNNs) and natural language processing (NLP), semantic communication systems have emerged as a pivotal direction in the evolution of sixth-generation (6 G) networks. However, the broadcast nature of wireless channels inherently introduces user privacy risks, underscoring the urgency for robust security solutions. Existing secure semantic communication approaches predominantly focus on network-layer encryption, which often entails complex operations and carries inherent risks of key leakage. In contrast, physical layer security (PLS) harnesses the intrinsic time-varying nature and randomness of wireless channels, obviating the need for cumbersome cryptographic mechanisms. To address this critical challenge, this paper proposes a channel capacity-constrained secure semantic communication framework. The core design objective is to maximize channel capacity for legitimate channels while confining the ergodic secrecy capacity within a predefined security threshold. By developing a differentiable mutual information estimator, we establish a dual-channel capacity estimation model-encompassing both legitimate users and eavesdroppers-that is jointly optimized with the semantic communication system. Furthermore, we introduce a secure sentence similarity metric to quantitatively assess the security performance of semantic communication. Simulation results validate that despite a slight degradation in transmission accuracy, the proposed scheme achieves a significant enhancement in communication security.