The information theoretic description of secrecy capacities and secrecy capacity regions (Liang et al., 2009) is an important research area to support a better understanding of security on the physical layer. Based on the secrecy capacity expressions or achievable secrecy rates, the transmit strategies, including power allocation, beamforming and subcarrier allocation, are optimized in order to choose a certain operating point. In this chapter, we focus on the optimization of the physical layer transmit strategies for typical wireless communication scenarios. In single-user scenarios, the system design is more complicated with additional secrecy constraints, since the secrecy capacity expressions are in general not concave or convex in the transmit strategies. The secrecy rate terms usually consist of a difference of two parts. The first one corresponds to the amount of data that can be reliably transmitted to the intended user and it is therefore concave in the transmit strategies. The second one corresponds to the amount of data that is overheard by the eavesdropper and it is thus also concave in the transmit strategies. The resulting transmit optimization problems are non-convex optimization problems since the difference of two concave functions is not necessarily convex or concave. However, in the multi-carrier case, the problem can be reduced to a convex optimization problem that can be efficiently computed. We conjecture that also the multiple antenna (MIMO) scenario will be completely solved in the very next future. In multi-user scenarios, the secrecy rate regions of all four elements of network information theory, the broadcast, the multiple access, the relay, and the interference channel were recently studied. The attacker models of the MAC and the relay case are more difficult than the wellmotivated ones of the broadcast and the interference channel. Therefore, we focus on the broadcast and interference channel. The resource allocation for the parallel broadcast channel without secrecy is involved due to a hard combinatorial problem the matching of carriers to users. Interestingly, with secrecy constraints, the resulting programming problem is much simpler and the optimal power and resource allocation can be solved efficiently. A similar
The information theoretic description of secrecy capacities and secrecy capacity regions (Liang et al., 2009) is an important research area to support a better understanding of security on the physical layer. Based on the secrecy capacity expressions or achievable secrecy rates, the transmit strategies, including power allocation, beamforming and subcarrier allocation, are optimized in order to choose a certain operating point. In this chapter, we focus on the optimization of the physical layer transmit strategies for typical wireless communication scenarios. In single-user scenarios, the system design is more complicated with additional secrecy constraints, since the secrecy capacity expressions are in general not concave or convex in the transmit strategies. The secrecy rate terms usually consist of a difference of two parts. The first one corresponds to the amount of data that can be reliably transmitted to the intended user and it is therefore concave in the transmit strategies. The second one corresponds to the amount of data that is overheard by the eavesdropper and it is thus also concave in the transmit strategies. The resulting transmit optimization problems are non-convex optimization problems since the difference of two concave functions is not necessarily convex or concave. However, in the multi-carrier case, the problem can be reduced to a convex optimization problem that can be efficiently computed. We conjecture that also the multiple antenna (MIMO) scenario will be completely solved in the very next future. In multi-user scenarios, the secrecy rate regions of all four elements of network information theory, the broadcast, the multiple access, the relay, and the interference channel were recently studied. The attacker models of the MAC and the relay case are more difficult than the wellmotivated ones of the broadcast and the interference channel. Therefore, we focus on the broadcast and interference channel. The resource allocation for the parallel broadcast channel without secrecy is involved due to a hard combinatorial problem the matching of carriers to users. Interestingly, with secrecy constraints, the resulting programming problem is much simpler and the optimal power and resource allocation can be solved efficiently. A similar
Inherent openness of the wireless medium imposes stronger challenges on the security of wireless communications. Information-theoretic security addresses these challenges at the physical layer by using tools from wireless communication theory, signal processing and information theory. In information-theoretic security, physical layer communication is intelligently designed to exploit the characteristics of the wireless medium, such as fading, interference, cooperation, and multi-dimensional signaling, in order to provide or improve security. In this dissertation, we study the security of several fundamental wireless network configurations from an information-theoretic perspective. First, we study the Gaussian multiple-input multiple-output (MIMO) wiretap channel. In this channel, the transmitter sends a common message to both the legitimate user and the eavesdropper. In addition to the common message, a private message is sent only to the legitimate user, which needs to be kept hidden as much as possible from the eavesdropper. We obtain the entire capacity-equivocation region for this channel model. In particular, we show the sufficiency of jointly Gaussian auxiliary random variables and channel input to evaluate the existing single-letter description of the capacity-equivocation region due to Csiszar-Korner. Next, we study the secure broadcasting problem, where a transmitter wants to have secure communication with multiple legitimate users in the presence of an external eavesdropper. We study several special cases of the secure broadcasting problem. First, we consider the degraded multi-receiver wiretap channel, and establish its secrecy capacity region. Second, we consider the parallel less noisy multi-receiver wiretap channel, and obtain its common message secrecy capacity and sum secrecy capacity. Third, we consider the parallel degraded multi-receiver wiretap channel for the two-user and two-sub-channel case, and obtain its entire secrecy capacity region. Finally, we consider a parallel channel model with two sub-channels, where the transmitter can use only one of the subchannels at any time, and characterize its secrecy capacity region. Then, we study the two-user Gaussian MIMO broadcast channel with common and confidential messages. In this channel model, the transmitter sends a common message to both users, and a confidential message to each user which needs to be kept perfectly secret from the other user. We obtain the entire capacity region of this channel. We also explore the connections between this channel model and its non-confidential counterpart, i.e., the Gaussian MIMO broadcast channel with common and private message. Next, we consider the Gaussian MIMO multi-receiver wiretap channel and obtain its secrecy capacity region for the most general case. We first show that even for the single-input single-output (SISO) case, existing converse techniques fall short of proving the secrecy capacity region, to emphasize the need for a new proof technique, which we develop by using the relationships between the Fisher information and the differential entropy. Using this new proof technique, we obtain the secrecy capacity region of the degraded MIMO channel. We then establish the secrecy capacity region of the general MIMO channel by using the channel enhancement technique in conjunction with the capacity result we obtained for the degraded MIMO channel. For the general MIMO channel, we show that dirty-paper coding (DPC) combined with stochastic encoding attains the entire secrecy capacity region. Then, we study the multi-receiver wiretap channel for a more general scenario, where, in addition to confidential messages, the transmitter sends public messages to the legitimate users, on which there are no secrecy constraints. First, we consider the degraded discrete memoryless channel, and obtain inner and outer bounds for the capacity region. These inner and outer bounds match for certain cases, providing the capacity region. Second, we obtain an inner bound for the general discrete memoryless channel by using Marton's inner bound. Third, we consider the degraded Gaussian MIMO channel, and show that jointly Gaussian auxiliary random variables and channel input are sufficient to exhaust the inner and outer bounds. Finally, we provide an inner bound for the capacity region of the general Gaussian MIMO channel. Next, we focus on the multiple access wiretap (MAC-WT) channel whose capacity region is unknown. We consider a special class of MAC-WT channels which we call the weak eavesdropper class, where each user's link to the legitimate receiver is stronger than its link to the eavesdropper. For this class of channels, we develop an outer bound for the secrecy capacity region, which partially matches the achievable region in an n-letter form. We evaluate a looser version of our outer bound for the Gaussian case, and show that our outer bound is within 0.5 bits/channel use of the achievable rates along the individual secrecy rates for all weak eavesdropper Gaussian MAC-WT. Then, we investigate the effects of user cooperation on the secrecy of broadcast channels by considering the cooperative relay broadcast channel (CRBC). We propose an achievable scheme that combines Marton's coding scheme for broadcast channels and Cover and El Gamal's compress-and-forward (CAF) scheme for relay channels. For the Gaussian CRBC, we show that both users can have positive secrecy rates, which is not possible for scalar Gaussian broadcast channels without cooperation. We further investigate the effects of user cooperation on secrecy by considering the multiple access channel with generalized feedback (MAC-GF), which can be viewed as the MAC-dual of the CRBC. We propose a CAF-based achievable secrecy rate region for the MAC-GF. Specializing our results to a Gaussian MAC-GF, we present numerical results which demonstrate that cooperation can improve secrecy for the MAC-GF. Next, we study the two-user one-eavesdropper discrete memoryless compound wiretap channel, and provide the best known lower bound for the secrecy capacity of this compound channel. We evaluate this achievable secrecy rate for the Gaussian MIMO case by using DPC. We show that this achievable secrecy rate achieves at least half of the secrecy capacity of this Gaussian MIMO compound wiretap channel, and also attains the secrecy capacity when the eavesdropper is degraded with respect to one of the two users. Then, we study the degraded compound multi-receiver wiretap channel (DCMRWC), which, in addition to a group of eavesdroppers, has two groups of users, namely the stronger group and the weaker group. We study two different communication scenarios for this channel. In the first scenario, there is only one eavesdropper, and the transmitter sends a confidential message to each group of legitimate users while keeping both messages secret from the eavesdropper. In the second scenario, we study the DCMRWC with layered messages without any restriction on the number of eavesdroppers. For both scenarios, we obtain the secrecy capacity region for the discrete memoryless channel, the parallel channel, and the Gaussian parallel channel. For the Gaussian MIMO channel, we obtain the secrecy capacity region when there is only one user in the second group. Next, we study the two-user fading broadcast channel and obtain its ergodic secrecy capacity region. We show that, thanks to fading, both users can have simultaneous secure communication with the transmitter, although this is not possible in the scalar non-fading Gaussian broadcast channel where only one user can have secure communication. This simultaneous secrecy of both users is achieved by an opportunistic communication scheme, in which, at each time instant, the transmitter communicates with the user having a better channel gain. Then, we study the secure lossy transmission of a vector Gaussian source to a legitimate user in the presence of an eavesdropper, where both the legitimate user and the eavesdropper have vector Gaussian side information. We obtain an outer bound for the rate, equivocation and distortion region. Moreover, we obtain the maximum equivocation at the eavesdropper when there is no constraint on the transmission rate. By using this maximum equivocation result, we show two facts. First, for this problem, in general, Wyner-Ziv scheme is suboptimal, although, it is optimal in the absence of an eavesdropper. And, second, even when there is no transmission rate constraint, an uncoded transmission scheme is suboptimal; the presence of an eavesdropper necessitates the use of a coded scheme to attain the maximum equivocation. Finally, we revisit the secure lossy source coding problem. In all works on this problem, either the equivocation of the source at the eavesdropper or the equivocation of the legitimate user's reconstruction of the source at the eavesdropper is used to measure secrecy. We first propose the relative equivocation of the source at the eavesdropper with respect to the legitimate user as a new secrecy measure. We argue that this new secrecy measure is the one that corresponds to the natural generalization of the equivocation in a wiretap channel to the context of secure lossy source coding. Under this new secrecy measure, we provide a single-letter description of the rate, relative equivocation and distortion region, as well as its specializations to degraded and reversely degraded cases. We investigate the relationships between the optimal scheme that attains this region and the Wyner-Ziv scheme.
Physical layer security is to enable confidential data transmission through wireless networks in the presence of illegitimate users, without basing on higher-layer encryption. The essence of physical layer security is to maximize the secrecy rate, that is the maxi- mum rate of information without intercepted by the eavesdroppers. Besides, the design of physical layer security considers the transmit power minimization. These two objectives conflict with each other. Consequently, the research on physical layer security designs often focuses on the two main classes of optimization problems: maximizing secrecy rate under the transmit power constraint and minimizing power consumption while guaranteeing the secrecy rate constraint. These problems are nonconvex, thus, hard to solve. In this thesis, we focus on developing optimization approaches to solve these two optimization problem classes. Our methods are based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) which well-known as one of the most powerful approaches in optimization. In the first part, we consider three classes of secrecy rate maximization problems (chapters 2, 3, 4). In particular, chapter 2 studies the secure information transmission in a multiple-input single-output (MISO) relay system by using joint beamforming and artificial noise strategy under the deterministic uncertainty channel models of all links. Without using a relay, chapter 3 addresses the problem of transfer wireless information and power simultaneously in MISO secure system where scenarios of perfect channel state information and deterministic uncertainty channel models are concerned. Transmit beamforming without artificial noise and that with artificial noise are investigated. Under the assumption of statistical channel state information to eavesdroppers, chapter 4 studies the probability constrained secrecy rate maximization problem in multiuser MISO simultaneous wireless information and power transfer (SWIPT) system. The unified approach based on DC programming and DCA is proposed to solve three classes of optimization problems. The optimization problem in chapter 2 is recast as two general DC programs. The general DCA schemes are proposed to solve these two DC programs. In chapter 3, we consider four optimization problems in accordance with four scenarios. Exploiting the special structures of these original optimization problems, we transform it into four general DC programs for which the corresponding general DCA based algorithms are developed. In chapter 4, we first transform the considered problem into a tractable form. We then develop an alternating scheme to solve the transformed problem. Two general DC programs are proposed in each step of the alternating scheme. For solving these DC programs, we study a variant of general DCA, namely, DCA−ρ scheme. The convergence of alternating general DCA−ρ scheme is proven. The second part studies the transmit power optimization problem under the probability constraints of secrecy rate and harvested energy in a MISO SWIPT system (chapter 5). We reformulate the original problem as three general DC programs for which the corresponding general DCA-based algorithms are investigated. Numerical results demonstrate the efficiency of the proposed algorithms.
The use of wireless networks has grown significantly in contemporary \ntimes, and continues to develop further. The broadcast nature of \nwireless communications, however, makes them particularly vulnerable \nto eavesdropping. Unlike traditional solutions, which usually handle \nsecurity at the application layer, the primary concern of this \ndissertation is to analyze and develop solutions based on coding \ntechniques at the physical-layer. \n \nFirst, in chapter 2, we consider a scenario where a source node \nwishes to broadcast two confidential messages to two receivers, \nwhile a wire-tapper also receives the transmitted signal. This model \nis motivated by wireless communications, where individual secure \nmessages are broadcast over open media and can be received by any \nillegitimate receiver. The secrecy level is measured by the \nequivocation rate at the eavesdropper. We first study the general \n(non-degraded) broadcast channel with an eavesdropper, and present \nan inner bound on the secrecy capacity region for this model. This \ninner bound is based on a combination of random binning, and the \nGelfand-Pinsker binning. We further study the situation in which the \nchannels are degraded. For the degraded broadcast channel with an \neavesdropper, we present the secrecy capacity region. Our achievable \ncoding scheme is based on Cover's superposition scheme and random \nbinning. We refer to this scheme as the Secret Superposition Scheme. \nOur converse proof is based on a combination of the converse proof \nof the conventional degraded broadcast channel and Csiszar Lemma. We \nthen assume that the channels are Additive White Gaussian Noise and \nshow that the Secret Superposition Scheme with Gaussian codebook is \noptimal. The converse proof is based on Costa's entropy power \ninequality. Finally, we use a broadcast strategy for the slowly \nfading wire-tap channel when only the eavesdropper's channel is \nfixed and known at the transmitter. We derive the optimum power \nallocation for the coding layers, which maximizes the total average \nrate. \n \n \nSecond, in chapter 3 , we consider the \nMultiple-Input-Multiple-Output (MIMO) scenario of a broadcast \nchannel where a wiretapper also receives the transmitted signal via \nanother MIMO channel. First, we assume that the channels are \ndegraded and the wiretapper has the worst channel. We establish the \ncapacity region of this scenario. Our achievability scheme is the \nSecret Superposition Coding. For the outerbound, we use notion of \nthe enhanced channels to show that the secret superposition of \nGaussian codes is optimal. We show that we only need to enhance the \nchannels of the legitimate receivers, and the channel of the \neavesdropper remains unchanged. We then extend the result of the \ndegraded case to a non-degraded case. We show that the secret \nsuperposition of Gaussian codes, along with successive decoding, \ncannot work when the channels are not degraded. We develop a Secret \nDirty Paper Coding scheme and show that it is optimal for this \nchannel. We then present a corollary generalizing the capacity \nregion of the two receivers case to the case of multiple receivers. \nFinally, we investigate a scenario which frequently occurs in the \npractice of wireless networks. In this scenario, the transmitter and \nthe eavesdropper have multiple antennae, while both intended \nreceivers have a single antenna (representing resource limited \nmobile units). We characterize the secrecy capacity region in terms \nof generalized eigenvalues of the receivers' channels and the \neavesdropper's channel. We refer to this configuration as the MISOME \ncase. We then present a corollary generalizing the results of the \ntwo receivers case to multiple receivers. In the high SNR regime, we \nshow that the capacity region is a convex closure of rectangular \nregions. \n \n \nFinally, in chapter 4, we consider a K-user secure Gaussian \nMultiple-Access-Channel with an external eavesdropper. We establish \nan achievable rate region for the secure discrete memoryless MAC. \nThereafter, we prove the secrecy sum capacity of the degraded \nGaussian MIMO MAC using Gaussian codebooks. For the non-degraded \nGaussian MIMO MAC, we propose an algorithm inspired by the \ninterference alignment technique to achieve the largest possible \ntotal Secure-Degrees-of-Freedom . When all the terminals are \nequipped with a single antenna, Gaussian codebooks have shown to be \ninefficient in providing a positive S-DoF. Instead, we propose a \nnovel secure coding scheme to achieve a positive S-DoF in the single \nantenna MAC. This scheme converts the single-antenna system into a \nmultiple-dimension system with fractional dimensions. The \nachievability scheme is based on the alignment of signals into a \nsmall sub-space at the eavesdropper, and the simultaneous separation \nof the signals at the intended receiver. We use tools from the field \nof Diophantine Approximation in number theory to analyze the \nprobability of error in the coding scheme. We prove that the total \nS-DoF of fracK−1K can be achieved for almost all channel \ngains. For the other channel gains, we propose a multi-layer coding \nscheme to achieve a positive S-DoF. As a function of channel gains, \ntherefore, the achievable S-DoF is discontinued.
In this dissertation, we present novel physical (PHY) and cross-layer design guidelines and resource adaptation algorithms to improve the security and user experience in the future wireless networks. Physical and cross-layer wireless security measures can provide stronger overall security with high efficiency and can also provide better flexibility in response to the typically time-varying wireless environment. To better utilize limited wireless resources, we present approaches and techniques based on new performance criteria for wireless system optimization by thoroughly exploiting user information including its Quality of Service (QoS) requests and channel state information (CSI) to improve network throughput and security.
In recent years, there has been a growing research interest in wireless system security from PHY layer perspective. In a wiretap channel model introduced in the seminal work by Wyner, a sender “Alice” wishes to transmit a secret message to the intended receiver “Bob” in presence of a passive eavesdropper “Eve”. Existing works have characterized maximum achievable secrecy rate or secrecy capacity for single- and multiple-antenna systems by applying Gaussian signaling and secrecy code. Despite the impracticality of Gaussian input, its compact closed-form expression of mutual information motivated the wide use of Gaussian input assumption in theoretical analysis. In contrast to the Gaussian codebook, practical wiretap codes must consist of finite-alphabet symbols. Because of this constraint, the achievable secrecy rate for a finite-alphabet input scenario would differ from the secrecy rate achievable by a Gaussian codebook. In this thesis, we quantify the effect of finite discrete-constellation on instantaneous and ergodic secrecy rate of multiple-antenna wire-tap channels. Our results demonstrate substantial performance difference between systems involving finite-alphabet inputs and systems with Gaussian inputs. In addition, we investigate the secrecy performance of multi-terminal wiretap systems when a codebook based transmission beamforming is implemented. We characterize the secrecy outage probability of a communication link under eavesdropping. We consider a limited feedback scenario where the transmitter uses a pre-defined codebook known to both the transmitter and the receiver for beamforming, and analyze the secrecy outage probability of the link when it being eavesdropped.
Next, we consider the security of wireless multi-hop networks. Certainly, network security is an overarching issue that transcends all layers of communication protocol stack. Therefore, we tackle the problem of network security by proper user scheduling, routing, and resource management. Our approach of secrecy is complementary to the traditional cryptography based techniques. Together, they can provide stronger overall security and flexibility in selecting the desired solution and can reduce bandwidth and energy overhead as much as possible. Multi-hop wireless networks frequently employ the traditional routing and resource allocation strategies. Such strategies often fail to take into account the additional need for protection against security vulnerabilities due to the wireless medium. Thus, to address security concerns, we investigate the problem of eavesdropping control by proposing security-aware power allocation strategies and routing algorithms for multi-hop networks.
The future wireless infrastructure will consists of a plethora of heterogeneous collections of applications and services. A differentiated service structure would therefore be an essential part of the future wireless infrastructure. In addition to the security consideration of wireless network, we focus on the heterogeneous structure of the network. We study the problem of cross-layer resource allocation and admission control in a multiuser heterogeneous OFDMA network providing both QoS-constrained user servicesand best-effort services. Such heterogeneous networks provide service to both High-Priority (HP) users and Best-Effort (BE) users. By clustering subcarriers, efficient algorithms for cluster and power allocation have been proposed. Our strategy maximizes the total network utility of the BE users while satisfying the QoS request for maximum number of HP users. The feasibility of the resource allocation problem depends on the number of HP users in the network. Since a large number of demanding HP users would render the resource allocation problem infeasible, a joint admission control and resource allocation scheme could be an efficient way of tackling both problems with less overhead. By incorporating admission control, we propose an efficient and optimal joint admission control and resource allocation algorithm.
This dissertation investigates the physical layer security of wireless communication networks using game-theoretic and optimization models. Considered is a wireless network of parallel channels that may be under eavesdropping attack. Game models are played between a friendly interferer (he) who can broadcast artificial noise and a strategic eavesdropper (she). The friendly interferer wants to maximize the secrecy level of the wireless communication network by decreasing the eavesdropper's listening capability with intentionally generated noise signals, while the eavesdropper wants to maximize her own capability and avoid interference. The game-theoretic models reflect the strategic confrontation between the friendly interferer and the eavesdropper in different communication environments. These models address some of the limitations in literature. Most existing resource allocation strategies for friendly interference adopt the conservative assumption that all parallel channels are under attack. Game-theoretic approaches provide more efficient strategies to the friendly interferer by anticipating the channels to be attacked by the eavesdropper. The first model considers the case in which the friendly interferer can flexibly allocate cooperative jamming power to all communication channels in the network under a total power constraint. When all channels used for secret communication are under the threat of eavesdropping attacks, the optimal power allocation policy results from solving a convex optimization problem. In this case, the optimal policy is unique and can be obtained via a water-filling scheme. When the eavesdropper can not attack all channels, the eavesdropper should behave strategically and may select the targets probabilistically. In this case, a non-zero-sum game helps the friendly interferer predict and concentrate on the targets selected by the eavesdropper. When the eavesdropper can only attack a single channel, an analytical form of the NE strategies is presented; while an iterative numerical method that guarantees the convergence to NE strategies is proposed if the eavesdropper can attack multiple channels. Compared to naive power allocation strategies, such as splitting the power equally to each channel, the game-theoretic power allocation strategy always delivers better expected total secrecy capacity. Next, an extension of the power allocation game for cooperative jamming models the case in which the state information of the eavesdropping channels may not be known completely. New utility functions for the friendly interferer and the eavesdropper are designed to include the distributional information of eavesdropping channels' state information. A non-zero sum game using the new utility functions is analyzed between the friendly interferer and the strategic eavesdropper who can only attack a limited number of channels. Note that the friendly interferer may not have the capability to send cooperative jamming signals to multiple channels simultaneously. For such a friendly interferer, a probabilistic cooperative jamming scheme is devised such that the defender focuses on protecting randomly selected channels. In the mean time, the eavesdropper selects the most vulnerable channels to attack. A linearly constrained non-zero-sum bimatrix game models the competition between the defender and the eavesdropper. The Nash equilibrium is shown to be equivalent to the solution of a quadratic programming problem. A closed form solution is derived for the special case in which the eavesdropper can only attack a single channel. We show that there exists a cut-off index determining the set of channels that will be covered in the equilibrium strategies. Furthermore, the adverse effect of the interference signal plays an important role in the value of the cut-off index and it represents the vulnerability of wireless communication networks against eavesdroppers Wireless communication networks of parallel channels actually share a lot of similarities with general cyber networks. Thus, the final model in this dissertation is a cost-efficient network security game for defensive resource allocation in a heterogeneous cyber network of N nodes. A defender, by allocating defensive resources to a networked node, can decrease the node's vulnerabilities. At the same time, the cost of using defensive resources is taken into consideration. Two non-zero-sum Nash games model the competition between the defender and two types of attackers, respectively. The first type tries to maximize damage based on the value of security assets related to networked nodes, while the second type aims at infiltrating the network. Consideration of the case that the defender does not know the attacker's type leads to a Bayesian game. For all three games, we derive conditions for the existence of a unique Nash equilibrium (NE), and obtain the equilibria and the value functions in closed form.
Security has become an increasingly significant and urgent issue in wireless networks. The growing computational capability of the eavesdropper (Eve) and more and more complicated secure key management in large scale and heterogeneous networks, pose more and more stringent requirement to the cryptographic protocols. As a new design paradigm, physical layer security has been recognised as a promising solution to enhance the security of wireless links relying on the channel properties and advanced signal processing, and can act as either an alternative or a complementary solution to the conventional cryptographic methods. Without complicated secure key generation and management, physical layer security is quite suitable for the large scale distributed networks. In this thesis, we aim to improve the secrecy performance in emerging wireless communication systems relying on different physical layer security techniques. Firstly, the security of an amplify-and-forward successive relaying network with multiple untrusted relay nodes is investigated, where the conventional detrimental inter-relay interference is exploited to jam the untrusted nodes without assistance of external helpers. Considering different complexity requirements, several relay selection schemes are proposed, and the closed-form expressions of the lower bound of the secrecy outage probability and the maximum secrecy diversity order are derived accordingly. It is shown that the maximum secrecy diversity order of N-1 can be achieved for an N relay nodes network. Secondly, the artificial noise (AN) aided secure transmission strategy for a multiple-input single-output multiple-Eve (MISOME) system with a secure user (Bob) and a normal user (NU) is investigated. The power allocations among Bob, NU and AN, as well as the wiretap code rates, are jointly optimized to maximize the effective secrecy throughput (EST), under the average throughput constraint of NU. An alternative optimization algorithm with guaranteed convergence is proposed to obtain the optimal parameters. It is shown that the EST increases with the transmitting power and the number of transmit antennas, and decreases with the throughput constraint of NU, and the EST can be improved through injecting AN and concurrent transmission of Bob and NU. Thirdly, physical layer security in a multi-antenna small-cell network is investigated, where the multi-antenna base stations (BSs), cellular users, and Eves are all randomly distributed according to independent Poisson point processes. Stochastic geometry is applied to analyze the connection and secrecy outage probabilities and the average achievable secrecy rate. The impact of different parameters, including power allocation, BS and Eve density, and the adaptive eavesdropping or jamming on the secrecy performance is analyzed. It is shown that the average secrecy rate is a quasi-concave function of the power allocation factor and monotonically decreases with the ratio of the Eve-BS density. Finally, we study the physical layer security in a large-scale heterogeneous network consisting of both sub-6 GHz massive multi-input multi-output (MIMO) macro cells and millimeter wave (mmWave) small cells. By considering pilot spoofing attacks from the Eves, the coverage and secrecy probabilities are derived using stochastic geometry and the conditions under which the millimeter wave tier outperforms the sub-6 GHz counterpart are discussed in terms of both coverage and secrecy.It is shown that the mmWave small cell can provide better coverage performance in the high transmission code-rate region, and the secrecy performance of the mmWave system outperforms the sub-6 GHz counterpart in the low redundant rate region, which reveals the advantage of using mmWave for secure communication.