Device-to-Device (D2D) communication is a promising technology for next generation cellular networks. The D2D communication is being considered for the LTE Advanced standards in 3GPP Release 12, as Proximity Services (ProSe) continue to be part of current 5G or beyond cellular networks to support diverse applications. To fulfil the QoS requirements and to overcome the challenges of beyond 5G (B5G) cellular networks, D2D communication plays an important role. However, to maximize the performance gain of D2D communication, there are many open challenges that need to be addressed. One of the main factors that influence the performance of D2D communication in cellular heterogeneous networks (HetNets) is interference management. Transmission mode selection, resource allocation, power control and small cell deployment strategies are key factors that contribute to interference for D2D communication in cellular HetNets. In this thesis, an empirical investigation of the key factors influencing interference for D2D communication in cellular HetNets is described and its results are reported. In the investigation, the key performance-limiting factors are identified and measured by simulation as well as analytically.

A new mode selection technique for improving D2D performance in HetNets is reported. The effect of various key performance-limiting factors, including DUEs positions and Signal-to-interference plus noise ratio (SINR) are analysed. Transmission mode is selected based on D2D user equipment (DUE), cellular user equipment (CUE) and evolved Node-B (eNB) locations and their mutual distances. The proposed mode selection scheme provides better outage probability and sum rate for D2D communication.

Another main contribution of this thesis is the development of a new dynamic algorithm for interference management in D2D-enabled cellular HetNets. Various parameters for defining mutual interference within the 3-tier cellular network are defined and their effects on system performance are investigated. Achieved small cell density, transmission power control and device locations help to manage interference leading to higher outage probability and system throughput.

A machine learning (ML)-based power control and resource allocation for interference management in a D2D-enabled cellular network is found to have a significant effect on achieving higher throughput and better quality of service (QoS). A deep Q-network (DQN) based deep reinforcement learning (DRL) algorithm is proposed to optimize resource allocation where D2D acts as an agent and take decision independently based on learned optimal policy from the environment. D2D-enabled cellular network design and deployment strategies are outlined and recommendations are made for various system design scenarios.