While cloud computing services have numerous advantages, they confront serious security threats. The complexity of cloud computing security tremendously exceeds traditional network security because cloud computing works in a dynamic environment with uncountable terminals in wired or wireless connections. New cyberattack techniques against cloud computing are continuously being invented. A review of literature reveals that heap overflow attacks, return-oriented programming (ROP) attacks, and cyberattacks launched inside virtual machines are the most notorious cyberattacks to compromise cloud computing. Although various defensive countermeasures against such attacks have been developed, malicious attackers can still circumvent many existing defensive countermeasures. In this thesis, an empirical investigation of the defensive countermeasures against three different cyberattack techniques is described, and its results are reported. In the investigation, the mixed method approach is chosen to collect qualitative data from cybersecurity experts and conduct experiments in a controlled environment to collect quantitative data for analysis.

The contributions of this thesis are to propose three novel defensive countermeasures: (1) Eight-tier Heap Overflow Prevention (EHOP) is a defensive countermeasure against heap overflow attacks. It is based on eliminating eight cyberattack approaches of function pointer modification. (2) Trie Graph of Monitoring Program (TGMP) is a defensive countermeasure of monitoring program control flow integrity and turning up the number of gadgets in each node of trie graph against ROP attacks. (3) Five-Tier Detection Mode (Five-TDM) prevents cyberattacks launched inside VMs. It is based on eliminating five cyberattack approaches inside VMs. These three defensive countermeasures are closely connected because heap overflow can meet one of the essential requirements of ROP attacks; ROP attacks that are launched inside VMs are more accessible to succeed than those launched outside the cloud.