Detection of Fileless Malware through Network Traffic Analysis

Abstract

The rapid growth of fileless malware raises a fundamental challenge to existing cybersecurity frameworks. These malwares operate entirely within a system’s volatile memory without creating malicious files on the disk. This research aims to overcome a critical gap in Network Intrusion Detection System (NIDS) by proposing a novel hybrid deep-learning framework. Traditional signature-based detection methods prove ineffective against these memory-resident threats, consequently this investigation details advanced feature extraction methodologies which can identify fileless malware using Network Packet Capture (PCAP) files. This study will employ Design Science Research (DSR) integrating it with a Design-Oriented Machine Learning (DS-ML) methodology which ensures systematic and rigorous development and evaluation process. Key contributions of this research will be: 1) holistic development of feature extraction mechanism that effectively captures fileless malware behavior within network traffic, 2) proposing a hybrid deep-learning model for optimizing the detection techniques for fileless malware, and 3) constituting specific evaluation metrics to measure the accuracy of detecting fileless malware. The resultant framework will discuss the limitations that are present in the existing approaches that primarily focus on detecting file-based malware.