Towards an optimised and adaptive automation of post-incident malware investigation: a novel reinforcement learning framework

Dunsin, Dipo (2024) Towards an optimised and adaptive automation of post-incident malware investigation: a novel reinforcement learning framework. Doctoral thesis, London Metropolitan University.

Abstract

In the rapidly evolving landscape of cybersecurity, the increasing sophistication of malware necessitates dynamic and adaptive solutions for effective post-incident investigations. Existing malware detection frameworks, predominantly reliant on heuristic and signature-based techniques, exhibit significant limitations in identifying polymorphic and evasive malware. Machine learning approaches, while promising, often struggle with adversarial attacks, false positives, and scalability challenges. To address these gaps, this research proposes a novel Reinforcement Learning (RL) Post-Incident Malware Investigation Framework. As a result of leveraging advanced Markov Decision Processes (MDPs), the framework enhances decision-making capabilities, enabling efficient and accurate analysis of malware in live memory dumps. This study introduces a unified investigation model that integrates static analysis, behavioural profiling, and machine-learning methodologies, surpassing traditional methods in adaptability and scalability. Key contributions include the development of an optimised RL architecture for efficient state-action mapping, the integration of live memory forensic tools such as Volatility and the AWK module, and the introduction of fault-tolerant techniques to handle real-world uncertainties. The results indicate that the RL-enhanced model significantly reduces the time required for post-incident malware forensics while maintaining a high accuracy of 94% in identifying malware. Furthermore, the framework’s effectiveness is rigorously tested and validated across diverse datasets and operational scenarios, demonstrating its ability to mitigate computational inefficiencies and improve malware detection accuracy. As a result of addressing critical gaps in dataset diversity, computational overhead, and the adaptability of RL-based methods, this research advances the state of post-incident malware forensics. Future work will explore the integration of Generative AI techniques through the introduction of specific Large Language Models (LLMs) and hybrid models to further improve the efficiency of post-incident malware forensic investigation. This work underscores the transformative potential of RL to create resilient defences against evolving malware threats, contributing not only to the theoretical understanding of reinforcement learning in cybersecurity but also to scalable and resource-efficient investigative practices.

Documents
10233:52021
[thumbnail of 16020881_Dipo-Dunsin.pdf]
16020881_Dipo-Dunsin.pdf - Published Version
Restricted to Repository staff only until 3 March 2026.

Download (16MB) | Request a copy
Details
Record
View Item View Item