Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks

Ghanem, Mohamed Chahine, Chen, Thomas and Nepomuceno, Erivelton G. (2022) Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks. Journal of Intelligent Information Systems, 2022. ISSN 0925-9902

Preview

Text s10844-022-00738-0.pdf - Published Version Available under License Creative Commons Attribution 4.0. Download (2MB) | Preview

Abstract / Description

Penetration testing (PT) is a method for assessing and evaluating the security of digital assets by planning, generating, and executing possible attacks that aim to discover and exploit vulnerabilities. In large networks, penetration testing becomes repetitive, complex and resource consuming despite the use of automated tools. This paper investigates reinforcement learning (RL) to make penetration testing more intelligent, targeted, and efficient. The proposed approach called Intelligent Automated Penetration Testing Framework (IAPTF) utilizes model-based RL to automate sequential decision making. Penetration testing tasks are treated as a partially observed Markov decision process (POMDP) which is solved with an external POMDP-solver using different algorithms to identify the most efficient options. A major difficulty encountered was solving large POMDPs resulting from large networks. This was overcome by representing networks hierarchically as a group of clusters and treating each cluster separately. This approach is tested through simulations of networks of various sizes. The results show that IAPTF with hierarchical network modeling outperforms previous approaches as well as human performance in terms of time, number of tested vectors and accuracy, and the advantage increases with the network size. Another advantage of IAPTF is the ease of repetition for retesting similar networks, which is often encountered in real PT. The results suggest that IAPTF is a promising approach to offload work from and ultimately replace human pen testing.

Item Type:	Article
Uncontrolled Keywords:	Penetration testing; Artificial intelligence; Machine learning; Reinforcement learning; Hierarchical reinforcement learning; Markov decision process; Vulnerability assessment
Subjects:	000 Computer science, information & general works
Department:	School of Computing and Digital Media
Depositing User:	Mohamed Ghanem
Date Deposited:	14 Oct 2022 08:20
Last Modified:	14 Oct 2022 11:00
URI:	https://repository.londonmet.ac.uk/id/eprint/7956

Downloads

Actions (login required)

View Item