Real-time cyber analytics data collection framework

Maosa, Herbert, Ouazzane, Karim and Sowinski-Mydlarz, Viktor (2022) Real-time cyber analytics data collection framework. International Journal of Information Security and Privacy (IJISP), 16 (1). pp. 1-10. ISSN 1930-1650

Real Time Cyber Analytics Data Collection Framework .pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives 4.0.

Download (418kB) | Preview
Official URL:

Abstract / Description

For effective security, it is critical that event data is collected in near real time as possible to enable early detection and response to threats. Performing analytics from event logs stored in databases slows down the response time due to the time cost of database insertion and retrieval operations. We present a data collection framework that minimizes the need for long term storage. Events are buffered in memory, up to a configurable threshold, before being streamed in real time using live streaming technologies. The framework deploys virtualized data collecting agents that ingest data from multiple sources including external Threat Intelligence. The framework enables the correlation of events from various sources, improving detection precision. We have tested the framework in a real time, machine-learning based threat detection system. Our results show a time gain of 300 milliseconds in transmission time from event capture to analytics system, compared with storage based collection frameworks. Threat detection was measured at 95%, which is comparable to the benchmark snort IDS.

Item Type: Article
Uncontrolled Keywords: data collection; event correlation; cyber event analytics; real time detection; log analysis
Subjects: 000 Computer science, information & general works
Department: School of Computing and Digital Media
Depositing User: Bal Virdee
Date Deposited: 07 Oct 2022 13:12
Last Modified: 11 Nov 2022 09:41


Downloads per month over past year

Downloads each year

Actions (login required)

View Item View Item