A Dynamic Rule Creation Based Anomaly Detection Method for Identifying Security Breaches in Log Records

• Main Authors: Breier, Jakub, Branišová, Jana
• Other Authors: Temasek Laboratories
• Format: Article
• Language: English
• Published: 2016
• Subjects: Network security; Log analysis
• Online Access: https://hdl.handle.net/10356/83422; http://hdl.handle.net/10220/41427
• Institution: Nanyang Technological University

Evidence of security breaches can be found in log files, created by various network devices in order to provide information about their operation. Huge amount of data contained within these files usually prevents to analyze them manually, therefore it is necessary to utilize automatic methods capable of revealing potential attacks. In this paper we propose a method for anomaly detection in log files, based on data mining techniques for dynamic rule creation. To support parallel processing, we employ Apache Hadoop framework, providing distributed storage and distributed processing of data. Outcomes of our testing show potential to discover new types of breaches and plausible error rates below 10 %. Also, rule generation and anomaly detection speeds are competitive to currently used algorithms, such as FP-growth and apriori.