Scalable analysis of Syslog data using Kibana from Elasticsearch

• Main Author: Muhammad Nafiz Hamza
• Other Authors: Lim Meng Hiot
• Format: Final Year Project
• Language: English
• Published: 2015
• Subjects: DRNTU::Engineering::Electrical and electronic engineering::Computer hardware, software and systems
• Online Access: http://hdl.handle.net/10356/63602
• Institution: Nanyang Technological University

Huge amount of data is being logged by network devices daily. As such, it is important to search through this huge amount of data through data mining. Data mining is like finding needle in a haystack and it can be very hard to extract useful information out of all the data that is being logged into the database. In this project, Elasticsearch is used as the database engine to query huge amount of data. It will be more meaningful if the chunks of data can be presented in a user-friendly way. The human mind is not good at processing data. But if data can be visualized as pie chart, bar chart, histogram and etc, then we can make meaningful assumptions about the data easily without having to see every part of it in the database. This is where an open source web interface called Kibana comes into play. This will make it easier to look at big streams of events since it can visualize the data either as chart, tables, histogram and many more, making it easier for users to interpret.