Scalable analysis of Syslog data using Kibana from Elasticsearch

Huge amount of data is being logged by network devices daily. As such, it is important to search through this huge amount of data through data mining. Data mining is like finding needle in a haystack and it can be very hard to extract useful information out of all the data that is being logged into...

Full description

Saved in:
Bibliographic Details
Main Author: Muhammad Nafiz Hamza
Other Authors: Lim Meng Hiot
Format: Final Year Project
Language:English
Published: 2015
Subjects:
Online Access:http://hdl.handle.net/10356/63602
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
Description
Summary:Huge amount of data is being logged by network devices daily. As such, it is important to search through this huge amount of data through data mining. Data mining is like finding needle in a haystack and it can be very hard to extract useful information out of all the data that is being logged into the database. In this project, Elasticsearch is used as the database engine to query huge amount of data. It will be more meaningful if the chunks of data can be presented in a user-friendly way. The human mind is not good at processing data. But if data can be visualized as pie chart, bar chart, histogram and etc, then we can make meaningful assumptions about the data easily without having to see every part of it in the database. This is where an open source web interface called Kibana comes into play. This will make it easier to look at big streams of events since it can visualize the data either as chart, tables, histogram and many more, making it easier for users to interpret.