Twitter archive system

• Main Author: Ong, Ann Aik.
• Other Authors: Sun Aixin
• Format: Final Year Project
• Language: English
• Published: 2011
• Subjects: DRNTU::Engineering::Computer science and engineering::Information systems::Information storage and retrieval
• Online Access: http://hdl.handle.net/10356/46456
• Institution: Nanyang Technological University

Twitter is a popular source of text data for mining and analysis as there is a large amount of free data available and easily accessible on Twitter. However, before data could be mined from Twitter, data has to be collected from Twitter. The purpose of this project is to design and develop a reliable data collector which will periodically collect data from selective Twitter users using a scheduler, based on the users’ pattern of tweeting and analyzes the collected data. The entire data collection and analysis process is fully automated and it is expected to be running 24/7/365. The java desktop application is developed using NetBeans IDE 6.7.1 with MySQL Server 5.0.91 as the data storage and Twitter4J as the java library to communicate with Twitter API. The testing of the data collector is spread over a period of 3 days, from 20th to 23rd September 2011. Within these 3 days of data collection, 56,961 users were captured. 20,779 of them are Singapore users while 36,182 are non Singapore users. Apart from that, 244,192 tweets were downloaded and 144,042 of follow relationships were found. The objective of this project has been met as the data collector was found to have successfully collected a large amount of data from Twitter within the 3 days of data collection. For optimal performance of the data collector, the implementation of a multithreaded scheduler is highly recommended for future improvement.