Development of a crawler to collect online game playing traces

• Main Author: Lee, Jia Hao
• Other Authors: Tang Xueyan
• Format: Final Year Project
• Language: English
• Published: 2018
• Subjects: DRNTU::Engineering::Computer science and engineering::Data
• Online Access: http://hdl.handle.net/10356/76161
• Institution: Nanyang Technological University

Over the years, the rise of multiplayer online games has seen games evolve from having to be played in the same room to being able to play with people across the world through an internet connection. Coupled with the upsurge in data science and analytics, data generated from these online games have naturally become an intriguing data source to be analysed. One technique to extract such data is through the development of a web crawler. However, there are currently no existing standalone applications that purely makes use of a web crawler to retrieve data from these online games. The purpose of this project is to develop an application which includes a GUI and the crawler to collect online gaming data. The online game chosen for this project is Defence of the Ancients 2, or DotA 2 for short, which is considered to be one of the greatest video games of all time [1]. To retrieve data from DotA 2 matches, Valve, which is the company responsible for creating DotA 2, provides official APIs for developers to utilize. In addition, Stratz eSports, which is a third-party website that provides more detailed data about the game, also provides APIs for people to use. Hence, the crawler implemented will be making use of APIs from both Valve and Stratz eSports to extract data related to DotA 2 matches. Also, a GUI is designed which allows the user to customize settings for the crawler. The user is able to select a range of dates from the GUI, which informs the crawler to retrieve data only within the specified date range. Subsequently, the user can also select specific region(s) - Singapore, Australia or China - to crawl the data from. Furthermore, the progress of the crawl is updated and displayed in real-time on the GUI. All the data retrieved are stored in an output file formatted in JSON. Following which, data analysis is implemented on the data stored in the output file. Players are classified as frequent or infrequent, based on the number of games they played during the specified date range. Matches are also classified into frequent or infrequent player dominant based on the respective number of players. More data analysis techniques are subsequently carried out on both groups of players and matches. Taken together, the application described herein encompasses a web crawler for automated retrieval of data from DotA 2 matches through official APIs, while also providing an intuitive GUI to facilitate the configuration and collection of data based on the user’s needs. Various statistical analyses are also performed to examine ways to improve the game.