A critical study on MovieLens dataset for recommender systems

• Main Author: Tan, Ernest Yan Heng
• Other Authors: Sun Aixin
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Information systems::Information storage and retrieval
• Online Access: https://hdl.handle.net/10356/171942
• Institution: Nanyang Technological University

The growth in recommendation systems (RecSys) research has led to the development of many toolkits, which provide users, who may have varying levels of knowledge in the field, with the necessary tools to build, test, evaluate and benchmark different algorithms. The MovieLens datasets have garnered widespread popularity as a benchmark dataset for RecSys research, but exploratory analysis has shown that the datasets elicit certain issues such as popularity bias and data sparsity as a result. Therefore, evaluation results of baseline algorithms trained on this dataset may pick up these inherent signals present in the data, and therefore should not be generalised across other recommendation scenarios. A comprehensive and consistent experiment involving 3 Python-based Top-N recommendation toolkits: LensKit, RecPack, and daisyRec have shown that toolkits are often built with different purposes or to solve specific issues, which leads to inconsistency in implementation methodology and hence evaluation results. This can be attributed to several main factors: (1) unclear or inconsistent definition of concepts such as evaluation metrics and (2) differences in default preprocessing and splitting strategies being the most significant. The experiments also highlight the disadvantages of using a global time-aware split on the MovieLens dataset, such as eliminating unseen users which are present in the test set but not in the train set. Additionally, analysis showed that having a low absolute number of train interactions, e.g., less than 15, is detrimental to the performance of a model than having a low train to test interaction ratio, with the evaluation metrics showing relatively poorer performance on 2 out of 3 of the toolkits discussed. Lastly, this study proposes some possible improvements to the toolkits based on the issues highlighted, such as clearly defined default dataset preprocessing, fully customisable hyperparameters, and frameworks which allow for quick development of algorithms and metrics, with a possible future work of producing an actively managed, open source toolkit which can solve the problems surfaced during this study.