The predictions of performance metrics in information retrieval: An experimental study

• Main Authors: Muwanei, Sinyinda, Ravana, Sri Devi, Hoo, Wai Lam, Kunda, Douglas
• Format: Article
• Published: Univ Malaya, Fac Computer Science & Information Tech 2021
• Subjects: QA75 Electronic computers. Computer science
• Online Access: http://eprints.um.edu.my/35384/
• Institution: Universiti Malaya

Information retrieval systems are widely used by people from all walks of life to meet diverse user needs. Hence, the ability of these retrieval systems to return the relevant information in response to user queries has been a matter of concern to the information retrieval research community. To address this concern, evaluations of these retrieval systems is extremely critical and the most popular way is the approach that employs test collections. This approach has been the popular evaluation approach in information retrieval for several decades. However, one of the limitations of this evaluation approach concerns the costly creation of relevance judgments. In recent research, this limitation was addressed by predicting performance metrics at the high cut-off depths of documents by using performance metrics computed at low cut-off depths. However, the challenge the research community is faced with is how to predict the precision and the non-cumulative gain performance metrics at the high cut-off depths of documents while using other performance metrics computed at the low cut-off depths of at most 30 documents. This study addresses this challenge by investigating the predictability of performance metrics and proposing two approaches that predict the precision and the non-cumulative discounted gain performance metrics. This study has shown that there exist dataset shifts in the performance metrics computed from different test collections. Furthermore, the proposed approaches have demonstrated better results of the ranked correlations of the predictions of performance metrics than existing research.