Evaluation of web-based search engines using user-effort measures
This paper presents a study of the applicability of three user-effort-sensitive evaluation measures —“first 20 full precision,” “search length,” and “rank correlation”—on four Web-based search engines (Google, AltaVista, Excite and Metacrawler). The authors argue that these measures are better al...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
2021
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/152490 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Institution: | Nanyang Technological University |
| Language: | English |
| Summary: | This paper presents a study of the applicability of three user-effort-sensitive evaluation measures
—“first 20 full precision,” “search length,” and “rank correlation”—on four Web-based search
engines (Google, AltaVista, Excite and Metacrawler). The authors argue that these measures are
better alternatives than precision and recall in Web search situations because of their emphasis on
the quality of ranking. Eight sets of search topics were collected from four Ph.D. students in four
different disciplines (biochemistry, industrial engineering, economics, and urban planning). Each
participant was asked to provide two topics along with the corresponding query terms. Their
relevance and credibility judgment of the Web pages were then used to compare the performance
of the search engines using these three measures. The results show consistency among these
three ranking evaluation measures, more so between “first 20 full precision” and search length
than between rank correlation and the other two measures. Possible reasons for rank correlation’s
disagreement with the other two measures are discussed. Possible future research to improve
these measures is also addressed. |
|---|