Asking clarifying questions: to benefit or to disturb users in web search?

Modern information-seeking systems are becoming more interactive, mainly through asking Clarifying Questions (CQs) to refine users’ information needs. System-generated CQs may be of different qualities. However, the impact of asking multiple CQs of different qualities in a search session remains und...

Full description

Saved in:
Bibliographic Details
Main Authors: Zou, Jie, Sun, Aixin, Long, Cheng, Mohammad Aliannejadi, Kanoulas, Evangelos
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2023
Subjects:
Online Access:https://hdl.handle.net/10356/164602
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-164602
record_format dspace
spelling sg-ntu-dr.10356-1646022023-02-06T06:06:02Z Asking clarifying questions: to benefit or to disturb users in web search? Zou, Jie Sun, Aixin Long, Cheng Mohammad Aliannejadi Kanoulas, Evangelos School of Computer Science and Engineering Singtel Cognitive and Artificial Intelligence Lab for Enterprises@NTU Engineering::Computer science and engineering User Study Information-Seeking Systems Modern information-seeking systems are becoming more interactive, mainly through asking Clarifying Questions (CQs) to refine users’ information needs. System-generated CQs may be of different qualities. However, the impact of asking multiple CQs of different qualities in a search session remains underexplored. Given the multi-turn nature of conversational information-seeking sessions, it is critical to understand and measure the impact of CQs of different qualities, when they are posed in various orders. In this paper, we conduct a user study on CQ quality trajectories, i.e., asking CQs of different qualities in chronological order. We aim to investigate to what extent the trajectory of CQs of different qualities affects user search behavior and satisfaction, on both query-level and session-level. Our user study is conducted with 89 participants as search engine users. Participants are asked to complete a set of Web search tasks. We find that the trajectory of CQs does affect the way users interact with Search Engine Result Pages (SERPs), e.g., a preceding high-quality CQ prompts the depth users to interact with SERPs, while a preceding low-quality CQ prevents such interaction. Our study also demonstrates that asking follow-up high-quality CQs improves the low search performance and user satisfaction caused by earlier low-quality CQs. In addition, only showing high-quality CQs while hiding other CQs receives better gains with less effort. That is, always showing all CQs may be risky and low-quality CQs do disturb users. Based on observations from our user study, we further propose a transformer-based model to predict which CQs to ask, to avoid disturbing users. In short, our study provides insights into the effects of trajectory of asking CQs, and our results will be helpful in designing more effective and enjoyable search clarification systems. This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU). This study is also supported by the NWO Smart Culture - Big Data/Digital Humanities (314-99-301), the NWO Innovational Research Incentives Scheme Vidi (016.Vidi.189.039), and the H2020- EU.3.4. - SOCIETAL CHALLENGES - Smart, Green, And Integrated Transport (814961). 2023-02-06T06:06:02Z 2023-02-06T06:06:02Z 2023 Journal Article Zou, J., Sun, A., Long, C., Mohammad Aliannejadi & Kanoulas, E. (2023). Asking clarifying questions: to benefit or to disturb users in web search?. Information Processing and Management, 60(2), 103176-. https://dx.doi.org/10.1016/j.ipm.2022.103176 0306-4573 https://hdl.handle.net/10356/164602 10.1016/j.ipm.2022.103176 2-s2.0-85142759494 2 60 103176 en IAF-ICP Information Processing and Management © 2022 Elsevier Ltd. All rights reserved.
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
User Study
Information-Seeking Systems
spellingShingle Engineering::Computer science and engineering
User Study
Information-Seeking Systems
Zou, Jie
Sun, Aixin
Long, Cheng
Mohammad Aliannejadi
Kanoulas, Evangelos
Asking clarifying questions: to benefit or to disturb users in web search?
description Modern information-seeking systems are becoming more interactive, mainly through asking Clarifying Questions (CQs) to refine users’ information needs. System-generated CQs may be of different qualities. However, the impact of asking multiple CQs of different qualities in a search session remains underexplored. Given the multi-turn nature of conversational information-seeking sessions, it is critical to understand and measure the impact of CQs of different qualities, when they are posed in various orders. In this paper, we conduct a user study on CQ quality trajectories, i.e., asking CQs of different qualities in chronological order. We aim to investigate to what extent the trajectory of CQs of different qualities affects user search behavior and satisfaction, on both query-level and session-level. Our user study is conducted with 89 participants as search engine users. Participants are asked to complete a set of Web search tasks. We find that the trajectory of CQs does affect the way users interact with Search Engine Result Pages (SERPs), e.g., a preceding high-quality CQ prompts the depth users to interact with SERPs, while a preceding low-quality CQ prevents such interaction. Our study also demonstrates that asking follow-up high-quality CQs improves the low search performance and user satisfaction caused by earlier low-quality CQs. In addition, only showing high-quality CQs while hiding other CQs receives better gains with less effort. That is, always showing all CQs may be risky and low-quality CQs do disturb users. Based on observations from our user study, we further propose a transformer-based model to predict which CQs to ask, to avoid disturbing users. In short, our study provides insights into the effects of trajectory of asking CQs, and our results will be helpful in designing more effective and enjoyable search clarification systems.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Zou, Jie
Sun, Aixin
Long, Cheng
Mohammad Aliannejadi
Kanoulas, Evangelos
format Article
author Zou, Jie
Sun, Aixin
Long, Cheng
Mohammad Aliannejadi
Kanoulas, Evangelos
author_sort Zou, Jie
title Asking clarifying questions: to benefit or to disturb users in web search?
title_short Asking clarifying questions: to benefit or to disturb users in web search?
title_full Asking clarifying questions: to benefit or to disturb users in web search?
title_fullStr Asking clarifying questions: to benefit or to disturb users in web search?
title_full_unstemmed Asking clarifying questions: to benefit or to disturb users in web search?
title_sort asking clarifying questions: to benefit or to disturb users in web search?
publishDate 2023
url https://hdl.handle.net/10356/164602
_version_ 1759058798151991296