A two-stage mechanism for ordinal peer assessment

Peer assessment is a major method for evaluating the performance of employee, accessing the contributions of individuals within a group, making social decisions and many other scenarios. The idea is to ask the individuals of the same group to assess the performance of the others. Scores or rankings...

Full description

Saved in:
Bibliographic Details
Main Authors: LI, Zhize, ZHANG, Le, FANG, Zhixuan, LI, Jian
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2018
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/8673
https://ink.library.smu.edu.sg/context/sis_research/article/9676/viewcontent/SAGT18_twostage.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-9676
record_format dspace
spelling sg-smu-ink.sis_research-96762024-03-28T09:09:55Z A two-stage mechanism for ordinal peer assessment LI, Zhize ZHANG, Le FANG, Zhixuan LI, Jian Peer assessment is a major method for evaluating the performance of employee, accessing the contributions of individuals within a group, making social decisions and many other scenarios. The idea is to ask the individuals of the same group to assess the performance of the others. Scores or rankings are then determined based on these evaluations. However, peer assessment can be biased and manipulated, especially when there is a conflict of interests. In this paper, we consider the problem of eliciting the underlying ordering (i.e. ground truth) of n strategic agents with respect to their performances, e.g., quality of work, contributions, scores, etc. We first prove that there is no deterministic mechanism which obtains the underlying ordering in dominant-strategy implementation. Then, we propose a Two-Stage Mechanism in which truth-telling is the unique strict Nash equilibrium yielding the underlying ordering. Moreover, we prove that our two-stage mechanism is asymptotically optimal, since it only needs $n + 1$ queries and we prove an $\Omega(n)$ lower bound on query complexity for any mechanism. Finally, we conduct experiments on several scenarios to demonstrate that the proposed two-stage mechanism is robust. 2018-09-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8673 info:doi/10.1007/978-3-319-99660-8_16 https://ink.library.smu.edu.sg/context/sis_research/article/9676/viewcontent/SAGT18_twostage.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Mechanism design Peer assessment Nash equilibrium Databases and Information Systems
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Mechanism design
Peer assessment
Nash equilibrium
Databases and Information Systems
spellingShingle Mechanism design
Peer assessment
Nash equilibrium
Databases and Information Systems
LI, Zhize
ZHANG, Le
FANG, Zhixuan
LI, Jian
A two-stage mechanism for ordinal peer assessment
description Peer assessment is a major method for evaluating the performance of employee, accessing the contributions of individuals within a group, making social decisions and many other scenarios. The idea is to ask the individuals of the same group to assess the performance of the others. Scores or rankings are then determined based on these evaluations. However, peer assessment can be biased and manipulated, especially when there is a conflict of interests. In this paper, we consider the problem of eliciting the underlying ordering (i.e. ground truth) of n strategic agents with respect to their performances, e.g., quality of work, contributions, scores, etc. We first prove that there is no deterministic mechanism which obtains the underlying ordering in dominant-strategy implementation. Then, we propose a Two-Stage Mechanism in which truth-telling is the unique strict Nash equilibrium yielding the underlying ordering. Moreover, we prove that our two-stage mechanism is asymptotically optimal, since it only needs $n + 1$ queries and we prove an $\Omega(n)$ lower bound on query complexity for any mechanism. Finally, we conduct experiments on several scenarios to demonstrate that the proposed two-stage mechanism is robust.
format text
author LI, Zhize
ZHANG, Le
FANG, Zhixuan
LI, Jian
author_facet LI, Zhize
ZHANG, Le
FANG, Zhixuan
LI, Jian
author_sort LI, Zhize
title A two-stage mechanism for ordinal peer assessment
title_short A two-stage mechanism for ordinal peer assessment
title_full A two-stage mechanism for ordinal peer assessment
title_fullStr A two-stage mechanism for ordinal peer assessment
title_full_unstemmed A two-stage mechanism for ordinal peer assessment
title_sort two-stage mechanism for ordinal peer assessment
publisher Institutional Knowledge at Singapore Management University
publishDate 2018
url https://ink.library.smu.edu.sg/sis_research/8673
https://ink.library.smu.edu.sg/context/sis_research/article/9676/viewcontent/SAGT18_twostage.pdf
_version_ 1795302169131352064