A two-stage mechanism for ordinal peer assessment
Peer assessment is a major method for evaluating the performance of employee, accessing the contributions of individuals within a group, making social decisions and many other scenarios. The idea is to ask the individuals of the same group to assess the performance of the others. Scores or rankings...
Saved in:
Main Authors: | , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2018
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/8673 https://ink.library.smu.edu.sg/context/sis_research/article/9676/viewcontent/SAGT18_twostage.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-9676 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-96762024-03-28T09:09:55Z A two-stage mechanism for ordinal peer assessment LI, Zhize ZHANG, Le FANG, Zhixuan LI, Jian Peer assessment is a major method for evaluating the performance of employee, accessing the contributions of individuals within a group, making social decisions and many other scenarios. The idea is to ask the individuals of the same group to assess the performance of the others. Scores or rankings are then determined based on these evaluations. However, peer assessment can be biased and manipulated, especially when there is a conflict of interests. In this paper, we consider the problem of eliciting the underlying ordering (i.e. ground truth) of n strategic agents with respect to their performances, e.g., quality of work, contributions, scores, etc. We first prove that there is no deterministic mechanism which obtains the underlying ordering in dominant-strategy implementation. Then, we propose a Two-Stage Mechanism in which truth-telling is the unique strict Nash equilibrium yielding the underlying ordering. Moreover, we prove that our two-stage mechanism is asymptotically optimal, since it only needs $n + 1$ queries and we prove an $\Omega(n)$ lower bound on query complexity for any mechanism. Finally, we conduct experiments on several scenarios to demonstrate that the proposed two-stage mechanism is robust. 2018-09-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8673 info:doi/10.1007/978-3-319-99660-8_16 https://ink.library.smu.edu.sg/context/sis_research/article/9676/viewcontent/SAGT18_twostage.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Mechanism design Peer assessment Nash equilibrium Databases and Information Systems |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Mechanism design Peer assessment Nash equilibrium Databases and Information Systems |
spellingShingle |
Mechanism design Peer assessment Nash equilibrium Databases and Information Systems LI, Zhize ZHANG, Le FANG, Zhixuan LI, Jian A two-stage mechanism for ordinal peer assessment |
description |
Peer assessment is a major method for evaluating the performance of employee, accessing the contributions of individuals within a group, making social decisions and many other scenarios. The idea is to ask the individuals of the same group to assess the performance of the others. Scores or rankings are then determined based on these evaluations. However, peer assessment can be biased and manipulated, especially when there is a conflict of interests. In this paper, we consider the problem of eliciting the underlying ordering (i.e. ground truth) of n strategic agents with respect to their performances, e.g., quality of work, contributions, scores, etc. We first prove that there is no deterministic mechanism which obtains the underlying ordering in dominant-strategy implementation. Then, we propose a Two-Stage Mechanism in which truth-telling is the unique strict Nash equilibrium yielding the underlying ordering. Moreover, we prove that our two-stage mechanism is asymptotically optimal, since it only needs $n + 1$ queries and we prove an $\Omega(n)$ lower bound on query complexity for any mechanism. Finally, we conduct experiments on several scenarios to demonstrate that the proposed two-stage mechanism is robust. |
format |
text |
author |
LI, Zhize ZHANG, Le FANG, Zhixuan LI, Jian |
author_facet |
LI, Zhize ZHANG, Le FANG, Zhixuan LI, Jian |
author_sort |
LI, Zhize |
title |
A two-stage mechanism for ordinal peer assessment |
title_short |
A two-stage mechanism for ordinal peer assessment |
title_full |
A two-stage mechanism for ordinal peer assessment |
title_fullStr |
A two-stage mechanism for ordinal peer assessment |
title_full_unstemmed |
A two-stage mechanism for ordinal peer assessment |
title_sort |
two-stage mechanism for ordinal peer assessment |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2018 |
url |
https://ink.library.smu.edu.sg/sis_research/8673 https://ink.library.smu.edu.sg/context/sis_research/article/9676/viewcontent/SAGT18_twostage.pdf |
_version_ |
1795302169131352064 |