Empirical evaluation of three common assumptions in building political media bias datasets

• Main Authors: GANGULY, Soumen, KULSHRESTHA, Juhi, AN, Jisun, KWAK, Haewoon
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: Numerical Analysis and Scientific Computing
• Online Access: https://ink.library.smu.edu.sg/sis_research/6088; https://ink.library.smu.edu.sg/context/sis_research/article/7091/viewcontent/7362_Article_Text_10592_1_10_20200601.pdf
• Institution: Singapore Management University

In this work, we empirically validate three common assumptions in building political media bias datasets, which are (i) labelers' political leanings do not affect labeling tasks, (ii) news articles follow their source outlet's political leaning, and (iii) political leaning of a news outlet is stable across different topics. We build a ground-truth dataset of manually annotated article-level political leaning and validate the three assumptions. Our findings warn that the three assumptions could be invalid even for a small dataset. We hope that our work calls attention to the (in)validity of common assumptions in building political media bias datasets.