Fairness testing of machine translation systems

Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach t...

Full description

Saved in:
Bibliographic Details
Main Authors: Sun,Zeyu, Chen, Zhenpeng, Zhang, Jie, Hao, Dan
Other Authors: College of Computing and Data Science
Format: Article
Language:English
Published: 2024
Subjects:
Online Access:https://hdl.handle.net/10356/180446
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-180446
record_format dspace
spelling sg-ntu-dr.10356-1804462024-10-07T08:38:08Z Fairness testing of machine translation systems Sun,Zeyu Chen, Zhenpeng Zhang, Jie Hao, Dan College of Computing and Data Science Computer and Information Science Fairness testing Metamorphic testing Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach tailored for machine translation systems. FairMT operates on the assumption that translations of semantically similar sentences, containing protected attributes from distinct demographic groups, should maintain comparable meanings. It comprises three key steps: (1) test input generation, producing inputs covering various demographic groups; (2) test oracle generation, identifying potential unfair translations based on semantic similarity measurements; and (3) regression, discerning genuine fairness issues from those caused by low-quality translation. Leveraging FairMT, we conduct an empirical study on three leading machine translation systems-Google Translate, T5, and Transformer. Our investigation uncovers up to 832, 1,984, and 2,627 unfair translations across the three systems, respectively. Intriguingly, we observe that fair translations tend to exhibit superior translation performance, challenging the conventional wisdom of a fairness-performance tradeoff prevalent in the fairness literature. Published version This work was supported by National Natural Science Foundation of China under Grant No. 62372005. 2024-10-07T08:38:08Z 2024-10-07T08:38:08Z 2024 Journal Article Sun, Z., Chen, Z., Zhang, J. & Hao, D. (2024). Fairness testing of machine translation systems. ACM Transactions On Software Engineering and Methodology, 33(6), 156-. https://dx.doi.org/10.1145/3664608 1049-331X https://hdl.handle.net/10356/180446 10.1145/3664608 2-s2.0-85198860702 6 33 156 en ACM Transactions on Software Engineering and Methodology © 2024 Copyright held by the owner/author(s). This work is licensed under a Creative Commons Attribution International 4.0 License. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Computer and Information Science
Fairness testing
Metamorphic testing
spellingShingle Computer and Information Science
Fairness testing
Metamorphic testing
Sun,Zeyu
Chen, Zhenpeng
Zhang, Jie
Hao, Dan
Fairness testing of machine translation systems
description Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach tailored for machine translation systems. FairMT operates on the assumption that translations of semantically similar sentences, containing protected attributes from distinct demographic groups, should maintain comparable meanings. It comprises three key steps: (1) test input generation, producing inputs covering various demographic groups; (2) test oracle generation, identifying potential unfair translations based on semantic similarity measurements; and (3) regression, discerning genuine fairness issues from those caused by low-quality translation. Leveraging FairMT, we conduct an empirical study on three leading machine translation systems-Google Translate, T5, and Transformer. Our investigation uncovers up to 832, 1,984, and 2,627 unfair translations across the three systems, respectively. Intriguingly, we observe that fair translations tend to exhibit superior translation performance, challenging the conventional wisdom of a fairness-performance tradeoff prevalent in the fairness literature.
author2 College of Computing and Data Science
author_facet College of Computing and Data Science
Sun,Zeyu
Chen, Zhenpeng
Zhang, Jie
Hao, Dan
format Article
author Sun,Zeyu
Chen, Zhenpeng
Zhang, Jie
Hao, Dan
author_sort Sun,Zeyu
title Fairness testing of machine translation systems
title_short Fairness testing of machine translation systems
title_full Fairness testing of machine translation systems
title_fullStr Fairness testing of machine translation systems
title_full_unstemmed Fairness testing of machine translation systems
title_sort fairness testing of machine translation systems
publishDate 2024
url https://hdl.handle.net/10356/180446
_version_ 1814047390196301824