Rater reliability of a localised CEFR descriptor for assessing writing at Malaysian Higher Education

• Main Authors: Abu Sufi, Mohd Khairul, Abu Kassim, Noor Lide, Mohd Bakri, Siti Khairunnisa
• Format: Proceeding Paper
• Language: English
• Published: 2023
• Subjects: LB2300 Higher Education
• Online Access: http://irep.iium.edu.my/108538/1/108538_rater%20reliability%20of%20a%20localised.pdf; http://irep.iium.edu.my/108538/
• Institution: Universiti Islam Antarabangsa Malaysia

The ability to master the English language writing has been recognised as a paramount importance at the Malaysian tertiary level education. Consequently, the Malaysian Ministry of Education introduced the CEFR-aligned English language curriculum in 2017 and the application of CEFR standards in assessing higher education students’ English language writing skill has become a ubiquitous practice in Malaysia. However, the CEFR Can Do statements only serve as a guiding principle towards a development of an assessment rubric that would conform to local Malaysian standards, whilst preserving the standards of the CEFR. This study explicates reliability of a localised CEFR assessment descriptor for writing. Using a Multi-facet Rasch Model (MFRM), a comparison between 2 sets of examiners are compared based on the raters’ severity level using the localised CEFR descriptor.10 experienced academics, as well as 3 CEFR experts from Malaysian higher educational institutions participated in semi-structured interviews, following a four-cyclical process of the language management theory. Qualitative findings revealed a positive reception of CEFR among participating academics. Despite positive feedback, the study underscores the necessity of localising CEFR within the Malaysian higher education system to align with the objectives of academics aiming to cultivate English-proficient students, leading towards the development of the localised CEFR writing descriptor. Additionally, quantitative findings of the study illuminated that as a group, there is no significance difference between the two groups of raters when assessing writing scripts using the localised CEFR descriptor. However, several raters from the pool of academics displayed a high severity level. Overall, it can be summarised that the localised CEFR descriptor for writing could be used to assess writing at higher education level, provided that rater training is conducted.