The generalizability theory as an approach to estimating the reliability and dependability of the academic language placement test

• Main Author: Fonacier, Elvira C.
• Format: text
• Language: English
• Published: Animo Repository 1997
• Subjects: Examinations > Design and construction; Educational tests and measurements; Language and languages > Examinations; English Language and Literature
• Online Access: https://animorepository.dlsu.edu.ph/etd_doctoral/1147
• Institution: De La Salle University

A particular approach that brings about more informative results than classical test theory is the Generalizability theory. Generalizability theory (referred to in this dissertation as G theory), an approach developed by Cronbach, Gleser, Nanda, and Ranjaratnam (1972), uses multiple factorial ANOVA as its basic formula to 1) identify the likely sources of error variance, 2) determine the magnitude of each source, and 3) provide information on the relationship these sources have with each other. This study aims to find differences between the power of Classical True Score Measurement Theory and Generalizability Theory in estimating the reliability of the Academic Language Placement Test (ALPT).Around 1,000 freshmen students from De La Salle University participated in the study by taking the ALPT. ALPT was a researcher-designed/adapted battery of tests which consisted of three timed-subtests: Reading, Listening, and Writing. Test results were presented with descriptive statistics, and analyzed using classical test theory and G theory approaches to test reliability.The analysis showed that G theory was more powerful than classical test theory in that it was able to provide estimates of the a) magnitude of multiple sources of variance, b) relationship between the different sources of variance, and c) scores through an analysis of combination of facets. Possible modifications in the number and combinations of facets that may bring about more dependable scores was also determined through G-study analyses which, in turn, provided insights on alternative designs for collecting large-scale assessment data for a D-study. Findings in the ALPT showed that sources of error variance can be determined through various G-study designs. For the purpose of this study, Reading and Listening Tests were analyzed using 'px[i:s]' design, where 'p' is person facet, 'i' for items, and 's' for subtest. Results showed that the largest component of the variance for Reading and Listening test was due to the subtest facet. For the writing test a 'pxtxr' G-study design was used to analyze test results. The largest component of the error variance was found to be due to the interaction effect between rater and task. Further, the study showed the relationships between the facets and their interaction with each other in the G-study designs. Such relationships led to further analyses of possible combinations of facets which were illustrated to have marked effects on the magnitude of the sources of error of variance and to the G-coefficient as a whole. The study also showed how G-study results could help estimate the likely results of Decision-study of D-study using the same test results.In conclusion, after analyzing ALPT test results through both Classical Test theory and Generalizability theory, the G theory was able to provide the researcher with more information about the likely sources of error variance in the test. The results helped determine the contributions made by each facet and their combined interaction which, when appropriately manipulated, could obtain variance component magnitudes and G-coefficients that may be acceptable at a particular test situation. Finally, the G-study designs allowed to obtain insight as to how the ALPT couldb improved for future administration or D-study purposes.