A Composite Framework for ESL Textbook Evaluation

• Main Author: Mukundan, Jayakaran
• Format: Thesis
• Language: English; English
• Published: 2004
• Subjects: English language - Textbooks for foreign speakers - Evaluation
• Online Access: http://psasir.upm.edu.my/id/eprint/10319/1/FBMK_2004_6.pdf; http://psasir.upm.edu.my/id/eprint/10319/
• Institution: Universiti Putra Malaysia

Textbook evaluation practices have not been critically examined to determine effectiveness and value in learning-teaching environments and this is probably the main reason why the literature suggests that textbooks selected have been more of a hindrance than a benefit to teaching. The assumption made by the researcher is that since much of the criticism in selection processes of textbooks is directed towards the checklist, which at this moment seems to be the only instrument used in textbook evaluation practices, then there would be a need to re-evaluate the usefulness of the checklist, identify weak areas and then develop a composite framework where the checklist will be supported by complementary instruments, namely the concordance software and the reflective journal. The researcher suggests a 4-phase procedure in the development of the composite framework. Phase 1 tests the Skierso Evaluation Checklist (SEC) for reliability and item difficulty. Phase 2 tests the capabilities of the concordance software (WordSmith Tools 3.0) to provide analysis of the patterns of presentation of vocabulary and structures in textbooks, to determine the extent to which the software will help discriminate between books in a selection process and to determine to what extent the analysis would provide greater illustration to responses required of by items in Section D of the SEC. Phase 3 tests the capabilities of the reflective journal in providing greater illustration to responses to items in Section E (Exercises and Activities) of the SEC. Finally, in Phase 4 the researcher will assemble aspects of the t, 0 complementary components into a framework which has the checklist as its main instrument. This framework will then be tested for reliability and item difficulty. In Phase 1, the findings revealed that while the overall reliability of the SEC was high, the difficulty analysis of items showed Section) and E of the checklist as having the largest number of difficult items. Phase 2 of the study found that the concordance software is capable of many useful functions it textbook evaluation and is able to provide greater illustration, through computation to 6 items in Section D of the SEC. Phase 3 of the investigation revealed that teaching reflections contributed to input that was beneficial to evaluation, especially the item in Section E of the SEC. The composite framework was assembled and created in Phase 4. It was then compared to the mono-instrument procedure (Phase 1) which consisted of the checklist (SEC). The comparison of the two procedures showed the composite framework to be more reliable at 0.9324 reliability as compared to 0.7675 reliability for the SEC as a standalone instrument. The difficulty analysis of items also showed marked improvement when comparisons were made. Only 4 items were considered difficult within the composite framework as opposed to 14 when the SEC was tested as a standalone. This study has provided an alternative to the checklist dominated procedure by proposing a framework which works on the combined effort of 3 distinct instruments, thus providing for much needed triangulation which is actually expected in an exercise as complex as textbook evaluation. The spin-offs to this research are the added value it provides by way of increased awareness of action research in textbook evaluation, to greater emphasis and attention to retrospective evaluation and adaptation. It has also led to the creation of the first Malaysian Corpus of the Language of Textbooks which has approximately 150,000 words. This corpus will expand when it accommodates the language of more textbooks within the school system.