Effectiveness of Options in Multiple-choice Items: the case of “none of the above”
Subject Areas : Educational PsychologyResa Nejati 1 , Mohammad Morady 2
1 - دانشیار دانشگاه تربیت معلم شهید رجایی
2 - دانش آموخته کارشناسی ارشد دانشگاه تربیت معلم شهید رجایی
Keywords: item response theory, none of the above, item fit, item difficulty, item discrimination, guess factor,
Abstract :
The present study examined the effectiveness of ‘none of the above’ (NOTA) as a test alternative in multiple- choice items.It intended to estimate item fit, item difficulty, itemdiscrimination, guess factor of such a choice and the reliability of the whole test. To this end, the researchersselected five passages of reading section of the Cambridge Key English Test known as KET (2010) and developed a parallel form of that test; test one did not include NOTA, whereas the second test, administered two weeks later, included NOTA. The two tests, 32 items each, were given to 142 high-school third graders. The results, analyzed through 3-parameter logistic model of item response theory (IRT), revealed that multiple-choice questions including the alternative NOTA were easier than their counterparts. In addition, NOTA option did not threaten item fit and item discrimination but increased the guess factor, which, in turn may threaten the reliability and validity of the test.
References:
Baker, F., B. (2001). The Basics of Item Response Theory. ERIC Clearinghouse.
Boynton, M. (1950). Inclusion of ‘none of these’ makes spelling items more difficult. Educational and Psychological Measurement, 10, 431-432.
Brown, J. D. (2005). Testing in Language programs: a Comprehensive Guide to English Language Assessment. New York: McGraw-Hill.
Burton, S. Sudweeks, Merrill, P. & Wood, B. (1991). How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University: Testing Services and the Department of Instructional Science.
Cambridge Key English Tests. (2010). Cambridge: Cambridge University Press.
Crehan, K. D., & Haladyna, T. M. (1991).The validity of two item-writing rules. The Journal of Experimental Education, 59(2), 183-192.
Crehan, K. D., Haladyna, T. M., & Brewer, E.W. (1993).Use of an inclusive option and the optimal number of options for multiple-choice items. Educational and Psychological Measurement, 53(1), 241-247.
DeMars, C., (2010). Item Response Theory. Oxford University Press.
Downing, S. M. (2002). Construct-irrelevant variance and flawed test questions: Do multiple-choice item-writing principles make any difference? Academic Medicine, 77(10), 103-104.
Dudycha, A. L., & Carpenter, J. B. (1973).Effects of item formats on item discrimination and difficulty. Journal of Applied Psychology, 58, 116-121.
Farhady, H., Jafarpur, A. & Birjandi, P. (1994). Testing language skills: from theory to practice, Tehran: the Center for Studying and Compiling University books in Humanities (SAMT).
Frary, R., B. (1991). The none-of-the-above option: An empirical study. Applied Measurement in Education, 4(2), 115-124.
Gronlund, N.E. (1988). How to construct Achievement Tests. Englewood Cliffs: Prentice Hall.
Haladyna, T. M. (1994). Developing and validating multiple-choice test items. Hillsdale, NJ: Lawrence Erlbaum Associates Publishers.
Haladyna, T.M. (2004). Developing and validating multiple-choice test items. (3rd). Mahwah, NJ: Lawrence Erlbaum Associates publishers.
Haladyna, T. M., & Downing, S. M. (1989a).A taxonomy of multiple-choice item- writing rules. Applied measurement in education, 2(1), 37-50.
Haladyna, T. M., & Downing, S. M. (1989b).Validity of taxonomy of multiple-choice item writing rules. Applied measurement in education, 2 (1), 51-78.
Haladyna, T. M., Downing, S. M., & Rodríguez, M.C. (2002). A review of
multiple-choice item-writing guidelines for classroom assessment. Applied
Measurement in Education, 15(3), 309-334.
Hughes, H. H., & Trimble, W. E. (1965).The use of complex alternatives in multiple-choice items. Educational and Psychological Measurement, 25(1), 117-126.
Knowles, S. L., & Welch, C. A. (1992).A meta-analytic review of item discrimination and difficulty in multiple-choice items using none-of-the-above. Educational and psychological measurement, 52, 571-577.
Kolstad, R. K., & Kolstad, R. A. (1991). The option “none of these” improves multiple-choice tests items. Journal of dental education, 55(2), 161-163.
Martínez, R., J., Moreno, R., Martin, I., Trigo, M. E. (2009). Evaluation of five guidelines for option development in multiple-choice item-writing. Psicothema21 (2), 326-30.
Mehrens, W. A., and Lehmann, I. J. (1984). Measurement and evaluation. Third edition. New York: CBS College Publishing.
Mehrens, W. A., and Lehmann, I. J. (1991). Measurement and evaluation in education and psychology. Orlando, FL: Harcourt Brace Jovanovich.
Mousavi, S. A. (2009). An encyclopedic dictionary of language testing. Tehran: Rahnama Publications.
Mueller, D. J. (1975).An assessment of the effectiveness of complex alternatives in multiple choice achievement test items. Educational and psychological measurement, 35, 135-141.
Mueller, J. (2011). Constructing good items. Retrieved on January 2012 from http:// jfmueller.faculty.noctrl.edu/toolbox/index.htm.
Odegard, T.N., Koen, J.D. (2007). "None of the above" as a correct and incorrect alternative on a multiple-choice test: implications for the testing effect. Memory,15(8), 873-85.
Oosterhof, A. C., & Coats, P. K. (1984). Comparison of difficulties and reliabilities of quantitative word problems in completion and multiple choice item formats. Applied psychological measurement, 8, 287-294.
Osterlind, S.J. (2002). Constructing test items: multiple-choice, constructed-response, performance and other formats. New York: Kluwer Academic Publishers.
Pashasharifi, H. & Keyamanesh, A. (1984). Shivehaye Arzeshyabi az Amookhtehaye Danesh Amoozan [Methods of assessing students knowledge].Tehran: Sherkat e chap va nashre iran.
Payne, D. A. (2003). Applied educational assessment (2nd Ed.). United States: Wadsworth.
Rich, C. E., & Johanson, G. A. (1990). An item-level analysis of “none of the above.” Paper presented at the annual meeting of the AERA, Boston, MA.
Rimland, B. (1960). The effects of varying time limits and of using Right answer not give in experimental forms of the U.S. Navy Arithmetic Test. Educational and psychological measurement, 20(3), 533-539.
Rodriguez, M. C. (1997). The art and science of item writing: A meta-analysis of multiple-choice item format effects. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational measurement: issues and practice, 24(2), 3–13.
Schmeiser, C. B., & Whitney, D. R. (1975). The effect of incomplete stems and “none of the above” foils on test and item characteristics. Paper presented at the annual meeting of the NCME, Washington, DC.
Seyf, A. A. (2009). Andazehgiri sanjeshva arzeshyabi e amoozeshi [Educational measurement assessment and evaluation]. Tehran: Nashr e Dawran.
Tarrant, M., and Ware, J., F. (2008).Impact of item-writing flaws in multiple- choice questions on student achievement in high-stakes nursing assessments.Medical education, 42 (2), 198-206.
Tollefson, N. (1987). A comparison of the item difficulty and item discrimination of multiple-choice items using none of the above and one correct response options. Educational and psychological measurement, 47(2), 377-383.
Tollefson, N., & Chen, J., S. (1986). A comparison of item difficulty and item discrimination of multiple-choice items using none of the above options. Paper presented at the annual meeting of the Midwest AERA, Chicago, IL.
Tollefson, N., & Tripp, A. (1983). The effect of item format on item difficulty and item discrimination. Paper presented at the annual meeting of the AERA, Montreal, Quebec.
Wesman, A. G., & Bennett, G. K. (1946).The use of ‘none of these’ as an option in test construction. Journal of Educational Psychology, 37, 541- 549.
Williamson, M. L., & Hopkins, K. D. (1967). The use of none-of-these versus homogeneous alternatives on multiple-choice tests: Experimental reliability and validity comparisons. Journal of educational measurement, 4(2), 53-58.
Zimmaro, D. M. (2010). Writing good multiple-choice exams. Retrieved from www.ctl.utexas.edu/Evaluation--Assessment/Writing Good Multiple Choice Exams-04-28-10-pdf.
