A Summative Evaluation of English for Iraq Series for High School: A Focus on Motivational Design
Subject Areas : Second Language Education
1 - College of Education, University of Shatrah
Keywords: Motivational Design, Summative Evaluation, Attention, Relevance, Confidence, Satisfaction ,
Abstract :
Motivation to learn, as a core concept in educational psychology, is a major determinant of academic achievement and productivity. Motivational design, as an aspect of instructional design focusing on strategies, principles, and processes that make instruction appealing, can contribute to the learners' goals and achievement. The present paper is an attempt to conduct a summative evaluation of the book series 'English for Iraq' for its motivational design. Accordingly, a mixed-method design was employed, relying on Keller's model. The participants included 403 Iraqi university students in the quantitative phase and 20 interviewees in the qualitative phase. Descriptive statistics and thematic coding were conducted for the data analysis. It was concluded that the overall motivational design of the book has been acceptable to the students. The analysis of the constructs in detail also showed that 'satisfaction,' followed by 'relevance,' received the highest scores. Moreover, based on the views of the students in the interviews, it was observed that most of them were motivated by the content of the book, although some had external motivation such as exam scores or achievement in exercises.
Al-Akraa, S. (2013). Teaching English in Iraq: An analysis of an EFL textbook (Published Master’s thesis). University of Central Florida.
Amin, M. Y. M. (2017). English language teaching methods and reforms in English curriculum in Iraq; an overview. Journal of University of Human Development, 3(3), 578-583.
Ayu, M., & Indrawati, R. (2019). EFL textbook evaluation: The analysis of tasks presented in English textbook. Teknosastik, 16(1), 21-25.
Ayu, L. (2020). Content Feasibility in English Textbook for Senior High School Grade XI (Doctoral dissertation). University of Muhammadiyah Sumatera Utara Medan.
Babula, M. (2013). Motivation, altruism, personality and social psychology. Palgrave Macmillan.
Bemani, M., & Jahangard, A., (2014). An Evaluative Analysis of Iranian Junior High School EFL Textbooks.Saabrucken.
Corlazzoli, V., & White, J. (2013). Practical Approaches to Theories of Change in Conflict, Security and Justice Programmes: Part II: Using Theories of Change in Monitoring and Evaluation(Search for Common Ground). Retrieved from https://issat.dcaf.ch/ara/Learn/Resource-Library/Policy-and-Research-Papers/Practical-Approaches-to-Theories-of-Change-in-Conflict-Security-and-Justice-Programmes-Part-II-Using-Theories-of-Change-in-Monitoring-and-Evaluation
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Plenum.
Dey, I. (2003). Qualitative data analysis: A user friendly guide for social scientists. Routledge.
Ezzy, D. (2013). Qualitative analysis. Routledge.
Gagne, M. (2014). The Oxford handbook of work engagement, motivation, and self-determination theory. Oxford University Press
Galletta, A. (2013). Mastering the semi-structured interview and beyond: From research design to analysis and publication. NYU press.
Huett, J. B. (2006). The effects of ARCS-based confidence strategies on learner confidence and performance in distance education (Unpublished PhD dissertation). Texas, University of North Texas.
Ibrahim, A. S. (2021). Evaluating Reading Activities Of Intermediate Grade-1 (English For Iraq Textbook) For In The Light Of Higher-Order Thinking Skills. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(7), 2809-2816.
Joseph, A. J., & Joy, J. J. L. (2019). An eclectic approach for teaching English: a key to solve the textbook puzzle. Impact Journals, 7(3), 130-133.
Keller, J.M. (1983a). Motivational design of instruction. In C.M. Reigeluth (Ed.), Instructional design theories and models: An overview of their current status. Lawrence Erlbaum Associates.
Keller, J.M. (1983b). Motivational design. IDD&E Working Paper No. 15. Syracuse University School of Education, Syracuse, New York.
Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of instructional development, 10(3), 2.
Keller, J. M. (2006). ARCS design process. Retrieved September 7, 2016. http://arcsmodel.com/Mot%20dsgn%20A%20prcss.htm.
Keller, J. M. (2009). Motivational design for learning and performance: The ARCS model approach. Springer Science & Business Media.
Keller, J. M. (2010). Motivational design for learning and performance: the ARCS model approach. Springer Science & Business Media.
Koshino, M., Kojima, Y., & Kanedera, N. (2013). Development and Evaluation of Educational Materials for Embedded Systems to Increase the Learning Motivation. ERIC Clearinghouse.
López-Medina, B. (2021). On the Development of a CLIL Textbook Evaluation Checklist: A Focus Group Study. TESL-EJ, 25(1).
Malik, S., & Asghar, M. Z. (2020). In-Service Early Childhood Education Teachers’ Training Program Evaluation through Kirkpatrick Model. Journal of Research, 14(2), 259-270.
Maslow, A. (2019). A theory of human motivation. Wilder Publications, Inc.
Mathi, M. B. (2020). Cultural Aspects in English for Iraq, Book 6. Journal of Talent Development and Excellence, 12(2s), 2539-2550.
Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C. Teddie (Eds), Handbook of mixed methods in social and behavioral research (pp. 241-272). Sage Publications.
Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis. SAGE publications.
Mishan, F., & Timmis, I. (2015). Materials Development for TESOL. University Press.
Nisak, N. M., Arifin, M. B. U. B., Fahyuni, E. F., & Rahmawati, I. M. (2021). The development of comic formatted Fiqh textbook for Islamic elementary school. European Journal of Education Studies, 8(1).
O'Neil, H., & Drillings, M. (2012). Motivation: Theory and research. Routledge.
Papakonstantinou, A., & Anastasiadou, A. (2021). Evaluating and Improving the Reading Component of an A1 Level EFL Textbook.
Petri, H. L. (2013). Motivation: Theory, research, and applications. Wadsworth Pub. Co.
Priscylio, G., Rochintaniawati, D., & Anwar, S. (2018, December). Needs of integrated science textbook for junior high school based on learning style (descriptive research). In International Conference on Mathematics and Science Education of Universitas Pendidikan Indonesia (Vol. 3, pp. 396-401).
Razzaq, H. A. (2021). Sociocultural Values In 2nd Intermediate Textbook" English For Iraq". Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(7), 2862-2867.
Richards, J. C. (2013). Creativity in language teaching. Iranian Journal of Language Teaching Research, 1(3 (Special Issue)), 19-43.
Roulston, K. & Choi, M. (2017). Qualitative interviews. In U. Flick. The SAGE Handbook of Qualitative Data Collection (pp. 233-249). Sage Reference.
Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations a systematic approach: a systematic approach to enhancing learning, performance, and change. Basic Books.
Tashakkori, A., Johnson, R. B., & Teddlie, C. (2020). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage publications.
Tok, H. (2010). TEFL textbook evaluatıon: From teachers’ perspectıves. Educational Research and Reviews, 5(9), 508-517.
Wuryani, M. T. (2018). Textbooks Thematic Based Character Education on Thematic Learning Primary School: An Influence. International Journal of Educational Methodology, 4(2), 75-81.
Zohoorian, Z., MatinSadr, N., & Shamabadi, F. (2018). A Summative Evaluation of 'Prospect 1': Employing the ARCS Model. International Journal of Instruction, 11(3), 449-462.
Zohrabi, M., Sabouri, H., & Kheradmand, M. (2014). Comparative study of Interchange1 and English book1 of Iranian high schools. Education International Journal of English, 3(2), 95-104.
Volume 1, Issue 1 (JSLP 2024)
|
Contents lists available at JSLP
Journal of Second Language Pedagogy
Journal homepage: https://www.sanad.iau.ir/journal/jslp |
A Summative Evaluation of English for Iraq Series for High School:
A Focus on Motivational Design
Mohammed Kareem Ali*1
1 M.A in TESOL, College of Education, University of Shatrah, Iraq
KEY TERMS | ABSTRACT | ||||||||
Motivational Design Summative Evaluation Attention, Relevance Confidence Satisfaction
ARTICLE TYPE
Original Research Paper
© The Author 2024 | Motivation to learn, as a core concept in educational psychology, is a major determinant of academic achievement and productivity. Motivational design, as an aspect of instructional design focusing on strategies, principles, and processes that make instruction appealing, can contribute to the learners' goals and achievement. The present paper is an attempt to conduct a summative evaluation of the book series 'English for Iraq' at high school level for its motivational design. Accordingly, a mixed-method design was employed, relying on Keller's model. The participants included 403 Iraqi university students selected through purposive sampling in the quantitative phase and 20 interviewees selected through random sampling in the qualitative phase. Descriptive statistics and thematic coding were conducted for the data analysis. It was concluded that the overall motivational design of the book series has been acceptable to the students. The analysis of the constructs in detail also showed that 'satisfaction,' followed by 'relevance,' received the highest scores. Moreover, based on the views of the students in the interviews, it was observed that most of them were motivated by the content of the book, although some had external motivation such as exam scores or achievement in exercises.
|
1. Introduction
Motivation to learn has long been considered a core concept in educational psychology and has been viewed by many scholars as one of the major determinants of academic achievement and work productivity, along with ability, instruction, and feedback (Gagne, 2014; Maslow, 2019; O'Neil & Drilling, 2012; Petri, 2012). There is, however, considerable diversity of opinion among scholars and educational psychologists regarding the definition of motivation. For example, Petri (2012, p. 2) defined motivation as "those processes that can: (a) arouse and instigate behavior, (b) give direction or purpose to behavior, (c) allow behavior to persist, and (d) lead to choosing or preferring a particular behavior." Maslow (2019) offered a similar definition of motivation as "those processes that influence the arousal, strength, or direction of behavior." Gagne (2014) described motivation as the factors that arouse, direct, and sustain increased performance.
On the other hand, Babula (2013) contends that motivation entails the forces within a person that compel the individual to satisfy their basic needs. The internal variable of motivation in Babula's definition is further elaborated by motivation and cognition as both internal and external to the learner. Gagne (2014) describes the internal condition of learning as the skills and knowledge that the learner brings to the learning situation, while the external conditions for learning are the experiences that the learner is exposed to in order to bring about learning.
The rise of technology has made it easy for students to replace books with media content to acquire the English language. However, according to Richards (2013), the educational textbooks used in the process of teaching English as a second language play a considerable role. A textbook is a key component of learning and plays a focal role in students' learning (López-Medina, 2021; Papakonstantinou & Anastasiadou, 2021; Razzaq, 2021). What is more important is the evaluation of the educational materials to make them suitable for learners (Kalfut, 2024). The type of evaluation must be identical to the evaluation of everyday life needs of the language learners (Koshino et al., 2013). The comprehensive evaluation and efficacy of the book are both essential to achieve the goal of teaching. The selection of suitable materials, as well as the evaluation of materials that need to be culturally and locally appropriate, is fundamental for providing learners with their needs in any teaching/learning context (Bemani & Jahangard, 2014). Teaching materials need to have a motivational effect, and it is possible to evaluate materials in this regard. One way to evaluate materials, including textbooks, is through summative evaluation, which is used to assess the learners' learning at the end of the course by comparing it against some standard goals. This approach helps in understanding the worth of the program being evaluated (Estrada-Araoz, et al., 2023). In addition, there is formative evaluation, which is conducted while a course is being designed or prepared (Russ-Eft & Preskill, 2009). According to Zohoorian et al. (2018), the findings of such an evaluation method give insight into whether the course should be adopted or modified. Summative evaluation, also known as external evaluation, "provides an assessment of the program's value, based on the evaluation. They take place towards or at the end of a project" (Corlazzoli and White, 2013, p.80) and can help both learners and teachers achieve their goals of language learning (Zohrabi et al., 2014). Motivational design in instruction relates to the arrangement of tools and processes to create motivational improvements, including the creation of teaching materials (Zhou, et al., 2023). One well-accepted model of motivational design is Keller's ARCS (Attention, Relevance, Confidence, Satisfaction) model, which is based on an integrated theory of motivation and instructional design (Keller, 1983a).
This model draws from various areas of empirical research on human motivation, such as expectancy-value theory, equity theory, reinforcement theory, social learning theory, achievement theory, and attribution theory. These theories are synthesized to describe the relationships between motivation and cognitive and environmental factors in the learning situation. Many researchers (more than 2000 according to the Google Scholar website) have used this model in various areas, including material evaluation and development. The process of motivation in the ARCS model involves capturing and sustaining learners' attention, establishing the relevance of the instruction, building learners' confidence, and providing learners with opportunities and environments for satisfying consequences in the learning situation (Alzahrani, 2023; Na, et al., 2024; Sung & Huang, 2024). Keller (2010) describes the influence of personal and environmental factors on three categories of responses: performance, effort, and consequence. In this model, attention, also referred to as "interest" (Keller, 1983b), encompasses all the characteristics of the individual that explain how concentration is achieved and sustained. Relevance refers to learners' perceived or related needs in instructional activity (Keller, 1983b). Satisfaction is defined as the relationship between the outcomes of performance and learners' expectations (Abd Aziz, et al., 2023). The extent to which the outcomes of the learning task align with learners' motivational elements influences motivation (Keller, 1987). Confidence describes the combined influences of fear of failure and the desire to succeed on motivation (Keller, 1983b).
Regarding the English for Iraq book series, existing studies have focused on sociocultural values (Razzaq, 2021), cultural aspects (Mathi, 2020), skill activities (Ibrahim, 2021), horizontal sequence of content (Mashkour & Hammadi, 2023), or based on skills model of Litz (Nafashi, et al., 2024). However, there is a lack of research on the motivational design of the books. To address this gap, the current study explores the strengths and weaknesses of the series by employing the ARCS model.
2. Methodology
The current study utilized a mixed method of the explanatory sequential design type in the form of a survey for the quantitative phase and in the form of an interview for the qualitative phase. A mixed method study includes the collection and analysis of both quantitative and qualitative data, and the qualitative data adds depth to the data collected through the quantitative phase (Tashakkori, et al., 2020). The sequential explanatory research is characterized by first collecting quantitative data followed by the qualitative data. Thus, quantitative data is given priority, while the two methods will finally integrate at the interpretation stage (Maxwell & Loomis, 2010).
As for the quantitative phase, descriptive statistics (percentages, means, and standard deviations) were used to gain detailed information for the overall measure and its constructs. As for the qualitative data, the recorded interviews were first transcribed. Then, a thematic coding procedure through an inductive approach was conducted to find the pre-specified themes for the first four interview questions, and emergent coding was used for the open-ended general question of the interview. To analyze the three stages of open coding, axial coding, and selective coding, were conducted (Ezzy, 2013; Roulston & Choi, 2017). According to Dey (2003), pre-specified themes are already known to the researcher based on already defined constructs or a theory, while emergent themes are explored through data analysis.
2.1 Participants and Setting
The target population for the present study consisted of Iraqi students at the high-school level in both private and governmental sectors. Accordingly, the accessible population for the present study included the students of Nasiriya. The sample included 403 students, with 203 males and 200 females from different schools in Nasiriya city. The students were in the 6th grade of high school and their ages ranged from 17 to 19 years old. The sample included students from 10 different high schools, and these schools gave the researcher official approval to conduct interviews with the students. The sampling procedure for the quantitative phase of the present study involved purposive sampling, also known as judgmental sampling, where the researcher selects the units to be studied based on their own judgment. This allows the researcher to generalize the findings (Sharma, 2017). For the second phase of the study, 20 students were interviewed. The sampling procedure for the qualitative phase was random, wherein the researcher chose interviewees randomly from the entire sample from the different schools (Wyse, et al., 2016). The number of interviewees in the explanatory mixed method is determined by the researcher according to the objectives of the study (Tashakkori, et al., 2020).
2.2 Instrumentation
The questionnaire used in this research study was developed by Keller (2009). This measure comprises 36 items on a 5-point Likert scale. The model incorporates four primary areas of attention, relevance, confidence, and satisfaction (Keller, 1987). Keller (2009) has reported the survey's internal consistency as acceptable, with a Cronbach's alpha index of 0.96 for the overall survey. Additionally, the reliability and validity of this measure are highly accepted by several researchers and scholars (Huett, 2006), while Zohoorian et al. (2018) report reliability indices of over 0.7 for the overall measure and its separate constructs.
For the qualitative data for the second phase of the study, semi-structured interviews were conducted. A semi-structured interview is conducted by asking "both open-ended and more theoretically driven questions," focusing on existing constructs. In the case of the present study, the four constructs of the study, as well as students' other motivational factors (open-ended) concerning the book, were the focus (Galletta, 2013, p. 45). Thus, the following theoretically-based relevant interview questions were designed to collect students' motivation in terms of the three constructs of attention, relevance, confidence, and satisfaction.
3. Data Analysis
This section presents the analysis of both the quantitative as well as the qualitative data collected.
3.1 Quantitative Data Analysis
Table 1 presents descriptive statistics of the four sub-constructs of the instructional materials motivation survey (attention, relevance, confidence, and satisfaction), including the mean, standard deviation, maximum, and minimum scores.
Table 1
Descriptive Statistics of Sub-constructs of Instructional Materials Motivation Survey
| N | Min | Max | Mean | Std. Deviation | Mean per item |
Attention | 403 | 14.00 | 60.00 | 37.84 | 6.58 | 3.15 |
Relevance | 403 | 10.00 | 45.00 | 30.72 | 5.78 | 3.41 |
Confidence | 403 | 10.00 | 45.00 | 28.48 | 5.14 | 3.16 |
Satisfaction | 403 | 6.00 | 30.00 | 21.16 | 4.88 | 3.52 |
OVERALL | 403 | 57.00 | 180.00 | 118.22 | 18.20 | 3.28 |
As can be seen in the last column, among the four subscales of the scale, Satisfaction has the highest mean score and Attention has the lowest mean score. Figure 1 presents the mean scores of the four sub-constructs.
Figure 1 The mean scores of four sub-constructs of instructional materials motivation
It can be observed that the mean scores for all four constructs were higher than 3. The order of acceptance of the constructs by the students is as follows: Satisfaction, Relevance, Confidence, and Attention. To answer the first research question ("To what extent is the textbook 'English for Iraq' motivating based on the ARCS model?"), frequency and percentage were used for each of the sub-constructs of the questionnaire to answer the minor research questions based on the ARCS model. The first minor question was concerned with attention.
Table 2
Frequencies and Percentages of the Items on the Attention Factor
Item | not true | slightly true | moderately true | mostly true | very true | |||||||||
| F | % | F | % | F | % | F | % | F | % | ||||
Q2 | 52 | 12.9 | 68 | 16.9 | 76 | 18.9 | 104 | 25.8 | 103 | 25.6 | ||||
Q8 | 25 | 6.2 | 39 | 9.7 | 129 | 32 | 101 | 25.1 | 109 | 27 | ||||
Q11 | 51 | 12.7 | 59 | 14.6 | 111 | 27.5 | 83 | 20.6 | 99 | 24.6 | ||||
Q12 | 84 | 20.8 | 71 | 17.6 | 129 | 32 | 66 | 16.4 | 53 | 13.2 | ||||
Q15 | 89 | 22.1 | 100 | 24.8 | 89 | 22.1 | 61 | 15.1 | 64 | 15.9 | ||||
Q17 | 63 | 15.6 | 55 | 13.6 | 86 | 21.3 | 91 | 22.6 | 108 | 26.8 | ||||
Q20 | 36 | 8.9 | 57 | 14.1 | 113 | 28 | 98 | 24.3 | 99 | 24.6 | ||||
Q22 | 105 | 26.1 | 77 | 19.1 | 109 | 27 | 60 | 14.9 | 52 | 12.9 | ||||
Q24 | 38 | 9.4 | 58 | 14.4 | 103 | 25.6 | 85 | 21.1 | 119 | 29.5 | ||||
Q28 | 61 | 15.1 | 42 | 10.4 | 83 | 20.6 | 73 | 18.1 | 144 | 35.7 | ||||
Q29 | 93 | 23.1 | 63 | 15.6 | 108 | 26.8 | 67 | 16.6 | 72 | 17.9 | ||||
Q31 | 92 | 22.8 | 83 | 20.6 | 114 | 28.3 | 66 | 16.4 | 48 | 11.9 | ||||
As Table 2 indicates, the majority of the participants selected "moderately true" for the Attention Factor. Based on the results, it can be concluded that this book has been successful in grabbing the students' attention. Minor question two was concerned with relevance. Table 3 displays frequencies and percentages of the items on the relevance factor.
Table 3
Frequencies and Percentages of the Items on the Relevance Factor
Item | not true | slightly true | moderately true | mostly true | very true | |||||
| F | % | F | % | F | % | F | % | F | % |
Q6 | 43 | 10.7 | 54 | 13.4 | 116 | 28.8 | 104 | 25.8 | 86 | 21.3 |
Q9 | 36 | 8.9 | 49 | 12.2 | 111 | 27.5 | 98 | 24.3 | 109 | 27 |
Q10 | 27 | 6.7 | 43 | 10.7 | 111 | 27.5 | 86 | 21.3 | 136 | 33.7 |
Q16 | 49 | 12.2 | 42 | 10.4 | 104 | 25.8 | 75 | 18.6 | 133 | 33 |
Q18 | 32 | 7.9 | 53 | 13.2 | 121 | 30 | 107 | 26.6 | 90 | 22.3 |
Q23 | 36 | 8.9 | 47 | 11.7 | 114 | 28.3 | 94 | 23.3 | 112 | 27.8 |
Q26 | 86 | 21.3 | 84 | 20.8 | 122 | 30.3 | 57 | 14.1 | 54 | 13.4 |
Q30 | 34 | 8.4 | 51 | 12.7 | 118 | 29.3 | 96 | 23.8 | 104 | 25.8 |
Q33 | 30 | 7.4 | 43 | 10.7 | 111 | 27.5 | 91 | 22.6 | 128 | 31.8 |
As indicated in Table 3, the majority of the participants selected "moderately true" for the Relevance Factor. The above table demonstrates that students believed the topics were relevant to their needs. Therefore, it can be concluded that the book has achieved a moderate level of success in providing relevant topics and examples. Minor question three focused on confidence. Table 4 presents the frequencies and percentages of the items related to the confidence factor.
Table 4
Frequencies and Percentages of the Items on the Confidence Factor
Item | not true | slightly true | moderately true | mostly true | very true | |||||
| F | % | F | % | F | % | F | % | F | % |
Q1 | 51 | 12.7 | 64 | 15.9 | 89 | 22.1 | 70 | 17.4 | 129 | 32 |
Q3 | 118 | 29.3 | 90 | 22.3 | 74 | 18.4 | 53 | 13.2 | 68 | 16.9 |
Q4 | 36 | 8.9 | 50 | 12.4 | 93 | 23.1 | 74 | 18.4 | 150 | 37.2 |
Q7 | 98 | 24.3 | 92 | 22.8 | 101 | 25.1 | 73 | 18.1 | 39 | 9.6 |
Q13 | 27 | 6.7 | 56 | 13.9 | 106 | 26.3 | 97 | 24.1 | 117 | 29 |
Q19 | 86 | 21.3 | 100 | 24.8 | 109 | 27 | 60 | 14.9 | 48 | 11.9 |
Q25 | 33 | 8.2 | 44 | 10.9 | 111 | 27.5 | 87 | 21.6 | 128 | 33.8 |
Q34 | 109 | 27 | 93 | 23.1 | 91 | 22.6 | 59 | 14.6 | 51 | 12.7 |
Q35 | 34 | 8.4 | 47 | 11.7 | 91 | 22.6 | 78 | 19.4 | 153 | 38 |
As indicated in Table 4, the majority of participants selected "moderately true" for the Confidence Factor. It can be concluded that the book has had moderate success in the area of confidence. Minor question four related to satisfaction. Table 5 presents the frequencies and percentages of the items on the satisfaction factor.
Table 5
Frequencies and Percentages of the Items on the Satisfaction Factor
Item | not true | slightly true | moderately true | mostly true | very true | |||||
| F | % | F | % | F | % | F | % | F | % |
Q5 | 67 | 16.6 | 46 | 11.4 | 94 | 23.3 | 86 | 21.3 | 110 | 27.3 |
Q14 | 39 | 9.7 | 72 | 17.9 | 97 | 24.1 | 89 | 22.1 | 106 | 26.3 |
Q21 | 40 | 9.9 | 42 | 10.4 | 112 | 27.8 | 93 | 23.1 | 116 | 28.8 |
Q27 | 38 | 9.4 | 48 | 11.9 | 114 | 28.3 | 90 | 22.3 | 113 | 28 |
Q32 | 23 | 5.7 | 36 | 8.9 | 120 | 29.8 | 89 | 22.1 | 135 | 33.5 |
Q36 | 31 | 7.7 | 52 | 12.9 | 69 | 17.1 | 60 | 14.9 | 191 | 47.4 |
As indicated in Table 5, the majority of the participants selected "very true" for the Satisfaction Factor. It can therefore be concluded that the book has succeeded well in the area of satisfaction. Furthermore, to answer the main question of the study, it can be concluded that since all the constructs received a mean score higher than 3, the overall motivational design of the book has been accepted by the students.
3.2 Qualitative Data Analysis
In the qualitative phase, the interview questions included the following, which were based on the constructs of Keller's motivational design questionnaire used in the quantitative phase:
Q1: Was the content of the book interesting and did it motivate you to study the content?
Q2: Were the topics presented in this book relevant and useful?
Q3: Was the content of this book easy for you to learn?
Q4: Were you satisfied with the amount of effort you put in and your achievements?
Since the questions were based on the already defined constructs, the analysis involved pre-specified thematic coding. The researcher sought to identify patterns and themes based on clustering and counting of the responses (Miles, et al., 2014).
The first question addressed the students' views on their 'attention' toward the book content. Based on the frequency count, it can be observed that 18 out of 20 interviewees mentioned that the book content attracted their attention. The reasons included the inclusion of pictures and descriptions of historic sites, as well as a focus on heritage (interviewees 2, 4, 5, 6, 10), poems and literature (interviewees 1, 5, 13, 19), and learning terms used in daily life, especially related to healthcare (interviewees 3, 5, 11, 17, 18). It was also found that some interviewees found the book easy (8 & 9) and relied on teachers' assistance and instructions (12, 15, and 18). The two students who found it demotivating and uninteresting mentioned that the design was dull and repeated throughout all the books in the series (interviewee 14), and that there was no guide book for the students (interviewee 12). Table 6 presents the responses to the first interview question.
Table 6
Interviewees’ Responses for the First Interview Question
Interviewee | Response |
1 | Literature |
2 | ancient pictures |
3 | new vocabulary such as healthcare |
4 | Iraqi history and civilization |
5 | Literature, vocabulary training, history |
6 | history and heritage |
7 | interaction with an actual native speaker in real life or in social media |
8 | the design is short and easy to learn |
9 | The book is simplified and easy to navigate through |
10 | information from the old times |
11 | phrases dealing with healthcare, new everyday terminology |
12 | cannot be understood and studied without the instructions of the teacher |
13 | Literature, literature |
14 | have the same designs/colors and sizes, boring design |
15 | The audio records |
16 | learning the tenses and grammatical rules |
17 | Vocabulary |
18 | common vocabulary, vocabulary related to health policy etc. |
19 | Literature |
20 | exciting writing style and the colorful pages |
The second question regarded the students’ views on the ‘relevance’ of the book content. Based on the frequency count it can be observed that 20 out of 20 interviewees referred to the book content as being relevant to them. The reasons for this were also categorized based on common answers. The interviewees stated that the “information was about our society”, “I happen to witness many of the situations and events that are relevant to the content of my English book” (interviewees 1, 2, 3, 4, 6, 16, 17, 18, 19), “…. can be used in our daily life, like using communicative materials to talk with English users….” (interviewees 1,3,7,9,18,19), “the book is useful to prepare us for a similar situation” (7,11, 14, 20), etc. The following table (Table 7) presents the responses to the second interview question.
Table 7
Interviewees’ Responses for the Second Interview Question
Interviewee | Response |
1 | Relevant to real situations from our daily lives, about our society |
2 | Many of the situations and events that are relevant to the content |
3 | Many topics were relevant and useful to daily life |
4 | many of the topics are relevant |
5 | some situations in which used book content |
6 | many topics in the book are relevant to our life |
7 | useful to prepare for a similar situation in daily life. |
8 | know how, when, and where to use the content in real-life situations. |
9 | useful vocabulary in daily life. |
10 | put them to use regular life for example applying for a job. |
11 | helpful content in preparing for similar situations |
12 | Content used in many circumstances and events |
13 | topics commonly known |
14 | Know how, when, and where to use the content in real-life situations. |
15 | Topics related to real life. |
16 | topics relevant and based on reality |
17 | content is useful and relevant |
18 | most of these topics relevant to my daily life |
19 | valuable and relevant to our daily needs |
20 | Use the content in right place |
The third question regarded the students’ views on their ‘confidence’. Based on the frequency count it can be observed that most of the students, 19 out of 20, believed that they had confidence in studying the book content. However, among these, 7 students (interviewees 2, 4, 7, 10, 11, 13, 18) relied on the teachers’ assistance and guidance. They mentioned that “with the assist of our competent teachers that direct us and simplify the problems that we face which make the content easy to learn and practice.”, … by talking about them with my colleges and by asking my teacher…”, “we work on it with our teacher on daily basis which makes the content of the book easy”, etc. other interviewees referred to the book as needing practice and focus (5, 7, 9, 12, 13, 14), and easy ( interviewees 8, 15, 17, 18 , 19). The following table (Table 8) presents the responses to the third interview question.
Table 8
Interviewees’ Responses for the Third Interview Question
Interviewee | Response |
1 | I can use the information |
2 | especially with the assist of our competent teachers |
3 | the style of the writing and the colorful pages |
4 | the topics were clarified talking about them with my colleges and asking my teacher |
5 | just with little focus in the class and after |
6 | Yes, when I manage to do my exercise by myself after the class and achieve learning goal |
7 | every day practice on daily basis |
8 | the content is easy and useful for daily use |
9 | some training needed |
10 | with the support of our professional teachers |
11 | tough without the teacher's guidance, hard to comprehend |
12 | with a little emphasis in the class |
13 | Work with our teacher on a regular basis |
14 | step by step practice, no difficulties in understanding and using |
15 | it was easy |
16 | I have a background about lots of the topics, not much difficulty |
17 | useful and easy to use |
18 | Some topics easy and even the hard ones were simplified by the teacher. |
19 | the majority were easy to understand and to use |
20 | never felt tired or bored |
The fourth question regarded the students’ views on their ‘satisfaction. Based on the frequency count it can be observed that only two students (interviewee 6 & 10) were not satisfied. Interviewee 6 stated that there had not been much effort so there was no satisfaction; and interviewee 10 mentioned the lack of spelling practice in the design of the book. Overall, most of the students (18) stated their satisfaction. However, 10 interviewees related their satisfaction to their achievement in exams and in doing exercises or getting good scores (interviewees 1, 2, 5, 8, 9, 11, 13, 14, 16, 17, 20). Others mentioned answering spontaneous and random questions in communication (interviewees 13, 17, 18), and understanding content (3, 8). The following table (Table 9) presents the responses to the fourth interview question.
Table 9
Interviewees’ Responses for the Fourth Interview Question
Interviewee | Response |
1 | hard effort given me so much in return. achieve a good score or answer the teacher correctly |
2 | solve my homework or I use any information |
3 | Have had Effort to understand the content |
4 | effort spent |
5 | If I get a full score |
6 | I haven’t even spent that much effort to gain |
7 | I feel satisfied with the effort |
8 | I can write read and answer correctly and efficiently |
9 | if I answer exam questions correctly, I feel I have reached goals |
10 | spent a lot of time for mastering spelling, the book didn’t have practice |
11 | feel a sense of accomplishment, understand the knowledge and respond correctly for exam and exercises |
12 | Much effort, pleased with current level and effort |
13 | sense of accomplishment, can answer or communicate random questions. |
14 | sense of satisfaction, address the book exercises |
15 | feeling the achievement |
16 | feeling of achievement answering exercises of the book |
17 | The sense of achievement happens when I can answer or communicate spontaneous questions |
18 | Achievement if I can communicate and answer |
19 | feel achievement when use information in real-life and network |
20 | sense of achievement in exams |
Based on the views of the students in interviews, it can be observed that most of them were motivated toward the content of the book although some had external motivation such as scores for exams or achievement in exercises.
4 Discussion and Conclusion
Considering the fact that textbook evaluation is a vital stage in successful education, especially when the textbook is employed nationwide, is undeniable. It is believed that "There can be no doubt that evaluating the official textbooks for learning English in Iraq is a task of great importance to the future success of English learning programs" (Al-Akraa, 2013, p.2). Based on the statistical analysis of the quantitative data, it was found out that the construct which received the highest score was satisfaction, followed by relevance. Also, attention, followed by confidence, received the lowest means.
As for the attention factor, the majority of the students believed that the book is attractive and the variety of the passages, exercises, and illustrations could keep them attentive. However, they believed that the lessons were abstract, which is confirmed through the data analysis as well.
As far as the relevance factor is concerned, more than 50% of the students believed that completion of the lesson has been important to them and through the explanations and examples, they could know how the ideas would be used by people. However, around 30% of the students believed that they already knew the materials. The reason for such a perception can be traced back to the fact that the Iraqi students study English from early classes in schools (Al-Akraa, 2013; Amin, 2017), and their English level might be higher than what is presented to them.
Regarding the confidence construct, it was found out that the majority of the students had confidence in what they are supposed to learn from the book. However, the importance of the tests was also confirmed.
As for the last construct, satisfaction, the majority of the students had the perception that the book is well-designed and they were pleased with it. This comprised about half of the students who had chosen 'very true' for item 36 (it was a pleasure to work on such a well-designed lesson).
In line with the findings of the quantitative phase, based on the findings of the qualitative phase of the study, including the interviews, several students have referred to the literature or historical themes related to Iraq, which have attracted their attention. The fact that such pictures, themes, or topics have been interesting and attracted students' attention can be related to the idea of localization. According to Mishan and Timmis (2015), there is a growing interest among developers to consider the geographical context. They refer to the localization of the materials as solving the problem of cultural familiarity, which can hinder learning and the attention of the learners. Another important theme found in the students' interviews was the daily life use of the language and its vocabulary, including healthcare. Tok (2010) highlights that the use of such materials can not only increase students' motivation but can also bring "greater realism and relevance to the ESL/EFL classroom" (p. 515). In the same vein, Ayu and Indrawati (2019) state that the use of realistic situations is in line with a student-centered approach advocated in the current era of education. Students like to study materials that are related to real life and include pictures (Derakhshan, 2024; Priscylio, et al., Anwar, 2018). Materials and situations related to daily life and the use of such real concepts can help students apply them in their real life as well (Ayu, 2020). In fact, it is believed that associating materials with daily life will make learning meaningful (Wuryani, 2018). Some students also highlighted the use of materials related to their societies and cultures. Joseph and Joy (2019) confirm that while making materials, "the content and the language elements should not alienate the learners from their life and society; it should resemble their daily life" (p. 133). Thus, materials that are learned should be applicable in the daily lives of the learners (Nisak, et al., 2021).
There are some units such as Unit 1 (the health service), Unit 5 (holidays), Unit 6 (banking and finance), and Unit 8 (the environment and recycling) that are related to the students' daily and routine lives. Topics such as "Let's Start with Diet" on page 12, "Airport Security" on page 18, "If You Could Choose Any Job" on page 28, "What's Your Personality Type" on page 39, "Why Are Holidays So Important" on page 51, "Making Money" on page 61, "Improve Your Computer Skills" on page 66, "Using the Library" on page 71, and "Recycling Waste" on page 75 are observed throughout the book, confirming the relevance aspect to what the students have stated.
Some other students referred to the relevance as they saw familiar site pictures. Such instances can be observed on page 78, "the Euphrates River." As for the attention aspect, where the students stated that the literature included in the book attracted their attention, an instance can be found on pages 85 and 86 where "Mohammed Khudhair," a writer, is introduced. Ten interviewees, on the other hand, attributed their satisfaction to their progress in tests and exercises, as well as high grades. Based on this, it can be concluded that their motivation has been partly due to external factors.
As for the 'confidence' element, it seems that a large group of the students relied on the teacher's support and guidance, as well as practice and focus. It is claimed that materials need to follow Krashen's input hypothesis (i+1), which states that it must be one level above the comprehensibility level (Mishan & Timmis, 2015). Based on the findings, it seems that many students seek to gain support from the teacher and are too reliant on teacher assistance. Similarly, there is too much emphasis on the part of the learners on gaining acceptable grades or passing tests. Thus, it seems that there must be modifications in the materials to enhance success opportunities strategies as well as learning requirements strategies to improve confidence.
As for the last element, satisfaction, Keller (2010) maintains that in order to promote satisfaction, strategies such as using personally meaningful and relevant topics or achieving a desirable level of success can be used. However, too much focus on scores and obtaining perfect grades is believed to be positive intrinsic satisfaction by Keller, which will be different from intrinsic satisfaction. Based on the findings of the study, it can be concluded that the students mainly had extrinsic satisfaction rather than intrinsic. However, it seems that intrinsic satisfaction is not enhanced through the book materials, which can be considered as a point of weakness.
While the confidence and attention factors were ranked lowly, it seems inevitable that some amendments to the book in future revised versions can increase the motivational design level of the book. As for the attention factor, raising the curiosity, concentration, and interest of the learners can solve this problem (Keller, 2006). Also, for the confidence factor, which can be described as the combined influences of fear of failure and the desire to succeed (Keller, 1983), the future versions must provide "positive expectations for achieving success" (Malik, 2014) by helping the learners to have better learning experiences. While Keller's model focuses on three aspects of expectancy for success, perceived competence, and personal control, it seems that the personal control factor is missing in the current books. Most of the students related their success to achieving high scores or having their teachers' support. Thus, it is recommended that policymakers modify the book in future versions so that there would be more intrinsic motivation (Deci & Ryan, 1985) to let the students feel more control over their learning. Positive aspects of the textbooks, such as the familiar topics (health, banking, security), and related topics to daily lives, should be preserved in the revision of the textbook.
References
Abd Aziz, N. N., Aziz, M. A., & Abd Rahman, N. A. S. (2023). The mediating effects of student satisfaction on technostress–performance expectancy relationship in university students. Journal of applied research in higher education, 15(1), 113-129. https://doi.org/10.1108/JARHE-03-2021-0117
Al-Akraa, S. (2013). Teaching English in Iraq: An analysis of an EFL textbook (Published Master’s thesis). University of Central Florida.
Alzahrani, R. (2023). Investigating the effect of motivational design on saudi university students’ motivation and l2 writing performance: an experimental mixed methods design using keller’s arcs model (Doctoral dissertation). https://hdl.handle.net/10125/107904
Amin, M. Y. M. (2017). English language teaching methods and reforms in English curriculum in Iraq; an overview. Journal of University of Human Development, 3(3), 578-583. DOI:
https://doi.org/10.21928/juhd.v3n3y2017.pp578-583
Ayu, M., & Indrawati, R. (2019). EFL textbook evaluation: The analysis of tasks presented in English textbook. Teknosastik, 16(1), 21-25.
Ayu, L. (2020). Content Feasibility in English Textbook for Senior High School Grade XI (Doctoral dissertation). University of Muhammadiyah Sumatera Utara Medan.
Babula, M. (2013). Motivation, altruism, personality and social psychology. Palgrave Macmillan.
Bemani, M., & Jahangard, A., (2014). An Evaluative Analysis of Iranian Junior High School EFL Textbooks. Saabrucken.
Corlazzoli, V., & White, J. (2013). Practical Approaches to Theories of Change in Conflict, Security and Justice Programmes: Part II: Using Theories of Change in Monitoring and Evaluation (Search for Common Ground). Retrieved from https://issat.dcaf.ch/ara/Learn/Resource-Library/Policy-and-Research-Papers/Practical-Approaches-to-Theories-of-Change-in-Conflict-Security-and-Justice-Programmes-Part-II-Using-Theories-of-Change-in-Monitoring-and-Evaluation
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Plenum. DOI: 10.1007/978-1-4899-2271-7
Derakhshan, A. (2024). ‘Should textbook images be merely decorative?’: Cultural representations in the Iranian EFL national textbook from the semiotic approach perspective. Language Teaching Research, 28(1), 79-113. https://doi.org/10.1177/1362168821992264
Dey, I. (2003). Qualitative data analysis: A user friendly guide for social scientists. Routledge. https://doi.org/10.4324/9780203412497
Estrada-Araoz, E. G., Sayed, B. T., Niyazova, G. G., & Lami, D. (2023). Comparing the effects of computerized formative assessment vs. computerized dynamic assessment on developing EFL learners’ reading motivation, reading self-concept, autonomy, and self-regulation. Language Testing in Asia, 13(1), 39. https://doi.org/10.1186/s40468‐023‐00253‐1
Ezzy, D. (2013). Qualitative analysis. Routledge. https://doi.org/10.4324/9781315015484
Gagne, M. (2014). The Oxford handbook of work engagement, motivation, and self-determination theory. Oxford University Press
Galletta, A. (2013). Mastering the semi-structured interview and beyond: From research design to analysis and publication. NYU press.
Huett, J. B. (2006). The effects of ARCS-based confidence strategies on learner confidence and performance in distance education (Unpublished PhD dissertation). Texas, University of North Texas.
Ibrahim, A. S. (2021). Evaluating Reading Activities Of Intermediate Grade-1 (English For Iraq Textbook) For In The Light Of Higher-Order Thinking Skills. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(7), 2809-2816.
Joseph, A. J., & Joy, J. J. L. (2019). An eclectic approach for teaching English: a key to solve the textbook puzzle. Impact Journals, 7(3), 130-133.
Kalfut, T. (2024). Approaches and Limitations of L2 Textbook Evaluation Research. International Journal of Language and Literary Studies, 6(1), 216-225. https://doi.org/10.36892/ijlls.v6i1.1617
Keller, J.M. (1983a). Motivational design of instruction. In C.M. Reigeluth (Ed.), Instructional design theories and models: An overview of their current status. Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203824283
Keller, J.M. (1983b). Motivational design. IDD&E Working Paper No. 15. Syracuse University School of Education, Syracuse, New York.
Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of instructional development, 10(3), 2.
Keller, J. M. (2006). ARCS design process. Retrieved September 7, 2016. http://arcsmodel.com/Mot%20dsgn%20A%20prcss.htm.
Keller, J. M. (2009). Motivational design for learning and performance: The ARCS model approach. Springer Science & Business Media. DOI: 10.1007/978-1-4419-1250-3
Keller, J. M. (2010). Motivational design for learning and performance: the ARCS model approach. Springer Science & Business Media. DOI: 10.1007/978-1-4419-1250-3
Koshino, M., Kojima, Y., & Kanedera, N. (2013). Development and Evaluation of Educational Materials for Embedded Systems to Increase the Learning Motivation. ERIC Clearinghouse.
López-Medina, B. (2021). On the Development of a CLIL Textbook Evaluation Checklist: A Focus Group Study. TESL-EJ, 25(1).
Malik, S., & Asghar, M. Z. (2020). In-Service Early Childhood Education Teachers’ Training Program Evaluation through Kirkpatrick Model. Journal of Research, 14(2), 259-270.
Mashkour, A. M., & Hammadi, M. D. (2023). Time Distribution in English for Iraq of Intermediate Schools: An Analysis of the Syllabus. Larq Journal for Philosophy, Linguistics & Social Sciences, 3(51). DOI: https://doi.org/10.31185/
Maslow, A. (2019). A theory of human motivation. Wilder Publications, Inc.
Mathi, M. B. (2020). Cultural Aspects in English for Iraq, Book 6. Journal of Talent Development and Excellence, 12(2s), 2539-2550.
Maxwell, J. A., & Loomis, D. M. (2003). Mixed methods design: An alternative approach. In A. Tashakkori & C. Teddie (Eds), Handbook of mixed methods in social and behavioral research (pp. 241-272). Sage Publications.
Miles, M. B., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis. SAGE publications.
Mishan, F., & Timmis, I. (2015). Materials Development for TESOL. University Press.
Na, K., Jeong, Y., & Lee, J. (2024). Exploring older adults' motivation to use public libraries in South Korea using the attention, relevance, confidence, and satisfaction (ARCS) motivation model. Library & Information Science Research, 46(2), 101296. https://doi.org/10.1016/j.lisr.2024.101296
Nafashi, N., Dabaghi, A., & Ketabi, S. (2024). Investigating Teachers' and Students’ Evaluation of the Textbook English for Iraq for First and Second Grades of High School Based on Litz's Checklist. Al-Adab Journal, (150), 13-44.
Nisak, N. M., Arifin, M. B. U. B., Fahyuni, E. F., & Rahmawati, I. M. (2021). The development of comic formatted Fiqh textbook for Islamic elementary school. European Journal of Education Studies, 8(1). DOI: 10.46827/ejes.v8i1.3513
O'Neil, H., & Drillings, M. (2012). Motivation: Theory and research. Routledge.
Papakonstantinou, A., & Anastasiadou, A. (2021). Evaluating and Improving the Reading Component of an A1 Level EFL Textbook.
Petri, H. L. (2013). Motivation: Theory, research, and applications. Wadsworth Pub. Co.
Priscylio, G., Rochintaniawati, D., & Anwar, S. (2018, December). Needs of integrated science textbook for junior high school based on learning style (descriptive research). In International Conference on Mathematics and Science Education of Universitas Pendidikan Indonesia (Vol. 3, pp. 396-401).
Razzaq, H. A. (2021). Sociocultural Values In 2nd Intermediate Textbook" English For Iraq". Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(7), 2862-2867.
Richards, J. C. (2013). Creativity in language teaching. Iranian Journal of Language Teaching Research, 1(3 (Special Issue)), 19-43.
Roulston, K. & Choi, M. (2017). Qualitative interviews. In U. Flick. The SAGE Handbook of Qualitative Data Collection (pp. 233-249). Sage Reference.
Sung, J. S., & Huang, W. H. D. (2024). Motivational Design for Inclusive Digital Learning: Women College Engineering Students' Motivation for Online STEM Learning. Contemporary Educational Technology, 16(1). https://doi.org/10.30935/cedtech/14047
Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations a systematic approach: a systematic approach to enhancing learning, performance, and change. Basic Books.
Tashakkori, A., Johnson, R. B., & Teddlie, C. (2020). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Sage publications.
Tok, H. (2010). TEFL textbook evaluatıon: From teachers’ perspectıves. Educational Research and Reviews, 5(9), 508-517.
Wuryani, M. T. (2018). Textbooks Thematic Based Character Education on Thematic Learning Primary School: An Influence. International Journal of Educational Methodology, 4(2), 75-81. doi: 10.12973/ijem.4.2.75
Zhou, X., Chen, S., Ohno, S., She, J., & Kameda, H. (2023). Motivational design for enhancing behavioral engagement in a flipped Chinese course. Asia Pacific Education Review, 1-15. https://doi.org/10.1007/s12564-023-09849-x
Zohoorian, Z., MatinSadr, N., & Shamabadi, F. (2018). A Summative Evaluation of 'Prospect 1': Employing the ARCS Model. International Journal of Instruction, 11(3), 449-462. https://doi.org/10.12973/iji.2018.11331a
Zohrabi, M., Sabouri, H., & Kheradmand, M. (2014). Comparative study of Interchange1 and English book1 of Iranian high schools. Education International Journal of English, 3(2), 95-104.