Examining Indicators of Validity in Online Formative Assessment: Insights from Iranian EFL Teachers
الموضوعات :علیرضا ملکی 1 , حسین خدابخش زاده 2 , میترا زراعت پیشه 3 , پوریا بقایی 4
1 - دانشگاه آزاد اسلامی واحد تربت حیدریه. ایران
2 - دانشگاه آزاد اسلامی واحد تربت حیدریه
3 - دانشگاه آزاد اسلامی مشهد. ایران
4 - دانشگاه آزاد اسلامی واحد مشهد. ایران
الکلمات المفتاحية: Validity, Online formative assessment, Indicators, Place of living, Observation,
ملخص المقالة :
ارزشیابی‌های تکوینی آنلاین معتبر برای اندازه گیری دقیق پیشرفت دانش‌آموزان و تصمیم‌گیری آموزشی موثر در یادگیری دیجیتال بسیار حائز اهمیت است. این مطالعه بر پایه رویکرد کمی دو هدف اصلی را دنبال کرد. ابتدا، هدف این مطالعه بررسی میزان استفاده معلمان زبان انگلیسی ایرانی شاغل در دانشگاه ها و موسسات زبان از نشانگرهای روایی ارزشیابی تکوینی آنلاین بود. بیست و یک کلاس آنلاین در سه جلسه با استفاده از یک مشاهده شدند. هدف دوم این مطالعه تعیین تأثیر محل اقامت معلمان زبان انگلیسی به عنوان زبان خارجه بر روایی ارزشیابی تکوینی آنلاین بود. به این منظور،از ۳۱۶ معلم زبان انگلیسی ایرانی از محیط‌های متنوع زبان انگلیسی، از جمله مدارس دولتی، مدارس خصوصی، موسسات زبان و دانشگاه‌ها، درخواست شد تا مقیاس اعتبار ارزشیابی تکوینی آنلاین توسعه یافته توسط ملکی و همکاران (۲۰۲۳) را تکمیل کنند. نمونه شامل معلمان زن و مرد با گروه‌های سنی و مدارک تحصیلی متفاوت بود. یافته‌های این مطالعه نشان داد که معلمان زبان انگلیسی ایرانی در دانشگاه ها و موسسات زبان نشانگرهای متمرکز بر دانش‌آموز در روایی ارزشیابی تکوینی آنلاین را نادیده می‌گیرند. علاوه بر این، آشکار شد که محل اقامت معلمان زبان انگلیسی می‌تواند بر روایی ارزشیابی تکوینی آنلاین تأثیرگذار باشد. این مطالعه تلویحات متعددی برای معلمان زبان انگلیسی و سیاست گذاران دارد. این یافته‌های بر ماهیت مبتنی بر زمینه در روایی ارزشیابی تکوینی آنلاین تأکید می‌کند. همچنین، به معلمان زبان انگلیسی ایرانی کمک می‌کند تا حوزه‌های خاصی که نیاز به توجه و بهبود بیشتر برای افزایش روایی ارزشیابی‌های تکوینی آنلاین آن‌ها دارند را شناسایی کنند.
Arefian, M. H. (2022). Perceptions of self-assessment literacy and self-directed reflection during online learning for Iranian EFL student teachers. Reflective Practice, 23(6), 623–634. https://doi.org/10.1080/14623943.2022.2096584
Baleni, Z. G. (2015). Online formative assessment in higher education: Its pros and cons. Electronic Journal of e-Learning, 13(4), 228-236.
Dörnyei, Z. (2007). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford University Press.
Elmahdi, I., Al-Hattami, A., & Fawzi, H. (2018). Using Technology for Formative Assessment to Improve Students' Learning. Turkish Online Journal of Educational Technology-TOJET, 17(2), 182-188.
Esfandiari, R., & Arefian, M. H. (2023). Developing collective eyes for Iranian EFL teachers’ computer-assisted language assessment literacy through internet-based collaborative reflection. Education and Information Technologies, 1-22. https://doi.org/10.1007/s10639-023-12201-w.
Ghanbari, N., & Nowroozi, S. (2021). The practice of online assessment in an EFL context amidst COVID-19 pandemic: views from teachers. Language Testing in Asia, 11(1), 1-18. https://doi.org/10.1186/s40468-021-00143-4.
Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers & Education, 57(4), 2333-2351. https://doi.org/10.1016/j.compedu.2011.06.004.
Kilis, S., & Yildirim, Z. (2019). Posting patterns of students’ social presence, cognitive presence, and teaching presence in online learning. Online Learning, 23(2), 179-195. https://doi.org/10.24059/olj.v23i2.1460.
Larson-Hall, J. (2015). A guide to doing statistics in second language research using SPSS. Routledge.
Liu, Y., & Zhang, H. (2022). Exploring the Influencing Factors and Validity of Formative Assessment in Online Learning. Journal of Education and e-Learning Research, 9(4), 278-287. https://doi.org/10.20448/jeelr.v9i4.4288.
Maleki, A., Khodabakhshzadeh, H., Zeraatpishe, M., & Baghaei, P. (2023). Online formative assessment validity: Theoretical perspectives, EFL teachers' beliefs, and practical implications. [Unpublished doctoral dissertation]. Islamic Azad University of Torbate Heydarieh. Iran.
Martin, F., Ritzhaupt, A., Kumar, S., & Budhrani, K. (2019). Award-winning faculty online teaching practices: Course design, assessment and evaluation, and facilitation. The Internet and Higher Education, 42, 34–43. https://doi.org/10.1016/j.iheduc.2019.04.001.
McLaughlin, T., & Yan, Z. (2017). Diverse delivery methods and strong psychological benefits: A review of online formative assessment. Journal of Computer Assisted Learning, 33(6), 562-574. https://doi.org/10.1111/jcal.12200.
Messick, S. (1989). Validity. In R. Linn (Ed.), Educational measurement (pp. 13–103). Macmillan.
Nayernia, A., Mohebbi, H. (2023). Teachers’ Online Language Assessment Literacy. In: Tajeddin, Z., Griffiths, C. (eds) Language Education Programs. Language Policy, vol 34. Springer, Cham. https://doi.org/10.1007/978-3-031-38754-8_8
Ogange, B. O., Agak, J. O., Okelo, K. O., & Kiprotich, P. (2018). Student perceptions of the effectiveness of formative assessment in an online learning environment. Open Praxis, 10(1), 29-39. https://search.informit.org/doi/10.3316/informit.423669258504414.
Pallant, J. (2016). SPSS survival manual: A step by step guide to data analysis using SPSS for windows. Open University Press.
Ryan, T., Henderson, M., Ryan, K., & Kennedy, G. (2021). Designing learner-centred text-based feedback: a rapid review and qualitative synthesis. Assessment & Evaluation in Higher Education, 46(6), 894-912. https://doi.org/10.1080/02602938.2020.1828819.
Shaw, S., & Crisp, V. (2011). Tracing the evolution of validity in educational measurement: Past issues and contemporary challenges Research Matters: A Cambridge Assessment publication, 11, 14-17.
Sudakova, N. E., Savina, T. N., Masalimova, A. R., Mikhaylovsky, M. N., Karandeeva, L. G., & Zhdanov, S. P. (2022). Online Formative Assessment in Higher Education: Bibliometric Analysis. Education Sciences, 12(3), 209. https://doi.org/10.3390/educsci12030209.
Waheeda, A., Shaheeda, F., Muna, F. & Shina, A. (2023). Online assessments in Maldivian higher education institutions: Challenges faced by students. Issues in Educational Research, 33(3), 1233-1249. http://www.iier.org.au/iier33/waheeda.pdf
Weleschuk, A., Dyjur, P., & Kelly, P. (2019). Online Assessment in Higher Education. Taylor Institute for Teaching and Learning Guide Series. Calgary, AB: Taylor Institute for Teaching and Learning at the University of Calgary. Retrieved from https://taylorinstitute.ucalgary.ca/resources/guides
The Journal of English Language Pedagogy and Practice
Vol. 16, No.33, Autumn and Winter 2024
Research Article
Examining Indicators of Validity in Online Formative Assessment: Insights from Iranian EFL Teachers
Alireza Maleki1, Hossein Khodabakhshzadeh*2, Mitra Zeraatpishe3, Purya Baghaei4
1,2Department of English, Torbat-e Heydarieh Branch, Islamic Azad University, Torbat-e Heydarieh, Iran
3,4Department of English, Mashhad Branch, Islamic Azad University, Mashhad, Iran
*Corresponding author: kh.phdtbt2015@gmail.com
(Received: 2024/01/22; Accepted: 2024/10/06)
Online publication: 2024/11/17
Abstract
Valid online formative assessments are crucial for accurate measurement of students' progress and effective pedagogical decision-making in digital learning. This quantitative-based study followed two primary aims. First, it aimed to investigate the extent to which Iranian EFL teachers working in universities and language institutes apply indicators of online formative assessment validity. Twenty-one online classrooms were observed in three sessions using a checklist. The second aim of this study was to determine the effect of EFL teachers’ place of living on the validity of online formative assessment. To this end, 316 Iranian EFL teachers from diverse EFL settings, including public schools, private schools, language institutes, and universities were asked to fill out online formative assessment validity scale developed by Maleki et al. (2023). The sample included both male and female teachers with varying age group ranges and academic degrees. The findings of the study indicated that Iranian EFL teachers in universities and language institutes tend to overlook indicators associated with the learner-centered aspects of online formative assessment validity. Furthermore, it was revealed that EFL teachers’ place of living could impact the validity of online formative assessment. This study has several implications for online EFL teachers and policymakers. The findings of this study emphasize the context-bound nature of validity in online formative assessment. Besides, it helps Iranian EFL teachers identify specific areas that need more attention and improvement in order to enhance the validity of their online formative assessments.
Keywords: Validity; Online formative assessment; Indicators; Place of living; Observation.
Introduction
Online assessment refers to any method used to assess and gauge students' accomplishments, provide feedback, and support their educational advancement in fully online credit courses (Weleschuk et al., 2019). Embedded assessment relies on formative assessment as a vital component, which entails ongoing evaluation of the progress of learning by actively participating in learning activities within a supportive social environment. This form of assessment is generally defined as a process that encourages opportunities for instruction and provides support throughout the learning process (Kilis & Yildirim, 2019; Martin et al., 2019). With the progress and widespread availability of technology, online assessments, including both summative and formative approaches, are quickly replacing traditional paper-based assessments. The educational landscape has been transformed by the advent of modern digital tools, which have introduced novel opportunities and approaches to teaching, learning, and assessment (Waheeda et al., 2023). The integration of information and communication technology in classrooms is essential for facilitating effective formative assessment of learning in the 21st century (Elmahdi et al., 2018).
Over the past few years, there has been a transition in higher education learning and teaching environments, moving the focus from Assessment for Learning (AfL) to online formative assessment (Baleni, 2015). The integration of research on formative assessment and computer-assisted assessments has given rise to the development of online formative assessment (OFA). In the 21st century, incorporating technology in classrooms has become crucial for facilitating effective teaching and learning practices. Technology serves as a valuable tool in supporting teaching and learning by facilitating the process of formative assessment, allowing for ongoing evaluation of learners' skills and knowledge during the instructional process (Elmahdi et al., 2018).
Drawing from the perspectives of Messick (1989) and Shaw and Crisp (2011), Gikandi et al. (2011) provided a definition of validity in relation to online formative assessment. According to their definition, validity in this context refers to the degree to which the assessment activities and processes effectively contribute to and support ongoing learning. Online formative assessment validity is associated with four attributes: the authenticity of assessment activities, the provision of effective formative feedback, the incorporation of multidimensional perspectives, and the availability of learner support (Maleki et al., 2023; Gikandi et al., 2011).
While previous studies have explored the validity of online formative assessment, most have been qualitative in nature, focusing on subjective perspectives rather than providing empirical, generalizable insights (Liu & Zhang, 2022; Ghanbari & Nowroozi, 2021). This leaves a significant gap in the literature, particularly regarding the quantitative measurement of how validity indicators are applied in practice. Additionally, there is limited research on the role of contextual factors, such as the teachers' place of living, in shaping the validity of online assessments—a gap highlighted by the context-dependent nature of validity as proposed by Messick (1989). This study addresses these gaps by quantitatively examining the extent to which Iranian EFL teachers apply online formative assessment validity indicators and investigating whether their geographical location impacts these practices. By adopting a quantitative approach, this research provides empirical evidence that complements existing qualitative findings and offers a more comprehensive understanding of the factors influencing the validity of online formative assessments in diverse educational contexts. This, in turn, can inform targeted interventions to enhance online assessment practices in different regions.
Validity, in the context of online formative assessment, has been debated among scholars over several years. Many scholars have discussed and argued about the issue of online formative assessment validity from different perspectives. One of the famous studies in the realm of OFA validity is a review study conducted by Gikandi et al. (2011). A bibliometric analysis study by Sudakova et al. (2022) revealed that the study by Gikandi et al. (2011) ranked first with 426 citations according to the total citations in the field of online formative assessment. According to Gikandi et al. (2011), the validity of online formative assessment is associated with the following attributes: (1) authenticity of assessment activities, (2) effective formative feedback, (3) multidimensional perspectives, and (4) learner support.
Authenticity of Online Assessment Activities
Numerous scholars have outlined key characteristics that define authentic online formative assessment (OFA) tasks. These tasks should promote deep knowledge construction and higher-order thinking (Kaya-Capocci et al., 2022; Wu et al., 2024), stimulate active problem-solving and logical reasoning, and engage students in meaningful discourse (Rajabalee & Santally, 2021). Additionally, they should involve students directly in the assessment process (Irons & Elkington, 2021), foster collaboration (Marden & Herrington, 2020), and offer flexibility (Kristiyanti et al., 2021). Authentic OFA tasks must also document and monitor students' achievements and progress (Sudakova et al., 2022; Veerasamy et al., 2021), engage learners in decision-making and problem-solving (Hazaymeh, 2021), and provide clear, analytical rubrics to help students understand the expected outcomes (Yousef & Sumner, 2021). Moreover, it is essential to inform students about assessment rubrics and offer multiple opportunities for reflection (Boss & Krauss, 2022).
Effective Online Formative Feedback
Panadero and Lipnevich (2022) characterized feedback as information provided by an agent about various aspects and levels of an individual's performance or understanding. They emphasized its central role in formative assessment (FA). In the context of online formative assessment (OFA), feedback must be both informative and supportive, offering detailed insights into students' performance and being delivered in a timely manner (Zhan et al., 2022). Additionally, peer feedback—both giving and receiving—is crucial in an OFA environment, contributing to the formative assessment process (Carless, 2022).
Moreover, online formative feedback should aim to enhance learners' self-regulation (Say et al., 2023). To support this, online educators need to provide personalized feedback through technological tools (Beardsley et al., 2021). Personalized feedback should identify both strengths and weaknesses in students' work (Mahapatra, 2021). Additionally, teachers should help students develop their online feedback literacy to effectively utilize and benefit from the feedback provided (Wongvorachan et al., 2022).
OFA includes dialogic feedback (Espasa et al., 2022). Since students and teachers are not physically together in an online setting to discuss feedback directly, it is essential to make feedback as interactive and dialogic as possible. Furthermore, online dynamic assessment is often employed by educators (Wang et al., 2021). This type of assessment can be conducted either by individuals or automated systems. Occasionally, students might require assistance to guide and scaffold their learning, helping them improve their performance in line with their capabilities.
Multidimensional Perspectives Towards OFA
Gikandi et al. (2011) emphasized the importance of incorporating multidimensional approaches into OFA. This involves integrating a variety of activities to enhance learner autonomy and flexibility (Kantaridou & Hatzopoulou, 2021). Another critical component is the knowledge and ongoing professional development of online teachers (Bragg et al., 2021). Online educators should engage in diverse aspects of online assessment and continuously refine their skills to effectively evaluate students. To achieve this, they should participate in professional development programs, stay current with advancements, and utilize technological assessment tools (Kristiyanti et al., 2021).
Online Learner Support
An effective learner support model must align with an institution’s values, objectives, strategic priorities, and teaching and learning philosophy. Providing robust student support is critical for overcoming challenges in online higher education, as it significantly enhances learner engagement, motivation, and success (Rotar, 2022). Online educators should serve as guides, helping students find tailored learning opportunities that fit their needs (Alamri et al., 2020). They should also support students in defining and pursuing their learning goals (Singh et al., 2021), assist them in developing clear learning plans (Lase & Zega, 2021), and ensure the creation of consistent course materials (Archambault et al., 2022).
In their paper, McCallum and Milner (2020) examined the effectiveness and implementation of formative e-assessments in first-year courses, focusing on both student perspectives and staff reflections. Their study aimed to assess how these assessments impacted student engagement and learning outcomes. According to questionnaires completed by students, formative e-assessments were perceived as beneficial for monitoring academic progress, motivating additional study, and enhancing overall learning and comprehension. Additionally, the findings provided academic staff with insights into the advantages of using formative e-assessments, such as fostering student engagement and enabling early intervention in the learning process.
In another study, Liu and Zhang (2022) employed a qualitative research methodology based on Grounded Theory to investigate the validity of formative assessment in online learning. Their study specifically focused on analyzing the factors inherent to online learning that impact its validity. The researchers also explored the extent to which these factors manifested in actual online practices. The study revealed that the validity of online formative assessment is influenced by several factors. These factors include the absence of functional alienation, which is caused by the emotional engagement and adaptability of the participants, the objective environment and technology, as well as poor interaction. In order to facilitate the integration of formative assessment into real classroom settings, the researchers developed two models based on their findings: the Formative Assessment Model (FAM) and the Online Formative Assessment Validity Model (OFAV). These models aimed to provide guidance and structure for the effective implementation of formative assessment in online educational contexts. To be more specific, the outcomes of their OFAV model demonstrated that the validity of online formative assessment (OFA) could be influenced by four distinct factors: the influence of agents, interaction, and feedback, the online teaching environment, and the reward and punishment mechanism.
The present study is based on the online formative assessment (OFA) validity scale Maleki et al. (2023) presented in their study. Their study on OFA validity was influenced by the work of Gikandi et al. (2011). To design a scale for OFA validity in EFL contexts, they first investigated the literature and then administered a Delphi method to extract indicators involved in the validity of online formative assessment. Their final online formative assessment (OFA) validity scale in the Iranian EFL (English as a Foreign Language) contexts included 27 indicators that fell into four broad categories: authenticity of online assessment activities, effective online formative feedback, multidimensional perspectives towards online formative assessment, and online learner support. As the first indicator of OFA validity, it was found that the authenticity of online assessment activities could be identified by ten sub-indicators. The second indicator of online OFA validity was effective online formative feedback, including ten sub-indicators. The third indicator of OFA validity was referred to as multidimensional perspectives towards OFA, encompassing three sub-indicators. The fourth indicator of OFA validity was online learner support, which comprised four sub-indicators.
Drawing from the findings of some studies (Esfandiari & Arefian, 2023; Nayernia & Mohebbi, 2023; Arefian, 2022; Ghanbari & Nowroozi, 2021), one can discern that Iranian EFL teachers lack the necessary knowledge and skills to conduct effective student assessment in an online learning environment. This is to say that they lack online assessment literacy. As Esfandiari and Arefian (2023) highlighted this, Iranian EFL teachers need to learn, unlearn, and improve their online language assessment literacy. Also, according to McLaughlin and Yan (2017), there is a lack of empirical research in the area of online formative assessment. Next, since the spread of COVID-19, as far as it was searched, there appears to be a limited number of studies available to examine issues related to validity indicators of online formative language assessment. Finally, as highlighted by Messick (1989), validity is context-dependent and can be influenced by various factors. In this study, we considered this notion and investigated whether the place of living, as a contextual factor, could impact the validity of online formative assessments. Therefore, this study aims to investigate the extent to which Iranian EFL teachers working in universities and language institutes apply indicators of online formative assessment validity in their classrooms. Also, it investigates whether EFL teachers’ place of living can affect the validity of online formative assessment. The following research questions were proposed in this research:
1. To what extent do Iranian EFL teachers in universities and language institutes apply indicators of online formative assessment validity in their classrooms?
2. Is there any significant difference in the validity of online formative assessment with regard to the Iranian EFL teachers’ place of living?
Method
Participants
In this study, the target population to fill out the OFAVS was 316 Iranian EFL teachers majoring in TEFL (Teaching English as a Foreign Language) from various EFL contexts, including public schools, private schools, language institutes, and universities. The sample was selected using a purposive sampling method and included both male and female participants from various age groups, ranging from 20 years old to over 50 years old. They had the minimum two years of online teaching experiences holding different degrees, namely B.A, M.A, and Ph.D. Table 1 displays the participants' demographic information in detail.
Demographic Characteristics of Respondents (n=316)
Demographic Characteristics | Category | Number |
Gender | Male Female | 152 164 |
Age | 20-30 31-40 41-50 More than 50 | 96 128 65 27 |
Teaching experiences | 2-5 years More than 5 years | 234 82 |
Educational degree | B.A M.A Ph.D. | 53 159 104 |
For the online observation, 21 Iranian EFL teachers were selected using convenience sampling, with 9 from universities and 12 from language institutes. Table 2 outlines the demographic details of the teachers, including their gender, workplace, age range, academic degree, and the provinces in which they are located.
Table 2
Demographic Information of EFL Teachers (n=21)
Demographic Characteristics | Category | Number |
Gender | Male | 13 |
Female | 8 | |
Workplace | University | 9 |
Language institute | 12 | |
Age | 25-35 | 3 |
36-45 | 11 | |
More than 45 | 7 | |
Academic degree | B.A | 3 |
M.A | 7 | |
Ph.D. | 11 | |
Location (province) | Tehran | 6 |
Khorasan-e Razavi | 5 | |
Golestan | 3 | |
Isfahan | 3 | |
Kerman | 2 | |
Fars | 2 |
Instrumentation
Online Formative Assessment Validity Scale (OFAVS)
This study administered an online formative assessment validity scale designed by Maleki et al. (2023). The effectiveness of this scale indicated to what extent EFL teachers' online formative assessment is valid. The items of their scale were elicited and extracted based on the Delphi method which they conducted previously. Their final online formative assessment (OFA) validity scale included 27 indicators that fell into four broad categories: authenticity of online assessment activities, effective online formative feedback, multidimensional perspectives towards online formative assessment, and online learner support. They designed a five-point Likert scale ranging from “Strongly Agree” to “Strongly Disagree”. The scale is presented in Appendix A. The reliability of the OFAVS was established with a high Cronbach's alpha coefficient of 0.935, indicating excellent internal consistency. To assess the validity of the scale, both Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were conducted. The EFA was used to identify the underlying factor structure of the scale, confirming that the items grouped into coherent factors consistent with the theoretical constructs. Following this, CFA was performed to validate the factor structure identified by the EFA, ensuring that the model fit the data well. The results from both EFA and CFA provide strong evidence for the construct validity of the OFAVS, supporting its use in this study.
Observation Checklist
In this study, observation was used to investigate the extent to which Iranian EFL teachers apply indicators of OFA validity in their classrooms. Twenty-one online classrooms were observed in 3 sessions. A checklist consisted of 27 indicators of OFA validity scale (Maleki et al., 2023) was used by the researcher to observe the online classrooms. The presence and application of each indicator were approved using "Yes" or "No" in the checklist by the researcher.
Procedure
Based on a scale designed by Maleki et al. (2023), 27 indicators are involved in the validity of OFA. During the first phase of the study, the researcher employed an observation checklist to assess the extent to which Iranian EFL teachers working in universities and language institutes applied the indicators of online formative assessment (OFA) validity in their classrooms. A total of 21 teachers were observed across three sessions using a checklist derived from Maleki et al. (2023)'s OFA validity scale (OFAVS). The number of participants at this stage was determined based on the saturation level of the observation results. For the second phase, the OFA validity scale was distributed among 316 Iranian EFL teachers, both in online and printed versions, to investigate if Iranian EFL teachers’ place of living can affect validity of online formative assessment. They were majoring in TEFL from different EFL contexts, namely public schools, private schools, language institutes, and universities. Also, the participants were asked to indicate their province, as this study aimed to investigate the effect of this variable on OFA validity.
Data Analysis
The data from the observation checklist were analyzed using descriptive statistics to determine the frequency with which each of the 27 indicators of OFA validity was applied by Iranian EFL teachers. Percentages were calculated to provide an overview of the most and least commonly observed indicators across the 21 classrooms.
For the OFA Validity Scale (OFAVS) data, both descriptive and inferential statistics were used. Mean scores and standard deviations were calculated for each indicator and the overall OFA validity score. To assess the impact of teachers' place of living on OFA validity, a one-way between-groups analysis of variance (ANOVA) was conducted to identify any statistically significant differences in scores among teachers from different provinces. Post hoc tests were applied as needed to explore these differences further.
Results
Assumptions for the Statistical Analysis
In this study, the researcher used ANOVA to analyze the data. Before conducting this statistical procedure, the researcher investigated their assumptions, including normality, equal variances, and linearity (Pallant, 2016; Larson-Hall, 2015).
The researcher employed descriptive analysis to demonstrate no abnormality in the data distribution, such as kurtosis or skewness. Table 3 shows the outcomes of descriptive statistics for the OFAVS, focusing specifically on ELT experts. The mean score obtained by ELT experts on the OFAVS was 52.20. Moreover, comparing the Mean and Trimmed Mean for ELT experts revealed a distinct resemblance between these values. As a result, these cases were included in the data file for further analysis (Pallant, 2016). Additionally, Table 3 provides information on the standard deviation of the survey data collected for the OFAVS.
Table 3
Descriptive Statistics for the ELT Experts Regarding Online Formative Assessment Validity Scale (OFAVS)
Statistic Std. Error
OFAVS Mean 52.2025 .69733
95% Confidence Interval for Mean Lower Bound 50.8305
95% Confidence Interval for Mean Upper Bound 53.5745
5% Trimmed Mean 52.3115
Median 54.0000
Variance 153.660
Std. Deviation 12.39598
Minimum 29.00
Maximum 74.00
Range 45.00
Interquartile Range 21.00
Skewness -.216 .137
Kurtosis -1.123 .273
Table 4 displays the findings of the Kolmogorov-Smirnov statistics, which aimed to evaluate the normality of the score distribution for ELT experts. The Sig. value is .000, indicating a deviation from normality, which is common in larger samples (Pallant, 2016).
Table 4
K-S Tests of Normality for Participants
Kolmogorov-Smirnova |
|
|
| Shapiro-Wilk |
| ||||
Statistic | df | Sig. | Statistic | df | Sig. | ||||
OFAVS .086 | 316 | .000 | .951 | 316 | .000 | ||||
a Lilliefors Significance Correction
The boxplots indicating a normal distribution support the assumption of homogeneity of variance, which means that conducting Multiple Regression is feasible. To check the assumption of linearity, a Q-Q plot was used, as mentioned in Appendix 1. Pallant (2016) suggested that a reasonably straight line on the Q-Q plot indicates a normal distribution. Based on these findings, it appears that the assumptions of homogeneity of variance and linearity have been met, allowing for the possibility of conducting Multiple Regression.
Results of Research Question 1: Observation
In this research question, the present study investigated the extent to which Iranian EFL teachers in universities and language institutes apply indicators of online formative assessment validity in their classrooms. To this end, the researcher observed 21 teachers in three sessions using a checklist and entered the data into SPSS 22. Using descriptive statistics, the researcher estimated the frequency of each indicator used by participants in their classrooms.
Table 5 illustrates the results of observations done by the researcher. The more observed indicators (more than 80%) are as follows: monitoring students’ progression to get real-time updates (item 7), providing students with informative and supportive online feedback (item 11), providing students with individualized feedback (item 15), giving feedback that identifies students’ strengths and weaknesses (item 16), advising learners on finding and selecting learning opportunities (item 24), and motivating students towards self-regulation. However, the least observed indicators (less than 25%) are related to items 3 and 19: involving students in the planning process of online assessment activities and developing dialogic feedback.
Table 5
The Extent Iranian EFL Teachers in Universities and Language Institutes Apply Indicators of Online Formative Assessment Validity in their Classrooms
Items Yes No
1. To involve students in deeper knowledge construction 33.3 66.7
2. To put emphasis on higher-order thinking skills 66.7 33.3
3. To involve students in the planning process of online assessment
activities 9.5 90.5
4. To be flexible and offer a variety of tools and options 61.9 38.1
5. To integrate online authentic activities across different subject areas 76.2 23.8
6. To document and monitor students’ achievements 58.7 14.3
7. To monitor students’ progression to get real-time updates 90.5 9.5
8. To offer online complex and authentic assessment activities 52.4 47.6
9. To include analytical and transparent rubrics in online assessment
activities 57.1 42.9
10. To empower learners reflect on their learning both individually and
socially 76.2 23.8
11. To provide students with informative and supportive online feedback 90.5 9.5
12. To provide students with timely online feedback 42.9 57.1
13. To promote dialogue by peer online formative feedback 33.3 66.7
14. To motivate students towards self-regulation 81.0 19.0
15. To provide students with individualized feedback 90.5 9.5
16. To give feedback which identifies students’ strengths and weaknesses 85.7 14.3
17. To develop students’ online feedback literacy 38.1 61.9
18. To develop teachers’ online feedback literacy 76.2 23.8
19. To develop dialogic feedback 23.8 76.2
20. To apply online dynamic assessment 47.6 52.4
21. To consider teachers’ knowledge, skills, and professional development 66.7 33.3
22. To have tools and further support to carry out high-quality assessment 66.7 33.3
23. To develop complex cognitive processes 42.9 57.1
24. To advise learners on finding and selecting learning opportunities 85.7 14.3
25. To help learners understand their learning orientations and strengths 61.9 38.1
26. To encourage learners to express their learning goals and plans 66.7 33.3
27. To mediate the standard and uniform elements of course materials 38.1 61.9
Results of Research Question 2: The Effect of Place of Living on OFAVS
The goal of the second research question was to determine if there were any notable differences in the effectiveness of online formative assessment among Iranian EFL teachers based on their location. To achieve this, a one-way between-groups analysis of variance (ANOVA) was conducted using the Online Formative Assessment Validity Scale (OFAVS) to measure the impact of location and place of living on online formative assessment. The participants were split into 20 groups based on their province of residence. The results in Table 6 indicate a statistically significant difference in OFAVS scores among the 20 groups, with a p-value < .05, F (19, 296) = 2.28, p = .002.
Table 6
ANOVA Results for OFAVS Regarding Place of Living
Sum of Squares | df | Mean Square |
| F |
| Sig. | ||||||||
Between Groups | 5552.017 | 19 | 292.211 |
| 2.288 | .002 | ||||||||
Within Groups | 37810.727 | 296 | 127.739 |
|
|
| ||||||||
Total | 43362.744 | 315 |
|
|
|
| ||||||||
Table 7 represents the data for the 20 provinces under study. This included information on the number of participants, the mean, and the standard deviation of each group. The Tukey HSD test was used for post hoc comparisons to assess differences between groups based on their place of living. The results indicated that no significant differences were found among the groups. While there was a significant difference in the validity of online formative assessment based on the Iranian EFL teachers' place of living, no significant difference was observed when comparing different locations for online formative assessment.
Table 7
Descriptive Statistics for EOFAIS Regarding Place of Living
N Mean Std. Deviation Std. Error 95% CI (Mean)
Lower Bound Upper Bound
Alborz 12 56.0833 10.95825 3.16338 49.1208 63.0459
Bushehr 11 47.6364 9.51076 2.86760 41.2469 54.0258
E. Azarbaeijan 13 48.1538 11.17919 3.10055 41.3983 54.9094
Fars 14 49.3571 12.56740 3.35878 42.1009 56.6133
Gilan 7 45.2857 3.81725 1.44279 41.7553 48.8161
Golestan 10 53.3000 11.05592 3.49619 45.3911 61.2089
Hamedan 7 47.5714 11.16329 4.21933 37.2471 57.8958
Isfahan 7 41.5714 8.05930 3.04613 34.1178 49.0250
Kerman 13 56.3077 7.88865 2.18792 51.5406 61.0748
Kermanshah 6 47.5000 14.43260 5.89209 32.3539 62.6461
Khorasan R. 86 50.8721 12.45396 1.34294 48.2020 53.5422
Khuzestan 18 44.3333 11.09849 2.61594 38.8142 49.8525
Kordestan 10 57.6000 6.18601 1.95619 53.1748 62.0252
Mazandaran 14 44.8571 9.60655 2.56746 39.3105 50.4038
No. Khorasan 6 57.0000 8.14862 3.32666 48.4485 65.5515
Semnan 9 53.0000 15.09139 5.03046 41.3997 64.6003
Sistan & B. 9 57.7778 10.23203 3.41068 49.9127 65.6428
Tabriz 2 48.5000 17.67767 12.50000 -110.3276 207.3276
Tehran 49 46.2449 11.45922 1.63703 42.9534 49.5364
Yazd 13 55.7692 11.14416 3.09083 49.0349 62.5036
Total 316 50.0285 11.73284 .66002 48.7299 51.3271
Discussion
Two primary aims were followed in the present study. First, this study investigated the extent to which Iranian EFL teachers in universities and language institutes apply indicators of OFA validity in their online classrooms. Next, it examined if Iranian EFL teachers’ place of living can affect the validity of OFA. To achieve these aims, this study used Maleki et al. (2023)’s OFA validity scale. They identified 27 OFA validity indicators that fell into four broad categories: authenticity of online assessment activities, effective online formative feedback, multidimensional perspectives towards online formative assessment, and online learner support.
In the present study, the first research question investigated the extent to which Iranian EFL teachers working in universities and language institutes apply the indicators of online formative assessment validity in their classrooms. Based on the results, it can be inferred that Iranian EFL teachers working in universities and language institutes possess a basic understanding of fundamental online formative assessment principles, which align with traditional formative assessment practices. The data suggests that these teachers are proficient in certain conventional aspects, such as monitoring student progress and providing individualized feedback. However, they seem less familiar with integrating more innovative, learner-centered approaches into their practice. This emphasis on traditional methods indicates a solid grounding in basic assessment principles but also suggests a potential resistance to, or lack of exposure to, the more collaborative and dynamic dimensions of online formative assessment.
The high frequency of observed practices like offering feedback that identifies students’ strengths and weaknesses, advising learners on learning opportunities, and motivating students towards self-regulation suggests that teachers are prioritizing immediate, teacher-driven interventions aimed at enhancing student performance. However, the neglect of indicators such as involving students in the planning of assessment activities and developing dialogic feedback points to a significant gap in fostering a learner-centered environment. This finding implies that while teachers are committed to providing support, they may not fully recognize or be equipped to implement practices that encourage students to take a more active role in their learning process. This is concerning because, as research by Bremner (2021) and Ryan et al. (2021) highlights, learner-centeredness is a critical component of effective online formative assessment. The omission of practices like involving students in assessment planning and offering dialogic feedback suggests that Iranian EFL teachers may be missing opportunities to engage students in deeper, more reflective learning processes. For example, the observation that only 33.3% of teachers apply online peer formative feedback indicates a limited use of peer-assisted learning strategies, which are known to foster critical thinking and collaborative learning. Additionally, with only 38.1% of teachers working to enhance students' feedback literacy, it appears that many are not adequately preparing their students to engage with feedback in a meaningful way.
These findings suggest that the current approach to online formative assessment in these settings may be too focused on teacher-led activities, which could limit students' opportunities for self-directed learning and reflection. The implications of this are substantial, suggesting that students may not be sufficiently equipped for independent learning or active participation in their educational journey—a critical goal in contemporary pedagogical approaches. Therefore, to address these gaps, it is crucial that professional development programs for Iranian EFL teachers emphasize the importance of learner-centered approaches and equip teachers with the tools and strategies to implement them effectively. By shifting towards a more balanced approach that includes both traditional and innovative assessment practices, teachers can create a richer, more engaging online learning environment that better supports student growth and development.
The second research question aimed to determine whether there were any notable differences in the validity of online formative assessment among Iranian EFL teachers based on their location, particularly their province. The findings revealed a statistically significant difference in OFA validity scale scores among teachers from 20 different provinces, suggesting that geographical location does indeed play a role in shaping how online formative assessments are perceived and implemented. This result underscores the context-bound nature of validity, aligning with Messick’s (1989) assertion that validity is inherently tied to the specific context in which an assessment is used. This context dependency suggests that various local factors—such as access to resources, teacher training, institutional support, and even cultural attitudes towards education—may influence how online formative assessments are conducted and perceived.
The lack of a significant difference when comparing different locations directly for online formative assessment validity might initially seem contradictory. However, this could be interpreted as evidence that while regional factors do influence assessment practices, these influences are complex and multifaceted, not easily captured by simple comparisons. The significant differences observed across provinces suggest that the influence of location is not uniform but rather shaped by a variety of interrelated factors that can differ substantially even within the same country. This complexity is consistent with Liu and Zhang’s (2022) findings in China and Taiwan, where location was identified as a crucial factor influencing the effectiveness of online formative assessment. However, unlike their study, which emphasized broader regional differences, the results of this research suggest that variability within a single country can be equally, if not more, significant.
These findings imply that efforts to improve the validity of online formative assessments should take into account the specific local contexts in which teachers operate. Professional development programs, for instance, may need to be tailored to address the unique challenges and opportunities present in different provinces. Additionally, policymakers should consider these regional differences when designing and implementing assessment policies, ensuring that they are flexible enough to accommodate the diverse educational environments across the country. This also raises important questions about equity in education, as it suggests that students’ experiences with and benefits from online formative assessments may vary depending on their geographical location. In conclusion, this finding highlights the importance of recognizing the context-dependent nature of validity in online formative assessments. The variability in OFA validity scores across different provinces suggests that a one-size-fits-all approach may not be effective in enhancing assessment practices across diverse educational settings. Instead, a more nuanced understanding of how local contexts influence assessment validity is necessary to ensure that online formative assessments are both fair and effective for all students, regardless of their location.
This study attempted to investigate the extent to which Iranian EFL teachers working in universities and language institutes apply indicators of OFA validity in their classrooms based on Maleki et al. (2023)’s OFA validity scale. Results indicated that the more observed indicators were monitoring students’ progression to get real-time updates, providing students with informative and supportive online feedback, providing students with individualized feedback, giving feedback that identifies students’ strengths and weaknesses, advising learners on finding and selecting learning opportunities, and motivating students towards self-regulation. On the other hand, the least observed indicators were involving students in the planning process of online assessment activities and developing dialogic feedback.
Furthermore, this study examined whether EFL teachers’ place of living can influence the validity of online formative assessment. The results revealed that EFL teachers' place of living could affect the validity of online formative assessment. This highlighted the context-bound nature of validity in online formative assessment.
The findings of this study have important implications for online EFL teachers and policymakers in Iran. Due to the spread of COVID-19, there has been a need to improve teachers' awareness of online assessment principles, specifically focusing on formative assessment. This study provides valuable insights for Iranian EFL teachers, enabling them to identify areas that require greater attention and enhancement in order to enhance the validity of their online formative assessments. Furthermore, assessment organizations and policymakers have the opportunity to design and implement professional development programs for Iranian EFL teachers, specifically targeting the areas that require improvement. Finally, given the contextual nature of validity, it is important for online teachers and assessment designers to consider these factors when developing and implementing formative assessments in the online learning environment. They should consider various factors and strive to create assessments that are aligned with learning goals, culturally sensitive, and provide meaningful feedback to students, ensuring the validity and fairness of the assessment process.
To further advance the existing scientific understanding of online formative assessment validity, there are several important avenues for future research. One potential direction is to explore the influence of various cultural and contextual factors on the validity of online formative assessments within EFL settings. For example, additional studies could examine the effects of factors such as EFL teachers' years of experience, the level they teach, their certification, age, or gender on the validity of online formative assessments. By investigating these factors, researchers can gain deeper insights into the complex dynamics that impact the validity of online formative assessment in EFL contexts. Also, in the observation phase of this study, participation was restricted to EFL university lecturers and language institute teachers. The inclusion of school teachers was not feasible due to the technical limitations of the Shad platform and the unavailability of online school classrooms. However, it is recommended that future studies incorporate and observe school teachers to obtain valuable insights from their perspective. Furthermore, conducting qualitative studies to observe the extent to which English teachers apply the indicators of online formative assessment (OFA) in various EFL or ESL contexts would be valuable. Comparative studies can be designed and implemented to compare and analyze the application of OFA indicators across different educational settings. Such research endeavors can provide valuable insights into the effectiveness and applicability of OFA indicators in diverse language learning environments.
Declaration of interest: none
References
Alamri, H. A., Watson, S., & Watson, W. (2020). Learning Technology Models that Support Personalization within Blended Learning Environments in Higher Education. TechTrends, 65(1), 62–78. https://doi.org/10.1007/s11528-020-00530-3
Archambault, L., Leary, H., & Rice, K. (2022). Pillars of online pedagogy: A framework for teaching in online learning environments. Educational Psychologist, 57(3), 178-191. https://doi.org/10.1080/00461520.2022.2051513
Arefian, M. H. (2022). Perceptions of self-assessment literacy and self-directed reflection during online learning for Iranian EFL student teachers. Reflective Practice, 23(6), 623–634. https://doi.org/10.1080/14623943.2022.2096584
Beardsley, M., Albó, L., Aragón, P., & Hernández‐Leo, D. (2021). Emergency education effects on teacher abilities and motivation to use digital technologies. British Journal of Educational Technology, 52(4), 1455-1477. https://doi.org/10.1111/bjet.13101
Boss, S., & Krauss, J. (2022). Reinventing project-based learning: Your field guide to real-world projects in the digital age. International Society for Technology in Education.
Bragg, L. A., Walsh, C., & Heyeres, M. (2021). Successful design and delivery of online professional development for teachers: A systematic review of the literature. Computers & Education, 166, 104158. https://doi.org/10.1016/j.compedu.2021.104158
Bremner, N. (2021). What is learner-centered education? A quantitative study of English language teachers’ perspectives. Tesl-Ej, 25(2). http://researchspace.bathspa.ac.uk/14223/
Carless, D. (2022). From teacher transmission of information to student feedback literacy: Activating the learner role in feedback processes. Active Learning in Higher Education, 23(2), 143-153.
Elmahdi, I., Al-Hattami, A., & Fawzi, H. (2018). Using Technology for Formative Assessment to Improve Students' Learning. Turkish Online Journal of Educational Technology-TOJET, 17(2), 182-188.
Esfandiari, R., & Arefian, M. H. (2023). Developing collective eyes for Iranian EFL teachers’ computer-assisted language assessment literacy through internet-based collaborative reflection. Education and Information Technologies, 1-22. https://doi.org/10.1007/s10639-023-12201-w.
Espasa, A., Mayordomo, R. M., Guasch, T., & Martinez-Melo, M. (2022). Does the type of feedback channel used in online learning environments matter? Students’ perceptions and impact on learning. Active Learning in Higher Education, 23(1), 49-63. https://doi.org/10.1177/1469787419891307.
Ghanbari, N., & Nowroozi, S. (2021). The practice of online assessment in an EFL context amidst COVID-19 pandemic: views from teachers. Language Testing in Asia, 11(1), 1-18. https://doi.org/10.1186/s40468-021-00143-4.
Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers & Education, 57(4), 2333-2351. https://doi.org/10.1016/j.compedu.2011.06.004.
Hazaymeh, W. A. (2021). EFL students’ perceptions of online distance learning for enhancing English language learning during Covid-19 pandemic. International Journal of Instruction, 14(3), 311-326. https://doi.org/10.29333/iji.2021.14320a.
Irons, A., & Elkington, S. (2021). Enhancing learning through formative assessment and feedback. Routledge.
Kantaridou, Z., & Hatzopoulou, C. (2021). The effect of alternative assessment methods on young learners’ autonomy and self-regulation: Α quasi-experimental study. Journal of Applied Linguistics, (34), 47-73. https://doi.org/10.2478/jal-2021-0003.
Kaya-Capocci, S., O’Leary, M., & Costello, E. (2022). Towards a Framework to Support the Implementation of Digital Formative Assessment in Higher Education. Education Sciences, 12(11), 823. https://doi.org/10.3390/educsci12110823
Kilis, S., & Yildirim, Z. (2019). Posting patterns of students’ social presence, cognitive presence, and teaching presence in online learning. Online Learning, 23(2), 179-195. https://doi.org/10.24059/olj.v23i2.1460.
Kristiyanti, K. L., Santosa, M. H., & Pratiwi, N. P. A. (2021). Students’ perception towards the implementation of online formative assessment in English learning. SAGA: Journal of English Language Teaching and Applied Linguistics, 2(2), 101-112. https://doi.org/10.21460/saga.2021.22.83.
Larson-Hall, J. (2015). A guide to doing statistics in second language research using SPSS. Routledge.
Lase, D., & Zega, T. G. C. (2021). How can teachers engage students in online learning? A Conceptual Framework. Technium Social Sciences Journal, 20(1), 143-155. Retrieved from https://techniumscience.com/index.php/socialsciences/article/view/3433.
Liu, Y., & Zhang, H. (2022). Exploring the Influencing Factors and Validity of Formative Assessment in Online Learning. Journal of Education and e-Learning Research, 9(4), 278-287. https://doi.org/10.20448/jeelr.v9i4.4288.
Mahapatra, S. K. (2021). Online formative assessment and feedback practices of ESL teachers in India, Bangladesh and Nepal: A multiple case study. The Asia-Pacific Education Researcher, 30(6), 519-530. https://doi.org/10.1007/s40299-021-00603-8
Maleki, A., Khodabakhshzadeh, H., Zeraatpishe, M., & Baghaei, P. (2023). Online formative assessment validity: Theoretical perspectives, EFL teachers' beliefs, and practical implications. [Unpublished doctoral dissertation]. Islamic Azad University of Torbate Heydarieh. Iran.
Marden, M. P., & Herrington, J. (2020). Design principles for integrating authentic activities in an online community of foreign language learners. Issues in Educational Research, 30(2), 635-654.
Martin, F., Ritzhaupt, A., Kumar, S., & Budhrani, K. (2019). Award-winning faculty online teaching practices: Course design, assessment and evaluation, and facilitation. The Internet and Higher Education, 42, 34–43. https://doi.org/10.1016/j.iheduc.2019.04.001.
McCallum, S., & Milner, M. M. (2020). The effectiveness of formative assessment: student views and staff reflections. Assessment & Evaluation in Higher Education, 46(1), 1–16. https://doi.org/10.1080/02602938.2020.1754761
McLaughlin, T., & Yan, Z. (2017). Diverse delivery methods and strong psychological benefits: A review of online formative assessment. Journal of Computer Assisted Learning, 33(6), 562-574. https://doi.org/10.1111/jcal.12200.
Messick, S. (1989). Validity. In R. Linn (Ed.), Educational measurement (pp. 13–103). Macmillan.
Nayernia, A., Mohebbi, H. (2023). Teachers’ Online Language Assessment Literacy. In: Tajeddin, Z., Griffiths, C. (eds) Language Education Programs. Language Policy, vol 34. Springer, Cham. https://doi.org/10.1007/978-3-031-38754-8_8
Pallant, J. (2016). SPSS survival manual: A step by step guide to data analysis using SPSS for windows. Open University Press.
Panadero, E., & Lipnevich, A. A. (2022). A review of feedback models and typologies: Towards an integrative model of feedback elements. Educational Research Review, 35, 100416. https://doi.org/10.1016/j.edurev.2021.100416
Rajabalee, Y. B., & Santally, M. I. (2021). Learner satisfaction, engagement and performances in an online module: Implications for institutional e-learning policy. Education and Information Technologies, 26(3), 2623-2656. https://doi.org/10.1007/s10639-020-10375-1.
Rotar, O. (2022). Online student support: A framework for embedding support interventions into the online learning cycle. Research and Practice in Technology Enhanced Learning, 17(1), 1-23. https://doi.org/10.1186/s41039-021-00178-4.
Ryan, T., Henderson, M., Ryan, K., & Kennedy, G. (2021). Designing learner-centred text-based feedback: a rapid review and qualitative synthesis. Assessment & Evaluation in Higher Education, 46(6), 894-912. https://doi.org/10.1080/02602938.2020.1828819.
Say, R., Visentin, D., Saunders, A., Atherton, I., Carr, A., & King, C. (2023). Where less is more: Limited feedback in formative online multiple‐choice tests improves student self‐regulation. Journal of Computer Assisted Learning, 40(1), 89–103. https://doi.org/10.1111/jcal.12868
Shaw, S., & Crisp, V. (2011). Tracing the evolution of validity in educational measurement: Past issues and contemporary challenges Research Matters: A Cambridge Assessment publication, 11, 14-17.
Singh, J., Steele, K., & Singh, L. (2021). Combining the best of online and face-to-face learning: Hybrid and blended learning approach for COVID-19, post vaccine, & post-pandemic world. Journal of Educational Technology Systems, 50(2), 140-171. https://doi.org/10.1177/00472395211047865.
Sudakova, N. E., Savina, T. N., Masalimova, A. R., Mikhaylovsky, M. N., Karandeeva, L. G., & Zhdanov, S. P. (2022). Online Formative Assessment in Higher Education: Bibliometric Analysis. Education Sciences, 12(3), 209. https://doi.org/10.3390/educsci12030209.
Veerasamy, A. K., Laakso, M. J., & D’Souza, D. (2021). Formative assessment tasks as indicators of student engagement for predicting at-risk students in programming courses. Informatics in Education. https://doi.org/10.15388/infedu.2022.15
Waheeda, A., Shaheeda, F., Muna, F. & Shina, A. (2023). Online assessments in Maldivian higher education institutions: Challenges faced by students. Issues in Educational Research, 33(3), 1233-1249. http://www.iier.org.au/iier33/waheeda.pdf
Wang, T. H., Sun, Y., & Huang, N. W. (2021). Implementation of web-based dynamic assessment in improving low English achievers’ learning effectiveness. Computer Assisted Language Learning, 1-27. https://doi.org/10.1080/09588221.2021.1998129.
Weleschuk, A., Dyjur, P., & Kelly, P. (2019). Online Assessment in Higher Education. Taylor Institute for Teaching and Learning Guide Series. Calgary, AB: Taylor Institute for Teaching and Learning at the University of Calgary. Retrieved from https://taylorinstitute.ucalgary.ca/resources/guides
Wongvorachan, T., Bulut, O., Tsai, Y. S., & Lindner, M. A. (2022). Improving Student Feedback Literacy in e-Assessments: A Framework for the Higher Education Context. Trends in Higher Education, 1(1), 16-29. https://doi.org/10.3390/higheredu1010002.
Wu, T. T., Sari, N. A. R. M., & Huang, Y. M. (2024). Integrating extended formative assessment in flipped jigsaw learning: Promoting learning engagement and higher-order thinking skills in international business education context. The International Journal of Management Education, 22(1), 100930. https://doi.org/10.1016/j.ijme.2024.100930
Yousef, A. M. F., & Sumner, T. (2021). Reflections on the last decade of MOOC research. Computer Applications in Engineering Education, 29(4), 648-665. https://doi.org/10.1002/cae.22334.
Zhan, Y., Wan, Z. H., & Sun, D. (2022). Online formative peer feedback in Chinese contexts at the tertiary Level: A critical review on its design, impacts and influencing factors. Computers & Education, 176, 104341. https://doi.org/10.1016/j.compedu.2021.104341
Appendix A
Online Formative Assessment Validity Scale (Maleki et al., 2023)
Instruction: The following is a list of statements related to online formative assessment validity indicators. Please rate how much you personally agree or disagree with these statements. Use the following scale:
1) Strongly agree
2) Agree
3) Neutral
4) Disagree
5) Strongly disagree
1. I believe online authentic tasks should involve students in deeper knowledge construction such as problem-solving and understanding of complex meanings.
2. I believe online authentic tasks should put emphasis on higher-order thinking skills as opposed to rote memorization of factual information.
3. I believe students should be involved in the planning process of online assessment activities in various roles.
4. I believe online assessment activities should be flexible and offer a variety of tools and options for various assessment methods and solutions.
5. I try to integrate online authentic activities and apply them across different subject areas to lead beyond a specific domain.
6. I try to continuously document and monitor students' achievements to help both students be self-sufficient and teachers be reflective about students' progress.
7. I try to monitor students' progression to get real-time updates on a student’s learning.
8. I try to offer online complex and authentic assessment activities that involve students in decision-making and problem solving.
9. I believe analytical and transparent rubrics should be part of online assessment activities to assist the students to perfectly understand the expected level of the outcomes.
10. I believe reflection on online authentic activities can empower learners reflect on their learning both individually and socially.
11. I try to provide students with informative and supportive online feedback rather than just giving a final score.
12. I try to provide students with timely online feedback since it can make changes in the teaching and learning process.
13. I believe peer online formative feedback is a way of promoting dialogue and stimulating online learners to actively engage in formative assessment processes.
14. I believe online formative feedback should motivate students towards self-regulation to make them responsible for their own learning.
15. I try to provide students with individualized feedback which optimizes learning.
16. I try to give feedback which identifies students' strengths and weaknesses.
17. I try to develop students' online feedback literacy to improve their self-regulation and awareness of e-Assessment aspects such as peer feedback.
18. I believe that teachers should develop their online feedback literacy to improve its effectiveness and make students more active.
19. I believe dialogic feedback is essential since lecturers and students do not share the same space or time in asynchronous online learning environments.
20. I believe teachers should apply online dynamic assessment to assist them in the construction of an assessment-centered e-Learning environment.
21. I believe that one significant aspect of online assessment effectiveness is with regard to teachers' knowledge, skills, and professional development.
22. I believe online teachers should have tools and further support to carry out high-quality assessment.
23. I try to develop complex cognitive processes that current students require to succeed.
24. I try to advise learners on finding and selecting learning opportunities that will meet their needs during the early development of their learning plans.
25. I try to help learners understand their learning orientations, strengths, and areas for improvement in the beginning of their programs.
26. I try to encourage learners to express their learning goals and plans.
27. I try to inspire and develop learning through the mediation of the standard and uniform elements of course materials and learning resources for individual students.
Appendix B
Q-Q Plot for Linearity Assumption, ELT Experts’ Perception of OFA
Biodata
Alireza Maleki, a PhD candidate in TEFL (Teaching English as a Foreign Language), is a certified teacher trainer, researcher, and university lecturer in Khorasan-e Razavi province, Iran. He has also published and presented papers in international and national journals and conferences. He is a journal reviewer and also has a keen interest in educational assessment concepts and subjects.
Hossein Khodabakhshzadeh is an Assistant professor at Islamic Azad University of Torbat-e Heydarieh, Iran. His research interests are in ELT, FLA, and SLA.
Mitra Zeraatpishe is an assistant professor at Islamic Azad University of Mashhad, Iran. Her research interests include SLA, linguistics, teaching methodologies, and teaching skills.
Purya Baghaei is an associate professor at Islamic Azad University of Mashhad, Iran. He has contributed to research in Rasch model & Item response theory.
بررسی نشانگرهای روایی در ارزشیابی تکوینی آنلاین: برداشت‌هایی از معلمان زبان انگلیسی ایرانی
ارزشیابی‌های تکوینی آنلاین معتبر برای اندازه گیری دقیق پیشرفت دانش‌آموزان و تصمیم‌گیری آموزشی موثر در یادگیری دیجیتال بسیار حائز اهمیت است. این مطالعه بر پایه رویکرد کمی دو هدف اصلی را دنبال کرد. ابتدا، هدف این مطالعه بررسی میزان استفاده معلمان زبان انگلیسی ایرانی شاغل در دانشگاه ها و موسسات زبان از نشانگرهای روایی ارزشیابی تکوینی آنلاین بود. بیست و یک کلاس آنلاین در سه جلسه با استفاده از یک مشاهده شدند. هدف دوم این مطالعه تعیین تأثیر محل اقامت معلمان زبان انگلیسی به عنوان زبان خارجه بر روایی ارزشیابی تکوینی آنلاین بود. به این منظور،از ۳۱۶ معلم زبان انگلیسی ایرانی از محیط‌های متنوع زبان انگلیسی، از جمله مدارس دولتی، مدارس خصوصی، موسسات زبان و دانشگاه‌ها، درخواست شد تا مقیاس توسعه یافته توسط ملکی و همکاران (۲۰۲۳) در خصوص روایی ارزشیابی تکوینی آنلاین را تکمیل کنند. نمونه شامل معلمان زن و مرد با گروه‌های سنی و مدارک تحصیلی متفاوت بود. یافته‌های این مطالعه نشان داد که معلمان زبان انگلیسی ایرانی در دانشگاه ها و موسسات زبان نشانگرهای متمرکز بر دانش‌آموز در روایی ارزشیابی تکوینی آنلاین را نادیده می‌گیرند. علاوه بر این، آشکار شد که محل اقامت معلمان زبان انگلیسی می‌تواند بر روایی ارزشیابی تکوینی آنلاین تأثیرگذار باشد. این مطالعه تلویحات متعددی برای معلمان زبان انگلیسی و سیاست گذاران دارد. این یافته‌های بر ماهیت مبتنی بر زمینه در روایی ارزشیابی تکوینی آنلاین تأکید می‌کند. همچنین، به معلمان زبان انگلیسی ایرانی کمک می‌کند تا حوزه‌های خاصی که نیاز به توجه و بهبود بیشتر برای افزایش روایی ارزشیابی‌های تکوینی آنلاین آن‌ها دارند را شناسایی کنند.
کلمات کلیدی: روایی، ارزشیابی تکوینی آنلاین، نشانگرها، محل اقامت، مشاهده.
