• الصفحة الرئيسية
  • A Critical Writing Scoring Rubric: Development and Validation for Iranian EFL Learners in Computer-Mediated Communication

شارک

عنوان URL للمقالة


رقم المقالة : 202502101199201 زيارة : 122 الصفحة: 85 - 104

https://doi.org/10.71962/jfl-2025-1199201

نوع المخطوط: ابحاث

A Critical Writing Scoring Rubric: Development and Validation for Iranian EFL Learners in Computer-Mediated Communication

الموضوعات :

Sara Samadi 1 , Mohammad Hashamdar 2 , Gholam Hassan Famil Khalili 3

1 -
2 - Department of English Language Teaching and Translation, K. C., Islamic Azad University, Karaj, Iran
3 - Department of English Language Teaching and Translation, K. C., Islamic Azad University, Karaj, Iran

تاريخ الإرسال : 11 الإثنين , شعبان, 1446 تاريخ التأكيد : 13 الخميس , رمضان, 1446 تاريخ الإصدار : 20 الثلاثاء , محرم, 1447

الکلمات المفتاحية: Computer-mediated communication, critical writing, intellectual standards, rubric,

ملخص المقالة :

Given the increasing prevalence of online learning and the absence of a validated instrument for assessing the critical writing skills of Iranian EFL learners in Computer-Mediated Communication (CMC) contexts, this study aimed to develop and validate an analytic rubric to enhance critical writing in CMC environments. The participants were 236 EFL students and 10 EFL/ESL teachers, representing a diverse range of demographics, experiences, and qualifications. The rubric development process was guided by a comprehensive literature review, the administration of a critical thinking questionnaire, expert feedback, thematic analysis of semi-structured interviews, integration of Paul and Elder’s (2019) Intellectual Standards, and the development and application of a Likert-scale critical writing questionnaire. Subsequent pilot testing facilitated refinements based on feedback. Accordingly, the finalized analytic rubric featured four principal components: (1) Clarity, Accuracy, and Precision (CAP); (2) Relevance and Logic (RL); (3) Depth and Significance (DS); and (4) Breadth and Fairness (BF). A rigorous validation process, including expert evaluation, factor analysis, and structural equation modeling (SEM), confirmed the rubric's reliability and validity. The findings demonstrated satisfactory internal consistency, as well as convergent and discriminant validity. This study offers valuable pedagogical implications for English as a Foreign Language (EFL) teachers, learners, researchers, and curriculum developers.

 

المصادر:

Alghizzi, T. M., & Alshahrani, T. M. (2024). Effects of grading rubrics on EFL learners’ writing in an EMI setting. Heliyon, 10(18).1-17. https://doi.org/10.1016/j.heliyon.2024.e36394
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Brookhart, S. (2013). How to create and use rubrics for formative assessment and grading.
ASCD. https://doi.org/10.4135/9781452218649.n15
Cohen, L., Manion, L., & Morrison, K. (2002). Research methods in education. Routledge. https://doi.org/10.4324/9780203224342
Dadakoğlu, S. C., & Özdemir, A. (2021). Development of analytical rubric for evaluating the effect of mobile design applications on artistic creativity: Validity-reliability study. İnönü Üniversitesi Eğitim Bilimleri Enstitüsü Dergisi, 8(16), 64-84. https://doi.org/10.29129/inujgse.897929
Dappen, L., Isernhagen, J., & Anderson, S. (2008). A statewide writing assessment model: Student proficiency and future implications. Assessing Writing, 13(1), 45-60. https://doi.org/10.1016/j.asw.2008.04.001.
Elbow, P. (2022). The democratization of writing and the role of cheating. Composition Studies, 50(1), 67-180.
Farzana, T. (2023). The standard measurement in online learning: a rubric as a focus on teaching-learning practices to move up quality education. EIKI Journal of Effective Teaching Methods, 1(3). https://doi.org/10.59652/jetm.v1i3.37
Ferris, D. (2009). Response to Student Writing: Implications for Second Language Students. New York, NY: Routledge. https://doi.org/sid.ir/paper/614509/en
Field, A. (2013). Discovering statistics using IBM SPSS statistics. Sage.
Gupta, B. L., & Gupta, P. B. (2021). Rubrics as versatile educational tool for outcome-based education. Journal of Engineering Technology Education, 15(2), 13-24
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2013). Multivariate data analysis: Pearson new international edition PDF eBook. Pearson Higher Ed.
Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43, 115-135. https://doi.org/10.1007/s11747-014-0403-8
Jackson, D. O. (2024). The longitudinal development of argumentative writing in an English for academic purposes course in Japan. System, 126, 103482. https://doi.org/10.1016/j.system.2024.103482
Kaiser, H. F. (1970). A second generation little jiffy. Psychometrika, 35(4), 401–415. https://doi.org/10.1007/BF02291817
Keller, S. D., Trüb, R., Raubach, E., Meyer, J., Jansen, T. & Fleckenstein, J. (2023). Designing and validating an assessment rubric for writing emails in English as a foreign language. RISTAL, 6, 16–48. https://doi.org/10.2478/ristal-2023-0002
Kobylarek, A., Błaszczyński, K., Ślósarz, L., & Madej, M. (2022). Critical Thinking Questionnaire (CThQ)–construction and application of critical thinking test tool. Andragogy Adult Education and Social Marketing, 2(2), 1-1. https://doi.org/10.15503/andr2022.1
Kohn, A. (2006). Speaking my mind: The trouble with rubrics. English Journal, 95(4), 12-15. https://doi.org/10.58680/ej20064950
Kuehner, A. V., & Hurley, J. (2019). How integrating reading and writing supports student success. Journal of Developmental Education, 20-26.
Kwan, J. L. Y., & Chan, W. (2011). Comparing standardized coefficients in structural equation modeling: A model reparameterization approach. Behavior Research Methods, 43(3), 730–745. https://doi.org/10.3758/s13428-011-0088-6
Le, X. M., Phuong, H. Y., Phan, Q. T., & Le, T. T. (2023). Impact of using analytic rubrics for peer assessment on EFL students’ writing performance: an experimental study. Multicultural Education, 9(3), 41-53. https://doi.org/10.5281/zenodo.7750831
Maatuk, A. M., Elberkawi, E. K., Aljawarneh, S., Rashaideh, H., & Alharbi, H. (2022). The COVID-19 pandemic and E-learning: challenges and opportunities from the perspective of students and instructors. Journal of Computing in Higher Education, 34(1), 21-38. https://doi.org/10.1007/s12528-021-09274-2
Panadero, E., & Jonsson, A. (2020). A critical review of the arguments against the use of rubrics. Educational Research Review, 30, 100329. https://doi.org/10.1016/j.edurev.2020.100329
Panadero, E., Jonsson, A., Pinedo, L., & Fernández-Castilla, B. (2023). Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analytic review. Educational Psychology Review, 35(4), 113.https://doi.org/10.1007/s10648-023-09823-4
Panadero, E., & Romero, M. (2014). To rubric or not to rubric? The effects of self-assessment on self-regulation, performance, and self-efficacy. Assessment in Education: Principles, Policy & Practice, 21(2), 133-148. https://doi.org/10.1080/0969594X.2013.877872.
Paul, R., & Elder, L. (2006). Critical thinking competency standards. Dillon Beach: Foundation for critical thinking.
Paul, R., & Elder, L. (2014). Critical thinking: Intellectual standards essential to reasoning well within every domain of human thought, Part 4. Journal of Developmental Education, 37(3), 34.
Paul, R., & Elder, L. (2019). The thinker's guide to intellectual standards: The words that name them and the criteria that define them. Rowman & Littlefield.
Piaget, J. (2015). The grasp of consciousness (psychology revivals): action and concept in the young child. Psychology Press. https://doi.org/10.2307/3120744
Reynolds-Keefer, L. (2019). Rubric-referenced assessment in teacher preparation: An opportunity to learn by using. Practical Assessment, Research, and Evaluation, 15(1), 8. https://doi.org/doi:10.7275/psk5-mf68.
Reynders, G., Lantz, J., Ruder, S. M., Stanford, C. L., & Cole, R. S. (2020). Rubrics to assess critical thinking and information processing in undergraduate STEM courses. International Journal of STEM Education, 7, 1-15. https://doi.org/10.1186/s40594-020-00208-5
Samadi, S., & Ghaemi, F. (2016). The impact of embedded story structures versus sequential story structures on critical thinking of Iranian intermediate EFL learners. International Journal of Applied Linguistics and English Literature, 5(5), 171-178. https://doi.org/10.7575/aiac.ijalel.v.5n.5p.171
Saxton, E., Belanger, S., & Becker, W. (2012). The Critical thinking analytic rubric (CTAR): investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments. Assessing Writing, 17(4), 251-270. http://dx.doi.org/10.1016/j.asw.2012.07.002
Sosik, J. J., Kahal, S. S., & Piovoso, M. J. (2009). Silver bullet or voodoo statistics? A primer for using the partial least squares data analytic technique in group and organization research. Group and Organization Management, 34(1), 5–36. https://doi.org/10.1177/1059601108329198.
Sword, H. (2019). Snowflakes, splinters, and cobblestones: Metaphors for writing. Innovations in Narrative and Metaphor: Methodologies and Practices, 39-55. https://doi.org/10.1007/978-981-13-6114-2_4
Tashtoush, M. A., Shirawia, N., & Rasheed, N. M. (2024). Scoring rubrics method in performance assessment and its effect of mathematical achievement. Athens Journal of Education, 11(1), 1-22. https://doi.org/10.30958/aje.X-Y-Z
Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: an exploration of undergraduate students’ understanding and perspectives. Assessment & Evaluation in Higher Education, 49(6), 799-809. https://doi.org/10.1080/02602938.2023.2299330
Torrance, H. (2012). Formative assessment at the crossroads: Conformative, deformative and transformative assessment. Oxford Review of Education, 38(3), 323-342. https://doi.org/10.1080/03054985.2012.689693
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (Vol. 86). Harvard University Press.
Yaffe, P. (2022). Why is expository writing so undervalued---and what to do about it? Ubiquity, 2022(July), 1-9. https://doi.org/10.1145/3542739
Yamanishi, H., Ono, M., & Hijikata, Y. (2019). Developing a scoring rubric for L2 summary writing: A hybrid approach combining analytic and holistic assessment. Language Testing in Asia, 9, 1-22. https://doi.org/10.1186/s40468-019-0087-6
Yancey, K. B. (2021). Follow the sources: Notes toward WEC’s contribution to disciplinary writing. Writing-enriched curricula: Models of faculty-driven and departmental transformation. The WAC Clearinghouse, 73-93. https://doi.org/10.37514/PER-B.2021.1299.2.03
Yu, Z. (2021). The effects of gender, educational level, and personality on online learning outcomes during the COVID-19 pandemic. International Journal of Educational Technology in Higher Education, 18(1), 14. https://doi.org/10.1186/s41239-021-00252.
نص كامل:


International Journal of Foreign Language Teaching and Research 

ISSN: 2322-3898-http://jfl.iaun.ac.ir/journal/about

© 2025- Published by Islamic Azad University, Najafabad Branch

img0 

img0 

 

Please cite this paper as follows:

Samadi, S., Hashamdar, M., & Famil Khalili, Gh. H. (2025). A Critical Writing Scoring Rubric: Development and Validation for Iranian EFL Learners in Computer-Mediated Communication. International Journal of Foreign Language Teaching and Research, 13 (54), 85-104.

 

 

 

A Critical Writing Scoring Rubric: Development and Validation for Iranian EFL Learners in Computer-Mediated Communication

 

Sara Samadi1, Mohammad Hashamdar2*, Gholam Hassan Famil Khalili3 

1Ph.D. Candidate, Department of English Language Teaching and Translation, K. C., Islamic Azad University, Karaj, Iran

sara.samadi@kiau.ac.ir

*2Assistant Professor, Department of English Language Teaching and Translation, K. C., Islamic Azad University, Karaj, Iran

mohammad.hashamdar@iau.ir

3Assistant Professor, Department of English Language Teaching and Translation, K. C., Islamic Azad University, Karaj, Iran

khalili@kiau.ac.ir

           

 

Abstract

With the increasing integration of digital platforms in education, assessing critical writing skills in Computer-Mediated Communication (CMC) contexts has become essential. Despite the widespread use of rubrics in educational assessment, there remains a lack of validated instruments specifically designed to evaluate Iranian EFL learners' critical writing skills in CMC environments. This study aims to develop and validate an analytic scoring rubric to assess critical writing effectively. The research involved 236 Iranian EFL learners and 10 EFL/ESL instructors, ensuring a diverse range of perspectives. The rubric development process was guided by Paul and Elder’s (2019) Intellectual Standards and refined through expert feedback, thematic analysis of semi-structured interviews, and pilot testing with iterative modifications. The final rubric comprises four key components: (1) Clarity, Accuracy, and Precision (CAP); (2) Relevance and Logic (RL); (3) Depth and Significance (DS); and (4) Breadth and Fairness (BF). Statistical analyses, including factor analysis and structural equation modeling (SEM), confirmed the rubric’s reliability and validity, demonstrating strong internal consistency and construct validity. The findings underscore the rubric’s effectiveness in fostering critical writing skills, offering valuable implications for language teachers, learners, and researchers seeking robust assessment tools in CMC-based education.

Keywords: Critical Writing, Computer-Mediated Communication, Intellectual Standards, Iranian EFL Learners, Rubric Development

 

 

 

روبریک نمره‌دهی نگارش انتقادی: توسعه و اعتبارسنجی برای زبان‌آموزان ایرانی زبان انگلیسی به عنوان زبان خارجی در ارتباطات با واسطه کامپیوتر

با افزایش ادغام پلتفرم‌های دیجیتال در آموزش، ارزیابی مهارت‌های نگارش انتقادی در زمینه‌های ارتباطات با واسطه کامپیوتر (CMC) ضروری شده است. علیرغم استفاده گسترده از روبریک‌ها در ارزیابی آموزشی، همچنان فقدان ابزارهای معتبری که به طور خاص برای ارزیابی مهارت‌های نگارش انتقادی زبان‌آموزان ایرانی زبان خارجی به عنوان زبان خارجی در محیط‌های CMC طراحی شده باشند، وجود دارد. هدف این مطالعه توسعه و اعتبارسنجی یک روبریک نمره‌دهی تحلیلی برای ارزیابی مؤثر نگارش انتقادی است. این تحقیق شامل 236 زبان‌آموز ایرانی زبان خارجی و 10 مربی زبان انگلیسی به عنوان زبان دوم و زبان دوم به عنوان زبان دوم بود که طیف متنوعی از دیدگاه‌ها را تضمین می‌کرد. فرآیند توسعه روبریک توسط استانداردهای فکری پاول و الدر (2019) هدایت شد و از طریق بازخورد متخصصان، تحلیل موضوعی مصاحبه‌های نیمه ساختار یافته و آزمایش آزمایشی با اصلاحات مکرر اصلاح شد. روبریک نهایی شامل چهار مؤلفه کلیدی است: (1) وضوح، دقت و صحت (CAP)؛ (2) ارتباط و منطق (RL)؛ (3) عمق و اهمیت (DS)؛ و (4) وسعت و انصاف (BF). تحلیل‌های آماری، از جمله تحلیل عاملی و مدل‌سازی معادلات ساختاری (SEM)، پایایی و اعتبار روبریک را تأیید کردند و همسانی درونی قوی و اعتبار سازه را نشان دادند. یافته‌ها بر اثربخشی روبریک در پرورش مهارت‌های نوشتاری انتقادی تأکید دارند و پیامدهای ارزشمندی را برای معلمان زبان، زبان‌آموزان و محققانی که به دنبال ابزارهای ارزیابی قوی در آموزش مبتنی بر CMC هستند، ارائه می‌دهند.

کلمات کلیدی: نگارش انتقادی، ارتباط با واسطه کامپیوتر، استانداردهای فکری، زبان‌آموزان ایرانی زبان انگلیسی به عنوان زبان خارجی، توسعه روبریک

 

Introduction

Over the past three decades, educators have utilized rubrics as influential assessment tools in higher education. Rubrics are cornerstones of academic success and are crucial in improving the quality of learners’ education. To enhance the quality of teaching and learning, teachers’ assessment literacy is essential for effective classroom assessment.

It is noteworthy that using a well-designed rubric offers a clear framework for activities, fosters peer and self-evaluation, expedites the grading process, serves as an authentic tool for teachers to provide meaningful feedback, and creates opportunities for learning and growth (Ragupathi & Lee, 2020). In the same vein, Yu (2021) put under the spotlight the significance of providing feedback on students' writing as a reflective experience that allows teachers to understand the importance of feedback in enhancing student writing, thereby supporting students' learning. Moreover, Chowdhury (2019) found that implementing rubrics is critical for higher education institutions seeking to shift from conventional evaluation methods to authentic assessment.

Furthermore, according to Paul and Elder (2006), the significance of objectivity in writing assessment cannot be understated, and critical writers should base their texts on essential criteria, including precision, logic, clarity, impartiality, and depth. Paul and Elder (2019) identified nine intellectual standards that are used to assess the quality of reasoning and critical thinking. These standards are widely applied in education to help students develop strong arguments and well-supported claims. According to Paul and Elder (2014), critical writing is a key component of critical thinking, and to express oneself effectively in language, it is essential to engage in the process of critical thinking. Thus, these criteria are directly connected to critical writing. Thus, drawing upon Paul and Elder’s seminal work, The Thinker’s Guide to Intellectual Standards: The words that name them and the criteria that define them (2019), the proposed critical writing rubric offers a set of criteria to assess different aspects of critical writing, such as clarity, accuracy, precision, relevance, logic, breadth, fairness, depth, and significance.

On the other hand, the integration of technology in education has changed traditional learning to online learning throughout the global COVID-19 pandemic. Although this shift in teaching methods has opened up many concerns and challenges, it provides an effective means of connecting students with comparable writing skills and critical thinking abilities (Bekar & Christiansen, 2021). In the same vein, Yu (2021) underscored that, notwithstanding challenges such as plagiarism, user trustworthiness, misuse of technology, and issues of reliability and validity, digital learning platforms have become pervasive modes of instruction in classrooms worldwide. Moreover, Maatuk et al. (2022) noted that online distance learning provides students with the opportunity to access high-quality education anytime and from vast distances. Gupta and Gupta (2021) argued that online platforms provide greater access to learning materials and peer feedback, but they also require well-designed rubrics to ensure the validity and reliability of assessment in these digital contexts.

Despite the substantial body of research investigating the effects of using rubrics to assess writing skills in language teaching (Keller et al., 2023), little attention has been given to the development and validation of critical writing rubrics in CMC contexts. To address this research gap, the present study aims to develop and validate an analytic rubric for assessing critical writing, providing a robust framework for educators and students to make informed judgments about the essential quality of writing skills in alignment with educational goals in CMC contexts.

 

Review of the Literature

The development and validation of rubrics have been widely explored in educational assessment as tools to provide objective and reliable scores of learners’ performance. A strong theoretical framework for designing critical writing rubrics is grounded in both critical thinking and educational frameworks for teaching writing. According to Paul and Elder (2019), critical thinking is guided by intellectual standards like clarity, accuracy, and fairness, which are crucial for evaluating writing skills effectively. Numerous researchers emphasized the significance of critical thinking in education, stating that students who can reason logically tend to succeed in both educational growth and future life (Samadi & Ghaemi, 2016). Bloom’s Taxonomy (1956) offers a valuable structure for creating critical writing rubrics, particularly when addressing higher-order thinking skills such as analysis, synthesis, and evaluation.

In the early 70s, as the process approach became popular in the United States, rubrics changed from simple assessment tools into frameworks that provide learners with feedback on how their writings align with specific criteria and offer suggestions for writing skills improvement (Ferris, 2009). Piaget (1976, as cited in The Grasp of Consciousness: Psychology Revivals, 2015) highlighted that writing is a way to express thoughts, allowing learners to demonstrate higher-order thinking skills, like reasoning, problem-solving, and reflection. Moreover, Vygotsky (1978) noted that learners develop writing and thinking skills more effectively with guidance from more knowledgeable others. He underscored the role of scaffolding, which refers to the support provided by teachers or peers, in helping learners accomplish tasks they cannot perform independently. Along the same lines, critical writing rubrics will encourage students not just to repeat information, but to apply critical thinking and develop well-reasoned arguments.

Brookhart (2013) asserted that rubrics are especially useful in assessing complex skills such as critical writing, which require multiple dimensions, including argumentation, organization, evidence, and critical thinking. In the same vein, Panadero et al. (2024) argued that, in the last decades, rubrics have gained widespread recognition and enhanced students’ metacognitive skills, academic performance, and self-regulatory strategies, especially in online learning environments. They also suggested that rubrics can help learners overcome challenges in the learning process. Tashtoush et al. (2024) highlighted that rubrics provide consistent evaluation, offer precise requirements, facilitate meaningful feedback, foster a deeper understanding of learners’ learning, engage learners in self-assessment, and minimize subjectivity in the evaluation processes. In addition, rubrics are essential for helping learners increase self-assessment and provide constructive feedback on one another's work (Keller et al., 2023).

Furthermore, Brookhart (2013) stated that a rubric is a comprehensible set of standards and scoring strategies that includes detailed explanations of different performance levels to evaluate students’ work in various fields and provide meaningful feedback to teachers and students. Thus, creating and applying rubrics are crucial in educational settings (Panadero & Jonsson, 2020). According to Farzana (2023), for evaluating students’ performance, there are four major types of rubrics: analytic, holistic, general, and task-specific; however, Tashtoush et al. (2024) noted that scoring rubrics in performance assessment are divided into holistic and analytic as they gather information about students’ performance levels (e.g., L2 writing assessment), and work to improve their performance. Holistic rubrics provide a single, overall score for students’ performance (Aloufi & Taani, 2022), while analytic rubrics offer more detailed diagnostic feedback, providing valuable insights that enhance both self-learning and peer-learning. Thus, analytic rubrics are more reliable than holistic ones (Gupta & Gupta, 2021). In addition, Dappen et al. (2008) argued that by using analytic rubrics, learners will improve their writing skills more effectively.

Apart from the importance of rubrics, educators must understand the crucial role of thinking in writing because the ability to think critically enables individuals to take targeted actions for improvement. If a product of intellectual work such as writing lacks logic, coherence, and organization, it cannot play a role in any academic discipline and is incomprehensible. Writing that lacks discipline and criticality tends to be vague and inconsistent. To achieve this quality, intellectual standards should guide the development of rubrics to ensure quality and coherence (Paul & Elder, 2019). Moreover, Saxton et al. (2012) pointed out that summarizing a student’s critical thinking ability into a single holistic score results in a loss of valuable diagnostic information. This information is crucial for guiding teachers’ instructional decisions. Therefore, analytic rubrics are more effective tools for assessing learners’ critical thinking subskills.

Several scholarly articles have explored and refined the application of rubrics in education. For instance, Reynders et al. (2020) developed two rubrics to assess undergraduate STM students’ critical thinking and information processing skills. Using ELIPSS rubrics allowed students to reflect on their work and understand their performance. Instructors also reported that these tools had facilitated their teaching process. In another research project, Yamanishi et al. (2019) developed a scoring rubric for L2 summary writing tailored to EFL students in Japanese universities. They examined the applicability of analytic and holistic five-dimensional scoring rubrics. This flexible combination of holistic and analytic assessments significantly influenced the evaluation and teaching of second language summary writing in the Japanese EFL context, addressing the diverse needs of teachers and the abilities of students. Likewise, Le et al. (2023) analyzed the writing skills of 22 university students using analytic rubrics for peer assessment. The results showed a significant positive difference in their writing performance, especially in the use of vocabulary and grammar structures. However, the intervention had little impact on learners with sufficient writing performance. Furthermore, Taylor, Kisby, and Reedy (2024) asserted that rubrics in higher education are widely acknowledged for clarifying assessment expectations for the intended recipients, which can enhance student confidence and reduce anxiety related to assessments.

Although students had positive attitudes toward rubrics, some perceived them as insights into teachers’ thought processes and expectations rather than guides to meet learners’ standards. Reynolds-Keefer (2019) cautions that some learners perceived rubrics as simple checklists, potentially causing them to miss significant learning objectives.

Finally, the rapid advancement of technology has significantly expanded the scope of CMC in teaching and learning contexts. CMC, defined as any communication that occurs either synchronously or asynchronously, has transformed the educational landscape, offering new possibilities for assessment (Bekar & Christiansen, 2021). In the same vein, Stevens and Levi (2013) proposed that continuous and real-time feedback, which is critical for student learning and development, can improve the effectiveness of assessment.

This literature underscores the significance of a rigorous approach to developing and validating a critical writing scoring rubric in CMC contexts. Rubrics should evolve beyond their traditional role as scoring tools to become dynamic instruments that provide ongoing feedback to improve learning processes and rresults, particularly when integrated with CMC technologies. Thus, an analytic critical writing rubric needs to be developed and validated to break down the writing process into discrete components and offer detailed diagnostic feedback. Moreover, incorporating CMC technologies into the development and use of critical writing rubrics facilitates more efficient communication, supports deeper engagement in critical writing, and enhances higher-order thinking skills. Through the use of technologies, the rubric’s criteria were continuously refined based on feedback collected from both instructors and students in real-time. The digital tools ensured that the rubric evolved in line with participants’ needs and the feedback provided. Google Docs enabled seamless collaboration between the research team and experts, ensuring that the rubric was relevant and comprehensive. Furthermore, Google Forms allowed the research team to gather feedback on the rubric from a large sample of participants, streamlining the evaluation process and making it possible to adapt the rubric iteratively. 

 

Objectives of the Study

The primary goal of this study is to develop and validate a comprehensive critical writing scoring rubric tailored to assess Iranian EFL learners’ writing performance in Computer-Mediated Communication (CMC) contexts. Given the increasing reliance on digital platforms for language learning and assessment, this research seeks to provide a standardized and reliable instrument that educators can use to evaluate students’ ability to engage in critical writing with depth, coherence, and analytical rigor.

To achieve this overarching aim, the study pursues the following detailed objectives:

--To identify the fundamental components of critical writing within CMC environments
This study aims to establish the essential criteria and dimensions of critical writing by drawing on Paul and Elder’s (2019) Intellectual Standards. Through an extensive literature review and expert consultations, the study identifies the most crucial aspects of critical thinking that should be reflected in EFL learners’ writing assessments.

--To construct an analytic rubric that effectively measures critical writing performance
The study seeks to develop a well-structured, multi-dimensional rubric that breaks down critical writing into distinct and measurable components. The rubric is designed to assess students’ writing in terms of Clarity, Accuracy, and Precision (CAP); Relevance and Logic (RL); Depth and Significance (DS); and Breadth and Fairness (BF). These categories align with recognized intellectual standards that contribute to high-quality academic writing.

--To ensure the validity and reliability of the developed rubric
A key objective is to establish the rubric’s statistical soundness through rigorous validation procedures. This includes conducting factor analysis and structural equation modeling (SEM) to confirm that the rubric accurately measures what it is intended to assess. Additionally, expert evaluations and pilot testing are employed to refine and enhance its effectiveness.

--To explore Iranian EFL learners’ and instructors’ perceptions of the rubric’s practicality and effectiveness. Understanding how students and educators perceive the newly developed rubric is crucial for its successful implementation. This study gathers qualitative feedback through semi-structured interviews, allowing participants to share their experiences and insights on how the rubric influences writing performance, self-assessment, and instructional practices.

--To investigate the impact of rubric-based assessment on students' critical writing skills
The study examines whether the use of the rubric leads to measurable improvements in students' writing. By analyzing students’ written work before and after rubric-based instruction, the research assesses how well learners adopt the intellectual standards of critical writing, particularly in terms of argumentation, logical reasoning, depth of analysis, and engagement with multiple perspectives.

--To provide pedagogical recommendations for EFL instructors and curriculum developers
Beyond the development of the rubric, this study seeks to offer practical guidance for educators on how to integrate rubric-based assessment into their teaching methodologies. Recommendations include strategies for using the rubric in peer review, formative assessment, self-assessment, and instructor feedback, ensuring its effective implementation in both online and traditional EFL classrooms.

--To contribute to the advancement of digital assessment tools in EFL education
Given the rapid transition to online learning, this research highlights the need for robust assessment tools that align with CMC-based instruction. The study explores how technology-enhanced feedback mechanisms, such as Google Docs and digital peer review platforms, can be integrated with the rubric to facilitate more interactive, reflective, and student-centered learning experiences.

 

Research Questions

This study seeks to address the following research questions:

RQ1. What are the principal components of the critical writing scoring rubric based on Paul and Elder’s (2019) Intellectual Standards?

RQ2.How effective is the newly developed rubric in assessing Iranian EFL learners' critical writing skills in CMC contexts?

RQ3.Does the rubric demonstrate validity and reliability as a standardized assessment tool for critical writing?

RQ4.To what extent do Iranian EFL learners and instructors perceive the rubric as a practical and effective tool for evaluating critical writing?

RQ5.What are the pedagogical implications of implementing this rubric in EFL writing assessment and instruction?

 

Research Hypotheses

Based on the research questions and previous literature, the following hypotheses were formulated:

H1: The developed critical writing rubric includes distinct and measurable components aligned with Paul and Elder’s (2019) Intellectual Standards.

H2: The rubric demonstrates high internal consistency and reliability in assessing Iranian EFL learners’ critical writing skills.

H3: The rubric exhibits strong construct validity as confirmed through factor analysis and structural equation modeling (SEM).

H4: Iranian EFL learners and instructors perceive the rubric as an effective tool for assessing and improving critical writing in CMC environments.

H5: The implementation of the rubric enhances learners’ ability to apply intellectual standards, resulting in improved clarity, coherence, and depth in writing.

 

Methodology

This study employed an exploratory mixed-methods approach to collect and analyze both qualitative and quantitative data in the development and evaluation of a critical writing rubric within CMC contexts.

 

Participants

The study was conducted in multiple phases, involving 236 advanced EFL learners (186 females, 50 males) and 10 EFL/ESL teachers (6 females, 4 males) with 10 to 16 years of expertise in teaching assessment and writing. Due to the focused nature of the research and the need for participants with specific experiences or knowledge, participants were selected through non-probability purposive sampling from various universities and institutes in Tehran and Karaj, Iran. 

 

Table 1

Demographic Information of the Teachers

Participants’ Characteristics

 

Frequency

Age range

34>50

10

Degree

M.A. (Ph.D. Candidate)

7

 

Ph.D.

3

Major of study

TEFL

10

 

Teaching experience

10

11

12

>16

4

1

2

3

Gender

 

Female

Male

6

4

Total

 

10


Table 2

Demographic Information of the Students

Participants’ Characteristics

 

Frequency

Age range

19>50

236

Degree

B.A. Students

B.A.

M.A. Students

 

148

72

16

Major of study

English Translation

TEFL

220

16

Gender

 

Female

Male

186

50

Total

 

236

 

Instruments

This study utilized multiple instruments at different stages to ensure comprehensive data collection. These included a thorough literature review, a Critical Thinking Questionnaire, a series of semi-structured interviews developed by the researchers, the newly developed Critical Writing Questionnaire, the newly developed Critical Writing Rubric, and computer-mediated forums for collaborative reflection. Each instrument is detailed as follows.

 

Procedure

First, a Critical Thinking Questionnaire was administered (Kobylarek et al. 2022) to evaluate instructors' and students' awareness and levels of critical thinking. For developing the Critical Writing Questionnaire, semi-structured interviews, consisting of nine questions, were conducted based on a comprehensive literature review and Paul and Elder’s (2019) nine intellectual standards.

In light of the rubric’s crucial role in ensuring consistency and fairness in assessment, expert opinions were sought on its items and elements to enhance validity. Each question in the semi-structured interviews addressed a specific intellectual standard. Since the interviews were semi-structured, the interviewer encouraged the interviewees (10 EFL/ESL instructors) to provide detailed responses, offering the researchers in-depth insights and enriched data (Cohen et al., 2002).

For convenience, the interviews were conducted in CMC contexts, utilizing Zoom and Telegram for real-time interaction and Google Docs for collaborative, written responses, and follow-ups. It is noteworthy that all participants provided informed consent after a thorough explanation of the study, which included recording permissions, confidentiality measures, and anonymity safeguards. Privacy was ensured during the interviews, and data were manually transcribed and coded to identify underlying themes. Following Braun and Clark’s (2006) thematic analysis, key themes were extracted. After the seventh interview and a thorough examination of the participants’ responses, it was found that the emergence of new themes became less frequent, and the data reached a saturation point. The analysis of the interview responses revealed valuable insights into teachers’ perspectives on critical writing and their application of the intellectual standards in their teaching practices (see Appendix A for the complete interview protocol).

The interviews were piloted under the same conditions with four experts who were representative of the research target population. This allowed the researchers to evaluate the effectiveness of the interview questions and identify areas for improvement. To ensure the reliability of the data collected from the semi-structured interviews, the researchers utilized low-inference descriptors, which included direct quotations from participants. Furthermore, member checking was conducted to validate the interpretations by comparing them with participants’ perspectives and statements (Taylor, Kisby, & Reedy, 2024). The semi-structured format of the interviews allowed the researchers to adjust the questions when earlier responses had already covered the core of a later question.

During the six-session intensive critical writing course, 236 EFL students were introduced to critical writing standards to enhance their awareness of high-quality critical writing processes. Critical writing standards were explicitly taught to students as a guide to help them understand the constituents of high-quality critical writing. CMC platforms facilitated resource sharing, allowing instructors to provide digital feedback on assignments and share electronic documents. The most useful Google products for this project were Google Forms and Google Docs. Google Forms is a tool that allows users to create online questionnaires that can be shared with others. On the other hand, Google Docs facilitates online cooperation and collaboration, allowing instructors and learners to work together within a shared space. It allows editing a single document synchronously and asynchronously (Blau & Caspi, 2009). Instructors and learners can write and edit critical writings in real-time or save them for later editing and revision, accessible from anywhere and at any time (Yang, 2010). These technologies were central in our efforts to make critical writing content accessible to students. Students were asked to write critically, and they received feedback from the instructors through Google Docs on different aspects of critical writing, such as grammatical accuracy and critical thinking. They could also post comments, ask questions, edit their classmates’ writings, and view their grades. 

Moreover, prior to developing the critical writing rubric, the researchers developed and validated a 50-item Likert-scale critical writing questionnaire (see Appendix B) to ensure the objectivity of the rubric’s statements. The construct validity of the instrument was evaluated by conducting exploratory and confirmatory factor analyses. The factor analysis results indicated that all items contributed to their respective components and were loaded onto four components: 1) clarity, accuracy, precision, 2) relevance, logic, 3) depth, significance, and 4) breadth and fairness. During the refinement process, several items were eliminated from the initial pool through a triangulation of evidence, factor analysis results, experts’ opinions, and an evaluation of the questionnaire’s overall goodness-of-fit. The final draft of the questionnaire, with 50 items, was distributed to 236 students via Google Forms after 2 weeks. All 236 participants responded to every item in the questionnaire. However, ten items were later found vague and inappropriate; therefore, they were excluded. Through a series of factor analytic procedures and structural equation modeling, the critical writing questionnaire with 40 Likert-scale items was validated to confirm the goodness-of-fit and objectivity of its statements. Drawing on the results of this instrument, a critical writing scoring rubric was developed (see Appendix C).

To ensure the validity of the critical writing rubric, four advanced writing professors were asked to evaluate the instrument. They provided feedback on its suitability for assessing the intended construct. Based on their comments, revisions were made, and the researchers developed an analytic rubric. Subsequently, the students were asked to write a critical essay, and their writings were evaluated using the newly developed critical writing rubric. The rubric’s reliability and validity were assessed using Composite Reliability (CR), Average Variance Extracted (AVE), and Maximum Shared Variance (MSV).

It should be noted that this study focuses solely on the development and validation process of the critical writing rubric. The evaluation of the rubric’s effectiveness and application in instructional setting was conducted in a separate study.

 

Results

Thematic Analysis

A variety of analytical techniques, including thematic analysis, EFA, CFA, and SEM, were implemented to provide valuable insights that support the reliability and validity of the study. In the qualitative phase of this study, the interviews were manually transcribed and coded using thematic analysis, which offered key insights into the development of the critical writing rubric. The results indicate that the majority of the respondents strongly agreed that a critical writing rubric is essential for understanding expectations and guiding learners on what and how to write critically. Nine major themes emerged from the responses to nine interview questions: 1) clarity, 2) precision, 3) accuracy, 4) relevance, 5) breadth, 6) depth, 7) fairness, 8) logic, and 9) significance were derived from the responses to nine interview questions.

In order to illustrate the key themes, two representative responses were carefully selected. In response to question one, one teacher stated, ‘I spend much of the class time explaining my expectations. I emphasize the importance of using clear and concise language to express ideas and enable effective communication, which is critical for achieving higher scores.’ This response highlights the importance of clarity in critical writing. For question five, another teacher explained, “I used various strategies to help my students develop deeper analytical skills in their critical writing. One of the most effective methods was engaging them in discussions to defend their ideas, address counterarguments, consider the potential consequences of the problem, and reflect on their initial thoughts.” This response emphasizes the importance of conducting an in-depth analysis and understanding of the broader context surrounding the problem.

Qualitative insights from the interviews revealed that teachers perceived the rubric as a practical and clarifying tool for identifying strengths and areas for improving critical writing skills in CMC. Participants noted that the rubric’s criteria foster structured, critical, and reflective writing practices.

The quantitative phase of this study aimed to address research questions by analyzing numerical data, as outlined in the following sections.

 

Reliability Measure

Before conducting statistical analyses, the researchers assessed the reliability of the data, obtaining Cronbach’s alpha of .990. This value was interpreted as demonstrating a remarkably high level of internal reliability across the 50 items in the questionnaire. It is important to note that an alpha value of this magnitude may indicate redundancy among some items. Consequently, all the items were retained and subjected to factor extraction analysis. Moreover, the excellent Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was calculated at 0.966, indicating that the data was highly suitable for factor analysis as it suggests a strong correlation among the variables.

 

The application of exploratory factor analysis (EFA)

To ensure the valid and unbiased development of the critical writing construct, an exploratory factor analysis (EFA) was accomplished despite the newly developed questionnaire being initially designed to align with Paul and Elder’s intellectual standards model (2019). The analysis employed Oblimin rotation and was based on responses from 236 participants. The adequacy of the dataset for factor analysis was measured through the Kaiser-Meyer-Olkin (KMO) measure, which evaluates the strength of correlations among variables. According to Kaiser (1970), a KMO value above .60 is acceptable, while Field (2013) identified 0.30 as the minimum threshold for sampling adequacy at the variable level. The overall KMO value for this study was 0.966, indicating excellent sampling adequacy and strong interrelationships among the variables, thus justifying the use of factor analysis.

The model fit values were evaluated by using a chi-square test. Fit indices suggest that values below 5 represent a moderate but acceptable fit, whereas values below 3 indicate a strong fit. In this study, the chi-square value (15,247.801, df = 1225, p < 0.001) demonstrated an acceptable fit for the model. Moreover, Bartlett’s test of sphericity validated the appropriateness of the data for factor analysis by identifying a significant difference (p = 0.00, < 0.05) between the observed correlation matrix and the identified matrix.

Furthermore, communality values were examined to ensure the robustness of the analysis. Since establishing a substantial dataset is frequently recommended in the literature for conducting EFA, the commonality values are always crucial. Based on Field’s (2013) recommendations, a cutoff value of 0.30 was applied. The results indicated sufficient shared variance among the variables, confirming the suitability of the data for EFA.

 

Factor extraction and retention

A parallel analysis (PA) was conducted, during which the eigenvalues from the exploratory factor analysis (EFA) were compared to a set of uncorrelated eigenvalues generated by the Monte Carlo algorithm. The results showed that all observed eigenvalues in the EFA matrix exceeded the corresponding uncorrelated eigenvalues produced by the Monte Carlo simulation, confirming the appropriateness and validity of the observed eigenvalues (see Table 3).

Nine components with eigenvalues greater than 1 were identified during the factor retention process, consistent with Kaiser’s criterion (Kaiser, 1970). However, factors with marginally different variances were excluded before proceeding with further statistical analyses. The scree plot (see Fig. 1) highlighted four dominant components, as the eigenvalues significantly declined after the fourth component. These four retained factors had eigenvalues exceeding 2, with the highest eigenvalue reaching 33.88. Collectively, the four components explained 78.62% of the total variance, indicating a substantial contribution to the overall data structure.

Given the significant variance explained by these components, the focus was placed on these four factors, and items with minimal factor loadings were removed to enhance the rubric’s quality. The scree plot provided additional confirmation, displaying a distinct elbow point after the fourth component, which supported the retention of four factors. The first factor accounted for the majority of the variance at 67.78%, while the remaining three contributed progressively smaller but still meaningful portions. In conclusion, as demonstrated by the scree plot and Kaiser’s criterion (eigenvalues > 1), retaining four factors was deemed appropriate for the analysis.

 

 

 

 

 

 

 

 

 

Fig. 1 

The distribution of the extracted factors

img0 

 

The component matrix was analyzed to identify problematic items and determine their contribution to variations within each component. Items exhibiting cross-loadings were carefully evaluated, and those with cross-loading values below 0.30 (Sosik et al. 2009) were excluded from the dataset. Following a second content analysis conducted by two instructors, the theoretical framework for the critical writing questionnaire was finalized through structural equation modeling (SEM) in IBM SPSS AMOS 26 (see Fig. 2).

During the final phase of the SEM analysis, ten additional items were removed, reducing the total number of items to 40. This refined version of the structural model retained 40 high-performing items, enhancing the construct validity of the instrument.

 

Developing the initial structural model with the remaining 40 items

The application of confirmatory factor analysis (CFA)

Although the initial structural model demonstrated an adequate goodness-of-fit index (GFI), it was further revised to enhance its overall fit. This process involved identifying more appropriate underlying components and refining the critical writing questionnaire’s pathways. Adjustments were made to the statistical framework by addressing fit issues and modifying specific items and components. These revisions resulted in the finalized critical writing rubric, incorporating new correlational pathways and underlying factors. In the last phase of the SEM analysis, ten items were removed, leaving a total of 40 items in the final model.

Following the removal of items with standardized regression weight estimates below 0.30 (Kwan & Chan, 2011), the first structural model, which comprised four key components, retained the remaining 40 items in the critical writing questionnaire. The model’s surface structure was designed to align with the nine components of Paul and Elder’s intellectual standards (2019). However, to verify the credibility of the model’s fit (see Fig. 2), a confirmatory factor analysis (CFA) was performed for both theoretical and statistical reasons.

 

Table 3

Total Variance Explained

Component

Initial Eigenvalues

Extraction Sums of Squared Loadings

Rotation Sums of Squared Loadings

Total

% of Variance

Cumulative %

Total

% of Variance

Cumulative %

Total

% of Variance

Cumulative %

1

33.889

67.777

67.777

33.889

67.777

67.777

14.057

28.114

28.114

2

2.564

5.128

72.905

2.564

5.128

72.905

11.028

22.055

50.169

3

1.472

2.944

75.849

1.472

2.944

75.849

9.926

19.851

70.021

4

1.385

2.771

78.620

1.385

2.771

78.620

4.300

8.599

78.620

5

.936

1.873

80.493

 

 

 

 

 

 

6

.844

1.687

82.180

 

 

 

 

 

 

7

.780

1.560

83.740

 

 

 

 

 

 

8

.682

1.364

85.104

 

 

 

 

 

 

9

.521

1.043

86.146

 

 

 

 

 

 

10

.490

.981

87.127

 

 

 

 

 

 

11

.444

.887

88.015

 

 

 

 

 

 

12

.399

.799

88.814

 

 

 

 

 

 

13

.372

.743

89.557

 

 

 

 

 

 

14

.355

.710

90.267

 

 

 

 

 

 

15

.326

.652

90.919

 

 

 

 

 

 

16

.324

.647

91.566

 

 

 

 

 

 

17

.305

.610

92.176

 

 

 

 

 

 

18

.280

.561

92.737

 

 

 

 

 

 

19

.272

.544

93.280

 

 

 

 

 

 

20

.234

.467

93.747

 

 

 

 

 

 

21

.224

.449

94.196

 

 

 

 

 

 

22

.211

.422

94.618

 

 

 

 

 

 

23

.201

.402

95.020

 

 

 

 

 

 

24

.185

.370

95.390

 

 

 

 

 

 

25

.175

.350

95.740

 

 

 

 

 

 

26

.165

.331

96.071

 

 

 

 

 

 

27

.157

.314

96.385

 

 

 

 

 

 

28

.146

.292

96.677

 

 

 

 

 

 

29

.138

.276

96.952

 

 

 

 

 

 

30

.132

.264

97.216

 

 

 

 

 

 

31

.124

.249

97.465

 

 

 

 

 

 

32

.119

.238

97.703

 

 

 

 

 

 

33

.109

.218

97.921

 

 

 

 

 

 

34

.107

.215

98.136

 

 

 

 

 

 

35

.100

.200

98.336

 

 

 

 

 

 

36

.089

.178

98.514

 

 

 

 

 

 

37

.085

.169

98.684

 

 

 

 

 

 

38

.079

.159

98.842

 

 

 

 

 

 

39

.073

.146

98.988

 

 

 

 

 

 

40

.068

.136

99.125

 

 

 

 

 

 

41

.064

.127

99.252

 

 

 

 

 

 

42

.059

.119

99.371

 

 

 

 

 

 

43

.051

.102

99.473

 

 

 

 

 

 

44

.050

.100

99.573

 

 

 

 

 

 

45

.045

.091

99.664

 

 

 

 

 

 

46

.039

.077

99.741

 

 

 

 

 

 

47

.038

.077

99.818

 

 

 

 

 

 

48

.037

.073

99.891

 

 

 

 

 

 

49

.032

.064

99.954

 

 

 

 

 

 

50

.023

.046

100.000

 

 

 

 

 

 

 

Construction of the Critical Writing Rubric with Four Latent Variables

The Critical Writing Rubric is the assessment tool linked to all four latent variables. CAP (construct 1) stands for “clarity, accuracy, and precision,” RL (construct 2) refers to “relevance and logic,” DS (construct 3) represents “depth and significance,” and BF (construct 4) signifies “breadth and fairness.”

 

Results of the finalized Critical Writing Rubric’s Goodness-of-Fit

According to Hair Jr. et al. (2013) guidelines, the researchers reassessed the goodness-of-fit of the Critical Writing Rubric. For a sample size of 236 participants or more, an acceptable factor loading should exceed 0.3. RMSEA was reported as 0.088, which is considered an acceptable fit as it is close to the permissible range (≤ 0.08), though it is slightly above the threshold. The confidence interval (0.083–0.093) shows that RMSEA is unlikely to fall below 0.08, supporting the fit quality. Moreover, other goodness-of-fit indices were above the critical value of 0.90. The CFI value of 0.918, the IFI value of 0.919, and the TLI value of 0.906 demonstrate that the model achieves an acceptable fit. In this analysis, the researchers successfully achieved a satisfactory measure of goodness-of-fit (GFI) exceeding 0.9.

 

Path analysis

In addition to performing factor analysis, a path analysis was conducted to assess the significance of the relationships among the four latent components and the overall construct of Critical Writing within the framework of structural equation modeling (SEM). Fig. 1 shows both direct and indirect paths between the main components and the construct, illustrated by arrows.

In the direct and indirect path models, the researchers deliberately removed unrelated paths from the equation to assess their effects separately. Among the components, depth and significance (DS) have the most substantial impact on the Critical Writing Rubric (0.49), suggesting that this construct plays the most significant role in predicting or influencing critical writing performance. Among the four path values, only components 1 (CAP: clarity, accuracy, and precision) and 2 (RL: relevance and logic) demonstrated moderate effects, with β values of 0.49 and 0.35 respectively. Components 3 (DS) and 4 (BF: breadth and fairness) exhibited comparatively weaker effects (β = 0.11, β =0.13). The standardized estimates of the covariance coefficients between the main components were computed with values of 0.86 between components 1 and 2, 0.98 between components 2 and 3, 0.91 between components 3 and 4, 0.89 between components 1 and 4, 0.86 between components 1 and 3, and 0.89 between components 2 and 4. The covariance values confirmed a strong covariance relationship between the components.

 

Fig. 2

Schematic Model of the Critical Writing Rubric

img0 

 

Validity and Composite Reliability (CR)

To evaluate the composite reliability (CR) of the components identified in the SEM model, the researchers calculated the standardized regression weights and the correlation
values. These measures are critical for evaluating construct reliability and validity. According to Hair Jr. et al. (2020), a cutoff point of 0.60 and above has been designated for CR. 

 

Table 4

Validity and Reliability Table

Component

CR

AVE

MSV

MaxR

Interpretation

CAP

0.95

0.75

0.79

0.96

High CR confirms excellent reliability. AVE > 0.5 shows good convergent validity.

RL

0.94

0.74

0.85

0.96

High CR and AVE indicate reliability and convergent validity, but MSV is high.

DS

0.92

0.69

0.85

0.94

Strong reliability and AVE, but MSV suggests overlap with RL and CAP.

BF

0.95

0.78

0.65

0.97

Excellent reliability and validity with lower MSV, ensuring discriminant validity.

 

The CR values for components 1, 2, 3, and 4 were above 0.70 (0.95, 0.94, 0.96, and 0.95, respectively), exceeding 0.9 and indicating excellent reliability.

 

Convergent Validity:

All constructs satisfied the AVE > 0.5 and CR > 0.7 threshold, which confirmed the effectiveness of the observed indicators in capturing their respective latent constructs.

 

Discriminant Validity:

RL and DS exhibit a high Maximum Shared Variance (MSV) of 0.85, which is close to their Average Variance Extracted (AVE) values. This indicated a significant overlap between the two components that suggested discriminant validity issues.

Components like CAP and BF demonstrated good discriminant validity as their AVE values are greater than MSV. Maximum shared variance (MSV) values were derived to assess the convergent validity. MSV represents the level of variance shared between constructs. For discriminant validity, MSV must be less than AVE (MSV < AVE).

RL and DS exhibited a high MSV of 0.85, which was very close to their AVE, while CAP and BF satisfied the MSV-AVE condition that supported discriminant validity.

Moreover, to assess discriminant validity, the researchers evaluated the average variance extracted (AVE). AVE measures the proportion of variance explained by a construct compared to the variance due to measurement error. In large sample sizes, the estimation typically leads to lower AVE values due to the sensitivity of indicator item loading (Henseler et al., 2015). Accordingly, the significance of discriminant validity was determined based on acceptable measures of CR AVE ≥ 0.5, which indicates good convergent validity. MaxR represents the upper bound of construct reliability. High MaxR values (> 0.90) reflect strong reliability. All constructs demonstrate excellent MaxR values, further supporting reliability.

 

Discussion

The current study aimed to develop and validate a critical writing rubric in CMC contexts. As a result of this study, the finalized critical writing rubric model consists of four main themes based on Paul and Elder’s (2019) intellectual standards. Theme 1 was labeled as clarity, accuracy, and precision (CAP), as it delves into how language learners actively engage with clear, accurate, and precise language in their writing. Theme 2 was labeled as depth and significance (DS), which helps learners produce meaningful written work while fostering a deeper understanding of the topics. Theme 3 was named relevance and logic (RL), which focuses on relevance and logic in addressing the topic. Theme 4 was regarded as breadth and fairness (BF), which investigates how learners demonstrate breadth and apply an unbiased approach to the problem.

Several studies have identified specific challenges related to the development and application of rubrics in EFL contexts. The findings align with existing research, emphasizing the importance of clear scoring criteria in supporting learning processes. For instance, Alghizzi and Alshahrani (2024) highlighted the impact of rubrics on students’ writing skills and IELTS scores in EFL contexts. The researchers found that students who utilized rubrics performed significantly better in writing tasks and achieved higher IELTS scores than those who did not. Another study by Dadakoğlu and Özdemir (2021) demonstrated that providing clear expectations and guidelines in rubrics enhances students’ writing skills in EFL contexts. Sword (2019) emphasizes the importance of clarity, highlighting techniques to improve reader comprehension.

Yaffe (2022) argued that precision in writing involves using concrete language and avoiding ambiguity, which allows readers to better grasp the writer’s intended meaning. Jackson (2022) discusses the importance of relevance in academic writing, emphasizing the need for a clear connection between the writer’s ideas and the topic at hand. Yancey (2021) encouraged writers to engage with their topics in-depth, emphasizing the value of critical reflection and analysis to produce thoughtful and compelling work. Elbow (2022) argued the importance of breadth in writing, suggesting that writers incorporate diverse perspectives, genres, and sources to create more engaging and well-rounded work. Kuehner and Hurley (2019) emphasized the significance of logical reasoning in writing, highlighting the need for clear argumentation and sound evidence to support claims. Moreover, representing diverse viewpoints and avoiding bias contributes to a more balanced and trustworthy work.

While rubrics offer a multitude of advantages, and this study aligns with these merits, Kohn (2006) pointed out that rubrics result in less depth of thought in students’ writing and provide a false sense of objectivity. The only way that a rubric can play a constructive role is that it is used as one of several sources and does not drive the instruction. Torrance (2012) also argued that using rubrics as explicit criteria may divert students’ attention away from deep learning and lead them toward surface learning. He asserted that the main goal of education is to foster students’ critical and independent thinking skills, rather than convergent thinking, which does not require significant creativity.

Despite the abundance of research on the use of writing rubrics in EFL contexts, there is a noticeable gap in studies focusing on developing EFL critical writing skills within CMC contexts. To address this gap, this study focused on developing and designing a valid and reliable critical writing rubric in CMC contexts to help teachers and students become more rational judges of the quality of writing based on educational goals in CMC contexts to confront the challenges of language teaching and learning in the 21st century.

 

Conclusion

In conclusion, this study developed and validated a critical writing rubric to examine and assess the critical writing skills of EFL learners in CMC contexts. The development and validation of this critical writing rubric represents a significant milestone in EFL language teaching and learning and provides a comprehensive tool for assessing critical writing skills in CMC contexts.

Moreover, the findings of this study provide pedagogical implications for curriculum developers, EFL/ESL language learners, teachers, and researchers. By integrating the newly developed critical writing rubric into their work, curriculum developers can create more effective learning experiences that account for various learning styles and abilities and contribute to ongoing improvements in educational practices. Furthermore, this critical writing rubric can establish transparent and well-defined criteria that support students’ growth as critical writers. It also helps learners understand the standards of effective critical writing, encourages students to engage in peer feedback, fosters collaboration, inspires self-reflection and revision, promotes critical thinking, facilitates fair assessment, and enables teachers to provide meaningful feedback.

Besides the contributions of the present study, several limitations were encountered throughout the development of the critical writing rubric. First, developing an effective critical writing rubric is a time-consuming and complex process. A longitudinal study with careful consideration of learning objectives and assessment priorities could offer a more in-depth understanding of the phenomena under investigation, especially when dealing with higher–order thinking skills. Moreover, given the nature of the research objectives, the participants of this research were recruited through non-probability purposive sampling to ensure in-depth insights into the phenomena under investigation, which may limit the generalizability of the findings. As the findings are primarily based on advanced learners, there is a need for caution in generalizing the results to all EFL learners. Therefore, utilizing an appropriate sampling method would enhance the applicability of the study results. Furthermore, because of limited access to a diverse range of participants, only a single dataset was used for validation. Thus, to increase the quality and credibility of the study, iterative data collection should be considered. Finally, practical application in classroom settings also presented potential limitations, as the rubric’s real-world usability and adaptability to various instructional contexts were not examined within the scope of this study. However, the practical application of the rubric was explored in a separate study to address these considerations more thoroughly.

 

Suggestions for Further Research

This study establishes a strong foundation for assessing critical writing in Computer-Mediated Communication (CMC) contexts, but several areas merit further exploration to refine and expand upon its findings. First, future research could focus on the long-term impact of implementing this rubric in educational settings through longitudinal studies. Tracking how students' critical writing skills develop over an extended period, such as a full academic year, would offer valuable insights into the rubric's effectiveness in fostering sustained improvements. Additionally, investigating the rubric’s applicability across different proficiency levels, particularly among intermediate and beginner EFL learners, would help adapt and scale it for a broader audience. Cross-cultural validation is another important area, as understanding how EFL learners from different linguistic and educational backgrounds respond to the rubric could highlight the need for adjustments to accommodate varying cultural expectations.

Moreover, the integration of AI-powered writing assistance presents a promising research avenue. Future studies could examine how automated writing evaluation systems align with human scoring and whether AI-generated feedback enhances learning outcomes. Exploring the rubric's impact on self-regulated learning is equally important; research could assess whether students become more adept at evaluating and improving their writing over time through repeated engagement with rubric-based feedback. A comparative analysis of rubric-based assessment and other methods, such as holistic grading and peer review, could provide insights into which approach most effectively promotes critical thinking and coherence in writing. Finally, investigating how teacher training influences rubric implementation and exploring its effectiveness in non-EFL contexts, such as humanities and STEM fields, would help refine its use across diverse academic settings. These research directions would ensure that rubrics remain adaptable and responsive to the evolving needs of EFL learners and educators in the digital age.

 

References

Alghizzi, T. M., & Alshahrani, T. M. (2024). Effects of grading rubrics on EFL learners’ writing in an EMI setting. Heliyon, 10(18).1-17. https://doi.org/10.1016/j.heliyon.2024.e36394

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa

Brookhart, S. (2013). How to create and use rubrics for formative assessment and grading.
ASCD. https://doi.org/10.4135/9781452218649.n15

Cohen, L., Manion, L., & Morrison, K. (2002). Research methods in education. Routledge. https://doi.org/10.4324/9780203224342

Dadakoğlu, S. C., & Özdemir, A. (2021). Development of analytical rubric for evaluating the effect of mobile design applications on artistic creativity: Validity-reliability study. İnönü Üniversitesi Eğitim Bilimleri Enstitüsü Dergisi, 8(16), 64-84. https://doi.org/10.29129/inujgse.897929

Dappen, L., Isernhagen, J., & Anderson, S. (2008). A statewide writing assessment model: Student proficiency and future implications. Assessing Writing, 13(1), 45-60. https://doi.org/10.1016/j.asw.2008.04.001.

Elbow, P. (2022). The democratization of writing and the role of cheating. Composition Studies, 50(1), 67-180.

Farzana, T. (2023). The standard measurement in online learning: a rubric as a focus on teaching-learning practices to move up quality education. EIKI Journal of Effective Teaching Methods, 1(3). https://doi.org/10.59652/jetm.v1i3.37

Ferris, D. (2009). Response to Student Writing: Implications for Second Language Students. New York, NY: Routledge. https://doi.org/sid.ir/paper/614509/en

Field, A. (2013). Discovering statistics using IBM SPSS statistics. Sage.

Gupta, B. L., & Gupta, P. B. (2021). Rubrics as versatile educational tool for outcome-based education. Journal of Engineering Technology Education, 15(2), 13-24

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2013). Multivariate data analysis: Pearson new international edition PDF eBook. Pearson Higher Ed.

Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43, 115-135. https://doi.org/10.1007/s11747-014-0403-8

Jackson, D. O. (2024). The longitudinal development of argumentative writing in an English for academic purposes course in Japan. System, 126, 103482. https://doi.org/10.1016/j.system.2024.103482

Kaiser, H. F. (1970). A second-generation little jiffy. Psychometrika, 35(4), 401–415. https://doi.org/10.1007/BF02291817

Keller, S. D., Trüb, R., Raubach, E., Meyer, J., Jansen, T. & Fleckenstein, J. (2023). Designing and validating an assessment rubric for writing emails in English as a foreign language. RISTAL, 6, 16–48. https://doi.org/10.2478/ristal-2023-0002

Kobylarek, A., Błaszczyński, K., Ślósarz, L., & Madej, M. (2022). Critical Thinking Questionnaire (CThQ)–construction and application of critical thinking test tool. Andragogy Adult Education and Social Marketing, 2(2), 1-1. https://doi.org/10.15503/andr2022.1

Kohn, A. (2006). Speaking my mind: The trouble with rubrics. English Journal, 95(4), 12-15. https://doi.org/10.58680/ej20064950

Kuehner, A. V., & Hurley, J. (2019). How integrating reading and writing supports student success. Journal of Developmental Education, 20-26.

Kwan, J. L. Y., & Chan, W. (2011). Comparing standardized coefficients in structural equation modeling: A model reparameterization approach. Behavior Research Methods, 43(3), 730–745. https://doi.org/10.3758/s13428-011-0088-6

Le, X. M., Phuong, H. Y., Phan, Q. T., & Le, T. T. (2023). Impact of using analytic rubrics for peer assessment on EFL students’ writing performance: an experimental study. Multicultural Education, 9(3), 41-53. https://doi.org/10.5281/zenodo.7750831

Maatuk, A. M., Elberkawi, E. K., Aljawarneh, S., Rashaideh, H., & Alharbi, H. (2022). The COVID-19 pandemic and E-learning: challenges and opportunities from the perspective of students and instructors. Journal of Computing in Higher Education, 34(1), 21-38. https://doi.org/10.1007/s12528-021-09274-2

Panadero, E., & Jonsson, A. (2020). A critical review of the arguments against the use of rubrics. Educational Research Review, 30, 100329. https://doi.org/10.1016/j.edurev.2020.100329

Panadero, E., Jonsson, A., Pinedo, L., & Fernández-Castilla, B. (2023). Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analytic review. Educational Psychology Review, 35(4), 113.https://doi.org/10.1007/s10648-023-09823-4

Panadero, E., & Romero, M. (2014). To rubric or not to rubric? The effects of self-assessment on self-regulation, performance, and self-efficacy. Assessment in Education: Principles, Policy & Practice, 21(2), 133-148. https://doi.org/10.1080/0969594X.2013.877872.

Paul, R., & Elder, L. (2006). Critical thinking competency standards. Dillon Beach: Foundation for critical thinking.

Paul, R., & Elder, L. (2014). Critical thinking: Intellectual standards essential to reasoning well within every domain of human thought, Part 4. Journal of Developmental Education, 37(3), 34.

Paul, R., & Elder, L. (2019). The thinker's guide to intellectual standards: The words that name them and the criteria that define them. Rowman & Littlefield.

Piaget, J. (2015). The grasp of consciousness (psychology revivals): action and concept in the young child. Psychology Press. https://doi.org/10.2307/3120744

Reynolds-Keefer, L. (2019). Rubric-referenced assessment in teacher preparation: An opportunity to learn by using. Practical Assessment, Research, and Evaluation, 15(1), 8. https://doi.org/doi:10.7275/psk5-mf68.

Reynders, G., Lantz, J., Ruder, S. M., Stanford, C. L., & Cole, R. S. (2020). Rubrics to assess critical thinking and information processing in undergraduate STEM courses. International Journal of STEM Education, 7, 1-15. https://doi.org/10.1186/s40594-020-00208-5

Samadi, S., & Ghaemi, F. (2016). The impact of embedded story structures versus sequential story structures on critical thinking of Iranian intermediate EFL learners. International Journal of Applied Linguistics and English Literature, 5(5), 171-178. https://doi.org/10.7575/aiac.ijalel.v.5n.5p.171

Saxton, E., Belanger, S., & Becker, W. (2012). The Critical thinking analytic rubric (CTAR): investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments. Assessing Writing, 17(4), 251-270. http://dx.doi.org/10.1016/j.asw.2012.07.002

Sosik, J. J., Kahal, S. S., & Piovoso, M. J. (2009). Silver bullet or voodoo statistics? A primer for using the partial least squares data analytic technique in group and organization research. Group and Organization Management, 34(1), 5–36. https://doi.org/10.1177/1059601108329198.

Sword, H. (2019). Snowflakes, splinters, and cobblestones: Metaphors for writing. Innovations in Narrative and Metaphor: Methodologies and Practices, 39-55. https://doi.org/10.1007/978-981-13-6114-2_4

Tashtoush, M. A., Shirawia, N., & Rasheed, N. M. (2024). Scoring rubrics method in performance assessment and its effect of mathematical achievement. Athens Journal of Education, 11(1), 1-22. https://doi.org/10.30958/aje.X-Y-Z

Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: an exploration of undergraduate students’ understanding and perspectives. Assessment & Evaluation in Higher Education, 49(6), 799-809. https://doi.org/10.1080/02602938.2023.2299330

Torrance, H. (2012). Formative assessment at the crossroads: Conformative, deformative and transformative assessment. Oxford Review of Education, 38(3), 323-342. https://doi.org/10.1080/03054985.2012.689693

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (Vol. 86). Harvard University Press.

Yaffe, P. (2022). Why is expository writing so undervalued---and what to do about it? Ubiquity, 2022(July), 1-9. https://doi.org/10.1145/3542739

Yamanishi, H., Ono, M., & Hijikata, Y. (2019). Developing a scoring rubric for L2 summary writing: A hybrid approach combining analytic and holistic assessment. Language Testing in Asia, 9, 1-22. https://doi.org/10.1186/s40468-019-0087-6

Yancey, K. B. (2021). Follow the sources: Notes toward WEC’s contribution to disciplinary writing. Writing-enriched curricula: Models of faculty-driven and departmental transformation. The WAC Clearinghouse, 73-93. https://doi.org/10.37514/PER-B.2021.1299.2.03

Yu, Z. (2021). The effects of gender, educational level, and personality on online learning rresults during the COVID-19 pandemic. International Journal of Educational Technology in Higher Education, 18(1), 14. https://doi.org/10.1186/s41239-021-00252.

 

Biodata

Sara Samadi is a university instructor and Ph.D. candidate in TEFL at Department of Teaching English and Translation, Karaj Branch, Islamic Azad University, Iran. She has over a decade of experience teaching English at various institutions and universities. Her research focuses on language teaching methodologies, assessment, and technology-assisted learning. She has published in peer-reviewed journals and has translated academic texts. She participates in conferences such as TELLSI and contributes to curriculum development.

Email: sara.samadi@kiau.ac.ir

 

Mohammad Hashamdar is an Assistant Professor in the Department of Teaching English and Translation, Karaj Branch, Islamic Azad University, Iran.  He earned his Ph.D. in Applied Linguistics from the Science and Research Branch of Islamic Azad University. His research interests encompass applied linguistics, sociolinguistics, and teacher education.

Email: mohammad.hashamdar@iau.ir

 

Gholamhassan Famil Khalili is an Assistant Professor in the Department of Teaching English and Translation, Karaj Branch, Islamic Azad University, Iran. He holds a Ph.D. in Teaching English as a Foreign Language (TEFL) and has contributed significantly to applied linguistics research. His scholarly work includes studies on translation quality assessment, language learning strategies, and the relationship between personality types and language proficiency among Iranian EFL learners.

Email: khalili@kiau.ac.ir

 

 

 

 

 

 

 

 

 

 

 

Creative Commons License © 2025 by the authors. Licensee International Journal of Foreign Language Teaching and Research, Najafabad Iran, Iran. This article is an open-access article distributed under the terms and conditions of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY NC 4.0 license). (http://creativecommons.org/licenses/by nc/4.0/).

International Journal of Foreign Language Teaching and Research, 13 (54), 2025        img0Islamic Azad University of Najafabad        


سند

Sanad is a platform for managing Azad University publications

الروابط

المراكز ذات الصلة

دعامة

الصفحات الرسمية

حقوق هذا الموقع الإلكتروني مملوكة لنظام SANAD Press Management. © 1442-1447

الصفحة الرئيسية| دخول| معلومات عنا| اتصل بنا|
[English] [فارسی] [en] [fa]