Translation Quality Assessment of Three MT Systems
Subject Areas : Journal of Teaching English Language StudiesAtefe Mazidi 1 , Guiti Mortazavizadeh 2
1 -
2 -
Keywords: Accuracy, Fluency, Idiom, MT quality assessment, Google translate, Chat GPT, Xerac,
Abstract :
This quantitative study performed a machine translation (MT) quality assessment on the accuracy and fluency of three MT systems: Google Translate, ChatGPT, and Xerac. The corpus of the study was The Alchemist, from which the total number of 173 idioms were chosen and translated by the MT systems. The translations were rated through Morgan’s (2004) rubrics by two experts in translation. The data were analyzed through SPSS 28. The findings indicated that Google Translate significantly outperformed the other two systems. It provided translations that were not only more relevant to the original text, but also exhibited higher fluency for readers in their first language. The evaluation highlighted that Google Translate effectively applied linguistic rules pertinent to the target language, resulting in translations that felt more natural and contextually appropriate. In contrast, both ChatGPT and Xerac fell short in meeting the standards set by Google Translate, with evaluators consistently favoring the former for its superior performance. This suggests that Google Translate can be the best reliable MT system for translating texts, making it a valuable tool for users seeking accurate and fluent translations. Overall, the study underscores the importance of evaluating MT systems based on both accuracy and fluency, providing insights into their relative effectiveness in real-world applications. The findings of the study could have implications for trasnaltors and translator course designers and trainers.
Key terms: Accuracy, Fluency, Idiom, MT quality assessment, Google translate, Chat GPT, Xerac
Aghai, M. (2024). ChatGPT vs. Google Translate: Comparative analysis of translation quality. Translation Studies, 22(85), 85-100. DOR: 20.1001.1.17350212.1403.22.1.9.2.
Al-Khresheh, M. H., & Almaaytah, S. A. (2018). English Proverbs into Arabic through Machine Translation. International Journal of Applied Linguistics & English Literature, 7, 158-166.
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1409.0473
Baker, M. (1991). In Other Words: A Coursebook on Translation. Routledge.
Bashir, A., Aleem, M., & Anjum, M. A. I. (2021). Pakistani translators’ strategies for translating English idioms. Ilkogretim Online, 20(5), 5110-5116.
Boulton, A., & Vyatkina, N. (2021). Thirty years of data-driven learning: Taking stock and charting new directions over time. Language Learning & Technology, 25(3), 66–89. https://psycnet.apa.org/record/2022-10654-005
Budianto, L. (2020). EFL learner’s perception about utilizing technology-driven learning in the midst of Covid-19 outbreak. International Journal of Innovation, Creativity and Change, 14 (4), 69-80. http://repository.uin-malang.ac.id/8023/
Ceramella, S. (2008). The role of translation in globalization. International Journal of Translation Studies, 2(3), 45-60.
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014a). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014).
Conneau, A., & Lample, G. (2019). Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, 32, 7057-7067. https://proceedings.neurips.cc/
Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Sage Publications.
Dabre, R., Cromieres, F., & Nakazawa, T. (2017). Neural machine translation: Basics, practical aspects and recent trends. Proceedings of the 8th International Joint Conference on Natural Language Processing, pages 11–13, Taipei, Taiwan.
Daxbock, B. (2010). Text and Context: The Role of Equivalence in Translation Quality Assessment. Journal of Translation Studies, 4(2), 123-137.
Dicks, B. (2018). The impact of AI-powered translation tools on language learning and practice. Language and Education, 32(1), 50-65. https://doi.org/10.32996/ijllt.2025.8.3.30
Dweik, B. S., & Thalji, M. B. (2016). Strategies for translating proverbs from &English into Arabic. Academic Research International, 7(2), 120-127.
Hadi, G., & Ghorbani, S. (2021). Comparative analysis of Google Translate and ChatGPT in translating idiomatic expressions. Journal of Applied Linguistics and Translation, 22(1), 89-105. DOI: https://doi.org/10.33474/j-reall.v6i1.23252
House, J. (2009). Translation. Oxford University Press.
Kafi, M., et al. (2018). Challenges in machine translation of Persian: A review. Persian Studies Journal, 12(4), 75-89.
Khoshafah, F. (2023). ChatGPT for Arabic-English translation: Evaluating the accuracy. https://doi.org/10.21203/rs.3.rs-2814154/v1
Liu, Q., & Liu, H. (2020). Improving machine translation quality with unsupervised pre-training. Journal of Artificial Intelligence Research, 68, 1143-1165.
Ma, J., & Huang, J. (2019). On the impact of training data quality on neural machine translation. Journal of Computer Science and Technology, 34(2), 245-263. https://doi.org/10.1007/s11390-019-1940-6
Madsen, H. (2009). The challenges of literary translation and machine assistance. Translation Journal, 13(4), 66-79.
Moon, R. (1998). Fixed expressions and idioms in English. Clarendon Press.
Popovic, M., & Ney, H. (2011). Towards the evaluation of machine translation quality: A study of the impact of fluency and fidelity. Computational Linguistics, 37(4), 799-806. https://doi.org/10.1162/COLI_a_00055
Qian, D., & Liu, Y. (2019). Evaluating Machine Translation of Cultural Terms: A Case Study of English-Persian. International Journal of Language Resources and Evaluation, 53(3), 367-383.
Rahman, F., & Saputra, N. (2021). English as international language revisited: Implications on South Korea’s ELT context. Journal of English Language Teaching, 6(1), 8-15. DOI: http://dx.doi.org/10.30998/scope.v6i1.9383
Ronagh Zadeh, S., et al. (2021). Challenges in translating idioms from English to Persian. Journal of Language and Cultural Studies, 15(2), 112-129.
Rushaid, A. (2010). The evolution of translation technology: From rule-based to neural machine translation. Journal of Language and Computer Science, 5(2), 77-92.
Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 1715-1725. https://doi.org/10.18653/v1/P16-1162
Shardlow, M. (2014). Neural Machine Translation: A Review. Journal of Machine Learning, 1(1), 1-10.
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems, 27, 238-245.
Taleghani, M., & Pazouki, E. (2018). Free Online Translators: A Comparative Assessment in Terms of Idioms and Phrasal Verbs. International Journal of English Language & Translation Studies. 6(1). 15-19.
Thao, N. (2023). An Investigation into Google Translates Translation of English Proverbs into Vietnamese. International Journal of Social Science and Human Research 7(2), 89-97.
Tiedemann, J. (2018). Neural machine translation: A review. Journal of Natural Language Engineering, 24(1), 1-30. https://doi.org/10.1017/S135132491800014X
Toral, A., & Apidianaki, M. (2016). The role of post-editing in the evaluation of machine translation systems. Machine Translation, 30(3), 257-274. https://doi.org/10.1007/s10590-016-9208
Vaswani, A., Shard, N., et al. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, 30, 1-19.
Wang, Y., & Wang, Y. (2019). Evaluating the fluency and adequacy of machine translation: A review and a new framework. ACM Transactions on Intelligent Systems and Technology, 10(4), 1-30. https://doi.org/10.1145/3310561
Wu, H., & Wang, Y. (2020). Challenges in literary machine translation. Translation Studies, 13(2), 210-226.
Zhang, M. (2024). A study on the translation quality of ChatGPT. International Journal of Educational Curriculum Management and Research, 5(4), 25-32. Doi: 10.38007/IJECMR.2024.050121.
