ChatGPT as a formative support tool in German A2 writing assessment: An exploratory study

Abstract

This study aimed to find out the feasibility of ChatGPT as an instrument for assessing German writing skills at the A2 CEFR level. This is motivated by the subjectivity and inconsistency often found in assessing writing skills in foreign language learning. ChatGPT, a popular generative artificial intelligence (AI) model, could serve as a solution to these challenges. This study focused on evaluating the consistency of ChatGPT assessments over time and comparing their accuracy with assessments from certified human raters. This study employed a mixed approach. Qualitative descriptive analysis was used to trace the assessment process by ChatGPT. Quantitative analysis with several relevant statistical tests was performed to examine the consistency of ChatGPT's assessments results and their level of agreement with the results from human raters. Writing samples were taken from 76 students of the German Language Education program at the Universitas Pendidikan Indonesia. The writing tasks and assessment rubrics were adapted from the Goethe Zertifikat A2 Modelsatz. The results showed that ChatGPT was able to provide consistent assessments over time, with moderate positive correlations and general agreement with human raters. Moreover, no significant differences were found in the average scores between ChatGPT's and human raters’ assessments. Thus, ChatGPT proved to be a reliable and efficient supplementary assessment tool for evaluating German writing skills at the A2 CEFR level. However, the presence of human raters is still essential to capture the full nuances of the language.