ChatGPT as a formative support tool in German A2 writing assessment: An exploratory study

Abstract

This study evaluates the feasibility of using ChatGPT as an instrument for assessing German writing skills at the CEFR A2 level. Motivated by concerns about subjectivity and inconsistency in traditional assessment, the study adopts an exploratory approach to examine ChatGPT’s operational role as an AI-assisted tool, specifically analyzing its scoring behavior and its alignment with human judgment. Adopting an exploratory perspective, this study investigates ChatGPT’s operational role as an AI-assisted assessment tool by examining its scoring behavior and alignment with human judgment. A mixed-methods complementary design was employed: qualitative descriptive analysis traced ChatGPT’s assessment process, while quantitative analysis, using several relevant statistical tests, examined the consistency of ChatGPT’s assessment results and their agreement with those from human raters. Writing samples were collected from 76 undergraduate students enrolled in a German-language education program at a public university in Indonesia. The writing tasks and assessment rubrics were adapted from the Goethe-Zertifikat A2 Modelsatz. The results indicate that ChatGPT demonstrated moderate consistency over time, reflected in significant positive correlations and general alignment with human raters. In addition, no significant differences were found in the mean scores assigned by ChatGPT and human raters. However, exact agreement on specific scoring decisions remained limited, revealing divergences in how AI and humans weigh linguistic nuances. The findings suggest that while ChatGPT can function as a valuable supplementary tool for formative assessment, human expertise remains essential for capturing the interpretive complexities of writing performance.