Evaluating the Effectiveness of Large Language Models in Identifying Communicatively Significant Errors in Papers of Students Learning Russian as a Foreign Language

Shulginov, V.A.; Klokova, K.S.; Yudinaa, T.A.; Obukhova, T.M.; Lebedeva, M.Y.

Авторы: Shulginov V.A., Klokova K.S., Yudinaa T.A., Obukhova T.M., Lebedeva M.Yu
Журнал: Doklady Mathematics
Том: 112
Номер: 3
Год издания: 2025
Издательство: Maik Nauka/Interperiodica Publishing
Местоположение издательства: Russian Federation
Первая страница: 565
Последняя страница: 571
DOI: 10.1134/S1064562425700577
Аннотация: This paper examines the capability of contemporary large language models (LLMs), such as GPT-5 and DeepSeek-R1, to identify and classify communicative errors in papers of students learning Russian as a foreign language (RFL). While existing tools primarily focus on formal errors, this study emphasizes the communicative aspect, evaluating the extent to which an error disrupts comprehension (i.e., communicatively significant errors) or merely affects linguistic norms (i.e., communicatively insignificant errors). To this end, a corpus of papers by B2-level students (TORFL-2) was created and annotated by experts, and a multi-stage pipeline for testing LLMs was developed, incorporating structured prompting and heuristic voting methods to enhance result reliability. The experiment revealed that while models can localize errors with certain accuracy, they experience considerable difficulties in their proper communicative classification. The models tend to systematically underestimate the degree of error impact on comprehension, confuse error types, and encounter challenges in identifying multiple errors within a single fragment. The study demonstrates both the potential and current limitations of LLMs as tools for automated, communicatively oriented feedback in educational technologies.Link: https://rdcu.be/e9mZC
Добавил в систему: Обухова Татьяна Михайловна

	ИСТИНА	Войти в систему Регистрация
	ИСТИНА ИНХС РАН
	Главная Поиск Статистика О проекте Помощь

ИСТИНА

ИСТИНА ИНХС РАН

Evaluating the Effectiveness of Large Language Models in Identifying Communicatively Significant Errors in Papers of Students Learning Russian as a Foreign Languageстатья

Прикрепленные файлы