|
ИСТИНА |
Войти в систему Регистрация |
ИСТИНА ИНХС РАН |
||
Impact of Tokenization on LLaMa Russian Adaptationстатья
- Авторы: Tikhomirov M., Chernyshev D.
- Сборник: Proceedings of Ivannikov ISPRAS Open Conference
- Год издания: 2023
- Аннотация: Latest instruction-tuned large language models (LLM) show great results on various tasks, however, they often face performance degradation for non-English input. There is evidence that the reason lies in inefficient tokenization caused by low language representation in pre-training data which hinders the comprehension of non-English instructions, limiting the potential of target language instruction-tuning. In this work we investigate the possibility of addressing the issue with vocabulary substitution in the context of LLaMa Russian language adaptation. We explore three variants of vocabulary adaptation and test their performance on Saiga instruction-tuning andfine-tuning on Russian Super Glue benchmark. The results of automatic evaluation show that vocabulary substitution not only improves the model’s quality in Russian but also accelerates finetuning (35%) and inference (up to 60%) while reducing memory consumption. Additional human evaluation of the instructiontuned models demonstrates that models with Russian-adaptedvocabulary generate answers with higher user preference than the original Saiga-LLaMa model.
- Добавил в систему: Тихомиров Михаил Михайлович