Evaluating Large Language Models for Multilingual and Typo-Tolerant Answer Validation in a Digital Quiz Application

We address the problem of automatically validating free-form text answers in a mobile quiz application, where users may answer with synonyms, minor typos, or in different languages. Traditional exact-string matching fails in these cases, so we explore whether large language models (LLMs) can effectively judge accuracy in alignment with human decisions. A comparative study of 19 LLMs (from the major model families) was conducted asking each model to evaluate the quiz answers (without the human annotation label) and scoring +1 for agreement or -1 for disagreement with human judgments. Each question in our dataset is independently judged by five humans to reduce bias. Our results show that the latest GPT-4o model has the highest agreement rate with human evaluators, substantially outperforming other LLMs. This suggests that advanced LLMs can robustly handle semantic equivalence and language variations in answer validation. We discuss implications for the deployment of LLMs in educational applications and relate our findings to previous work on semantic text similarity, fuzzy matching, and multilingual evaluation.

International Conference on Smart and Sustainable Technologies (SpliTech), 1–5