A MODEL FOR ASSESSING THE LINGUOCULTURAL ADEQUACY OF AI-GENERATED FOLKLORE TEXTS
Keywords:
linguocultural adequacy, evaluation model, AI-generated texts, folklore, fairy tales, translation quality assessment, cultural bias, large language models, English, Uzbek.Abstract
Large language models (LLMs) such as ChatGPT, Claude and Gemini now retell, translate and invent folk tales in seconds, and their texts are already used in classrooms, children’s books and online media. Yet there is no established way to judge whether such a text is culturally adequate. Standard measures of machine translation quality, such as BLEU or general fluency ratings, can tell us whether a text is grammatical and close to a reference, but not whether the Dev is still a Dev, whether Zumrad’s sabr is still a virtue, or whether the hero still greets the monster as bobo. Research on AI-generated folklore has shown that cultural losses occur at many levels, from names and idioms to metaphors, values, emotions, speech etiquette, narrative structure and visual imagery, but these findings have not been integrated into a practical evaluation tool.
References
1.Adilazuarda, M. F., Mukherjee, S., Lavania, P., Singh, S., Aji, A. F., O’Neill, J., Modi, A., & Choudhury, M. (2024). Towards measuring and modeling “culture” in LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 15763–15784).
2.AlKhamissi, B., ElNokrashy, M., AlKhamissi, M., & Diab, M. (2024). Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (pp. 12404–12422).
3.Brown, P., & Levinson, S. C. (1987). Politeness: Some universals in language usage. Cambridge: Cambridge University Press.
4.Cao, Y., Zhou, L., Lee, S., Cabello, L., Chen, M., & Hershcovich, D. (2023). Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) (pp. 53–67).
5.Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., & Macherey, W. (2021). Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9, 1460–1474.
6.Hermans, T. (1988). On translating proper names, with reference to De Witte and Max Havelaar. In M. J. Wintle (Ed.), Modern Dutch studies (pp. 11–24). London: Athlone Press.
7.Hershcovich, D., Frank, S., Lent, H., de Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Cabello Piqueras, L., Chalkidis, I., Cui, R., Fierro, C., Margatina, K., Rust, P., & Søgaard, A. (2022). Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (pp. 6997–7013).
8.House, J. (2015). Translation quality assessment: Past and present. London: Routledge.
9.Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12(10), 1–8.
10.Ide, S. (1989). Formal forms and discernment: Two neglected aspects of universals of linguistic politeness. Multilingua, 8(2–3), 223–248.
11.Karasik, V. I. (2002). Yazykovoy krug: lichnost’, kontsepty, diskurs [The language circle: Personality, concepts, discourse]. Volgograd: Peremena.
12.Kövecses, Z. (2000). Metaphor and emotion: Language, culture, and body in human feeling. Cambridge: Cambridge University Press.
13.Kövecses, Z. (2005). Metaphor in culture: Universality and variation. Cambridge: Cambridge University Press.
14.Krasnykh, V. V. (2002). Etnopsikholingvistika i lingvokul’turologiya [Ethnopsycholinguistics and linguoculturology]. Moscow: Gnozis.
15.Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). London: Routledge.
16.Krippendorff, K. (2018). Content analysis: An introduction to its methodology (4th ed.). Thousand Oaks, CA: SAGE.
17.Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press.
18.Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 12, 455–463.
19.Lüthi, M. (1982). The European folktale: Form and nature (J. D. Niles, Trans.). Philadelphia: Institute for the Study of Human Issues.
20.Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385.
21.Naous, T., Ryan, M. J., Ritter, A., & Xu, W. (2024). Having beer after prayer? Measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (pp. 16366–16393).
22.Nida, E. A. (1964). Toward a science of translating. Leiden: E. J. Brill.
23.Olrik, A. (1965). Epic laws of folk narrative. In A. Dundes (Ed.), The study of folklore (pp. 129–141). Englewood Cliffs, NJ: Prentice-Hall. (Original work published 1909)
24.Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318).
25.Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.
26.Propp, V. (1968). Morphology of the folktale (L. Scott, Trans.; 2nd ed.). Austin: University of Texas Press. (Original work published 1928)
27.Reiss, K., & Vermeer, H. J. (2013). Towards a general theory of translational action: Skopos theory explained (C. Nord, Trans.). London: Routledge. (Original work published 1984)
28.Robinson, N. R., Ogayo, P., Mortensen, D. R., & Neubig, G. (2023). ChatGPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation (pp. 392–418).
29.Sharifian, F. (2017). Cultural linguistics: Cultural conceptualisations and language. Amsterdam: John Benjamins.
30.Tao, Y., Viberg, O., Baker, R. S., & Kizilcec, R. F. (2024). Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9), pgae346. https://doi.org/10.1093/pnasnexus/pgae346
31.Toury, G. (1995). Descriptive translation studies and beyond. Amsterdam: John Benjamins.
32.Venuti, L. (1995). The translator’s invisibility: A history of translation. London: Routledge.
33.Vorob’ev, V. V. (1997). Lingvokul’turologiya: teoriya i metody [Linguoculturology: Theory and methods]. Moscow: RUDN Publishing.
34.Wierzbicka, A. (1997). Understanding cultures through their key words: English, Russian, Polish, German, and Japanese. New York: Oxford University Press.