Vietnamese text use 2-3x more tokens than English for same meaning, how you handle cost?
Hi all, first post here. I build support chatbot for an e-commerce company, customers write in Vietnamese and English. I notice our token cost per conversation is much higher for Vietnamese. I test same FAQ answer in both language, the Vietnamese version is around 2.3x tokens of the English version with the tokenizer of our model. It make two problem: cost per conversation is higher our RAG chunk size is set by…