Journal · Language
Tokenisation pitfalls in Thai in-app search
English-centric search stacks arrive in Bangkok with a quiet arrogance: split on whitespace, lowercase, drop stopwords, stem a little, ship. Thai writing does not obligingly insert spaces between words. Users still type English brand names, Arabic numerals, and emoji in the same query. In-app search analytics that ignore this will report healthy recall on Latin test sets and miss the actual log.
This essay is a field guide, not a computational linguistics paper. Agent Flowcore’s two-day Thai Query Normalisation Workshop exists because the Lab cannot spend eleven weeks on analysers alone. If you need a research tokenizer, hire researchers. If you need to stop deleting meaning, read on.
Whitespace is not a token boundary
Many Thai queries in super-apps arrive as a single string. A naive whitespace splitter yields one token. Downstream IDF weights become nonsense. Word-break libraries help and also hallucinate. We teach teams to log both the raw string and the analyser output, then spot-check disagreements on high-volume queries. The disagreement list is more valuable than the library’s README.
Spaces do appear — often around English brands or prices. “sku B2 แดง” is a different shape from a fully Thai known-item string. Your taxonomy should notice the mix instead of forcing one cluster name.
Stopword lists that steal the query
English stopword lists still sneak into Thai pipelines via default analyser configs. Words like “on” or “to” look harmless until a brand or a station code collides. Thai function words, when stripped aggressively, can leave a query that matches half the catalogue. We ask alumni to print the stopword file and initial it. If nobody on the team can explain a removal, it does not ship.
If your analyser output is shorter than the user’s intent, you did not simplify. You guessed.
Transliteration and nicknames
Users type Romanised Thai, official English names, and nicknames for the same biller, station, or dish. A synonym table owned by product — not a one-off spreadsheet on someone’s laptop — is the unglamorous fix. Analytics should report how often a click happens only after a reformulation that looks like a human synonym. That is a coverage gap, not engagement. See also reformulation chains.
What we refuse to sell
We will not email you a tokenizer. We will not certify your NLP vendor. We will sit with your query dump, mark failure modes, and leave you with a list your vendor cannot politely ignore. That limitation is stated again in the Lab FAQ. Thailand’s language reality is why the studio is here; it is not a marketing motif. For the wider analytic stack, read in-app search analytics or write to the studio.