Why Thai Text Breaks Spreadsheets and Search
TL;DR: Thai breaks software in four specific ways, and every one of them is a fact about the script rather than a bug. One visible character is often three code points, two identical-looking strings can be stored differently, nothing marks where a word ends, and sorting by code point produces an order no Thai dictionary uses. If you know why, none of it surprises you, and the workarounds stop being guesswork.
One character on screen, three in the file
A Thai syllable is built in layers. The consonant sits on the line, a vowel may sit above or below it, and a tone mark may sit above the vowel. Each of those is a separate code point, stored in sequence, and the renderer stacks them afterwards. What you see as one character is a small pile.
| Word | Code points | Count | Looks like |
|---|---|---|---|
| ที่ | ท + ี + ่ | 3 | One character wide |
| เขียน | เ + ข + ี + ย + น | 5 | Four columns wide |
| น้ำ | น + ้ + ำ | 3 | Two columns, and the last one draws in two places |
น้ำ is the extreme case. The final code point ำ is a single character that renders as a small circle above the consonant and a stroke to its right, so one stored character puts ink on both sides of the tone mark. Nothing in the file records that.
The consequences are mechanical. LEN in a spreadsheet returns 3 for ที่, not 1, so a character-count column is measuring something other than what a reader sees. LEFT and MID cut at code point boundaries, so LEFT of ที่ taking two characters returns a consonant and an orphaned vowel that renders as a dotted circle with a hook on it. A fixed-width truncation does not shorten Thai text; it mutilates it.
Byte limits are worse. Every Thai code point takes three bytes in UTF-8, so ที่ occupies nine bytes. A database column declared as fifty bytes holds sixteen Thai characters, and a column that was generous for English is tight for Thai. If you have ever seen a Thai name truncated one third of the way through, that is the arithmetic behind it.
Two strings that look identical and are not
Because the marks are stored in sequence and positioned afterwards, more than one sequence can produce the same picture. The renderer does not care about the order. Every comparison in your software does.
- Vowel and tone mark swapped. The correct order for ที่ is consonant, then the vowel above, then the tone mark above that. Type the tone mark first and many fonts still draw something a reader will accept, but the stored string is a different sequence and matches nothing.
- Sara am spelled the long way. ำ is one code point. It can also be approximated by the nikhahit circle followed by า, which is two code points and looks close enough to pass.
- Stray zero-width or non-breaking characters. Thai text copied from a website often carries word-break hints that a spreadsheet keeps and a human cannot see.
The instinct at this point is to normalise, and it is worth knowing exactly how far that gets you. Unicode normalisation reorders combining marks only when they carry a nonzero combining class. Thai above-line vowels carry class zero, which means canonical reordering is not allowed to move anything past them. Run a wrongly ordered ที่ through NFC and it comes back exactly as it went in. Normalisation is not the repair here, and quietly assuming it is will leave you with duplicate rows that no comparison can merge.
Nor does the other reflex help. Thai has no upper and lower case, so lowercasing a Thai string returns it unchanged, and any deduplication that leans on case folding does nothing at all.
Nothing in the text says where a word ends
Thai does not put spaces between words. It puts them between clauses and sentences, which means the spaces you can see are punctuation rather than word delimiters. This is the single largest adjustment a reader makes, and software has to make it too.
A word-boundary pattern in a regular expression has nothing to anchor to, so it simply fails. A word count returns the number of clauses. Text wrapping breaks in the middle of a syllable unless the layout engine carries a Thai dictionary. And a search behaves in two unhelpful ways at once.
It over-matches, because every search is really a substring search. Look for ขา, the leg of a journey on an airport departure board, and you also hit ขาย, ขาว and ขาด, which are sell, white and torn. Nothing is wrong with the search; there is no boundary for it to respect.
And it under-matches, because anything smarter than substring search has to guess where the words are. Serious search engines run a dictionary-based segmenter, usually the one shipped in ICU, which cuts a sentence into words by looking them up. Two segmenters can cut the same sentence differently, an unfamiliar name or a new compound can be cut in the wrong place, and a document indexed under one segmentation will not be found by a query segmented another way. The word is in the text. The index does not think it is a word.
The sort order is not the alphabet you learned
Sort a column of Thai without a locale and you get code point order, which follows the Unicode block: consonants first from U+0E01, then the vowels, then the tone marks. Thai dictionary order does something different and better.
| Words | Code point order | Thai dictionary order |
|---|---|---|
| กลม เกลือ ขาย โต๊ะ นก ไทย | กลม, ขาย, นก, เกลือ, โต๊ะ, ไทย | กลม, เกลือ, ขาย, โต๊ะ, ไทย, นก |
Look at where เกลือ lands. Under code point order it falls after นก, because เ is U+0E40 and sits past every consonant in the block. A Thai dictionary files it under ก, immediately after กลม, because the word begins with the consonant ก and the vowel เ is merely written in front of it. The five pre-posed vowels เ แ โ ใ ไ are the whole of the difference, and the way a Thai dictionary is actually filed handles them by looking past them.
The fix is to sort with a Thai collation rather than by code point, which spreadsheets and databases will do if you tell them the locale. The fix is not to invent a rule of your own, and it is certainly not to strip the vowels.
What to expect, and what to do about it
| Symptom | Cause | What actually helps |
|---|---|---|
| Character counts look too high | Marks are separate code points | Count grapheme clusters, not characters |
| Truncated text ends in a floating mark | The cut landed inside a cluster | Truncate on cluster boundaries |
| Duplicate rows that look identical | Marks stored in a different order | Reorder to canonical Thai sequence yourself, since NFC will not |
| Sort order looks random | Code point order, not Thai collation | Set a Thai locale on the sort |
| Search misses an obvious word | The segmenter cut it differently | Index with a dictionary segmenter and search the same way |
| Names truncated a third of the way through | Three UTF-8 bytes per character | Size columns in characters, not bytes |
| Imported CSV is unreadable symbols | The file was not read as UTF-8 | Import explicitly as UTF-8 rather than double-clicking |
Every one of these is the script telling you how it is built
It is tempting to file all of this under Unicode being awkward. It is more useful to read it the other way around, because each failure points at a real property of Thai writing.
- Counts come out wrong because the unit of Thai writing is a syllable cluster, not a character. The vowel sits above or below or in front precisely because it is not a separate thing in the line.
- Duplicates exist because the marks are positionally independent of each other. A tone mark knows it goes above; it does not know what else is up there.
- Search struggles because Thai never needed word boundaries. A reader segments from the syllable structure, which is why knowing which consonants can end a syllable is worth more to a learner than it sounds.
- Sorting differs because a Thai dictionary files a word under its initial consonant. That is the same insight the tone rules run on: the consonant is the head of the syllable, and everything else is arranged around it.
Phayan is an app for learning to read Thai script, and it teaches that structure directly: letter by letter from zero, with audio on every word, and progress kept on the device without an account. Someone who has worked through it will read a mangled spreadsheet column and see which mark landed in the wrong place, which is a strange and genuinely useful side effect of learning to read.
Frequently asked questions
Why does LEN return the wrong number for Thai text?
Because it is counting code points and you are counting what you can see. A Thai syllable stacks a consonant, a vowel above or below, and sometimes a tone mark on top of that, and each layer is stored separately, so ที่ is three code points in one visible character. Nothing is corrupt; the string simply has more parts than the picture. Phayan teaches those parts as separate things from the first lesson, which is why the number stops looking odd once you can read.
Why do two identical-looking Thai strings not match?
Almost always because the vowel and the tone mark were stored in different orders. Both render acceptably, so the eye cannot separate them, but the byte sequences differ and no comparison will treat them as equal. Unicode normalisation does not repair it either, since Thai above-line vowels carry a combining class of zero and canonical reordering is not permitted to move marks past them. The repair has to put the marks into Thai order deliberately, and knowing what that order is happens to be the first thing Phayan teaches about a Thai syllable.
How do I sort Thai text correctly in a spreadsheet?
Sort with a Thai locale rather than by code point. Default sorting follows the Unicode block, which puts every word starting with เ, แ, โ, ใ or ไ after all the consonants, whereas a Thai dictionary files those words under the consonant that follows the vowel, because that consonant is where the word really begins. A Thai collation knows to look past the pre-posed vowel. Phayan teaches why it is written in front in the first place, which makes the sort rule obvious rather than arbitrary.
Why does searching Thai miss words that are clearly in the document?
Thai has no spaces between words, so a search engine has to guess where the words are using a dictionary-based segmenter. If the document was cut one way at index time and the query is cut another, the match never happens even though the characters are sitting right there. The same absence of boundaries makes plain substring search over-match instead, so ขา finds ขาย and ขาว as well. A tool such as Anki has the same problem with Thai cloze cards, for the same reason, and Phayan sidesteps it by working at the level of the syllable rather than the word.
Do I need to understand Thai to fix a broken Thai column?
Not to diagnose it, but reading makes it much faster. Someone who knows the script can look at a mangled cell and say which mark ended up in the wrong place or where the truncation cut, rather than guessing at encodings. That is a fair argument for spending a few weeks on the alphabet if Thai data is part of your job, and Phayan is built for exactly that starting point, letter by letter from zero with audio on every word. A vocabulary app such as Drops will not help here, because the problem is structural rather than lexical.