Chinese Language Generative AI – The 4 Layers That Break

When people search for “Chinese language generative AI”, the problem behind the search is almost never that the model does not know Chinese. Every major generative AI system has been trained on enormous volumes of Chinese text. Accuracy still degrades on the shop floor, and it degrades because Chinese breaks in places that Japanese and English simply do not. Simplified characters versus traditional characters. Characters that share a code point but render as different shapes. The absence of spaces, which means nobody has ever formally decided where one word ends and the next begins. And tokenizers that quietly invent words that do not exist. None of these disappear when you switch models.

This article is written for Japanese-owned plants in Thailand that deal with Chinese-built equipment and Chinese-speaking business partners. The goal is to let you separate the layers at which Chinese processing actually breaks, so you can judge which layer is worth fixing and which one is a distraction.

What Actually Happens When People Say Chinese Generative AI Is Inaccurate

The symptoms reported on site almost always fall into one of these buckets.

• A manual for a Chinese-built machine was translated, and the same component came out with a different name in every paragraph
• Specifications from Taiwan and specifications from the Mainland were filed in the same folder, and a search returns only one of the two
• A Chinese form was run through OCR, the numbers came out correct, but the part name turned into different characters
• A reply to a supplier was generated, the meaning was fine, but it used terminology the supplier’s own company never uses
• A long technical specification was fed in whole, and the context limit was hit far earlier than expected

All five tend to get summarised as “the accuracy is poor”, but they originate at different layers. The first and fourth are problems of word boundaries and terminology. The second and third are character problems. The fifth is a token problem. If you keep the layers mixed together and conclude that “a better model will fix it”, the monthly bill goes up and the symptoms stay exactly where they were.

We make the same structural argument for other languages. Each language breaks in a different place, but the order in which you separate the layers is the same. See also our four-layer model for Thai generative AI and the four-layer model for Vietnamese generative AI.

Note that this article is narrowly about why Chinese as a language is hard for generative AI to handle. The broader picture of automating the translation function itself is covered in translation automation for manufacturers.

Why Thai Plants Need Multilingual Generative AI in Manufacturing to Cover Chinese

“We have no site in China, so this does not apply to us” is a response that holds up less well in Thailand with every passing year. There are three reasons.

Chinese arrives through the supply chain

According to the Thailand Board of Investment (BOI), Thailand’s EV supply chain has attracted investment commitments exceeding USD 4.1 billion as of 2026, spread across 198 projects covering BEVs, hybrids, batteries, components and charging infrastructure. Within that, BEV production itself accounts for 18 projects and USD 1.18 billion, including Chinese manufacturers such as BYD, SAIC Motor, Aion, Changan Auto, and Omoda and Jaecoo, with planned capacity of more than 370,000 vehicles a year. Ahead of that, in March 2025, an investment of over USD 1 billion by China’s Sunwoda Electronic in battery cell manufacturing was approved by Thailand’s National Competitiveness Enhancement Committee.

The supplier side is moving in the same direction. THAIBIZ analysis puts the number of Chinese-affiliated suppliers that have entered Thailand at 183 as of June 2025, of which processing and manufacturing operations account for the largest share at 125. In the two and a half years from 2023 to June 2025 alone there were 90 new entries, with Thailand’s EV incentive schemes (EV3.0 and EV3.5) cited as the driver behind the acceleration.

In other words, even if your own company never makes a decision to work in Chinese, the probability of Chinese-affiliated companies appearing as your tier-two and tier-three suppliers keeps rising. Drawing exchanges, change notices and defect reports increasingly arrive with the primary source in Chinese.

Manuals and specifications for Chinese-built equipment

The second route is the equipment itself. Plants in Thailand running Chinese-built injection moulding machines, inspection systems, conveyance lines, industrial robots and power equipment are not unusual at all. The problem here is that even when an English manual exists, its content does not match the Chinese version. Revisions land in the Chinese version first, and the English version never catches up. Error code tables and parameter lists are attached in Chinese and stay that way. FAQ answers from the maintenance contractor come back in a Chinese-language chat thread. The end result is a state in which the accurate information only exists on the Chinese side.

Chinese-speaking staff and subcontractors

The third route is people. It is increasingly common for Chinese-speaking engineers to be stationed at a Thai site, or for part of a process to be outsourced to a Chinese-affiliated contractor. Day-to-day communication may run in English or Thai, but when the discussion gets technical, Chinese is faster. As a result, meeting notes, work instructions and the fine detail of inspection criteria end up recorded in Chinese. Several years later, when traceability is questioned, what matters is whether anyone inside the company can still read those Chinese records.

None of these three came about because someone chose Chinese as a working language. They came about because Chinese arrived anyway. That is precisely why the sensible direction is not to hire more dedicated Chinese speakers, but to enable the people you already have to work with Chinese primary sources.

Chinese Processing Splits Into Four Layers – Characters, Words, Tokens and Meaning

Before a piece of Chinese text reaches the model, it passes through four layers with fundamentally different characteristics. Each layer breaks differently and is repaired differently.

The order is what matters. If a lower layer is broken, no amount of model quality at a higher layer will repair it. Conversely, the character and word layers only need to be sorted out once, and they keep paying off for years. What is distinctive about Chinese, compared with what a Japanese speaker would intuitively expect, is how much weight sits at the character layer.

Layer 1 Characters – Simplified Traditional Chinese AI Problems Start Here

Simplified and traditional characters do not map one to one

The assumption that “simplified and traditional are just a conversion away” does not survive contact with real documents. During the simplification process, many distinct traditional characters were merged into a single simplified character.

Converting from simplified back to traditional means deciding, from context, which traditional character is correct. Handle that with a naive character substitution and you get errors such as 理发 becoming 理發. The correct form is 理髮. The reverse direction, traditional to simplified, is largely a many-to-one mapping and is therefore comparatively safe, but even there proper nouns and personal names throw up exceptions.

The practical consequence is that unless you decide which script your internal master documents use, both search and aggregation will fork into two incompatible halves.

The same Unicode code point can render as a different glyph

A second issue that is easy to overlook is Unicode Han unification. Characters used in Mainland China, Taiwan, Hong Kong, Japan and Korea are, as a rule, unified into a single code point even where glyph shapes differ, with the exception of characters that were separately encoded in the originating national character sets. Which shape actually appears on screen or on paper is determined by the font and the language tag.

Characters such as 骨, 直, 次, 令 and 者 render with visibly different shapes depending on whether a Chinese font or a Japanese font is applied. Because the byte sequence is identical, searches still match, but printed output and PDFs look different. And from the OCR side, this matters a great deal. Feed a document printed with Chinese glyph shapes to a model trained on the glyph conventions of the Japanese-language region, and recognition accuracy moves in the wrong direction.

On top of that, characters carrying an Ideographic Variation Selector (IVS) are almost indistinguishable on screen without a supporting font, yet their byte sequence differs from the base character alone. If the part-name column of an Excel file received from a business partner contains IVS-tagged characters mixed in with plain ones, exact-match reconciliation fails silently.

Vocabulary differs across the Mainland, Hong Kong, Taiwan and Singapore

It is not only the script that diverges. Vocabulary does too. Here are terms that come up constantly in manufacturing.

Singapore uses simplified characters as its script, but has vocabulary of its own. Hong Kong uses traditional characters while sometimes importing Mainland vocabulary.

Here is what that means in practice. When you search for a laser welding specification, documents from Mainland suppliers are written as 激光焊接 and documents from Taiwanese suppliers as 雷射焊接. Search on only one of the two without a synonym dictionary, and the other set will never surface. This sits on the boundary between Layer 1 and Layer 2, and changing models does nothing about it.

Punctuation and symbols

Chinese uses full-width punctuation (，。；：) and an enumeration comma (、) for lists. Mix half-width commas and periods into that, and both segmentation and sentence splitting are disturbed. When Chinese documents are pulled in from Excel or email, half-width symbols creeping in is an everyday occurrence. Character-layer normalisation needs to include unifying these symbols as well.

Layer 2 Words – Nobody Has Decided Where a Word Ends in a Language Without Spaces

Chinese has no space-based word boundaries the way English does. Characters simply run one after another. Where a word is cut is therefore decided by the processing system, and that decision is inherently ambiguous.

The ambiguity is structural, not accidental

A frequently cited example is 部分居民生活水平. The correct segmentation is 部分 / 居民 / 生活 / 水平, meaning “the standard of living of some residents”. However, that same string also contains 分居 (to live separately) and 民生 (people’s livelihood), each of which is a perfectly valid word in its own right. Dictionary lookup alone therefore produces multiple simultaneously valid segmentations.

This class of ambiguity has been known in Chinese language processing research for decades, and is broadly divided into overlapping ambiguity, where candidate segmentations cross over each other, and combinational ambiguity, where a string makes sense whether you keep it together or split it. The classic example of the latter is 才能, which reads as “talent” as a single word, and as 才 / 能 (“only then can one”) when split into two. Which one is correct can only be determined from the surrounding context.

What a segmentation tool is actually doing

A Chinese segmentation tool such as jieba answers this ambiguity through a combination of dictionary matching and statistical modelling. Broadly, it first uses a prefix dictionary to collect every possible word candidate from the string and builds a directed acyclic graph (DAG), then uses dynamic programming to select the highest-probability segmentation path. For words not present in the dictionary, so-called out-of-vocabulary words, it uses a hidden Markov model (HMM) with the Viterbi algorithm to estimate the role of each character position and carve out the word.

This is the point that matters operationally. Out-of-vocabulary handling is left to statistics, which means that your own part numbers, component names, equipment names and internal abbreviations will, left alone, be segmented incorrectly with high probability. A string such as MX-3200型伺服压机 has no guarantee of surviving as a meaningful unit unless it is registered in a user dictionary.

jieba provides a mechanism for loading user dictionaries, where you register a word along with its part of speech and frequency. pkuseg, HanLP, LTP and other tools offer equivalent mechanisms. What needs to be done is simple: build a list of the proper nouns your company uses and register them. The hard part is not the technology. It is deciding who builds the list and who keeps updating it.

What breaks downstream when the word layer breaks

Word segmentation is the foundation for search, chunking, keyword matching, classification and aggregation. When it breaks, everything stacked above it shifts. The impact is especially large if you are building RAG, because when a chunk boundary lands in the middle of a word, that word can no longer be retrieved at all. Building RAG itself is covered in the cost and approach for RAG implementation.

Layer 3 Tokens – What Chinese NLP Tokenization Does to Your Text

Why one Chinese character is not always one token

Today’s major LLMs use byte-level BPE (Byte Pair Encoding) tokenizers. BPE repeatedly merges the byte pairs that most frequently sit next to each other in the training data, then freezes those merge rules.

A characteristic of Chinese comes into play here. In UTF-8, a typical Chinese character occupies three bytes. Consequently, unless enough merge rules have been learned for that character, a single Chinese character splits into several tokens. Conversely, frequently occurring two-character words do have merge rules, and can come out as a single token for two characters. In short, two Chinese documents of the same character count can produce different token counts depending on the tokenizer.

Tokenizers from the cl100k_base generation onwards are said to contain considerably more Chinese vocabulary than earlier generations, though still not at the efficiency level they achieve for English. Tokenizers belonging to models that were explicitly strengthened for Chinese tend to carry a thicker Chinese vocabulary again. Because the specific multiplier varies by document domain and script, counting on your own real documents is the only reliable method. Start by running three to five representative internal documents through the tokenizers of your candidate models and counting the tokens. That alone dramatically improves the accuracy of both your monthly cost forecast and your context-length design.

Wrong merges happen

There is a further phenomenon reported specifically for Chinese. In The Digital Orientalist article “To Merge or Not to Merge – The Pitfalls of Chinese Tokenization in General-Purpose LLMs”, the example sentence 他是学科技的，他不相信神秘和超自然的事物 was run through the tokenizers of several models. Across multiple models, the authors report cases where a sequence that is not actually a word gets merged into a single token (the 的事 merge reported for LLaMA-family models) and cases of context-blind merging (the 他不 and 的事物 merges observed with DeepSeek).

The cause lies in the mechanics of BPE itself. Because BPE freezes merge rules in order of training-data frequency, it does not look at context at inference time. It applies a merge that is wrong in context simply because that merge is frequent overall. 的 is one of the most common characters in Chinese, so rules merging it with the following character naturally rank high.

The practical implication is limited, but not zero. When token boundaries diverge from semantic word boundaries, the model has a harder time treating a word as a single unit of meaning. This shows up most in use cases that process large numbers of short fragments, such as part-name reconciliation, short-text classification and keyword extraction. You should not choose a model on this basis alone, but it is worth remembering as a candidate explanation when short-text processing keeps producing inexplicable errors.

Layer 4 Meaning – Only Now Do You Choose a Chinese LLM

Once you have sorted out the character and word layers and formed a view on cost and context length at the token layer, and only then, does the conversation turn to model selection. The following are the models whose names come up most often for Chinese work as of 2026. We describe qualitative tendencies only. Published benchmark scores vary enormously with measurement conditions, so this article does not declare a winner on the basis of specific numbers.

Beyond these, Western frontier models also handle Chinese to a high standard. For use cases that go back and forth between Japanese and Chinese, the quality on the Japanese side is under scrutiny at the same time, so a Chinese-specialist model is not automatically the best choice.

What you should be looking at during selection is not a composite benchmark score, but the following.

• Whether your own documents come out at the same quality in both simplified and traditional script
• Whether a supplied glossary is honoured all the way through a document, or whether adherence collapses partway into a long text
• Whether the specified output format (tables, JSON, a fixed heading structure) is respected
• Whether Chinese translated into Japanese reads naturally as manufacturing terminology on the Japanese side
• Where the data is processed, which ties directly into the PDPA question discussed later

And the single most important thing is to have your own evaluation set for making these judgements. Pick 30 to 50 items from your real documents and produce the correct answers by hand. Without that, nobody can say whether a model change made things better or worse.

By Use Case – Translation, Chinese OCR AI and Part-Name Matching

Layer talk tends towards the abstract, so here it is mapped onto use cases.

Reading it is straightforward. Find the use case where your symptom occurs, and start work at the dominant layer named on that row. The rightmost column lists measures that get tried frequently and rarely deliver.

If you are building a mechanism for handling internal enquiries that arrive in Chinese, see also automating the internal helpdesk, and if you are putting a multilingual chatbot in front of external users, see the cost and approach for chatbot implementation.

An Original Estimate – How Cost and Benefit Change With the Approach

What follows is an estimate based on a fictional model plant. The EV investment figures and supplier entry counts cited earlier are real published data, but every number below is a model value set for the purposes of explanation, and does not represent the actual results of any real company. Every intermediate calculation is written out so you can substitute your own numbers and check the arithmetic.

The model plant

The volume of work requiring Chinese, and the time it currently takes with people alone, is as follows.

The current monthly cost is 220 hours × 600 THB = 132,000 THB.

Three configurations

Configuration A – use a general-purpose generative AI as is (fix no layers)

Run on the existing general-purpose generative AI licence alone (3,000 THB per month), with no glossary and no normalisation. Element count is 1.

Configuration B – fix the character and word layers

On top of Configuration A, implement a simplified, traditional and Japanese glossary of 1,200 equipment terms, plus normalisation rules for simplified and traditional script. Element count is 3 (general-purpose AI, glossary, normalisation rules).

Configuration C – work on all four layers

On top of Configuration B, build OCR templates for Chinese forms and select a Chinese-capable LLM after measuring actual token counts on candidate models. Element count is 5 (normalisation rules, glossary, form OCR, token design, Chinese-capable LLM plus evaluation set). In Configuration C the 3,000 THB general-purpose AI licence is replaced by the cost of the Chinese-capable LLM, so it is not counted twice.

How the hours change

Form reading is unchanged in Configurations A and B because that task depends on the Layer 1 OCR implementation, and is not shortened by using a general-purpose AI or a terminology dictionary. It drops from 9 minutes to 3 minutes in Configuration C because OCR templates and simplified-traditional post-processing narrow the range of what a human has to check visually.

Cost and benefit

The initial cost breakdown is mapped onto the layers. Each cost line is allocated to the time saving it produces, down to the minute.

The 56 hours saved in Configuration A come purely from the zero-initial-cost general-purpose AI licence (translation 25 → 16 minutes, email 12 → 8 minutes). Check that the allocation above reconciles with the hours saved shown earlier. What Configuration B adds on top of A is 17 + 17 = 34 hours, matching 90 − 56 = 34 hours. What Configuration C adds on top of B is 40 + 8 + 9 = 57 hours, matching 147 − 90 = 57 hours. In other words, cut any cost line and the time saving written on that line disappears with it.

Here is the monthly cost breakdown.

Converting the hours saved into money gives the net benefit.

The interesting part of this table is that over 12 months the gap between the three configurations is smaller than people expect (367,200 / 403,200 / 439,200 THB). The gap only opens up in year two, and at 24 months Configuration C is about 1.7 times Configuration A. If you put Configuration C into an approval request written on the assumption that initial costs are recovered within one year, the numbers will look weak. It is safer to flag up front that the conclusion changes with the evaluation horizon you choose.

Sensitivity – what if only 70 percent of the savings materialise

The estimate of hours saved is the most uncertain assumption in the whole model. Here we drop that one variable to 70 percent, hold costs unchanged, and recalculate. The 600 THB rate and every cost item stay as they were.

The ranking at the 12-month mark inverts. If the benefit only reaches 70 percent of the assumption, Configuration A is the most favourable over 12 months and Configuration C comes last. Only when you look out to 24 months does Configuration C finally edge past Configuration B (603,360 versus 597,600 THB).

This result does not mean you should abandon Configuration C. It means that you should verify, on a small scale and before committing the investment, whether the hours saved actually materialise. The 90-day approach described next is sequenced to put that verification first.

Extra Considerations for Sites in Thailand and ASEAN

PDPA and cross-border data transfers

Thailand’s Personal Data Protection Act (PDPA) imposes conditions on transferring personal data out of the country. Chinese forms and correspondence with business partners routinely contain personal data such as contact names, contact details and signatures. If you are sending that to an overseas API, you need to know which operator in which country processes it. If you choose a service based inside China for its Chinese capability, Chinese data-related legislation applies in addition to Thailand’s PDPA, so treat the choice of processing location as a legal decision rather than a technical one.

A practical mitigation is a pre-processing step that strips personal data before transmission, replacing names and contact details with masked placeholders. That is not technically difficult, but it presupposes that you have agreed the definition of what must be stripped with your legal function.

Governing language clauses in contracts and specifications

Contracts with Chinese-affiliated suppliers are frequently drawn up in both Chinese and English. Check whether a clause exists specifying which version prevails. Without one, there is nothing to fall back on when interpretations diverge.

Where this bites in a generative AI context is the line between a translated document and an official internal reference document. A Japanese version produced by AI is, contractually, no more than a reference translation. If you intend to use it as a work instruction or an inspection standard, you need an operating practice that records who reviewed the content and who approved it.

Choosing simplified or traditional as your master

This is a matter of agreement rather than technology, but its effects reach across the whole character layer. If most of your suppliers are Mainland-based, simplified is the natural choice, and if most are Taiwanese, traditional is. What matters is that you choose one. Having chosen, normalise to the master script at ingestion and retain the originals separately as originals. With that practice in place, neither search nor aggregation forks.

Operate for several years without deciding, with both mixed together, and the cost of normalising retrospectively grows in proportion to your document volume. This is the kind of decision that gets cheaper the earlier you make it.

Storage and traceability for Chinese documents

If you receive quality records and inspection certificates in Chinese, an audit will ask who verified the content of those records. Where AI translation sits in the chain, design the system to retain the correspondence between the Chinese original and the translation, the person who checked the translation, and the date of that check. A practice that keeps no translation history cannot be reconstructed afterwards.

What to Do in the First 90 Days

Days 1 to 30 – concentrate on counting

You implement nothing during this period. You count what exists.

• Collect the documents arriving in Chinese by type, and count monthly volumes and page counts
• Classify senders as Mainland, Taiwanese or Hong Kong based, and establish how mixed your scripts actually are
• Measure the time currently spent on processing, task by task, by actual measurement rather than estimation
• Extract 50 to 100 of your most frequent terms and draft a correspondence table across simplified, traditional and Japanese
• Run three to five representative documents through the tokenizers of candidate models and count the tokens

The deliverable of these 30 days is the estimate table from the previous section with the “monthly volume” and “time per item” columns filled in with your own numbers. Move on without filling those in, and there is no basis for an investment decision.

Days 31 to 60 – verify on a small scale

Confirm with minimal investment whether the hours saved actually appear.

• Build a provisional dictionary of around 50 terms, apply it to draft translation, and re-measure the time taken
• Apply simplified and traditional normalisation to a single document type and see whether search behaviour changes
• For OCR, prototype a template on one form type with around 20 sheets, and measure the time spent on visual verification
• As the seed of an evaluation set, select 30 items from real documents and produce the correct answers by hand

By day 60 you will know whether the 70 percent figure in the sensitivity analysis is realistic, or whether the true number is higher or lower. If the numbers do not appear, you are in a position to narrow the scope or restrict the target tasks.

Days 61 to 90 – fix the scope and lock it down

• Decide the master script and write it into your document management procedures
• Decide who owns updating the terminology dictionary and how often
• Agree the scope of personal data pre-processing with your legal function
• Decide, based on the verification results, whether to proceed with Configuration A, B or C
• Cut a version of the evaluation set and agree that all future model changes will be compared against that version

The deliverable at day 90 is not a working system. It is a set of settled assumptions. With Chinese, the more things you have left undecided, the more expensive it gets later.

Common Failure Patterns and How to Avoid Them

Starting from model selection

This is the most common failure. Begin by researching which LLM is strongest in Chinese, and you end up swapping out Layer 4 while the character and word layers remain broken. Symptoms such as drifting terminology and searches that miss survive any model change, so you lose time migrating repeatedly. The avoidance is to map your symptom onto the use case table above and identify the dominant layer first.

Implementing simplified-to-traditional conversion as a naive character substitution

Write simplified-to-traditional conversion as a one-to-one substitution table and you will pin 发 to 發, breaking every word relating to hair. The avoidance is either to standardise conversion in one direction only, towards simplified, or to use a conversion library that considers context while excluding proper nouns from the conversion scope.

Building a terminology dictionary and then stopping the updates

Even if you build a 1,200-term dictionary up front, new equipment arrives and new suppliers appear, and the dictionary goes stale. A dictionary is a mechanism for enforcing translations, so an un-updated dictionary keeps enforcing outdated ones indefinitely. The avoidance is to decide the owner and the update frequency at the day-90 mark. The estimate assumes 4 to 6 hours a month of maintenance effort, and that is not a cost you can strip out.

Proceeding on a feeling that accuracy has improved, with no evaluation set

Without an evaluation set, you cannot debate whether a model change was good or bad. The judgement rests on one person’s impression, and it cannot be handed over to their successor. The avoidance is to build correct answers for around 30 real documents and freeze the version. Even 30 is decisively different from none.

Processing documents from Taiwan and Hong Kong with Mainland settings

Vocabulary differs, not just script, so a synonym dictionary built for the Mainland will not retrieve Taiwanese documents. The avoidance is to give the dictionary a column mapping Mainland vocabulary to Taiwanese vocabulary. The terminology table above is the minimal form of that.

Starting to send data to an API before deciding how personal data is handled

It is common for a practice to take hold, on the grounds that it is only a PoC, of throwing real partner emails straight at an external API. The avoidance is to build personal data pre-processing in from the verification stage, or to use only documents containing no personal data for verification.

Frequently Asked Questions

What does Chinese language generative AI actually mean?

It refers to any use of generative AI that involves Chinese text on the input or output side. It is not synonymous with using a Chinese-built model. Handling Chinese with a Western frontier model falls under the same heading. This article organises the topic not by where the model comes from, but by which layer of processing Chinese as a language breaks at.

How accurate is Chinese OCR in practice?

Chinese OCR has become highly accurate for both simplified and traditional script, and several technical write-ups report systems achieving recognition rates above 95 percent. That said, traditional characters have more complex shapes than simplified ones, so the risk of misrecognition is considered relatively higher. Numbers of this kind also swing heavily with print quality, font, layout and scan resolution, so we recommend measuring on your own forms. In practice, what matters more than the raw recognition rate is designing where an error can be caught once it occurs.

Can simplified and traditional characters simply be converted by AI?

No. A single simplified character can correspond to several traditional characters, and which one is correct is determined by context. Conversion from simplified to traditional is error-prone, and the reverse direction is safer. In practice, the most stable arrangement is to pick one master script and normalise to it at ingestion.

Should a Chinese LLM mean a Chinese-built model?

It depends on the use case and on how the data is handled. If you work only in Chinese and the fine detail of Chinese naturalness matters, there are situations where a Chinese-built model has the advantage. On the other hand, use cases that go back and forth between Japanese and Chinese put the quality on the Japanese side under scrutiny at the same time, and if you handle personal data at a Thai site, PDPA and processing location come first. Do not decide on a composite benchmark score alone. Compare on your own evaluation set.

Our Chinese forms are a mixture of simplified and traditional. Where should we start?

Start by deciding which one is your master. Then put in place processing that normalises to the master script at ingestion. Those two steps alone will substantially reduce both missed search results and double counting in aggregation. Building OCR templates can come after that.

Summary

When someone reports that accuracy is poor in Chinese, the problem is usually not the model. Simplified and traditional characters do not map one to one. The same Unicode code point renders as different glyphs in different regions. The absence of spaces makes word segmentation structurally ambiguous. And BPE freezes merge rules without ever looking at context. All of this happens at the character, word and token layers, and none of it disappears when you swap a model at Layer 4.

For Japanese-owned plants in Thailand, Chinese is not a language you choose to adopt. It is a language that arrives, through the EV supply chain and through Chinese-built equipment. That is exactly why the design worth pursuing is not hiring more dedicated Chinese speakers, but enabling the people you already have to work with Chinese primary sources.

As the estimate shows, working on all four layers produces the largest benefit, but the gap in year one is small, and the ranking inverts if the benefit comes in below assumption. So spend the first 90 days not on implementation, but on counting and on verifying at small scale. The things you must decide are the master script, the owner of dictionary updates, the scope of personal data pre-processing, and the version of the evaluation set. With those four settled, you will not go far wrong whichever configuration you proceed with.

Talking It Through at the Consideration Stage

A problem that presents as “accuracy is poor in Chinese” can be tackled by separating out which layer it is happening at. If you would like to look once at where your Chinese equipment manuals or Chinese forms are actually breaking, or to think through how far simplified and traditional normalisation should go, or simply to talk about how to build an evaluation set, the consideration stage is a perfectly good time to get in touch. You can reach us through the contact page.

References

• Thailand Secures $4.1 Billion in EV Chain Investments as Southeast Asia Auto Hub Pivots to Next-Gen Tech – Thailand Board of Investment — Official announcement covering the USD 4.1 billion of committed investment in Thailand’s EV supply chain, 198 projects, and 18 BEV production projects worth USD 1.18 billion with capacity above 370,000 vehicles a year (3 July 2026)
• Thailand approves Sunwoda to invest $1B in EV battery cell plants – TechNode Global — Report on the approval of Sunwoda Electronic’s investment in battery cell manufacturing (17 March 2025)
• Chinese supplier expansion scenarios for Thailand and ASEAN – THAIBIZ, in Japanese — Analysis giving the breakdown of 183 Chinese-affiliated suppliers entering Thailand, 125 of them in processing and manufacturing, and 90 new entries between 2023 and June 2025 (Nomura Research Institute group, February 2026)
• To Merge or Not to Merge – The Pitfalls of Chinese Tokenization in General-Purpose LLMs – The Digital Orientalist — Worked examples of incorrect Chinese merges in general-purpose LLM tokenizers, with an explanation of the BPE mechanics behind them (4 February 2025)
• jieba – Chinese text segmentation — Implementation of the prefix dictionary, directed acyclic graph (DAG), dynamic programming, and HMM with Viterbi for out-of-vocabulary words, plus how to register user dictionaries
• China LLM Guide 2026 – Top Chinese LLMs Compared – Swift Horse AI — Use-case comparison of the tendencies of Chinese-built LLMs, centred on DeepSeek, Qwen, Kimi and GLM (June 2026)
• Parsing Through Boundaries in Chinese Word Segmentation – arXiv — Research on how the placement of Chinese word segmentation boundaries affects downstream parsing (March 2025)
• Maximum Overlapping Ambiguous String dataset – Tsinghua University Natural Language Processing Group — Evaluation dataset for Chinese word segmentation collecting cases of overlapping ambiguity
• Personal Data Protection Act B.E. 2562 – Office of the Personal Data Protection Committee Thailand — The text of Thailand’s Personal Data Protection Act
• Cross-border personal data transfer under Section 29 – Office of the Personal Data Protection Committee Thailand — Requirements for cross-border data transfer under appropriate safeguards