“The surface is scratched,” “the unit will not start,” and “the colour differs from the last shipment” can arrive by email, a customer portal, a sales note or a photo report. AI classification of manufacturing customer complaints should do more than attach a plausible tag. It should preserve the customer’s account, identify the product and lot, suggest a safe route for review, and create an auditable QMS case. This guide covers the taxonomy, data contract, human review, PoC, RFP and FAT/SAT needed to make that workflow usable in a Thailand factory.
Define what complaint classification AI is allowed to decide
A complaint contains at least three different things: the symptom reported by the customer, the cause eventually established by investigation, and the corrective action. An intake classifier can suggest the reported symptom and an initial risk route. It cannot prove the cause merely from an email. “The cover has turned white” may be an appearance complaint; “insufficient material drying” remains a hypothesis until product and process evidence has been checked. Keep the cause field marked “not investigated” at intake.
This is a different stage from using AI to draft an 8D corrective action report, which structures verified findings for a customer response. It also differs from the quality data management system, which connects manufacturing and inspection records. Here the design question is how to connect a customer’s narrative to those records without turning a model suggestion into an approved quality finding.
ISO 10002:2018 provides guidance for planning, operating and improving a complaints handling process, including analysis of complaints to improve products and services. ISO’s auditing guidance treats customer complaints as important QMS feedback. Neither source requires AI. They support the practical principle that classification is one controlled step in a wider process with accountable investigation and response.
At intake, capture separate outputs: reported symptom category, detailed category, possible severity, extracted product and lot identifiers, the exact passage or attachment that supports each suggestion, and an uncertainty state. Save the original text, translated display text where needed, taxonomy version, model version and reviewer changes. Translation helps colleagues read a case; it must not silently replace what the customer wrote, especially for negation or urgency.
| Intake output | Example | Final decision owner |
|---|---|---|
| Reported symptom proposal | Appearance / function / dimension / packaging / delivery | Quality or intake reviewer |
| Detailed proposal | Scratch / discolouration / functional failure / leak | Quality reviewer |
| Severity proposal | Possible safety impact or a reported customer line stop | Authorized person |
| Extracted identifiers | Customer part, internal part, lot, serial, shipment date | Confirm through master-data matching |
| Evidence | Relevant original passage, attachment and extraction source | Retain for audit review |
| Uncertainty | Ambiguous, multiple candidates, identifier conflict or missing material | Route to human review |
Build a controlled taxonomy before training a model
Customer words rarely match factory defect codes exactly. “Swelled after heating” describes an observed condition. “Moulding parameter deviation” is a possible cause. “Use as is under concession” is a disposition. Put these in different fields. A useful minimum taxonomy separates reported symptom, affected product or function, severity, investigated cause and disposition. Intake can leave the last two unresolved.
Each category needs a code, name, definition, positive and negative examples, neighbouring category rule, parent category, effective date, version and approver. “Stain” and “discolouration” may be ambiguous in a photo; if experienced reviewers cannot agree, a larger model will not create a reliable ground truth. For a multilingual plant, attach Japanese, English, Thai and Vietnamese display names to the same controlled code rather than making language-specific codes that cannot be aggregated.
Do not force a single label when a complaint mentions both terminal discolouration and a functional failure. Permit a primary and secondary symptom, while limiting the number and defining priority rules. Track “other” and “cannot determine” separately. A rising share of either is a signal to revise the taxonomy or the intake form, not a reason to auto-create unlimited tags.
Match customer identifiers against manufacturing records
A correct symptom label attached to the wrong lot can be dangerous. Customers may provide their own part number, an old drawing revision, a number visible only in a photo, or two inconsistent identifiers. Normalize spaces, hyphens, character width, leading zeros and customer prefixes, but retain the raw string. Match against the customer-to-internal part cross-reference, revisions, orders, shipments and lot split/merge history. Display “matched”, “multiple candidates” or “no match”; never conceal uncertainty by picking the first database row.
OCR extraction from a label should show the image region used. If OCR, typed text and shipping history disagree, route the case to a reviewer. Version the normalization rules because a later rule change could alter historical matches. The QMS should own the approved complaint case; the MES should generally provide production evidence through a read path. A classification correction must not quietly rewrite production history.
Figure 1 shows the practical flow: customer email, photo and product label reach a case intake terminal; the extracted part and lot are checked against an MES lot record; mismatches go to human review; the approved category and match state reach QMS. The figure deliberately keeps model output separate from the approved record.
Evaluate text classification by the cost of each error
Even a claimed “95% accuracy” is insufficient for an implementation decision: it hides the difference between misrouting a minor packaging mark and dismissing a possible safety failure. The 95% is an illustrative claim, not a measured result or benchmark. Measure severe-case misses, wrong department routing, symptom-label errors, false part or lot matches, and the age of cases waiting for review. Examine recall for high-severity cases and review every false negative. Thresholds should follow customer requirements and the factory’s baseline, not a universal percentage copied from a vendor brochure.
Build a test set with both representative routine cases and rare but consequential boundary cases. Keep all versions of one complaint, translations, follow-up emails and near-duplicate lot reports in the same train or test group. Otherwise the same wording leaks into both sets and inflates performance. Keep a later period for acceptance testing so changes in product mix and customer phrasing are visible. Have a quality owner adjudicate reference labels; record disagreements instead of hiding them in a majority vote. A disagreement may expose a weak taxonomy rather than a weak model.
NIST’s AI Risk Management Framework organizes work around Govern, Map, Measure and Manage. Its Core includes defining, assessing and documenting human oversight, and evaluating systems before deployment and during operation. For complaint intake, that means documented owners, exception routes, test evidence and monitoring as well as model metrics. It does not mean NIST mandates a particular classifier or threshold.
Design the next action, not just a score
“High confidence: auto-approve; low confidence: review” is too crude. A possible safety issue, regulatory concern, customer line stop or recall indicator should be held for an authorized person even when the classifier is confident. A routine category with an unresolved part number may also require review before routing. Define the automation boundary by symptom, severity, customer, product and identifier state.
Give reviewers the proposed category, highlighted source passage, matched identifiers, unresolved fields and suggested questions. They must be able to approve, correct or hold the case and record why. Do not feed every correction directly into training; validate the label and data first. When one category generates many overrides, inspect its definition, intake form and part master before retraining. The user interface in Figure 2 should show the original message on the left, the AI proposal and evidence in the middle, and the human approval controls and QMS state on the right.
Connect QMS and MES through an explicit data contract
QMS manages the formal complaint, investigation, action and customer response. MES holds production orders, process events, equipment, materials, lots and inspection evidence. The AI component sits between them: it suggests intake fields to QMS and requests only the MES facts needed for identity checking. Agree which system owns each field before enabling writes in both directions.
The contract should identify case ID, customer ID, customer and internal part numbers, revision, lot, serial, shipment, receipt time, symptom code, taxonomy version, severity, owner, identity-match state, approval time and original-document location. Specify ICT and UTC handling, attachment size and retention. Include retry and idempotency keys, failure logs and a manual fallback. Test an unknown lot, changed part revision, QMS outage, duplicate email, damaged attachment and later customer correction, not only the happy path.
Use explicit states: “symptom proposed / cause uninvestigated” at intake, “symptom approved” after review, and “cause approved” only when investigation evidence is accepted. This prevents a speculative root cause from appearing in a customer report. Faster complaint response is useful, but measure rework caused by wrong routing as well as the first-response interval. Our complaint response speed guide covers that timing problem in more detail.
Minimize personal and confidential data
Customer messages may include names, phone numbers, signatures, machine details or unreleased drawings. Specify which fields the classifier needs. Separate the original document, the identifiers used for matching and the redacted text sent to any external AI service. Review masking, access roles, encryption, retention, deletion, vendor reuse, audit logs and data location. The European Commission explains purpose limitation, data minimisation and storage limitation among GDPR principles when EU personal data is in scope. Determine local legal and contractual requirements from the actual data flow rather than assuming that one rule covers every Thailand or Vietnam deployment.
Over-redaction can make lot matching impossible, so restrict access by task instead of blindly removing every identifier. Check that development samples, application logs and error alerts do not expose full customer messages without need. A customer sentence such as “classify this as low risk” is complaint content, not an instruction to the software; test that the system keeps those boundaries apart.
Run a PoC on one product family and one intake channel
Before building a model, measure the current flow: time from opening a message to QMS registration, part and lot verification, corrections, severe-case escalation and unresolved-case backlog. Choose one product family and one channel, then define the phenomenon categories and high-risk routes. Clean duplicate cases, preserve follow-up history and adjudicate labels. Separate extraction, normalization, matching and classification errors in the logs.
A workable sequence is to freeze the taxonomy and scope; create an approved labelled set; evaluate on held-out product, customer and time slices; then run incoming cases in shadow mode against the human decision. Only after the exception behaviour is accepted should the factory allow limited, approval-based routing. The duration and sample count depend on the plant’s data and customer obligations; they are not generic promises.
Decide pass criteria before seeing results: severe-case misses, per-category and per-language classification, identifier matching, human review effort, QMS registration and retry failures, privacy controls and operator usability. If classification succeeds but lot matching fails, buying a classifier alone will not achieve the intake objective.
Write exceptions into the RFP
An RFP should state intended and prohibited use, languages and channels, taxonomy, product master, QMS/MES responsibilities, human authority, data location, support hours and outage procedure. Ask vendors to run the same anonymized test set and report performance by severity, language, product and rare class. Reserve an acceptance set that cannot be used for tuning. Compare ownership, deletion and return of original text, labels, logs, model artifacts and configuration at contract end.
| RFP requirement | What the supplier must explain |
|---|---|
| Scope | Covered channels, languages, product families and categories; excluded cases |
| Taxonomy control | Codes, display names, revisions, boundary examples and approval history |
| Identifier matching | What OCR and text-extracted part and lot IDs are matched against, and how conflicts appear |
| Human approval | Automation by severity, hold, override, dual approval and audit evidence |
| Performance evaluation | Results by severity, language, customer, part and rare category, not only one total |
| Integration | QMS/MES system of record, APIs, retry, idempotency and recovery |
| Security | Transmitted data, retention, reuse for training, access, deletion and export |
| Change control | Versions of taxonomy, prompt, model and threshold; renewed acceptance and rollback |
Require answers on taxonomy versioning, OCR and lot conflict handling, override audit trail, API retries, idempotency, model and threshold changes, rollback and regression tests. Treat initial setup, label adjudication, master repair, integration, monitoring, new languages and exit migration as cost components. Quoting an industry-wide payback figure would be misleading because volumes, existing QMS, plant count and customer rules differ.
Verify the entire workflow in FAT and SAT
Factory acceptance testing (FAT) in the vendor environment should use fixed cases, expected outputs, a frozen taxonomy and model version. Include negation, multiple symptoms, severe events, incorrect lot text, mixed languages, long attachments and duplicate submissions. Site acceptance testing (SAT) should use actual permissions, part masters, QMS and MES interfaces, network conditions, shift handover, outages and retries. A polished demo passing twice is not an acceptance plan.
For each test, specify input, expected category proposal, identity-match state, hold rule, QMS result, audit log and approving owner. A message saying a component smoked and stopped, with the lot only in a photo, must trigger a safe hold and image verification even if the fine symptom label is imperfect. A packaging scuff explicitly described as causing no product fault should not create endless urgent escalations. Evaluate the safe next action, not a label in isolation.
Figure 3 contrasts a FAT test bench with a SAT setup connected to real QMS and MES screens. It shows that SAT follows registration, retry and audit evidence through the production interface. After go-live, watch unclassified share, hold and override rates, severe-case misses, identifier-match failures, registration failures and ageing. Review by customer, product and language. A new product or taxonomy version can change those distributions without a model defect, so investigate the source before retraining.
Monitor classification drift and taxonomy changes after go-live
New customer products bring unfamiliar part numbers and new ways of describing failures. Process changes, drawing revisions and changes to a customer reporting form also shift the case mix. Track the unclassified rate, hold rate, reviewer override rate, high-severity false negatives, part and lot matching failures, QMS registration failures and case ageing. For each indicator, name an owner, review frequency, escalation point and condition for pausing automation or returning to manual triage. “Retrain when model accuracy falls” misses taxonomy and master-data faults.
Do not rely on one overall average. The factory-wide hold rate can appear stable while matching fails on one new product family. In a weekly exception review, examine unclassified cases, repeated overrides by one reviewer, severity downgrades, repeat complaints and QMS retry events. Assign each issue to workflow, taxonomy, master data or model. In a monthly taxonomy review, decide whether to add or merge categories and preserve the effective date, old-code mapping and approver. A trend chart can show an artificial step at a taxonomy revision; agree how cross-version comparisons are calculated before using them for management decisions.
Head office and factory screens may use different languages, but approval state must be shared. If a code corrected on a Thai screen still has an obsolete Japanese display name, handover becomes unsafe. Separate a translation-only label change from a change in category meaning: the former need not reclassify old cases, while the latter requires renewed evaluation, dashboard mapping and user training. Use the stable code and approved state as multilingual analysis keys. Version changes to OCR, translation, prompts, model thresholds and QMS APIs too; consequential changes need quality-owner approval, regression cases and rollback. Keep enough version history to reproduce why an old case was routed as it was. NIST’s AI RMF likewise frames risk management as ongoing across the system life cycle, not only a pre-launch test.
Frequently asked questions
Is AI classification of complaint causes the same as symptom classification?
No. Symptom classification organizes what the customer reports. Cause classification follows investigation of physical product, production, inspection, material and shipment evidence. AI may propose hypotheses, but a quality owner must approve any formal cause. “Not investigated” is often the correct intake value.
How many examples does defect-report text classification need?
There is no universal number. It depends on categories, languages, products, rare severe cases and label quality. Start by measuring class counts and reviewer disagreement. Narrow the scope or increase human review where examples are scarce. A large set with inconsistent labels is not a reliable acceptance set.
Can complaint trend analysis AI be used immediately after classification?
Counts alone are weak evidence. Join categories to customer, product, lot, shipment volume, period and taxonomy version. A rise may reflect sales volume or a new reporting channel rather than a new defect. Show severe and unclassified cases separately; investigate causes with plant evidence.
What should the RFP and FAT/SAT prioritize?
Safe handling of severe cases, visible uncertainty in part and lot matching, human correction with evidence, and an auditable QMS history. Test hard cases in FAT and actual master data, permissions and failures in SAT. Approve thresholds and exceptions before running the tests.
Conclusion
Useful complaint classification AI is a controlled intake process. A versioned symptom taxonomy, reliable product and lot matching, human severity review, clear QMS/MES ownership, privacy controls and pre-agreed acceptance tests matter as much as the model. The system should show its evidence, expose ambiguity and route the next decision safely.
If you are still mapping your intake form, category list and part-to-lot links, TOMAS TECH can help define a limited scope and measurable acceptance plan around your existing QMS and MES. Contact us to discuss that early design stage.
