When an AI implementation is failing, rushing into more development can make the situation worse. The PoC looked good but production accuracy fell; operators stopped using the tool; ERP or machine integration stalled; the vendor keeps proposing another round of tuning. At this point, the organization does not need another generic AI adoption roadmap. It needs a 30-day recovery diagnosis that separates symptoms from causes and uses evidence to choose one of four outcomes: continue, reduce scope, redesign, or stop.
This guide is for AI projects at Thai operations that have stalled or missed expectations. It does not use weak failure-rate statistics, fictional fees, or invented case studies. It explains how to reassess baselines, test sets, holdouts, business KPIs, model metrics, data drift, human override, IT/OT integration, logs, stop rules, and contract/exit materials, then turn the evidence into an approvable decision memo.
Do not let “AI implementation failure” remain only a symptom
“Accuracy is bad,” “it is too slow,” and “the factory does not use it” are symptoms, not root causes. Lower accuracy may come from changed input distributions, incorrect labels, a new business definition, different preprocessing, permissions, model versions, thresholds, interface design, or training. Asking a vendor to “improve accuracy” from the symptom alone often produces endless model tuning while workflow and data defects remain.
In the first diagnostic meeting, translate opinions into observations.
| Symptom statement | Diagnostic fact to capture |
|---|---|
| Accuracy is poor | Which period, site, product, language, class, and metric changed relative to which baseline? |
| People do not use it | Eligible users, opportunities, actual use, abandonment point, corrections, and fallback |
| It is slow | End-to-end delay split into preprocessing, inference, API, UI, and approval |
| Integration does not work | Exact system of record, API, network, identity, schema, or owner at the failure point |
| There is no ROI | Comparable baseline, post-AI business difference, added operating work, and time window |
Do not start by assigning blame. Start by asking which hypothesis can be rejected with which evidence. Responsibility can later be reconciled with the contract; lost logs and overwritten configurations may never be recovered.
Preserve evidence before the 30-day recovery diagnosis
Do not retrain or make sweeping configuration changes on day one. The current state may be the only usable comparator. Declare a diagnostic freeze: except for urgent safety controls, no one changes the model, prompt, threshold, preprocessing, retrieval index, or data connection without approval.
What to preserve in the first 48 hours
- Production and PoC versions of models, prompts, rules, libraries, and containers.
- IDs, extraction conditions, hashes, and creation dates for training, validation, test, and holdout data.
- Input, output, confidence, citation, error, latency, and human-override logs.
- ERP, MES, SCADA, data-lake, and API-gateway interfaces and change histories.
- Requirements, acceptance criteria, PoC reports, minutes, change requests, incidents, and vendor reports.
- Agreement, SOW, data terms, SLA, IP, transition support, and subprocessors.
Avoid creating uncontrolled diagnostic copies, especially where personal data or trade secrets may be present. Use ETDA’s official Thai PDPA materials as a starting point and review purpose, access, retention, and processing roles with legal counsel or the DPO. This is not legal advice.
The 30-day AI project recovery sprint
Thirty days is not a promise to repair everything. It is a timebox for collecting sufficient evidence to prevent another ambiguous investment decision.
Days 1–5: preserve facts and map symptoms
Freeze contracts, requirements, versions, data, and logs. Classify symptoms into business, model, data, integration, operations, and governance. Include representatives from management, the business, users, IT, security, legal, and the vendor. Build one timeline: what the project originally intended, what changed, when it changed, and who approved it.
Deliverables are an evidence inventory, symptom map, change timeline, and missing-evidence list. Do not select a solution yet.
Days 6–12: rebuild the baseline and evaluation design
If the original success criterion was vague, remeasure the current business baseline. Verify the separation of test and holdout data, label quality, target population, period, and segments by product, language, site, or shift. Determine whether PoC data represented production conditions.
Deliverables are a metric dictionary, data lineage, reproducible evaluation script, evaluated version, and known bias/limitations.
Days 13–20: test causal hypotheses with limited experiments
For each hypothesis, change the smallest possible factor. Changing model, prompt, data, and threshold together hides causality. Prefer comparisons such as the same model with the previous preprocessor, the same test set with one model version changed, or business outcomes before and after human override.
Record hypothesis, experiment, result, uncertainty, and reproduction conditions.
Days 21–26: assess operational, integration, and contractual recoverability
Even a repairable model cannot reach production if data rights, APIs, OT safety, operations, or vendor dependency are unrecoverable. Separately assess named operators, monitoring, incident response, cost drivers, and whether code, settings, and evidence can be transferred.
Days 27–30: approve a four-option decision memo
Compare continue, reduce scope, redesign, and stop in the same format: benefit, residual risk, resources, next experiment, stop rule, and contract impact. The final day must produce a decision with an owner, deadline, and evidence gate—not another “continue reviewing” status.
Rebuild the baseline
A frequent failure in recovery work is the absence of a valid comparator. If volume, product mix, shift, staff experience, or demand changed between pre- and post-AI periods, a simple before/after comparison is misleading. A baseline is the measured current or alternative process under comparable conditions, not an idealized “no-AI” number.
Three baseline layers
- Business baseline: handling time, rework, first resolution, defect escape, or downtime.
- Method baseline: rules, search, statistical model, or human decision against which AI competes.
- Risk baseline: severity of errors, privacy exposure, unapproved actions, and recovery time.
A model score may improve while extra review makes the workflow slower. Conversely, a modest average improvement may still create value if severe cases are reliably escalated. Place business and model evidence in the same table.
Check test-set and holdout contamination
When a PoC performs well and production does not, investigate whether evaluation data leaked into development. Repeatedly adjusting prompts or thresholds while viewing test results turns the test set into development data. Restore clear roles:
- Training/development: used to train or improve prompts and logic.
- Validation: used to choose approaches, thresholds, and hyperparameters.
- Test: used to assess the selected approach, not for repeated optimization.
- Holdout: untouched until the final decision gate.
- Production shadow: inputs sampled from production to observe distribution without automatic action.
Small datasets still require separation discipline. Random splitting may be wrong when the same machine, document family, customer, lot, or adjacent time records appear on both sides. Split by a meaningful unit and record duplicates, derivatives, labelers, and versions.
The MEASURE function in NIST AI RMF Core calls for documenting test sets, metrics, and TEVV tools, measurement in deployment-like conditions, and production monitoring. The Core is not a checklist or a mandatory sequence. Its GOVERN, MAP, MEASURE, and MANAGE functions are intended to be applied iteratively according to context. A recovery diagnosis must make the measurement method reproducible, not merely report a score.
Reassess business KPIs separately from model metrics
Model metrics are necessary but insufficient for a continuation decision. Map each model/system metric to the business outcome it can affect.
| Use case | Model/system metrics | Business KPIs |
|---|---|---|
| Document answers | factuality, citation, abstention, latency | search time, first resolution, misinformation, review time |
| Visual inspection | recall/precision by defect, throughput | escaped defects, false-alarm checks, takt impact |
| Demand forecast | error distribution, bias, critical-period error | stockouts, excess, plan changes, emergency freight |
| Maintenance support | detection, lead time, false alarms | unplanned downtime, inspection effort, unnecessary parts |
Segment by product, equipment, language, user, shift, and risk class rather than relying on averages. Check whether severe minority classes are hidden. When comparing AI users and non-users, also examine selection bias.
Separate data drift from context drift
Data drift is a change in input distribution, but it is not the only source of missed expectations. There may be concept drift in the relationship between input and label, context drift in purpose or workflow, pipeline drift caused by ETL/API changes, or changed user behavior.
Drift diagnosis table
| Type | Example | Evidence | Candidate response |
|---|---|---|---|
| Data drift | product mix, language, lighting | time distribution, segment metrics | monitor, resample |
| Concept drift | business definition of correct changed | old/new label policy, expert agreement | relabel, reevaluate |
| Pipeline drift | unit, imputation, API changed | schema, ETL version, interface logs | repair preprocessing, contract tests |
| Context drift | user or decision changed | SOP, permissions, use logs | reduce scope, redesign workflow |
“Retrain whenever drift is detected” is unsafe. If the problem is a label policy or ETL defect, retraining embeds the error. Define detection, impact review, approval, reevaluation, and release.
NIST’s AI Metrology Center connects metrics, methods, and tools to AI RMF characteristics and lifecycle stages. Its page explicitly says inclusion does not constitute NIST endorsement, validation, or suitability. A recovery team must establish that a metric fits its use case rather than adopting it because it appears in a repository.
Verify that human override actually worked
A specification may say “the human makes the final decision” even when operators cannot meaningfully override AI. An AI recommendation can become a default; rejection may require extra justification; the response window may be too short; users may lack permission; disagreement may harm performance ratings.
Do not judge override rates as simply high or low. Examine the situation, reason, outcome, time required, AI confidence, and user role. Rare correct overrides differ from necessary overrides that were impossible. Test stop authority, restart approval, manual procedure, appeal, and log preservation.
Diagnose integration and IT/OT separately from the model
In Thai factories, a healthy model can fail at ERP, MES, SCADA, PLC, camera, network, or time synchronization. Before retraining, trace representative records end to end from input to business action.
Check the system of record, acquisition time, units, missing values, retries, ordering, identifier mapping, permissions, timeout, buffering, and manual mode. For vision, verify camera configuration, lighting, lens, trigger, and compression. For sensors, inspect calibration and replacement histories. When AI can influence OT control, test whether it can return to read-only, whether equipment has a fail-safe state, and whether changes trigger quality or customer approval.
The diagnosis must include equipment, controls, quality, production, maintenance, and cybersecurity—not only the AI team. If ownership stops at the model API, assign an owner for the end-to-end KPI.
Use logs and reproducibility to review the AI vendor
An AI vendor review does not assume that the relationship must end. It tests reproducibility and transferability as contracted deliverables. For a diagnostic sample, determine whether the buyer or an independent team can reproduce results with the same version, input, and configuration.
Minimum reproduction package
- Architecture/data flow and version matrix for model, prompt, rules, and index.
- Runtime, dependencies, configuration, and secret-reference method.
- Evaluation-data generation, label definition, metric code, and expected results.
- Deployment, rollback, monitoring, incident, backup, and decommissioning runbooks.
- Third-party APIs, OSS, licenses, terms, and subprocessors.
- Known limitations, unresolved issues, technical debt, and backlog.
Do not accept “proprietary technology” as a complete reason for non-reproducibility. Even when IP cannot be disclosed, parties can agree on input conditions, output, interfaces, performance evidence, monitoring, and export at exit. Define the black-box scope and the buyer’s verifiable boundary.
Write stop rules before further repair
Recovery becomes endless when “one more adjustment may work” has no deadline. Approve stop rules before the recovery sprint.
Examples include failure to meet a critical safety-class minimum on holdout data, reproducible unauthorized-data exposure, inability to establish data rights by a fixed date, failure to execute the manual fallback in time, missing required logs, or improvement below a pre-agreed minimum effect. Set thresholds from project risk tolerance; do not copy generic values.
Stopping does not always mean eliminating the whole system. Stop autonomous action but retain decision support, limit products, remove sensitive data, or continue only one language. Prepare a scope-reduction option beside every stop rule.
Four choices: continue, reduce scope, redesign, or stop
Continue
Choose this only when the cause is identified, improvement repeats in a limited experiment, and residual risk is acceptable. Attach the next review date, monitoring owner, change approval, and stop rules. Continuation by inertia is not a decision.
Reduce scope
Use this when value exists in selected segments but not across every site, product, language, or user. Define the new scope, permission, manual work, and KPIs. This is not a return to an open-ended PoC; it is a narrow production scope with demonstrated value.
Redesign
Use this when the objective remains valid but the architecture, data pipeline, human workflow, or evaluation design is fundamentally wrong. Separate reusable and discarded assets and approve the redesign as a distinct project. Do not relabel the original project as successful simply to roll its budget forward.
Stop
Stop when there is no value for the stated objective, critical risk cannot be controlled, data or rights are unavailable, or a non-AI alternative is more rational. Include user notice, manual process, data return/deletion, access revocation, settlement, asset retention, and lessons learned.
NIST AI RMF Core’s MANAGE function includes determining whether intended objectives are achieved and whether development or deployment should proceed. It also covers mechanisms to supersede, disengage, or deactivate inconsistent systems. Stopping is a normal risk-management option, not a concealment of failure.
Decision memo template
The day-30 output should be a short approvable memo with evidence in appendices:
- Original objective and current symptoms: scope, users, expectation, observations.
- Causes and confidence: separate confirmed, probable, and unresolved.
- Reevaluation: baseline, holdout, segments, business KPIs, risks, integration.
- Four-option comparison: benefit, residual risk, resource, time, contract impact.
- Recommended decision: one choice and its rationale.
- Conditions and stop rules: next gate, owner, evidence, and deadline.
- Dissent: unresolved objections and why a decision is still justified.
The U.S. GAO AI Accountability Framework organizes accountability around governance, data, performance, and monitoring. Although originally created for federal agencies and other entities, it provides useful questions for breaking an existing AI system into auditable evidence. It is not a Thai legal requirement or a direct corporate mandate.
Diagnose contract and exit materials
Contracts can constrain recovery more than technology. If rights to models or code, data export, log access, third-party services, minimum commitments, and transition support are unclear, redesign and vendor transition cannot be compared.
Review the master agreement, SOW, change orders, DPA, SLA, licenses, acceptance records, invoice basis, subprocessors, and termination terms. Where contract and reality differ, record facts and escalate to legal counsel.
Exit-readiness checks
- Can data, labels, prompts, configuration, logs, and evaluation results be received in machine-readable formats?
- Can customer, vendor, and third-party assets be separated?
- Can credentials and connections be revoked and deletion—including backups—be evidenced?
- Will a successor receive interfaces, schema, runbooks, and known limitations?
- Is there a safe parallel-transition period and a defined shutdown point?
ISO/IEC 42001:2023 specifies requirements to establish, implement, maintain, and continually improve an AI management system. Do not decide recovery from certification alone. Check whether accountability, risk assessment, change, monitoring, corrective action, and continual improvement are actually documented for this system.
How to use 2026 TEVV material
On 7 August 2026, NIST sought comment on the initial public draft of NIST AI 200-2, the TEVV-Athlon Framework for evaluating AI systems, through 6 October 2026. It is not a finalized standard. Use it as evolving material for thinking about test, evaluation, verification, and validation across diverse AI use cases—not as a mandatory certification or contractual conformity standard.
The draft introduces a customizable assessment concept based on organizational TEVV objectives. For recovery, that supports designing evidence around operational goals rather than reducing the system to one score. The memo should state that the terminology and structure may change.
ETDA’s Generative AI Governance Guideline similarly provides useful organizational context across benefits, limitations, risks, application modes, and governance. Treat it as guidance and distinguish it from Thai law and contractual duties.
FAQ about AI implementation failure
Does AI project recovery mean replacing the vendor?
No. If requirements, data, or internal operations caused the failure, changing vendors may repeat it. First test evidence and reproducibility, then compare correction with the current vendor, reduced scope, redesign, and transition on equal terms.
Can an AI PoC reevaluation reuse the old test set?
If the set was repeatedly viewed during tuning, it may not be independent. Review its usage history and create an untouched holdout where possible. With limited data, split by time, equipment, customer, or blinded review to reduce leakage. For establishing cost drivers and success gates early, see AI PoC cost and success criteria in Thailand.
Is improved model accuracy enough to continue?
No. Recheck business KPIs, critical classes, permissions, human override, latency, integration, operating burden, and residual risk. See AI quality management and acceptance testing in Thailand for more on acceptance design.
If operators do not use AI, is training the problem?
Not necessarily. Slow output, difficult correction, unclear accountability, conflict with SOPs, ineffective override, or a login-based adoption KPI may be design failures. Use logs and field observation to locate abandonment. See AI adoption support in Thailand for sustainable operating practices.
Can the system be fixed in 30 days?
The 30 days in this guide are a diagnostic timebox, not a repair guarantee. Limited repairs may be tested, but the primary output is evidence on causes, reevaluation, options, stop rules, and contract implications.
Does stopping mean the investment was wasted?
Stopping can prevent additional loss and preserve data, evaluation methods, interfaces, and lessons for future work. Decide from future benefit and risk, not sunk cost.
Conclusion: restore the ability to decide before trying to repair
When AI implementation is failing, preserve versions, data, logs, and contracts before more tuning. Separate symptoms from causes. In a 30-day recovery sprint, reassess baseline, test/holdout separation, business KPIs, drift, human override, IT/OT, reproducibility, and stop rules, then compare continue, reduce scope, redesign, and stop in one decision memo.
If an AI project at a Thai operation has stalled and you need to separate model issues from workflow and IT/OT issues, contact TOMAS TECH. We can support the diagnostic phase without assuming that continuation is the only acceptable result.
Primary sources checked on 2 September 2026
- NIST AIRC, AI RMF Core
- NIST, AI Resource Center
- NIST AIRC, AI Metrology Center
- NIST, The TEVV-Athlon Framework for Evaluating AI Systems
- U.S. GAO, Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities
- ISO, ISO/IEC 42001:2023 AI management systems
- ETDA, Generative AI Governance Guideline for Organizations
- ETDA, Thailand Personal Data Protection Act resources
