Factory Productivity with AI: Implementing Delivery Delay Prediction and Acceptance Tests

Even if you implement an AI system that predicts delivery delays, simply displaying “70% delay probability” on a screen will not protect your shipments. To truly improve productivity in a factory using AI, you must design not only when the prediction is made, but also who can take action on which orders, and when and by whom customers will be contacted. This article provides concrete guidance on RFP creation and acceptance testing for delivery delay prediction AI, aimed at production management leaders, factory IT, and sales/purchasing staff at manufacturing sites using ERP and MES—including those in Thailand. All figures and thresholds below are illustrative and not industry averages or proven improvement rates.

Translating “AI Productivity Improvement in Factories” into Delivery Decision-Making

Factory productivity cannot be measured by equipment utilization alone. Even if you detect delivery risks early, unless that information leads to actionable decisions by responsible staff, you may only increase overtime, expedited shipping, or work-in-process inventory. Predictive AI is only useful when there are still options available to meet promised delivery dates, and actions can be taken on a per-order basis. For example, can the factory consider alternative lines before sales contacts the customer? If there’s a material shortage, can purchasing arrange alternative sourcing? This time margin must be included in your acceptance criteria.

If you want a broader overview of production planning AI types and implementation issues, see our AI Production Planning Explained. For general approaches to progress visualization and delay prevention, refer to Delivery Delay Prevention. This article goes a step further, focusing on the practicalities of “fixing the prediction point in time, verifying predictions, and tracking the results of interventions” for each order and manufacturing instruction.

Don’t Reduce the Prediction Target to Just “Delivery Date”

The promised date to the customer, the requested delivery date in ERP, the factory’s planned completion date, warehouse shipping date, carrier pickup date, and customer receipt date are all different. If you train your AI without clarifying which of these is the prediction target, the AI may label orders as delayed even when the shop floor considers them on time, and vice versa. First, define the “delivery delay” ground truth label across sales, logistics, and production management. For example, you might define a delay as any order line where inspection, packing, and shipping confirmation are not all completed by the agreed shipping deadline. However, if the contractual delivery point is the customer’s warehouse, factory shipment alone does not count as fulfillment. Capture differences by customer, product, and contract terms in your specifications.

If an order has multiple lines and partial shipments are allowed, you must also define whether you treat the entire order as one unit or pair each line with its promised date—this affects your denominator. Decide how to handle cancellations, renegotiated deadlines, manual changes by sales, and urgent special approvals. If you change the ground truth after the fact for convenience, your model may look good on paper, but your customer response will not improve.

Linking ERP, MES, and Shop Floor Events to the Same Order Line

ISA-95 is a framework for organizing information exchange between enterprise business systems and manufacturing operations. It describes ERP as closer to business planning/logistics and MES as closer to manufacturing operations. However, ISA-95 does not specify particular AI models or database products. In this context, it serves as a guide for organizing identifiers, meaning, and update responsibility when connecting ERP commitments, MES progress, and equipment events.

The first integration failures often occur when similar numbers are assumed to mean the same thing. Do not assume customer order numbers and internal order numbers, manufacturing orders and rework orders, or parent and split lots are one-to-one. Use intermediate mapping tables and revision histories, and count unmatched rows as missing data rather than silently dropping them. If you display “low risk” for unmatched items, the most problematic cases may disappear from your dashboard.

Manage Three Types of Timestamps

At minimum, distinguish between the event occurrence time on the shop floor, the registration time when the system receives the data, and the prediction time when the model makes its judgment. For example, if night shift staff enter paper records the next morning, even if you backdate the occurrence time, the AI running the previous night could not have known that result. For training and validation features, use only information available to the system by the prediction time. When spanning sites in Thailand and Japan, normalize times internally to UTC or similar, and clearly display local time and time zone to users. Do not infer times from date-only columns.

Assign Data Owners for Each Integration Item

“Integrate ERP and MES” is not a sufficient RFP specification. For each data item, specify the original source, updater, update frequency, delay tolerance, revision method, and display behavior when matching fails. For example, if only sales can update the ERP promise date, the AI must not overwrite the MES requested delivery date arbitrarily. If the reason for equipment downtime is not yet confirmed, do not use it as a finalized failure code for training. By clarifying input responsibility and correction contacts, you can distinguish model errors from data operation errors.

Fixing the Prediction Point and Preventing Data Leakage

Delivery delay prediction is prone to “hindsight bias”—using information that would not have been available at the time. If you train using shipping confirmation dates, inspection results at completion, or expedited shipping flags set after a delay is known, your model may appear highly accurate on historical data. However, this information is unavailable to the person making decisions the day before, so it cannot be reproduced in operation. This is known as data leakage, and must be checked. In acceptance testing, save for each prediction row: the prediction time, data version used, feature acquisition time, and model version, so that auditors can reproduce each case.

For example, suppose you run predictions every day at 08:00 for orders due in the next 7 days. Use the ERP order version as of 08:00 and MES process events received by 08:00; do not use results entered at 09:30 for the previous day in that day’s prediction. If results are corrected later, do not automatically overwrite past predictions with “what was known at the time.” Use the corrected version for the next prediction, and keep the old version for audit. This is an illustrative design—08:00 and 7 days are not industry standards.

For training data, split by time series. If you mix earlier process steps of the same order in training and later steps in validation, you risk leaking results from the same case. Whether simple random splits are acceptable depends on the independence of your cases; at minimum, use validation that preserves future periods. Include busy seasons, new products, equipment changes, and customer order pattern shifts, and check for differences from operational conditions. NIST’s industrial AI guidance also emphasizes the importance of evaluating input data for relevance, accuracy, representativeness, missingness, and rare events.

Allow for “Unable to Predict” Outputs

If material arrival forecasts are blank, MES is offline, or orders and manufacturing instructions cannot be matched, it is safer to output “unable to predict/needs review” than to force a 0% or 100% probability. Display not only the missing rate, but also which decision-critical items are missing. For new products or lines with little historical data, do not output probabilities with the same confidence as for established products. Define rules for out-of-scope predictions and escalation paths for human review before selecting a model.

Separate Delay Probability from Action Recommendations

The prediction model’s job is to indicate, at the defined prediction point, the likelihood that a given order will miss its deadline. Deciding which countermeasures to take, however, is a separate problem involving quality constraints, equipment capacity, personnel, materials, customer priority, cost, and impact on other orders. IBM distinguishes between predictive analytics and prescriptive analytics, with the latter linked to action selection in planning, scheduling, inventory, and logistics. However, do not cite IBM’s statements as proof that any specific delay prediction product will necessarily deliver results.

For initial implementation, it is practical not to promise automatic plan changes, but to design the prediction screen to include rationale and intervention options. Staff can select options such as “expedite materials,” “consider alternative lines,” “switch shipping carrier,” or “propose partial delivery to customer,” and check feasibility. The reasons shown by AI are not proof of causality. For example, do not assume that “material not received is common among high-risk cases” means that expediting materials will always prevent delays.

Tailor Alerts by Role and Deadline

Even for the same risk, production management looks at process sequence, purchasing at material arrival, maintenance at equipment recovery, and sales at customer agreement terms. The screen should display the relevant order line, promise date, prediction time, current progress, missing data, responsible person, and next decision deadline together. Simply sorting by risk level mixes cases that can and cannot be addressed. Define state transitions, such as moving cases with no intervention options left to customer contact and alternative proposal review.

Customer contact requires separate approval. Do not implement AI that automatically notifies customers of delays without confirming contract terms and sales judgment. In testing, verify who approves revised estimates, who communicates externally, and where contact time, content, and renegotiated dates are recorded. Also establish correction paths for mistaken predictions. Do not combine the operation to close an alert with the operation to complete a customer commitment.

Acceptance Testing: Go Beyond “Accuracy” Alone

AI evaluation metrics are necessary, but a single aggregated accuracy figure is not enough for order management decisions. In periods with few delays, simply answering “on time” for all cases may yield high accuracy. List missed cases, false alarms, usefulness of top alerts, probability calibration, and time from prediction to action, and review both shop floor workload and customer impact. ISO 22400-1 provides a neutral framework for defining, structuring, and using manufacturing operations KPIs, but does not specify pass/fail thresholds for delivery prediction AI. Set thresholds based on your own delivery, cost, and staffing realities.

The first stage of acceptance testing is historical replay. Restore the data visible at each prediction time and check that no future information has leaked in. The second stage is shadow operation on the shop floor. The AI records predictions only, without changing official plans or customer communications. Staff continue normal operations and record what information they used for decisions. In the third stage, try human-approved interventions on limited lines and cases. When expanding scope, use not only model performance but also data integration stability, response time, false alarm burden, and audit trails.

Constructing Reproduction Tests for Each Judgment Point

In reproduction testing, it is not sufficient to simply review delayed cases after the fact. For each order line item, the judgment time is fixed, and only the versions of ERP and MES registered up to that time are reloaded. The test record should list the order number and line item number, the version of the promised date, the manufacturing instruction and process version, the occurrence and registration times of each event, the model version, the prediction result, and the final result. For systems where input data cannot be reverted to the same point in time, at minimum, the acceptance criteria must require a design that retains daily snapshots and revision logs. If either is missing, even if good results are obtained from past data, it does not constitute verification of “predictions as they were at the time.”

As a concrete test case, prepare a hypothetical timeline where AI makes a judgment at 08:00 on Monday, sales renegotiates the delivery date with the customer at 10:00 the same day, and the shop floor retroactively registers Sunday’s process results at 11:00. Confirm that the prediction at 08:00 on Monday uses the old promised date and does not use the new promised date at 10:00 or the actual results at 11:00. In Tuesday’s prediction, use the renegotiated promised date and the actual results that have already been received. An implementation that overwrites past records with the new promised date and even changes the correct label is unacceptable. It is required to save the state before and after renegotiation as separate versions and to be able to track which version the delay was against.

Another case to prepare is where MES communication is down from 07:30 to 09:00 on Monday, but the equipment itself recovers at 07:45. At 08:00, the recovery event has not yet reached the system, so the AI cannot treat the recovery as known. The 08:00 screen should indicate “information is outdated due to communication failure,” and whether to issue an alert or mark as indeterminable should follow pre-defined business rules. After communication resumes at 09:00, the latest event is incorporated, and any changes to the previous alert are recorded. Such tests should also be extended to partial shipments, split instructions, changes in material delivery schedules, and re-evaluation of inspections.

Comparing Missed and False Alarms Within the Same Processing Framework

When comparing models, it is important that only AI does not selectively choose favorable days or cases. Both the baseline conventional rules and the AI must be run over the same period, with the same order line items, at the same judgment times, and using the same definition of delay for correct answers. The number of alerts that a person in charge can check per day should be determined in advance, and within that limit, compare how many actual delayed cases were detected, how many days in advance they were detected, and how many false alarms occurred. This is to avoid comparisons that favor methods capable of issuing a large number of alerts.

Suppose, for example, that the person in charge can only check 20 cases per day. This 20 cases is a hypothetical assumption for explanation purposes. Both the conventional rules and the AI should be limited to the top 20 cases, and the number of detected delayed cases and those that could have been intervened upon should be compared. Even if the AI detects delays earlier, if all are cases where the shipping deadline has already passed, the effect is limited. Conversely, even false alarms that lead to the discovery and correction of data entry errors can be recorded as a separate operational benefit from prediction performance. Absences and shift differences of the person in charge on each day should also be recorded in the test record, to confirm the difference between the theoretical processing capacity and the actual processing capability.

If the alert threshold is adjusted after the fact to fit the test data, that same data must not be used as the basis for final pass/fail judgment. Separate periods should be used for threshold determination and for final confirmation. Any bias, such as alerts being concentrated only for specific customers or missed only for new products, should also be tabulated. Pass/fail decisions should not be based solely on “AI accuracy above a certain percentage,” but should also take into account the speed of actionable intervention on the shop floor, the volume of alerts that can be processed, and the completion of customer notifications.

Replace Numeric Examples with Your Own Data

Suppose you have 1,000 order lines per month, 50 actual delays, and 20 alerts that staff can check daily. These numbers are for illustration only. Before deciding on implementation, check your own actual delay rate, number of cases staff can review, and lead time required for intervention. For model comparison, use the same month and prediction point, and set a simple rule such as “alert if planned completion date exceeds promise date” as a baseline. Verify whether AI can find actionable cases earlier or with fewer unnecessary alerts than this baseline.

For example, if you issue hundreds of alerts daily to “catch as many delays as possible,” staff will be overwhelmed. If you filter too much, you may miss critical customer cases. Separate misses by priority customer, product, and promise condition, and do not decide pass/fail on overall averages alone. Adjust thresholds and staff processing capacity together to find the operational balance point.

Don’t Confuse “Prediction Effectiveness” with “Intervention Effectiveness”

If AI predicts a delay, staff expedite materials, and the deadline is met, the prediction appears wrong. However, the intervention may have changed the outcome. Record the risk at prediction, intervention details and timing, comparable cases without intervention, and final results separately. If you measure model accuracy only on post-AI outcomes, your evaluation will be skewed. For implementation impact, it is easier to explain if you separately track early detection, intervention execution, delivery results, and quality of customer communication.

Specifications and Responsibilities to Include in Your RFP

In your RFP, describe which order lines are judged when, what information is used, and who takes action—before showing dashboard screen samples. “ERP/MES integration possible” or “high AI accuracy” alone are not enough to compare vendor proposals. Provide anonymized real data samples, and have vendors solve test cases under the same conditions, including missing data, revisions, partial shipments, and rework. Do not set a fixed required history period or cost for training; instead, request estimates based on data granularity and rate of change.

Even if a vendor says an item is “predictable,” check for API, license, or history retention limitations on your existing ERP/MES. If some processes cannot provide data, define a first-phase scope excluding those processes. Rather than postponing integration because AI is difficult, first visualize what cannot be integrated. NIST’s AI Risk Management Framework is a voluntary framework for considering trustworthiness from design through operation and evaluation. The test table in this article is a practical proposal referencing that framework, not a regulatory requirement.

Points to Confirm When Implementing in Thai Factories

At Thai manufacturing sites, shop floor screens may mix Thai for local staff and Japanese/English for headquarters. Translating only the risk display is not enough—if definitions of promise dates, reasons for equipment downtime, or notifications to responsible staff are inconsistent, the same case may be treated as a different status. Prepare a glossary with original terms, business meanings, and screen labels, and create test sheets in languages readable by approvers. Also test escalation across shift changes and holidays using actual work schedules.

When connecting from a Thai factory to headquarters ERP, always display the last sync time if communication is lost. Do not show alerts based on outdated data as if they are new predictions. Separate access rights for sales viewing customer information, production management viewing instructions, and external vendors handling validation data. For handling of personal or contract information, check your own company’s policies and legal requirements individually.

After Implementation: Monitor Both Model and Operations

Post-launch, monitor not only the prediction model but also ERP/MES join rates, number of late-arriving events, cases unable to be judged, time to open alerts, intervention approval wait times, and time to customer contact. When new products, process changes, or supplier changes occur, compare with past data and revise thresholds or models as needed. When the model version changes, be able to explain output differences for the same case.

In improvement meetings, review not just “was it accurate,” but “who was able to do what.” If no action could be taken on a high-risk case, was it due to equipment capacity, approval delays, or customer negotiation deadlines? The next investment area will differ. Sometimes revising shop floor input rules, material commitment methods, or customer contact responsibilities is more effective than retraining the AI.

Compare Model Changes Using the Same Yardstick

When introducing a new model, do not just swap out prediction results—run both old and new versions in parallel on the same input at the same prediction point. Check which order lines saw increases or decreases in alerts, and whether “unable to predict” increased for any product group. Even if overall accuracy rises in a month with product mix changes, if misses increase for key customers, it is not a pass. Record reasons for changes, approvers, start times, and rollback procedures, and retain alert history from before the model update.

To avoid blaming operational issues on the model, check for input data delays and integration failures daily. If the ERP promise date is updated but the MES instruction remains on the old version, the screen will be inconsistent even if the AI works correctly. Weekly, check alert processing and intervention results; monthly, review thresholds and scope—adjust meeting cycles to your business rhythm. These frequencies are examples; set them according to your system and customer contracts.

Clearly Define a Small Initial Scope

For the first proof of concept, it is easier to evaluate if you limit the scope to product groups/lines where promise date definitions are consistent and process results can be tracked. However, if you select only areas with almost no delays, you will not know if it works for difficult cases. Include both normal and delayed cases, as well as examples of late entry and rework, and clearly state which lines or products are out of scope. Do not extrapolate results directly to all factories; check whether data definitions, work shifts, and customer terms are the same in the next expansion area.

Also set exit criteria in advance. For example, if unmatched cases are more frequent than expected, return to improving ID mapping rather than model tuning. If staff cannot process all alerts, revise notification targets or prioritization. Even with high prediction performance, if customer contact is stalled awaiting approval, change the operational flow. By including both success and stop conditions in your approval and acceptance documents, you prevent ambiguous pilots from moving to production.

NIST’s 2026 roadmap for AI/ML in smart manufacturing also highlights the complexity of industrial data, integration with heterogeneous sensing/control systems, and challenges of reliability and explainability. This does not guarantee any specific method or improvement rate for delivery delay prediction. When making implementation decisions, document how you addressed these challenges in your own data contracts and operational acceptance tests.

FAQ: How Many Years of History Are Needed to Implement Delivery Delay Prediction AI?

There is no universal answer. Check whether you have enough cases for each product group, busy season, process change, and actual delay for both training and validation. Representative data recorded with the same meaning as current processes is more important than large volumes of old history. First, define your ground truth labels and prediction points, build a baseline with available data, and plan to collect more if needed.

FAQ: Can We Start with ERP Only?

In some cases, you can create simple alerts from ERP alone—using promise dates, order quantities, material allocations, and planned completion dates. However, if process results are delayed, early prediction and explanatory power are limited. Start small with a clearly defined scope, and compare missed and false alerts under the same test conditions when you add MES process events.

FAQ: Should High Probability Automatically Trigger Customer Notification?

No. Probability is an input for decision-making, not authority to change customer commitments. Confirm delivery terms, alternatives, internal approvals, and customer acceptability, and have an authorized person make contact. In acceptance testing, separate drafting notification messages from approval to send, and check logs and correction paths.

Conclusion: Place Acceptance Criteria in Business Operations

Delivery delay prediction AI is not valuable just for showing delay probabilities. You must verify, through a series of tests, the definition of commitments, prediction timing, prevention of data leakage, ERP/MES integration, handling of “unable to predict,” staff interventions, and customer contact. Evaluate under the same conditions as your baseline, and judge whether actionable cases can be found within the number and time your staff can handle. AI investments aimed at factory productivity improvement can only be evaluated by linking model metrics to business outcomes.

We can support you from the stage of organizing RFP items and small-scale acceptance tests based on your own order lines and process results. Even if you have not yet finalized your target systems or delivery definitions, please contact TOMAS TECH.

References

1. ISA, ISA-95 Standard: Enterprise-Control System Integration — A framework for information exchange between ERP and manufacturing operations. Does not specify individual AI requirements.
2. ISO, ISO 22400-1:2014 — Overview of concepts, terminology, and use of KPIs in manufacturing operations management. Does not specify pass/fail values for specific AI.
3. NIST, How to Find the Right Balance of Data for Your Industrial AI System — Considerations for relevance, accuracy, representativeness, and missingness in industrial AI data.
4. NIST, AI Risk Management Framework — Voluntary risk management framework for AI design, use, and evaluation.
5. IBM, Prescriptive analytics: Manage manufacturing costs and improve ROI — Vendor perspective on the distinction between prediction and action selection.
6. IBM Research, Continuous prediction of manufacturing performance throughout the production lifecycle — Research example of updating predictions during manufacturing processes. Note: Focuses on product quality, not proven delivery delay effects.
7. NIST, 2026 Roadmap on Artificial Intelligence and Machine Learning for Smart Manufacturing — 2026 primary source summarizing challenges of industrial data, heterogeneous system integration, and reliability.

*Editorial note: All scenarios, times, quantities, and thresholds in this article are illustrative for design explanation. They are not guaranteed statistical results.*