When selecting a production management system, manufacturers often ask how quickly the application can be restored. A more useful question for a plant is what happens to materials, work orders and quality decisions while the application is unavailable. Production may continue on paper, but a duplicate completion after recovery can inflate inventory, and an unchecked quality hold can release the wrong lot. This guide explains how to design manual continuity during a production management system outage, then reconcile paper records, system transactions and physical stock before normal operation resumes. It addresses an operational acceptance test, rather than another general comparison of software features.
Application recovery is not the same as operational recovery
A backup can reopen an application. It cannot automatically reconstruct everything that physically happened while the plant was offline. NIST’s contingency planning guidance includes temporary manual processing among the ways to continue affected business processes. The US Ready.gov business continuity template asks organizations to document the forms and resources required for manual workarounds. These are useful design references, not a claim that US guidance imposes a legal requirement on Thai factories.
If a MES is unavailable, a press might still run. Yet the plant needs evidence of the approved work-order revision, material lot, first-piece inspection, good and reject quantities, and handover to the next step. A paper production count without these relationships cannot safely become a confirmed transaction. Conversely, a device may have sent a completion just before losing its connection. Re-entering its paper copy after recovery would count it twice. The continuity objective is an explainable chain of product, quantity, quality state and authorization.
Define three operational states: normal, restricted manual operation, and reconciled normal operation. Do not mark the third state merely because users can log in again. Return only after outstanding records, duplicate candidates and quality holds have assigned owners and have been checked. A service provider’s availability target should not substitute for the plant’s own return-to-operation criteria.
Classify the outage before choosing a workaround
Separate at least four events: the production application is down; the plant network or site link is down; only a scanner or shared tablet is down; or the ERP-MES-WMS interface is delayed while the individual applications are running. Each changes the available master data, the visibility of work orders, and the location at which a transaction may already have been committed.
For each event, specify detection, declaration authority, affected lines, permitted work, prohibited work and the next review time. A useful rule is: “Existing approved MES orders may continue; new orders and engineering-change products must wait while the ERP-to-MES link is unavailable.” If malware is suspected, IT may need to isolate devices rather than reconnect cached clients. A continuity plan should never instruct operators to bypass machine safety or move uncontrolled data by USB. NIST’s operational technology security guidance is a reference for considering safety and availability together.
Decide which processes may continue and which must stop
Evaluate human safety, customer and regulatory requirements, inspection evidence, lot traceability, wrong-material prevention, stock accuracy and delivery impact. If a mandatory condition cannot be demonstrated manually, stop that operation. A paper form is not itself a substitute for quality assurance. If a process requires measurements captured from an instrument, the quality owner must approve a valid alternative before the step begins.
One practical classification is A: an approved order with available materials and complete manual traceability; B: conditional continuation with finished goods held; and C: stop safely and identify work in process. This classification applies to the condition at the time of the outage, not permanently to a product. An A item can still stop at a later electronic approval gate. A B item needs a physical hold tag and a system hold when service returns. A C item needs a safe-stop procedure and a record of partially processed material.
Set a maximum duration for manual work based on the capacity to record and reconcile transactions, not simply the capacity to run equipment. For illustration only, if the plant creates 100 paper events per hour and cannot reconcile them during the outage, a five-hour outage leaves 500 events outstanding. At a post-recovery reconciliation rate of 40 per hour, clearing them takes 12.5 hours if no new paper events arrive. If new paper events continue to arrive, subtract that inflow from the processing capacity when estimating clearance time; at 40 or more new events per hour, the backlog will not shrink. Replace these hypothetical figures with measured values. Define the backlog threshold at which the plant narrows the product scope or stops.
Measure RTO, RPO and manual capacity separately
Recovery time objective and recovery point objective describe targets for restoring service and limiting data loss. A cloud platform’s RTO/RPO may not cover the plant network, integrations, paper entries or quality release. Microsoft’s business continuity guidance says organizations should document stakeholders, processes and specific steps in their own plans. Record separate targets for application availability, time to start manual work, maximum manual duration, completion of backlog reconciliation and quality release.
A system could recover in two hours while entry of 200 paper transactions and release of held lots take another six. The numbers are an illustration, not a promise. The business return time is longer than the software recovery time. Also identify “boundary transactions” sent just before the interruption: the operator saw no confirmation, but the server may have committed them. Compare transaction identifiers, device send history, interface queues, printed document numbers and physical stock before deciding whether to re-enter them.
Design paper forms for later reconciliation
A paper copy of every screen is too cumbersome; an item and quantity alone is too thin. Give every form a unique number and capture event time and time zone, plant and work center, work order and revision, product and material lots, good and reject quantities, unit, quality state, operator, checker and reason. Use employee identifiers where possible instead of unnecessary personal details. A number structure such as plant-date-form type-line-sequence helps avoid collisions across shifts and lines. Preserve voided and missing numbers in the register. Corrections should retain the original entry, date and approver.
Use separate forms for material issue, production completion and quality decision because their owners and confirmation points differ. Keep the form number as an external reference in the application. The system should allow a user to trace from paper to transaction and back. Decide in advance where blank forms, pens, hold tags and copies will be stored, including when a printer is unavailable. Ready.gov explicitly calls out the resources needed for manual workarounds, because a template alone is not an operating capability.
Keep “awaiting inspection” distinct from “failed inspection”
Define four quality states during an outage: accepted, rejected, inspection pending, and evidence missing. Pending means a test was done but awaits authorization; missing means the plant must determine whether the test happened at all. Do not merge them into “enter later.” Mark held pallets or containers and block allocation and shipment until the quality owner releases them. A hold that exists only in MES but not in WMS will not stop a warehouse user after systems reconnect.
Production permission and shipment permission are separate gates. Exceptions such as a customer concession require the plant’s formal process, identifier and approval. This article does not set a universal rule for release, because customer and sector requirements differ. Its recommendation is to make the authorization path explicit before the outage.
Reconcile paper, transactions and physical stock in three directions
Start by establishing the last confirmed transaction and the state of delayed queues. Put uncertain boundary transactions aside. Register each paper line using form number, line number and transaction type as a unique business key. Check for that key before entry; reject a second entry where the software supports idempotent references. If it does not, use a controlled serial entry queue and independent before-and-after review. Then compare system balances and lot identities with physical material.
Classify differences as duplicate, missing, wrong quantity, wrong item or lot, wrong quality state, or wrong event order. Do not silently overwrite a discrepancy. A lot substitution may leave total quantity unchanged while breaking traceability. Compare both the production completion and the material consumption. A discrepancy in quality state keeps the product held until the quality team resolves it. Microsoft’s guidance for a WMS linked to an external ERP explicitly addresses inventory update logs and reconciliation; the actual log and replay features must be verified for the product being purchased.
Before entry, establish a product-specific recovery order. First pause automatic replay and snapshot the queue and last committed transactions. Confirm the versions of orders and item masters. Count paper records and physical containers. Then enter material issues and returns, operation completions, quality decisions and warehouse moves in the dependency order agreed for this ERP-MES-WMS configuration. The sequence must be tested against the actual product; it is not universal. If a completion automatically consumes material, independently entering the same material issue can deduct inventory twice. If held finished goods are first posted as unrestricted stock, WMS may allocate them. At each gate, compare counts, quantities, lots and quality states; hold differences before advancing. Resume replay only after matching paper entries against the old queue. Use corrective transactions with an audit trail rather than deleting the original record.
A closeout checklist should confirm that all form numbers are collected, voided numbers explained, good plus reject plus remaining WIP reconciles, ERP-MES-WMS queues have no unowned messages, and every held lot has an owner. If a difference remains, isolate the affected lots and record an exception rather than automatically releasing everything or stopping unrelated products.
Make one system the owner of each transaction
ERP, MES and WMS may each show an “inventory” number, but they observe different business moments. Create a responsibility matrix for order release, material issue, operation completion, quality decision, location move and shipment confirmation. For each, identify the normal system of record, the manual source, recovery entry point, approver and downstream replay source.
For example, if a paper location move is entered into WMS and WMS then sends it to ERP, entering the same move independently in ERP would duplicate it. If finance needs a temporary ERP entry, specify how it will be tagged and reversed or matched to the later integration event. Distinguish “sent,” “received” and “applied to business records” in interface logs. A successful send is not proof that the downstream inventory was updated. Our related guide to production system implementation failures discusses wider project risks; the issue here is the outage boundary and recovery transactions.
Put a continuity exercise into procurement acceptance
“Backup included” and “incident support available” are not test cases. Ask the supplier to simulate an outage, issue and collect manual forms, restore service, re-enter transactions and produce an exception report. Start in a safe test environment and include a representative work order, lot switch, reject, partial shipment and engineering change. Agree beforehand who authorizes the test and which plant equipment must remain untouched.
Four scenarios expose common defects. A worker presses Submit when the connection drops; demonstrate how to learn whether the server committed the event. A material lot changes on paper; show cancellation of the old issue and authorization of the new one. Inspection stays pending while MES returns; prove WMS cannot allocate the lot. One side of the ERP-MES interface replays a message twice; demonstrate detection or idempotent rejection. Use a factory acceptance test (FAT) to prove the software and interface logic in a controlled environment, then a site acceptance test (SAT) to verify the plant network, forms, shift roles and physical hold locations. These are proposed contract gates, not universal industry time limits. Evidence for acceptance includes timestamps, issued and returned form register, backlog trend, duplicate log, physical reconciliation, quality releases and signatures for returning to normal operation. Microsoft’s go-live guidance calls for completed integration, user acceptance and performance tests as well as migration, cutover, training and access roles. The proposed exercise applies those broad checks to this factory-specific failure mode.
Compare eight items in a supplier proposal
| Item | Evidence to request |
|---|---|
| Detection and contact | Monitored components, escalation path and out-of-hours language coverage |
| Manual declaration | Authorized decision maker, prohibited work and review interval |
| Forms and numbering | Version control, issue register, corrections and loss handling |
| Quality hold | Physical isolation, electronic allocation block and release authority |
| Recovery and replay | Backup test, interface queue visibility and duplicate control |
| Reconciliation | Paper-system-physical exception report and owner |
| Training | Shift, weekend and deputy participation in exercises |
| Closure | Return-to-normal sign-off and management of unresolved differences |
Ask what is included in the price: software recovery, paper-form design, data entry, interface replay, reconciliation report, training, annual revision and out-of-hours support may be different scopes. Do not treat an unpriced line as free. For the broader investment decision, see production management system advantages and disadvantages. The additional procurement question here is who will prove that materials and data match after an outage.
A worked night-shift example
Imagine an ERP-to-MES work-order link fails at 22:00. Existing approved orders remain visible and WMS is available. The shift leader assigns an incident number and restricts work to orders with a confirmed revision. IT checks that the interface queue will not automatically replay messages without review. A scheduled material-lot substitution moves the affected products to conditional category B. Operators record the actual issued lot on a numbered form, while quality places finished containers on hold. The next shift receives the form register, missing-form list, held lot IDs and queue status.
When the link returns, the team first checks whether the ERP order revision arriving in MES matches the one used during production. A mismatch is referred for engineering and quality assessment, not attached to a convenient order. Then the team compares paper issues and completions with MES, WMS and physical containers. If software is restored in 30 minutes but reconciliation takes another 90, operational recovery took two hours in this hypothetical example. Exercises should record the real timings and bottlenecks.
A reconciliation sheet should include form and line IDs, event time, transaction type, order, item and lot, quantity, unit, quality state, system transaction ID, entrant, reviewer, difference category and correction date. “Entered” and “reconciled” are separate statuses. Equal total quantity can hide a wrong lot. Keep both the overall count and line-level check in the approval record. If the software assigns a new message ID on each retry, retain the original business reference alongside every technical message ID.
Make the procedure usable at a Thai plant
A procedure written only for the Japanese head office may fail during a Thai night shift. Provide a short Thai decision card at each affected work center, with Japanese or English alongside where the supervising team needs it. Translate the operational states, stop reasons, quality hold, release authority and discrepancy categories—not only application buttons. The glossary must distinguish an IT outage from a machine breakdown and an inspection failure. Each calls for a different owner and may require a different stop decision.
At shift handover, transfer the unused and issued form-number ranges, unreturned sheets, held lots and the number of interface messages still awaiting review. Saying “we are on paper” is insufficient when the prior shift may have a boundary transaction awaiting confirmation. Put the current system state, incident number, last confirmed order revision and contact list in a handover register. Ask the incoming lead to read back the restrictions in Thai. If forms could be damaged by oil or water, select a suitable writing surface and storage location in advance.
A phone photograph can assist evidence collection, but it is not automatically an authorized replacement for a controlled form. Decide access rights, retention, upload method and how the photograph links to the original numbered record. Keep a procedure for power loss when printers and chargers are unavailable. These details determine whether the manual workflow works at the workstation, rather than only in a conference-room plan.
Include night and weekend shifts, production, quality, stores and dispatch in an exercise. Name deputies for each role, because the person who designed the procedure may be away when the incident happens. NIST’s 2026 OT backup guide recommends integrating backups with change management, testing them, and reviewing them during recovery exercises. If a technology restore drill and business reconciliation drill occur together, the plant can measure the gap between application recovery and release of usable stock.
Design the reconciliation worksheet before the outage
The recovery team cannot reliably “find differences” without agreeing what to compare. Make one line per form number, line number and transaction type. Include event time, order and revision, item and lot, quantity and unit, quality state, system transaction ID, entering user, reviewer, discrepancy type, disposition and correction date. Check that the chosen package can export or display the necessary system fields. If a required worksheet field does not exist on the paper form, recovery will require interviews with operators; if a system field cannot be obtained, the table will remain incomplete.
Report forms issued, missing and voided separately from forms entered and forms reconciled. Entry is not proof of reconciliation. A correct total may conceal a wrong material lot, so show both totals and line detail. The reviewer should sign off on the material-consumption record as well as the finished-output record. Keep an exception list for unresolved items, with lot, owner, action and release restriction. Do not allow a green “backlog zero” indicator to erase a quality hold.
For interface retries, store the original business reference and every technical message ID. Some products generate a new message ID on each attempt; an apparently new technical message can still represent the same movement. Ask the supplier to demonstrate what happens if one numbered form is entered twice, and how users can query a transaction when a connection disappears before the response. Where the product lacks idempotent entry, the manual queue and independent check become explicit acceptance controls.
Assign deputies and sequence the return decision
List people, not only departments: who declares the incident, diagnoses the technology, decides which products continue, distributes and collects forms, enters recovery transactions, reviews discrepancies and releases held goods. Add a deputy, language and reachable contact for every role. For high-risk entries, separate the person typing from the person approving. A small factory that cannot staff both simultaneously can require review by the next shift or a manager before releasing the affected lot.
The order of approvals matters. IT reporting “service restored” does not release a quality hold. Collection of every paper form does not prove physical stock is correct. ERP-MES synchronization does not justify closing an order when WIP location is unknown. Use one closeout sheet with distinct checks and signatures by production, quality, stores and IT. An unresolved exception can stay assigned and held while unaffected products resume, if the approved scope permits it.
Give the same owner matrix to the vendor and local integrator. Identify whether the software supplier’s service ends at application/database recovery or extends to interface replay and reconciliation reporting; identify whether the local implementation partner assists with paper-entry recovery and exercises. Agree contact changes and weekend support before procurement closes. Otherwise, the factory may discover during an incident that each party assumed another party owned the disputed transaction.
A practical 90-day design path
In the first month, map affected operations, previous outages, transaction ownership and stop/continue thresholds. In the second, prototype forms, numbering, hold tags and exception reports. Time how long an operator wearing normal protective equipment needs to fill a form, and test duplicate entry and late device sends at a desk. In the third, run a safely scoped exercise from outage declaration through a zero-unowned-backlog decision. Revise forms and assign versions. Ninety days is a planning example, not a universal implementation duration. The meaningful milestone is one complete exercise before accepting the system.
Frequently asked questions
Must every line stop when the production system goes down?
No universal rule applies. Continue only the processes whose safety, quality, customer requirements and lot traceability can be evidenced manually, and only within the approved scope and duration. Stop any step for which mandatory evidence cannot be secured. Production and shipment decisions remain separate.
What fields belong on a manual production record?
Use a unique form and line number, event time and time zone, plant and work center, approved order and revision, product and material lots, good and reject quantities, unit, quality state, operator and checker. Add process-specific measurement values and hold reasons. The test is whether the record can be matched one-to-one to a system transaction after recovery.
Does a backup eliminate the need for paper continuity?
No. A backup restores an earlier system state; it does not create a record of physical production or movement during the outage. Uncertain transactions sent just before failure also need log and physical checks.
How do we prevent duplicate production counts after recovery?
Assign a unique form and line number, check the external reference before entry, separate uncertain boundary transactions, inspect replay queues, then reconcile paper, system and physical goods. If the product lacks idempotent entry, control the entry queue and review it independently.
What should a supplier quote include?
Separate configuration and software recovery from form design, quality holds, queue replay, reconciliation, exercises, training, revisions and out-of-hours support. Compare quotes using the same outage acceptance scenarios rather than an assumed fixed market price.
Conclusion
Manual continuity for a production management system outage is complete only when the plant can decide what to stop, keep traceable evidence for permitted work, reconcile every paper event with system transactions and physical stock, and retain quality holds until authorized release. Put that full cycle in the acceptance test. The practical procurement question is not only how fast the application comes back, but who demonstrates that the factory’s goods and records agree.
If your Thai factory is still selecting a production system or drafting an RFP, contact TOMAS TECH to discuss a limited outage scenario and the ERP, MES and WMS boundaries that need a recovery owner.
