AI OCR Accuracy: PoC and Acceptance Testing for Multilingual Documents

AI OCR accuracy cannot be compared responsibly by placing one advertised “recognition rate” beside another. A number measured on a vendor’s test set does not tell you whether the system can process your Japanese, English, Thai, and Vietnamese documents, preserve dates and currencies, extract tax IDs and line items, or route uncertain results to a reviewer. This guide shows how to run an AI OCR implementation PoC and acceptance test using field-level precision, recall and F1; critical-field exact match; document exact match; straight-through processing (STP); critical-error leakage; processing failure; repeatability; and version-change regression.

The practical goal is not to find the product with the highest headline rate. It is to create a measurable operating contract that says what may pass automatically, what must be reviewed, and what evidence is required before production acceptance.

Why a single recognition rate is not an OCR accuracy comparison

“99% OCR recognition” sounds precise, but it is not comparable unless the denominator, matching rule, test population, preprocessing and exclusions are disclosed. Was the rate calculated per character, word, field or document? Were empty optional fields counted as correct? How were repeating line items aligned? Were unreadable images and API failures removed from the denominator? Different answers can produce very different percentages from the same output.

Business impact also varies by field. A missing punctuation mark in a note does not create the same risk as a wrong beneficiary account, tax number, currency, total or decimal point. The purpose of document OCR is not merely to produce text. It is to shorten downstream registration, matching, approval, payment and audit work without losing control. The evaluation design should therefore start from error cost and process risk.

Google Document AI’s official evaluation material explains precision, recall, F1 and confidence thresholds, and shows why precision/recall-based evaluation is useful when documents contain optional fields or repeating line items. Microsoft’s Document Intelligence transparency material emphasizes use-case-specific evaluation, confidence-informed handling and human review. Amazon Textract expresses confidence on a 0–100 scale, while 0–1 scores are encountered in other services. Raw scores must not be treated as vendor-neutral probabilities: 0.95 and 95 may look convertible, but calibration differs by model, processor, version and data population.

Eight measures for document OCR accuracy

An acceptance test should calculate at least the following measures from the same frozen test set. One metric alone can hide a serious failure in another part of the workflow.

Precision is TP ÷ (TP + FP) and recall is TP ÷ (TP + FN). A true positive (TP) is a correctly extracted field. A false positive (FP) is a nonexistent or wrong value emitted as a valid candidate. A false negative (FN) is an expected value that was missed. F1 is 2 × precision × recall ÷ (precision + recall).

The counting contract must be frozen before the test. If an optional field is absent in ground truth and the system also omits it, that result is a true negative. Extraction tasks can contain vast numbers of true negatives, so adding them to a simple accuracy denominator can make results look artificially high. Precision and recall expose both hallucinated optional fields and missed line items without allowing empty space to dominate the score.

Building a stratified AI OCR PoC test set

The PoC set should represent production conditions, not merely documents that are convenient to collect. An evaluation based only on clean digital PDFs often fails when deployment receives scans from multifunction printers, mobile phones, fax workflows or compressed email attachments.

Stratify by language, document type and image quality

Map the input population by language, document type, supplier, template, capture channel, image quality, handwriting, stamp overlap and page count. A high volume of Japanese invoices should not hide failures in lower-volume but high-risk Thai tax documents or Vietnamese supplier documents. Maintain both a production-representative set and a challenge set that deliberately over-samples rare and high-impact conditions.

Use at least three quality strata—good, medium and poor—and tag skew, blur, shadow, folds, bleed-through, compression artifacts, faint print, patterned backgrounds, cropping and rotation. AWS best-practice material treats input quality as a material factor and recommends handling confidence according to the use case. A quality tag makes that sensitivity measurable rather than anecdotal.

Separate tuning data from the acceptance holdout

The implementation team may inspect a development set while adjusting preprocessing, extraction schemas, templates or prompts. The final acceptance holdout must remain unseen until configuration is frozen. Reusing the same documents for tuning and acceptance turns template memorization into apparent improvement. Near duplicates, rescans of the same original and consecutive documents from one template must stay on the same side of the split.

Audit the ground truth

Ground truth can itself be wrong. Thai tone marks, Vietnamese diacritics, Japanese full-width and half-width forms, tax-ID separators, and ambiguous 0/O or 1/I characters create annotation disagreement. Use independent double entry and adjudication for critical fields, sampled quality control for general fields, and retain annotator, reviewer and revision history.

Keep failed inputs in scope

Timeouts, empty responses, encrypted PDFs, page-limit errors, files labeled corrupt and API failures must not disappear from the accuracy report. “96% on processed documents” is incomplete if 5% of the workload could not be processed. Track initial failure, recovery after retry, permanent failure and manual fallback separately.

Normalization rules for Japanese, English, Thai and Vietnamese

Document OCR accuracy depends on whether output is compared as raw text or as a normalized business value. Normalization is legitimate when it maps semantically equivalent representations to a declared form; it becomes misleading when it silently repairs model errors or erases meaningful distinctions.

Unicode Standard Annex #15 defines canonical equivalence and normalization forms. NFC can be a practical evaluation baseline, but NFKC should not be applied globally without field-specific approval because compatibility normalization can alter distinctions. Retain the source image, raw OCR output and normalized value so every transformation is auditable.

For mixed-language pages, retain the language of each field as well as the primary document language. A Thai address on an English form should not disappear inside the English-document average. Microsoft’s published OCR language information includes Thai and Vietnamese, but model, feature, region, API version, and preview/general-availability status can change. Treat the linked material as referenced on 26 August 2026 and re-check current official specifications with your exact inputs before procurement.

Use exact match and business constraints for critical fields

Character similarity is not sufficient for money or identifiers. 1,000.00 and 10,000.00 are textually similar but materially different in payment processing. Critical fields should normally use exact match after approved normalization, supplemented by business controls such as:

• line subtotal plus tax equals document total;
• currency symbol agrees with currency code;
• tax and registration IDs meet declared length or checksum rules;
• invoice date precedes or reasonably relates to due date;
• purchase order exists in the ERP candidate set;
• supplier identity and registered account agree with master data.

Report rule detection separately from OCR correctness. If master-data matching catches a wrong account, the control is effective, but the OCR result is still wrong. Separate “correct at OCR,” “detected by rule,” and “corrected by human” to identify where investment is needed.

Set confidence thresholds from measured risk, not a universal standard

A confidence score is not a guaranteed probability of correctness. Measure the relationship between score and observed correctness on the exact documents you plan to use. Build calibration tables by score band, then segment them by language, document type, image quality and criticality. A score that is reliable on clean English forms may behave differently on a poor-quality Thai scan.

Do not search for one industry-standard threshold. Choose thresholds from error cost and reviewer capacity. A beneficiary account or total may require a high threshold, a business rule and human review, while an unused note field may need a lower threshold or no extraction at all. As described in Google’s evaluation guidance, moving a confidence threshold changes the precision/recall trade-off. Raising it commonly reduces automatic passage and improves precision, but it may reduce recall and STP.

Practical threshold procedure

1. Classify fields as critical, controlled or informational.
2. Sweep candidate thresholds in 0.01 increments or another appropriate interval.
3. Recalculate precision, recall, F1, STP, leakage and review volume at every threshold.
4. Inspect the worst language, document and image-quality strata.
5. Overlay reviewer capacity and processing SLA.
6. Record the selected threshold, rationale, approver and model version.

If only low-confidence reviewed cases are audited, the evaluation becomes selection-biased. Retain random sampling of high-confidence auto-passed documents and targeted sampling of high-risk strata. That is the only way to estimate critical-error leakage outside the review queue.

Worked AI OCR accuracy example

Every number in this section is an illustrative assumption for explaining the calculation. It is not measured vendor performance, a TOMAS TECH guarantee, or a forecast of business benefit.

Assume a test set with four language groups—Japanese, English, Thai and Vietnamese—three quality strata—good, medium and poor—and 100 documents per cell. The total is 4 × 3 × 100 = 1,200 documents. Across them, ground truth contains 9,600 target field instances. Suppose the run produces TP 9,024, FP 216 and FN 576:

• Precision = 9,024 ÷ (9,024 + 216) = 97.66%
• Recall = 9,024 ÷ (9,024 + 576) = 94.00%
• F1 = 2 × 97.66% × 94.00% ÷ (97.66% + 94.00%) = 95.80% after rounding

If 3,480 of 3,600 critical fields are exact after normalization, critical-field exact match is 3,480 ÷ 3,600 = 96.67%. If all required fields are exact in 684 documents, document exact match is 684 ÷ 1,200 = 57.00%. This shows why high field metrics do not automatically translate into correction-free documents.

If 870 documents pass without human intervention, STP is 870 ÷ 1,200 = 72.50%. If 9 of those 870 auto-passed documents contain at least one critical error, critical-error leakage is 9 ÷ 870 = 1.03%. If 18 inputs cannot be processed, the failure rate is 18 ÷ 1,200 = 1.50%. If repeated runs produce identical output on 1,188 documents, repeatability is 1,188 ÷ 1,200 = 99.00%.

Now assume a model-version comparison changes 44 of 9,600 field results. Version delta is 44 ÷ 9,600 = 0.46%; 31 changes improve the result and 13 regress, which reconciles to 44. A net improvement does not automatically justify release. If the 13 regressions are concentrated in totals or tax IDs, the change may still be unacceptable.

Turn acceptance criteria into independent gates

For the same illustrative assumption, define gates of F1 at least 95.00%, critical exact match at least 99.50%, STP at least 70.00%, leakage no more than 0.20%, processing failure no more than 1.00%, and repeatability at least 99.00%.

The correct overall decision is not “approve because the average is high.” It is conditional hold; remediate and retest. Critical-field errors, leakage and failures breach their gates. Remediation can be split into poor-scan preprocessing, critical-field thresholds, master-data controls, review routing and failure retry. Run the frozen holdout again and add challenge cases that cover the identified root causes.

Include repeatability and model-version regression in acceptance

Cloud AI services may update processors, models, APIs and defaults under a familiar product name. Consult official resources such as Google’s processor list and Microsoft’s What’s new page, and record the model ID, API version, region, configuration and preprocessing code used for every test. Never rely on remembered language or preview/GA availability; verify current status immediately before a procurement or release decision.

For repeatability, process identical files multiple times under the same setting and compare raw values, normalized values, confidence, geometry and line alignment. Separate harmless formatting variation from changes that alter critical business values. For a version update, run old and new versions in parallel on the same regression set and review every changed case, not only aggregate metrics.

The minimum evidence package should include:

• model, processor, model version, API version and region;
• file hashes, test-set version and ground-truth version;
• versions of preprocessing, normalization, schema, threshold and business rules;
• run time, retries, error responses and processing duration;
• aggregate and stratified metrics, changed-case list and approval record.

The NIST AI Risk Management Framework treats measurement, monitoring, risk response and human–AI role allocation as continuing governance activities. For AI OCR, a PoC score should not become a permanent assurance claim. Production sampling, drift monitoring, incident review and release regression gates belong in the operating design.

Six phases for an AI OCR implementation PoC

Phase 1: Define process goals and error cost

Inventory document types, monthly and peak volume, languages, capture channels, downstream systems and retention requirements. Score each field by impact, detectability and correction effort. State the business goal, such as reducing invoice-registration lead time while preventing erroneous payment, instead of merely “improving OCR accuracy.”

Phase 2: Write the evaluation contract

Document matching rules, normalization, TP/FP/FN definitions, line-item alignment, failure treatment, strata, threshold sweep and acceptance gates. Ask each vendor whether it can return field value, raw text, confidence, bounding region, model/version and error information in a form accepted by the same evaluation script.

Phase 3: Build a narrow end-to-end path

Do not begin by auto-posting every document into ERP. Build a narrow path covering extraction, normalization, rule validation, reviewer interface and audit log. Reviewers should see the original image, extracted value, confidence, differences and master-data candidates together; otherwise review time may erase the benefit of automation.

Phase 4: Measure the holdout

Freeze configuration and run the untouched holdout once. Report overall results and breakdowns by language, document type, quality, supplier and field criticality. If the average passes but one operationally important stratum fails, restrict scope or hold acceptance.

Phase 5: Run in shadow mode

Use production inputs while the existing process remains the system of record. Observe daily volume, STP candidates, review load, leakage, failure, latency and API cost. Validate sensitive-data handling, storage location, access control, log retention and deletion conditions at the same time.

Phase 6: Accept, deploy and monitor

Approve gates, exceptions, residual risk, manual fallback, SLA, incident contact and model-update notification. After launch, continue random audits of auto-passed documents, root-cause analysis of critical errors, monthly stratified metrics and regression testing before version changes.

Questions for an OCR accuracy comparison or RFP

1. Can the candidate be tested under identical conditions on the specified Japanese, English, Thai and Vietnamese documents?
2. Can it return field value, raw text, confidence, bounding region, page, processor and version?
3. How are optional fields and repeating line items aligned and evaluated?
4. What does confidence mean, how is it calibrated, and how can a customer set measured thresholds?
5. What constraints apply to poor images, rotation, multiple pages, mixed languages, handwriting and stamps?
6. How are API errors, partial extraction, timeout, retry and duplicate processing handled?
7. What are the terms for storage, training use, encryption, region, access logs and deletion?
8. Are model-update notice, version pinning, rollback and regression test mechanisms available?
9. How will end-to-end time and cost, including human review and audit logs, be measured?
10. Will the PoC deliver the test-set definition, evaluation code, error inventory and operating design?

For commercial planning, see our guide to AI OCR pricing and costs in Thailand. If your scope includes handwriting or highly variable forms, first separate those strata using the handwritten OCR and form digitization guide.

Common mistakes and their controls

Comparing demonstrations on ten clean pages

A demonstration helps users understand the interface; it is not statistical acceptance evidence. Use a stratified population, a protected holdout and explicit processing-failure measurement.

Treating vendor confidence as a common scale

Converting 0–1 to 0–100 does not align model calibration. Measure observed correctness by score band for each candidate and establish independent thresholds.

Auditing only documents sent to human review

That hides errors in the auto-passed population. Keep random high-confidence audits and targeted audits of high-risk strata to measure leakage.

Saving only normalized values

That makes Unicode and transformation errors hard to trace. Retain the source, raw output, normalized value and normalization-rule version.

Approving multilingual processing from one overall average

High-volume languages dominate the mean. Set minimum gates by language, image quality, document type and critical field.

FAQ about AI OCR accuracy and recognition rate

What AI OCR accuracy is high enough for implementation?

There is no universal percentage. It depends on error cost, field criticality, reviewer capacity and downstream controls. Use F1, critical exact match, leakage, failure and STP as independent gates on your own documents.

Should an OCR accuracy comparison prioritize precision or recall?

Prioritize precision when sending a wrong value downstream is most costly, and recall when missing a required value is most costly. In practice, combine both through F1 and keep critical fields under exact-match controls.

How should blank optional fields be counted?

Avoid inflating simple accuracy with a large number of absent fields. Define TP, FP and FN explicitly, and count an invented value for an absent optional field as FP.

Can Japanese, English, Thai and Vietnamese be evaluated in one PoC?

Yes, but report language, field-language and quality strata separately. Service support varies by model, feature, API version, region and release status, so re-check current official documentation at evaluation time.

What is the standard confidence threshold?

There is no vendor-neutral standard threshold. Sweep candidate values by field criticality and measure precision, recall, STP, leakage and reviewer volume on your own data.

Does an AI OCR PoC need production integration?

Full production rollout is unnecessary, but extraction-only testing cannot measure STP or reviewer effort. A narrow end-to-end path through normalization, rules, review and audit logs produces a more realistic decision.

Conclusion: Measure AI OCR accuracy as your acceptance contract

Do not select AI OCR by comparing advertised recognition rates. Build a stratified holdout across four languages, document types and image quality, then measure precision, recall, F1, critical exact match, document exact match, STP, critical-error leakage, failure, repeatability and version delta. Freeze normalization for dates, currency, tax IDs, Unicode and line-item alignment. Set confidence thresholds from measured criticality and review capacity. If a critical gate fails, hold acceptance even when the average looks good, remediate the cause and retest under the same contract.

TOMAS TECH can help while you are still defining multilingual test data, ground truth, PoC scorecards and acceptance gates; a product decision is not required first. If you want an evaluation design that compares candidates on your own documents, contact TOMAS TECH.

References

• Google Cloud: Evaluate a processor
• Google Cloud: Shared.Types.Metrics
• Google Cloud: Processor list
• Microsoft: Transparency note for Document Intelligence
• Microsoft: OCR language support
• Microsoft: What’s new in Document Intelligence
• AWS: Amazon Textract best practices
• AWS: Responsible AI Service Card—Textract AnalyzeID
• Unicode Consortium: UAX #15 Unicode Normalization Forms
• NIST: Artificial Intelligence Risk Management Framework 1.0

Service support, processors, models, API versions, regions and preview/general-availability status may change. The primary materials above were referenced on 26 August 2026; verify the latest official information before selection or contracting.