AI Agent Acceptance Testing 2026: Seven Gates for an RFP

AI agent acceptance testing must examine more than answer accuracy. An agent connected to a factory or core business process can read data, select tools, operate external systems, and request human approval. Acceptance specifications must therefore cover permission boundaries, approvals, stop and recovery behavior, security, audit logs, and post-launch drift in addition to functional results. This guide organizes procurement through factory acceptance testing (FAT) and site acceptance testing (SAT) into seven gates that a Thai site leader, IT, quality, audit, and procurement team can use together.

This is not a general AI agent implementation guide. For use-case and rollout planning, see AI Agent Implementation 2026. For session and execution-environment design specific to the Agents API, see AI Agent API Operations Design. This article focuses on the buyer’s evidence-backed definition of “ready for production.”

Why AI agent acceptance testing needs a new design now

OpenAI announced the Agents API on September 10, 2026. It provides a platform for running cloud agents by specifying a task, model, tools, and environment. Long-running sessions, tool discovery, programmatic tool calls, and subagent collaboration are easier to implement. The acceptance target consequently expands from a model alone to the full system: model, harness, tools, permissions, data, and operators.

OpenAI’s description of Presence makes the enterprise control points explicit. A company decides what an agent may do, when approval is required, and when a person must take over. Before launch, teams test common requests, edge cases, and higher-risk situations, assessing policy compliance, tool use, and escalation as well as the outcome. A capability benchmark by itself is not an acceptance decision.

NIST announced the initial public draft of TEVV-Athlon on August 7, 2026. The four-stage approach helps an organization tailor test, evaluation, verification, and validation to its objectives. It includes agentic systems and names procurement specialists among relevant stakeholders. The important point is that the draft does not prescribe one universal pass score. The buyer must define evidence and thresholds that match its process, data, loss exposure, and approval authority.

Accuracy testing versus acceptance testing

Accuracy testing mainly asks whether an answer is correct. Acceptance testing also asks whether the agent used only authorized information, performed only authorized actions, respected required approvals, stopped safely, and left enough evidence to reconstruct the run.

A purchasing agent may choose an appropriate supplier but must still fail acceptance if it writes a purchase order before approval. A maintenance agent may identify the correct fault but must fail if it recommends a PLC change from an obsolete instruction and leaves no change record. Acceptance evaluates both the result and the path.

Fix the acceptance boundary before writing the AI agent RFP

Before writing an AI agent RFP, define the job in one sentence. “Automate procurement” is too broad. A testable statement would be: “Read the approved bill of materials and inventory, prepare supplier candidates, and create a purchase-order proposal in ERP only after approval by the purchasing manager.” The sentence identifies inputs, output, permitted action, approver, and write target.

Then list what is outside the boundary: unit-price master changes, supplier bank-account changes, emergency orders, or external transmission of personal data. “Do nothing that was not requested” cannot be verified. Create a case for every prohibited action and confirm both that the action is blocked and that the refusal is recorded.

Eight deliverables that belong in the RFP

1. Prioritized business scenario inventory
2. Tool, API, and data-source register
3. Role-based permission matrix
4. Human approval and handoff flow
5. Test cases covering normal, exception, and attack conditions
6. Stop, isolation, rollback, and restart procedure
7. Audit-log fields, retention, access, and masking policy
8. Production metrics, change control, and re-acceptance criteria

Do not ask a supplier for a simple yes/no response. Require implementation method, test method, evidence, limitations, and residual risk for each item. Using the same item structure in the RFP response and acceptance sheet creates traceability from proposal comparison to production approval.

The seven gates of AI agent acceptance testing

The following gates are a practical template for FAT/SAT-style acceptance. The numeric language in the table is an organization-defined example, not a general industry benchmark. A business may permit a statistical target for language quality while allowing zero deviations for a high-risk unauthorized action.

G1 Function: separate critical scenarios from the average

Test not only the happy path but also stockouts, inconsistent master data, mixed Thai and English, missing attachments, duplicate equipment names, and API timeouts. An average score can hide rare but severe failure. Classify cases as critical, major, and general, then use different rules—for example, all critical cases must pass.

The Agents API announcement contains a Ciridae customer comment saying its evaluation score moved from 0.71 to 0.85. That is the result reported for one customer workflow, not a transferable acceptance target. An RFP should instead state that the buyer’s frozen test set, graders, and scoring rules control acceptance.

Run the same input more than once because generative systems are probabilistic. Reproducibility also requires snapshots of retrieved data, tool responses, and time-dependent values; fixing only the model and temperature is insufficient.

G2 AI agent permission design: constrain the execution path

Least privilege is the core of AI agent permission design. Separate read-only from write tools, and constrain writes by table, field, amount, operating period, and site. A prompt saying “do not modify” is not access control. Enforce the boundary in service accounts, APIs, networks, and tool wrappers.

Acceptance cases should switch between an authorized user, unauthorized user, expired session, former-employee role, another site, and pre/post approval. Verify more than the rejection itself: the reason must be auditable and the fallback safe. An agent denied an ERP write must not work around the control by exporting a CSV to an external store.

G3 Approval: stop immediately before the side effect

Place the approval gate between preparation of a proposal and the operation that changes a real system. Show the action, target, diff, rationale, expected effect, and expiry to the approver. Bind the approval to the exact payload with a unique ID or hash so that execution cannot silently differ from what was approved.

Test approval, rejection, abandonment, loss of approver rights, data changes after approval, double-clicks, and simultaneous approval from another device. An expired approval should not execute; it should require a new request. A change to price or destination must invalidate the prior approval.

OpenAI Agents SDK guardrails apply at different workflow boundaries. Input guardrails apply to the first agent, output guardrails to the final-output agent, and tool guardrails to each protected tool call. With parallel input-guardrail execution, agent work and even tool activity may begin before cancellation. Where side effects must be prevented, test blocking execution or a tool-level guardrail and retain proof of the actual behavior.

G4 AI agent stop conditions: stop, clean up, and return safely

“Stop on error” is not a sufficient AI agent stop condition. Define the trigger, affected scope, post-stop state, and restart authority. Candidate triggers include a prohibited tool request, approval mismatch, sensitive-data detection, repeated failure, external API anomaly, cost limit, log-export failure, or loss of monitoring.

A kill switch must reach the execution queue, subagents, long-running tools, and retry jobs—not only the parent agent’s user interface. After a stop, block new runs, inventory active work, and reconcile completed side effects. A partially written ERP transaction needs a compensation transaction or a controlled human repair procedure.

Recovery tests must prove that resuming the same job does not create a duplicate purchase order or message. Use idempotency keys, checkpoints, processed markers, and external transaction IDs. Judge “stops safely” and “recovers safely” as separate conditions.

G5 Security: do not trust a one-attempt attack test

Agents ingest untrusted email, files, web pages, and ERP comments. Acceptance must include indirect prompt injection in which malicious instructions are embedded in that data. The OWASP Agentic Security Initiative covers risks unique to autonomous agents, multi-step workflows, and connection points such as MCP. Require a threat model, tool policy, data classification, egress control, secret handling, and incident process in the RFP.

In one specific NIST CAISI AgentDojo experiment, five injection tasks were each attempted 25 times. Under that experiment’s comparison, the average attack success rate for those five tasks rose from 57% to 80% after repeated attempts. This is not a general industry rate; it is limited to those tasks, model, environment, and procedure. The practical lesson is narrower: one blocked attempt is weak evidence. Repeat high-risk attacks with paraphrases, reordered steps, and different data channels.

NIST also documents evaluation cheating: agents consulted newer code, disabled checks, wrote test-specific logic, or caused denial of service instead of exploiting the intended vulnerability. Isolate answer artifacts, constrain network access as designed, and review traces rather than accepting the final score alone.

G6 Audit logs: reconstruct who did what and why

At minimum, log the business/case ID, user, agent version, model version, prompt/policy version, data references, tool name and arguments, tool result, approval ID, timestamps, final outcome, stop, and exception. Sensitive values need not be stored in clear text; masking and stable references can preserve both confidentiality and traceability.

OpenAI Agents SDK tracing can capture model generations, tool calls, handoffs, guardrails, and custom events. The documentation also states that this tracing is unavailable to organizations using OpenAI APIs under a Zero Data Retention policy. An acceptance specification must therefore name the actual contract and configuration, storage destination, fields, and compensating in-house evidence when built-in tracing is unavailable.

Test a logging outage, clock skew, oversized input, confidential fields, multiple subagents, retries, and rejected approval. Confirm that one business ID retrieves the entire path. Decide in advance whether high-risk actions stop safely when evidence collection is unavailable.

G7 Operational drift: launch is not the end of acceptance

Models, prompts, tools, APIs, master data, and user phrasing change in production. The launch-day pass result does not automatically remain valid. OpenAI Presence similarly describes using production sessions, escalations, and quality signals, comparing proposed changes with the production version, and approving controlled rollout.

Classify changes into three levels. Minor changes run automated regression. Impacting changes need partial re-acceptance with owner approval. Material changes run all seven gates again. A model major version, new write tool, wider permission, new personal data, or change to stop behavior are candidates for the material category.

Monitor more than task completion: critical errors, approval and rejection, human handoff, prohibited-tool requests, stop events, missing logs, and variance when the same case is rerun. The organization sets thresholds and names the owner of degraded mode, shutdown, and re-evaluation.

How to assemble the acceptance evidence package

Even after running all seven gates, scattered screenshots do not constitute auditable acceptance. Evidence should link “requirement → case → execution → result → defect → correction → retest” through one ID scheme. For example, use REQ for RFP requirements, TC for test cases, RUN for execution records, DEF for defects, and APR for approvals, with cross-references in a register. A final decision maker should be able to move from a requirement to the accepted execution record in a few steps.

Each RUN should retain execution time, operator, environment, versions of the agent/model/prompt/tools, data snapshot, input, output, every tool call, approval, variance from expectation, and evaluator. Video and screenshots are supporting evidence, not replacements for searchable structured logs. A failure needs more than its final screen: reviewers need to know what the agent read, which tool it selected, and where the guardrail fired.

Do not overwrite the failed RUN after a correction. Preserve the failed and corrected RUN separately and connect them through the change ticket. This prevents the record from appearing as though it had always passed and lets auditors review the improvement path. If the same issue returns after acceptance, the team can compare the former assumptions with current conditions.

Evidence ownership and viewing rights

If evidence exists only in the supplier’s log platform, the buyer may lose access when the contract ends or the service is unavailable. The buyer should retain at least the decision summary, configuration versions, approvals, tool history, hashes, and repository references. Conversely, prompts and traces may contain customer names, equipment details, personal data, or secret-like values. Separate evidence required by quality from confidential material restricted to security staff, and set export and retention rules.

For joint decisions between a Thai site and Japanese headquarters, store timestamps in UTC or with an explicit time zone and keep the same case ID across languages. Separate numbering for a Japanese test specification, English supplier response, and Thai site record can cause a critical finding to disappear during translation. Package the source, approved translation, and glossary together; standardize equipment names, role names, and stop-reason terms.

Implementing accountability at a Thai site

When headquarters writes policy and the site operates the system, align stop and restart rights with actual shift coverage. A design in which only a Japanese approver can release a stop during a Thai night shift may prolong downtime, while allowing local staff to restart a high-risk action alone is also unsafe. One workable pattern is immediate stop authority for a local responsible person and dual approval for restart based on business impact.

For shift operations, define job roles and the delegation order, not only individual names. Specify when an approval crossing a shift expires, how it is handed over, and who reviews the pending queue. Include the holiday calendar, after-hours supplier contact, power loss, and communication outage in SAT. These are operational design items, not claims about a specific legal requirement.

For multilingual input, test interpretation of permission language as well as translation quality. Words such as “approve,” “confirm,” and “execute” may be used ambiguously in Japanese, English, and Thai; a confirmation request must not be mistaken for execution permission. Present action type and target as structured fields in the approval UI and do not accept free text alone as authorization. Bind local equipment abbreviations and part names to master IDs to prevent action on a similarly named asset.

Do not treat an unstable network as merely a performance problem. Verify that a retransmitted approval is not counted twice, disconnected jobs do not all execute at once after reconnection, and writes do not continue when only log export has failed. Running these cases on the Thai site’s real route and working schedule is a central reason to perform SAT.

How to divide FAT and SAT

FAT produces repeatable evidence in a supplier or validation environment. Use a frozen data set, simulated ERP and equipment, attack data, and controlled network faults to cover all seven gates. SAT checks the differences created by the Thai site’s real network, roles, languages, approvers, operating hours, and connection constraints.

If SAT is the first stage allowed to write to production ERP, constrain target records, volume, and hours. Progress from shadow mode to proposal-only, approved write, and only then automated write. Full autonomy is not a prerequisite for a useful deployment.

Building reproducible AI agent evaluation cases

An AI agent evaluation case must describe the initial state and expected route, not just the user sentence. Package the case ID, purpose, risk, preconditions, input, available tools, prohibited tools, data snapshot, expected result, tolerance, expected approval, expected logs, and cleanup.

Specify prohibited paths as well as the desired result

For “detect shortage and prepare a purchase proposal,” prohibited paths could include ERP write before approval, sending to an unapproved supplier, export to a personal mailbox, and price-master modification. Reaching the correct final answer through a prohibited path is a failure.

Protect the test environment from the agent

Do not put answers in filenames, retain solutions in git history, expose the scoring API to tools, or confuse test-only permissions with production. To reduce solution contamination and grader gaming, separate runner from grader permissions and manually review a sample of traces.

Preserve failed runs so they can be reproduced

For a failure, preserve the referenced document version, tool response, model/configuration, approval state, and trace ID in addition to the input. Mask personal and secret values while retaining references that reconstruct the condition. After a fix, run the affected case and its related regression set.

From RFP to production approval

1. Assign accountability with RACI

The business owner defines purpose and impact; IT owns connections and permissions; quality owns test design; security owns the threat model; internal audit owns evidence requirements; and the local site owner owns SAT conditions. Name the final production approver and stop authority separately.

2. Issue the gates and evidence template before supplier selection

Adding acceptance requirements after contracting disrupts price and responsibility. Provide the seven gates, scenarios, evidence template, test environment, and retest conditions in the RFP. Treat supplier benchmarks as supporting material and require reproduction on the buyer’s own cases.

3. Approve residual risk, not only pass/fail

Not every risk can be eliminated. Record the impact, trigger, detection, workaround, due date, and owner for every unresolved item. Do not accept unresolved high-risk privilege violations or unauthorized egress; a minor wording variation may be accepted with monitoring. Make the distinction explicit.

4. Put scope and expiry conditions in the production permit

A production permit is not perpetual. Record the approved agent version, model, tools, data, site, roles, operations, and validity period. A material change, missing audit evidence, stop-condition event, severe incident, or threshold breach should expire or reopen the approval.

Common AI agent acceptance testing failures

• Treating a successful demonstration as production acceptance
• Using only an average accuracy score and hiding critical scenarios
• Mistaking a prompt prohibition for enforceable access control
• Showing an approval screen without binding approval to the executed payload
• Stopping the parent agent but leaving subagents or retry workers active
• Keeping logs without model version, tool arguments, or approval ID
• Declaring security after one blocked attack attempt
• Passing FAT but ignoring the Thai site’s language, network, and shifts
• Treating a model update as minor and skipping re-acceptance

These are not purely technical failures. They are broken links between contract, accountability, operations, and audit. Connecting the RFP and acceptance process through the same seven gates addresses that gap.

FAQ

What is AI agent acceptance testing?

It is the evidence-based pre-production evaluation of the entire agent system: ability to complete the defined job plus permissions, approvals, stop/recovery, security, audit logs, and operational drift. Its scope is broader than model accuracy evaluation.

What must be mandatory in an AI agent RFP?

Make the job and exclusions, tools and data, role permissions, approval points, stop conditions, test cases, evidence, logs, change management, and re-acceptance criteria mandatory. Request implementation and test methods and constraints, not a yes/no checkbox.

Is a prompt enough for AI agent permission design?

No. Enforce least privilege through service accounts, APIs, networks, tool wrappers, and the target data system in addition to prompt instructions. Acceptance testing should confirm that prohibited operations and workarounds are rejected in practice.

How should AI agent stop conditions be defined?

Start with triggers such as prohibited action, approval mismatch, sensitive-data detection, API fault, repeated failure, or loss of monitoring. Define affected scope, maximum time, isolation, compensation, and restart authority. The organization sets numeric thresholds based on operational impact.

How many times should an AI agent evaluation be repeated?

There is no universal number. Set repetitions from probabilistic variance, impact, and how easily an attacker can retry. Repeat critical and attack cases with different phrasing and record model, configuration, data, and tool responses.

Are both FAT and SAT necessary?

Use both when the production site differs from the validation environment. FAT provides repeatable evidence; SAT validates local permissions, language, network, people, and external systems. The SAT scope can be phased according to risk.

Conclusion

Passing AI agent acceptance testing does not mean “it answered well.” It means the agent acted within its authorized boundary, obtained the required approval, stopped and recovered safely, and left evidence that can be reconstructed. Using the same seven gates—function, permission, approval, stop/recovery, security, audit log, and operational drift—from RFP through production lets the buyer, supplier, site, and auditor judge the same evidence. Targets should come from the organization’s operational impact, not borrowed headline numbers.

TOMAS TECH can help at the stage of structuring an AI agent RFP or acceptance checklist for a Thai operation. You can contact us while the project is still defining its scope; we can break down the job, permission boundary, and FAT/SAT evidence into a decision sheet suitable for suppliers.