Before allowing an AI agent to create or modify records, evaluate it on representative work, inspect the actions it attempts, and verify the result in the destination system. Agree in advance which failures block release. The decision should rest on evidence from the specific workflow and on access controls that have been tested.
What exact responsibility are you evaluating?
Define a bounded task. In a hypothetical service workflow, an agent might read a request, identify the required information, and prepare a ticket. Creating that ticket, changing its priority, and replying to the customer are separate actions that should be evaluated separately.
For each action, record the information it needs, the required permission, and the expected outcome. Specify when the agent should ask a question, stop, or hand the case to a person.
This brief also helps compare proposals. “Handles service requests” leaves significant questions open. “Prepares a ticket using approved fields and creates it after the required review” gives you behavior you can test.
Which cases belong in the evaluation?
Collect authorized workflow examples and minimize personal information. Include routine requests, known exceptions, and cases where taking no action is the correct outcome. Hold some examples back for the final review rather than repeatedly using every example to tune the system.
OpenAI’s evaluation guidance recommends task-specific tests, typical and difficult examples, and human review to calibrate evaluation. A useful set should reflect the conditions of your operation.
For the ticket example, test incomplete information, similar customer names, requests outside the agreed scope, conflicting instructions, and an unavailable tool. Include different ways of expressing the same need. Ask the process owner to record the expected result before seeing the agent’s output.
How will you test access boundaries?
Use identities and permissions equivalent to those planned for production. Check that a request about one customer does not allow the system to read or modify another customer’s records without authorization.
OWASP identifies excessive functionality, permissions, and autonomy as agent-system risks. Its guidance includes limiting tools and access and requiring approval for high-impact actions.
Turn those principles into observable tests for this pilot: missing permission blocks a write, an approval applies to the specific proposed action, and rejection prevents execution. Check the receiving service as well as the model’s response. An instruction telling the agent to respect a rule does not demonstrate that the system enforces it.
What if a document tries to issue instructions?
Include a test document containing an instruction unrelated to the task, such as changing a recipient or bypassing an approval. OWASP describes indirect prompt injection through external content, including files and pages retrieved by a model.
Make the expected outcome explicit: the content can provide information but cannot grant new permissions. Inspect both the response and any attempted tool calls. This evaluation provides evidence about the tested cases; it cannot guarantee protection against every possible attack.
Which results should you measure separately?
Create an execution record containing the input, evaluated version, decision, proposed fields, authorized action, and final state. Avoid retaining sensitive information that is unnecessary for the review.
Separate at least these dimensions:
An average score can hide a consequential failure. Decide which errors block release even when other answers are useful. For example, modifying a record without the required authorization should be investigated before access is expanded.
When should you allow the first writes?
Begin with action preparation in a controlled environment. Review proposals and demonstrate that the controls work before enabling a narrowly scoped write. Identify who can pause the workflow, how failures will be detected, and what recovery is possible for each action.
Agree to repeat relevant tests when the model, instructions, tools, or business rules change. Add discovered incidents to the evaluation and reconsider whether the authorized scope remains appropriate.
For a project discussion with QuantixCode, bring a specific task, permitted examples, and a list of unacceptable failures. That foundation helps define what the agent can prepare, what it can execute, and what evidence is needed before granting additional autonomy.