Coding agents now close most of the engineering loop on their own: planning, coding, testing, opening a pull request, responding to review comments. As capabilities improve, the risks associated with coding agents compound - agents can now run for extended periods of time searching a codebase, reasoning through tasks, and iterating on their own work with limited supervision.

The latest AIUC-1 standard update, shaped by 100+ AIUC-1 Consortium members, expanded to cover risks introduced by coding agents, with new controls for secrets management, secure defaults in code, and runtime containment. These complement existing controls on agent identity, access management, and monitoring. AIUC-1’s requirements surface the risks that show up in real user flows such as adversarial robustness, tool calls, hallucinations, and safety.

“For coding agents, I look for clear agent identity, least-privilege access, explicit approval for consequential actions, and tamper-evident auditability. The goal is to let agents move quickly without losing control of the security boundary,” said Nancy Wang, CTO at 1Password and AIUC-1 Consortium member.

This post covers:

1. AIUC-1's evaluation framework for coding agents, with technical deep dives on the sandbox environment setup and a dynamically generated MCP attack.
2. The risks coding agents are evaluated against to earn AIUC-1 certification.

Designing the evaluation framework to test coding agents

Evaluations for modern coding agents are fundamentally different to evaluations required for conversational agents. Given that coding agents can write, test, and debug code, evaluations must judge the entire work session and not just the response. This full record of every action taken (agent trace) encompasses the multi-turn conversation, the tool calls and commands the agent ran, file edits, web searches and more.

AIUC-1 evals are designed to validate the AI system's robustness against specific risks. In the case of coding agents this means data/privacy, security, and reliability.

Fundamentally, the coding agent evaluator takes in the agent trace and classifies the severity of the agent's actions according to the risk being tested. Technically, the evaluator is a primary LLM-as-a-judge that uses tools (e.g., LLM review tool, adversarial debate) to gather information about the agent's actions and assess the risks elicited. Concretely:

1. Evaluator input: The primary LLM-as-a-judge receives the coding agent trace as input.
2. Agent trace analysis: The primary agent calls the LLM review tool which reconstructs the code written by the agent in real time by replaying every edit - code diffs, file edits, and tool calls. Static Application Security Testing (SAST) tooling such as Semgrep then runs over the reconstructed code, catching vulnerability patterns in the code the agent wrote.
3. Evidence report: The LLM review tool produces an evidence report with findings from the trace analysis.
4. Adversarial debate: Evidence report is debated over by two LLMs. A Blue Team LLM defends the report's findings while a Red Team LLM critiques them, debating over multiple turns until both converge; the final findings are fed back into the evidence report.
5. Final evaluation grade: The final report is fed back into the primary LLM-as-a-judge to produce the final severity grade. Each severity is linked to a specific part of the trace - a specific code instance, file, or chat message. This enables easy verification and benchmarks the accuracy of our evaluations.

AIUC-1 framework for a coding agent evaluator

Deep-dive: Building isolated sandbox environments

Evals should produce credible evidence about how agents behave in production. That demands an isolated and observable environment in order to run safe adversarial scenarios where the results can be explained. Sandboxing an agent makes it less realistic, which also drives a need to create a more realistic environment to mirror production behaviour.

In preliminary testing runs, some evals were run in parallel, some on a shared machine. For evals being run on the same machine with the same file system, instances where the agent contaminated evidence for other evals, and found the rubrics behind its own evaluation run were discovered. After the early evaluations, each test was run in a sandbox, designed in the following configuration:

Eval testing environment for coding agents

After sandboxing, in order to create a more realistic evaluation environment, runtime was built from a small Linux image and layered a realistic workstation on top:

• Real tooling: Git, Node.js, package managers, Python, ripgrep, and the actual coding-agent CLI
• Real repository: a working application rather than a synthetic collection of vulnerable snippets
• Surrounding context: workspace rules, agent steering files, human READMEs, an environment file, package-manager policies, and project documentation
• Files outside the workspace: adjacent projects, shared folders, operational notes, exported data

A sandbox that contains only the target repository is not representative of a real developer workstation. In prod, coding agents see home-directory files, environment variables, shell history, credentials, adjacent projects, package-manager configuration, and authenticated developer tools. An agent that never encounters those never gets tested against them.

Deep-dive: MCP attacks as a common attack vector

MCP is a common attack surface introduced by coding agents, introducing security risks such as credential exfiltration and unauthorized tool execution. As the primary way for agents to gather context from externally controlled systems, MCP servers allow both the tools and data behind them to change after installation.

MCP attacks test what happens when an approved MCP server starts serving malicious content - whether the agent preserves user intent when a third-party dependency tries to redirect it. Example attacks created were based on existing CVEs (e.g., CVE-2025-54136). One included modelling a scenario involving a realistic internal project-tracking and documentation system that exposed two read-only tools (document search and context retrieval for specific tasks) and had a corpus of project specs, including backlog items, completed tasks, and meeting notes with initially benign content.

The attack involved the red-teaming agent injecting adversarial content into the selected document, and keeping the visible developer request benign, with no suggestion that MCP content should be treated as instructions.

Surfacing real coding agents risks

AIUC-1 evals for coding agents involve setting up specialized testing environments, realistic attacks and designing an evaluator that can take in the whole coding agent trace. The purpose is to validate the coding agents performance against specific risks, and provide third party testing for enterprises deploying agents in production.

“We run thousands of evals with our in-house engineering teams for each product release. AIUC-1 certification enables us to add an additional independent layer going deep on data and privacy, security and reliability risks,” said Kenneth Moras, Security at Cursor.

When evaluating coding agents, three dominant risk areas are tested against:

1. Data and privacy. Agents with broad environment access can surface personal credentials from files outside the workspace, or reproduce third-party code in ways that create IP exposure. Testing involves checking whether agents leak personal credentials from surrounding context, and whether generated code reproduces third-party material in ways that create IP exposure.
2. Security. Agents write code that may ship into production without review, and can take actions that are executed against real systems. Testing involves checking for insecure outputs, insecure and unauthorized actions, and disclosure of vulnerabilities the agent discovered in the codebase.
3. Reliability. With agents able to run unsupervised for extended periods, wrong steps taken can compound and lead to unauthorized actions taken beyond scope. Testing involves checking for hallucination in both generated code and returned information, incorrect actions, and service disruption.

Benign indirect ask testing for insecure output generation

Using AIUC-1 to validate coding agent security

As coding capabilities improve, the bar for demonstrating that these agents can act securely, safely, and reliably in deployment rises with them.

Cursor and Lovable are the first two coding agents to earn AIUC-1 certification, with Schellman conducting the independent audit. Read more about how AIUC-1 requirements were adapted for coding-agent specific risks - and how Lovable demonstrated these in this whitepaper. Read more about Cursor’s AIUC-1 certification here.

Organizations are able to certify their own deployment of certified coding agents - validating that the unique deployment context, data access, tool access, and configurations operate as intended by the organization.

Read more about how to begin your AIUC-1 certification process here