Evaluating Coding Agents

A coding benchmark tells you how an agent performed on someone else’s task distribution.

It does not tell you whether the agent can:

• follow your repository conventions
• solve your users’ common problems
• work with your tools and permissions
• avoid unnecessary changes
• produce code your maintainers will accept
• remain reliable after a model, prompt, or tool change

To answer those questions, you need an evaluation suite built around your environment.

An evaluation, usually shortened to eval, is a repeatable test of an AI system. It gives the agent a task and environment, records what happens, and applies one or more graders to the resulting code, state, and execution trace.

You Are Evaluating A System

A coding agent is more than a model.

Its behavior also depends on:

• the system prompt and project instructions
• the agent loop or harness
• available tools and their schemas
• repository state
• execution environment
• permissions and sandboxing
• model choice and reasoning settings
• context management
• stop conditions

flowchart LR
    T["Task and repository state"] --> A["Agent system"]
    M["Model"] --> A
    H["Harness and instructions"] --> A
    U["Tools and permissions"] --> A
    A --> C["Code changes and final response"]
    A --> R["Trajectory of tool calls and observations"]
    C --> G["Outcome graders"]
    R --> P["Process graders and metrics"]
    G --> S["Evaluation result"]
    P --> S

Changing only the model can change performance. So can changing a file-edit tool, compaction policy, permission mode, or test environment.

Record the complete configuration for every evaluation run.

Core Evaluation Terms

Term	Meaning
Task	One problem with inputs, environment, and success criteria
Trial	One attempt at a task
Grader	Logic that scores some part of the result
Assertion	One specific condition checked by a grader
Trajectory	The sequence of messages, tool calls, observations, and intermediate states
Outcome	The final state of the repository or environment
Harness	Infrastructure that runs tasks, captures traces, and applies graders
Suite	A collection of tasks measuring related capabilities or regressions

Because model behavior varies, one task should often be run through several trials. One successful demonstration is not proof of consistent behavior.

Start With The Real Outcome

The final answer may say:

The bug is fixed and all tests pass.

That statement is not the outcome.

The outcome is the repository state:

• is the bug actually fixed?
• do the relevant tests pass?
• do existing tests still pass?
• is the public contract preserved?
• did the agent modify only allowed files?
• is the patch buildable and reviewable?

For coding agents, deterministic checks are usually the strongest first graders.

A Layered Evaluation Model

No single score captures agent quality. Use several layers.

flowchart TB
    L1["1. Tool contract tests"] --> L2["2. Decision and checkpoint tests"]
    L2 --> L3["3. End-to-end task outcomes"]
    L3 --> L4["4. Trajectory and efficiency analysis"]
    L4 --> L5["5. Human and production evidence"]

Each layer answers a different question.

Layer 1: Tool Contract Tests

Test deterministic tools as normal software.

Examples:

• file-edit tool applies the expected patch
• search tool respects path restrictions
• test runner returns structured status
• deployment tool rejects an unapproved target
• API tool validates typed arguments
• side-effecting tool uses an idempotency key

Tool tests should cover:

• valid inputs
• malformed inputs
• permission failures
• timeouts
• partial failures
• bounded output
• retries
• side-effect safety

If the agent receives misleading or ambiguous tool output, better prompting may not solve the problem.

Layer 2: Decision And Checkpoint Tests

Sometimes you need to test one decision without running a complete task.

Create a snapshot containing:

• the task
• relevant conversation history
• current repository state
• recent tool results
• available tools

Then check what the agent does next.

Examples:

• after a failing test, does it inspect the error instead of claiming success?
• when credentials are required, does it request approval rather than search the filesystem?
• after discovering a generated file, does it edit the source template instead?
• when the requested behavior conflicts with an API contract, does it surface the conflict?

Checkpoint tests are faster and easier to diagnose than full agent runs. They are also more artificial, so do not use them as a substitute for end-to-end outcomes.

Layer 3: End-To-End Task Outcomes

An end-to-end task gives the agent a realistic repository and lets it work until completion or a limit.

A useful coding task includes:

• a clear problem statement
• a pinned repository revision
• setup instructions
• an isolated workspace
• the tools and permissions available in production
• acceptance tests hidden from the agent where appropriate
• regression tests
• time, turn, and spend limits
• a reference solution or proof that the task is solvable

Example outcome graders:

Grader	What it checks
Focused test	Required behavior now works
Regression suite	Existing behavior remains intact
Build/type check	Contracts remain valid
Static analysis	Lint, security, or policy requirements
File-state check	Required artifact exists in the right location
Diff policy	Forbidden files or dependencies were not changed
Human review	Patch is maintainable and matches architecture

Do not let the agent author the only test that grades its own work. It may unintentionally create a test that matches its implementation rather than the requirement.

Layer 4: Trajectory And Efficiency

Two agents may reach the same correct outcome through very different paths.

Trajectory analysis can reveal:

• repeated tool calls
• unnecessary retries
• reading the same file many times
• editing before gathering context
• broad commands where focused commands were available
• permissions repeatedly requested without progress
• excessive turns or token usage
• a final answer unsupported by the observed results

These are useful diagnostic signals, but avoid treating one exact trajectory as the only correct path.

For example, requiring the agent to call grep before every edit may reject a valid solution that used code intelligence instead. Grade process steps only when they represent a real requirement:

• a safety control
• a required approval
• a cost or latency limit
• an audit obligation
• evidence needed to trust the outcome

Layer 5: Human And Production Evidence

Automated checks cannot fully measure:

• architectural fit
• maintainability
• clarity of the diff
• unnecessary complexity
• user experience
• review burden
• whether the agent solved the real problem

Useful human and production signals include:

• maintainer accept/reject decisions
• requested review changes
• time spent reviewing and correcting
• escaped regressions
• rollback rate
• user-reported task success
• total time from request to accepted change

Do not use generated lines of code as a productivity metric. More code can mean more unnecessary work.

Capability And Regression Suites

Maintain two kinds of suites.

Capability suite

Question:

What difficult tasks can the agent solve now?

These tasks should challenge the system. A low initial pass rate is acceptable because the suite identifies opportunities for improvement.

Examples:

• unfamiliar multi-module bugs
• migrations with implicit dependencies
• architecture-sensitive fixes
• tasks requiring several tools

Regression suite

Question:

Can the agent still perform behavior we already depend on?

These tasks should pass consistently and run whenever you change:

• model
• prompt
• tool definitions
• permissions
• context or compaction behavior
• agent harness

When a capability task becomes reliable and important, promote it into the regression suite.

Build The Initial Dataset

You do not need hundreds of tasks to start.

Anthropic’s current agent-evaluation guidance recommends beginning with roughly 20 to 50 tasks drawn from real behavior and failures, then expanding as the system matures.

Good sources include:

• bugs users reported
• tasks developers repeatedly try
• pull requests that required substantial correction
• production incidents
• security-review findings
• model-upgrade regressions
• manual checks used before release

For each task, record:

Task ID:
User-visible goal:
Starting repository revision:
Setup:
Allowed tools and permissions:
Required outcomes:
Forbidden outcomes:
Graders:
Limits:
Reference solution:
Why this task matters:

Write Solvable, Unambiguous Tasks

A broken task creates misleading failure data.

Before using a task:

1. ask whether two maintainers would agree on pass or fail
2. run the reference solution through every grader
3. verify setup from a clean environment
4. make every grader requirement visible in the task unless it is intentionally a hidden behavioral test
5. remove assumptions about paths, tools, or package versions

If a task asks for a script but the grader silently expects one exact filename, a correct agent may fail because of the evaluation rather than the implementation.

Include Positive And Negative Cases

One-sided evals produce one-sided agents.

If you test only:

The agent should search when current information is required.

the system may learn to search every time.

Also test:

The agent should not search when the answer is available in the repository.

Other balanced pairs:

• edit when needed / do not edit for an explanation-only request
• ask for approval / proceed without unnecessary approval
• retry a transient failure / do not retry a validation failure
• delegate a large independent task / keep a small task in the main context
• add a regression test / do not create meaningless tests for a typo

Use Several Grader Types

Code-based graders

Examples:

• unit and integration tests
• exact or regex checks
• compiler and type checker
• static security analysis
• repository-state inspection
• tool-call and argument checks

Strengths:

• fast
• reproducible
• inexpensive
• easy to debug

Limitations:

• may accept a narrow test-specific patch
• can reject valid alternative outputs
• cannot judge every maintenance concern

Model-based graders

Examples:

• rubric-based code quality
• pairwise patch comparison
• instruction-following review
• explanation groundedness

Strengths:

• handle open-ended outputs
• capture nuanced criteria
• scale beyond manual review

Limitations:

• nondeterministic
• sensitive to the rubric
• can share blind spots with the agent
• require calibration against human judgment

Human graders

Examples:

• maintainer review
• domain-expert review
• blind patch comparison
• sampled audit

Strengths:

• understand product and architectural context
• expose flaws in automated graders
• provide the quality standard that model graders should approximate

Limitations:

• slow
• expensive
• may disagree

Use deterministic graders for objective conditions, model graders for bounded qualitative criteria, and humans for calibration and high-impact judgment.

Example Evaluation Task

task:
  id: auth-refresh-regression-01
  goal: preserve the session after a valid token refresh
  repository_revision: 4f2a8c1
  limits:
    max_turns: 40
    max_minutes: 25
  permissions:
    network: false
    writable_paths:
      - src/auth/**
      - tests/auth/**
  graders:
    - type: deterministic_test
      command: npm test -- refresh-session.test.ts
    - type: regression_test
      command: npm test -- auth
    - type: typecheck
      command: npm run typecheck
    - type: diff_policy
      forbidden_paths:
        - migrations/**
        - package-lock.json
    - type: human_rubric
      criteria:
        - fixes the root cause
        - preserves expiry validation
        - does not add an unnecessary retry
  metrics:
    - turns
    - tool_calls
    - elapsed_seconds
    - input_tokens
    - output_tokens

The exact schema depends on your harness. The design matters more than the file format.

Evaluate More Than One Trial

Agent behavior is probabilistic.

If an agent succeeds once and fails four times, reporting the successful example is misleading.

Useful views include:

• pass@1: first-attempt success rate
• pass@k: probability that at least one of several attempts succeeds
• pass^k: probability that all of several attempts succeed

The product requirement determines which matters:

Product behavior	Useful view
Developer may choose among several generated patches	pass@k
Unattended action must work every time	pass^k and failure analysis
Interactive assistant gets one normal attempt	pass@1

A single average can hide a task that fails consistently or a model that is highly variable.

Historical Replay And Backtesting

Historical tasks are valuable because they represent real failures.

A replay process can:

1. capture the original task and repository revision
2. remove the original fix
3. run the current agent in an isolated workspace
4. apply current graders
5. compare results with the accepted patch

Replay helps answer:

• did the new model fix cases the old one missed?
• did a prompt change break an existing workflow?
• did a new tool reduce repeated failures?
• did improved success increase cost or review burden?

Keep a held-out set that prompt and tool authors do not repeatedly optimize against. Otherwise the agent may improve on the visible suite without becoming more useful.

Environment Reproducibility

An agent eval is unreliable if the environment changes between trials.

Pin:

• repository commit
• dependency lockfiles
• runtime and tool versions
• environment variables
• available network access
• seeded test data
• time and locale where relevant
• model identifier and settings
• agent harness version

Use a fresh container, VM, worktree, or other isolated workspace for each trial. One trial’s files, caches, or services must not influence another.

Metrics To Track

Quality

• task success rate
• focused and regression-test pass rate
• maintainer acceptance rate
• severity of review findings
• escaped regression rate

Reliability

• success distribution across repeated trials
• timeout rate
• tool failure rate
• permission-denial rate
• incomplete-run rate

Efficiency

• elapsed time
• model turns
• tool calls
• input and output tokens
• cost per accepted task
• repeated reads, edits, or tests

Human effort

• review time
• number of requested changes
• correction time
• percentage of work rewritten by a maintainer

Optimize for accepted outcomes, not the smallest token count or fewest tool calls in isolation.

Common Evaluation Failures

Failure	Consequence	Better approach
Only evaluate the final text	Agent can claim work it did not complete	Inspect repository and environment state
Only run happy-path tests	Regressions and unsafe behavior remain hidden	Add negative and boundary cases
Require one exact trajectory	Valid strategies are rejected	Grade process only when required
Run one trial	Random success looks reliable	Run repeated trials
Let tasks drift	Results cannot be compared	Pin revisions and environments
Use only an LLM judge	Shared blind spots and grader variance	Combine code, model, and human graders
Tune repeatedly on the entire suite	Evaluation becomes training data	Keep a held-out set
Ignore review effort	Fast generation appears productive	Measure accepted change and correction cost
Keep broken tasks	Agent quality is underestimated	Verify with reference solutions
Report only an aggregate	Consistent failures disappear	Review per-task and per-category results

Release Gate Example

Before changing the production model or harness:

1. run the regression suite on the old system
2. run the same tasks and trial count on the candidate
3. compare quality, reliability, cost, and latency
4. inspect changed trajectories for important tasks
5. review failures and unexpectedly large improvements
6. calibrate model graders with sampled human review
7. canary the change on limited real traffic
8. monitor production outcomes and rollback criteria

Do not promote a candidate because one benchmark score improved.

Checklist

Dataset:

• Tasks come from real user behavior and failures.
• Each task has clear success and failure criteria.
• A reference solution passes every grader.
• Positive, negative, and boundary cases are represented.
• A held-out set exists.

Environment:

• Repository, dependencies, tools, and model are pinned.
• Every trial starts from a clean isolated state.
• Permissions and network access match the intended product.
• Time, turns, tokens, and spend are bounded.

Graders:

• Outcomes are checked independently of the final response.
• Focused and regression tests are included.
• Qualitative graders have explicit rubrics.
• Model graders are calibrated against humans.
• Process checks represent real requirements.

Reporting:

• Several trials are run where reliability matters.
• Per-task results remain visible.
• Quality, reliability, cost, latency, and review effort are separated.
• Failures include trajectories and environment details.

Key Takeaways

• Evaluate the model, harness, tools, environment, and policy as one system.
• The final repository state matters more than the agent’s claim of completion.
• Combine tool tests, checkpoint tests, end-to-end outcomes, trajectory analysis, and human review.
• Keep capability and regression suites separate.
• Start with real tasks and failures instead of invented benchmark prompts.
• Run multiple trials because agent behavior varies.
• Grade trajectories only when the path itself affects safety, cost, or trust.
• Pin environments and keep a held-out suite.
• Measure accepted work and review effort, not generated code volume.

Resources

• Source talk: How Claude Code Works
• Demystifying Evals for AI Agents
• Building Effective AI Agents
• Understanding Coding-Agent Benchmarks
• Verification-Driven Agentic Coding
• DSPy Evaluation and Optimization