Evaluating Coding Agents

A coding benchmark tells you how an agent performed on someone else’s task distribution.

It does not tell you whether the agent can:

• follow your repository conventions
• solve your users’ common problems
• work with your tools and permissions
• avoid unnecessary changes
• produce code your maintainers will accept
• remain reliable after a model, prompt, or tool change

To answer those questions, you need an evaluation suite built around your environment.

An evaluation, usually shortened to eval, is a repeatable test of an AI system. It gives the agent a task and environment, records what happens, and applies one or more graders to the resulting code, state, and execution trace.

You Are Evaluating A System

A coding agent is more than a model.

Its behavior also depends on:

• the system prompt and project instructions
• the agent loop or harness
• available tools and their schemas
• repository state
• execution environment
• permissions and sandboxing
• model choice and reasoning settings
• context management
• stop conditions

flowchart LR
    T["Task and repository state"] --> A["Agent system"]
    M["Model"] --> A
    H["Harness and instructions"] --> A
    U["Tools and permissions"] --> A
    A --> C["Code changes and final response"]
    A --> R["Trajectory of tool calls and observations"]
    C --> G["Outcome graders"]
    R --> P["Process graders and metrics"]
    G --> S["Evaluation result"]
    P --> S

Changing only the model can change performance. So can changing a file-edit tool, compaction policy, permission mode, or test environment.

Record the complete configuration for every evaluation run.

Core Evaluation Terms

Term	Meaning
Task	One problem with inputs, environment, and success criteria
Trial	One attempt at a task
Grader	Logic that scores some part of the result
Assertion	One specific condition checked by a grader
Trajectory	The sequence of messages, tool calls, observations, and intermediate states
Outcome	The final state of the repository or environment
Harness	Infrastructure that runs tasks, captures traces, and applies graders
Suite	A collection of tasks measuring related capabilities or regressions

Because model behavior varies, one task should often be run through several trials. One successful demonstration is not proof of consistent behavior.

Start With The Real Outcome

The final answer may say:

The bug is fixed and all tests pass.

That statement is not the outcome.

The outcome is the repository state:

• is the bug actually fixed?
• do the relevant tests pass?
• do existing tests still pass?
• is the public contract preserved?
• did the agent modify only allowed files?
• is the patch buildable and reviewable?

For coding agents, deterministic checks are usually the strongest first graders.

A Layered Evaluation Model

No single score captures agent quality. Use several layers.

flowchart TB
    L1["1. Tool contract tests"] --> L2["2. Decision and checkpoint tests"]
    L2 --> L3["3. End-to-end task outcomes"]
    L3 --> L4["4. Trajectory and efficiency analysis"]
    L4 --> L5["5. Human and production evidence"]

Each layer answers a different question.

Layer 1: Tool Contract Tests

Test deterministic tools as normal software.

Examples:

• file-edit tool applies the expected patch
• search tool respects path restrictions
• test runner returns structured status
• deployment tool rejects an unapproved target
• API tool validates typed arguments
• side-effecting tool uses an idempotency key

Tool tests should cover:

• valid inputs
• malformed inputs
• permission failures
• timeouts
• partial failures
• bounded output
• retries
• side-effect safety

If the agent receives misleading or ambiguous tool output, better prompting may not solve the problem.

Layer 2: Decision And Checkpoint Tests

Sometimes you need to test one decision without running a complete task.

Create a snapshot containing:

• the task
• relevant conversation history
• current repository state
• recent tool results
• available tools

Then check what the agent does next.

Examples:

• after a failing test, does it inspect the error instead of claiming success?
• when credentials are required, does it request approval rather than search the filesystem?
• after discovering a generated file, does it edit the source template instead?
• when the requested behavior conflicts with an API contract, does it surface the conflict?

Checkpoint tests are faster and easier to diagnose than full agent runs. They are also more artificial, so do not use them as a substitute for end-to-end outcomes.

Layer 3: End-To-End Task Outcomes

An end-to-end task gives the agent a realistic repository and lets it work until completion or a limit.

A useful coding task includes:

• a clear problem statement
• a pinned repository revision
• setup instructions
• an isolated workspace
• the tools and permissions available in production
• acceptance tests hidden from the agent where appropriate
• regression tests
• time, turn, and spend limits
• a reference solution or proof that the task is solvable

Example outcome graders:

Grader	What it checks
Focused test	Required behavior now works
Regression suite	Existing behavior remains intact
Build/type check	Contracts remain valid
Static analysis	Lint, security, or policy requirements
File-state check	Required artifact exists in the right location
Diff policy	Forbidden files or dependencies were not changed
Human review	Patch is maintainable and matches architecture

Do not let the agent author the only test that grades its own work. It may unintentionally create a test that matches its implementation rather than the requirement.

Layer 4: Trajectory And Efficiency

Two agents may reach the same correct outcome through very different paths.

Trajectory analysis can reveal:

• repeated tool calls
• unnecessary retries
• reading the same file many times
• editing before gathering context
• broad commands where focused commands were available
• permissions repeatedly requested without progress
• excessive turns or token usage
• a final answer unsupported by the observed results

These are useful diagnostic signals, but avoid treating one exact trajectory as the only correct path.

For example, requiring the agent to call grep before every edit may reject a valid solution that used code intelligence instead. Grade process steps only when they represent a real requirement:

• a safety control
• a required approval
• a cost or latency limit
• an audit obligation
• evidence needed to trust the outcome

Layer 5: Human And Production Evidence

Automated checks cannot fully measure:

• architectural fit
• maintainability
• clarity of the diff
• unnecessary complexity
• user experience
• review burden
• whether the agent solved the real problem

Useful human and production signals include:

• maintainer accept/reject decisions
• requested review changes
• time spent reviewing and correcting
• escaped regressions
• rollback rate
• user-reported task success
• total time from request to accepted change

Do not use generated lines of code as a productivity metric. More code can mean more unnecessary work.

Grade Progress On Long-Horizon Tasks

A binary final result works well when a task takes a few minutes and either passes or fails. It becomes too sparse when an agent works for hours on a migration, translation, or repository-wide refactor.

Suppose an agent translates most of a library correctly but reaches its time limit before completing the last module. A final score of zero hides meaningful progress. Another agent may modify many files while leaving the repository unbuildable. Counting changed files would make that activity look like progress even though it produced no usable intermediate state.

Long-horizon evaluation therefore needs two distinct measurements:

1. intermediate progress, which shows how far the agent moved toward the goal
2. final acceptance, which confirms that the completed system satisfies the full contract

flowchart TB
    A["Pinned starting state"] --> B["Checkpoint 1<br/>repository still builds"]
    B --> C["Checkpoint 2<br/>some components satisfy new contract"]
    C --> D["Checkpoint 3<br/>integration behavior preserved"]
    D --> E["Final acceptance<br/>complete task contract"]

    B --> P["Progress record"]
    C --> P
    D --> P
    E --> F["Final pass or fail"]

The final grader remains authoritative. Intermediate graders explain partial capability and failure location; they do not turn an incomplete production change into a successful one.

Choose milestones that represent usable state

A good progress signal is externally observable and connected to the requirement. It should not reward activity for its own sake.

Long task	Useful intermediate evidence	Final acceptance
Language migration	Modules compiling in the target language; translated components passing behavioral tests	Legacy implementation removed where required; complete test and compatibility suite passes
API migration	Call sites moved to the new contract; compatibility tests passing by subsystem	Old API no longer used; integration and deployment checks pass
Large refactor	Behavior-preserving commits; dependency boundary progressively enforced	Architecture rule holds across the repository without regressions
Performance optimization	Correctness retained at each checkpoint; measured bottlenecks reduced	Representative workload beats the agreed baseline without semantic shortcuts
Security remediation	Vulnerable paths removed by component; focused exploit tests blocked	Full attack path is closed and regression controls pass

The Syzygy research on translating a complete C library into safe Rust illustrates the shape of this problem. End-to-end equivalence is essential, but a single final bit cannot explain whether an agent failed during code translation, test translation, refactoring, or final integration.

Do not grade one prescribed path

Intermediate grading can accidentally become a hidden implementation plan. For example, assigning points for editing files in a fixed order will penalize a valid strategy that changes dependency foundations first.

Prefer state-based milestones:

• percentage of independently testable components satisfying the new contract
• number of old API references removed without breaking the build
• behavioral test groups passing
• type, memory-safety, or policy invariants established
• performance improvement on held-out representative workloads
• known blockers remaining at the checkpoint

Avoid activity metrics such as:

• number of files changed
• generated lines of code
• tool-call count interpreted as progress
• whether the agent copied the reference solution’s sequence

A useful checkpoint record includes:

Elapsed time and spend:
Repository revision or snapshot:
Build and test status:
Completed contract milestones:
Regressions introduced:
Remaining blockers:
Evidence produced:

Capture checkpoints at defined intervals or after meaningful state transitions. This creates a progress curve: how much verified work the system completes as time and inference budget increase. The curve distinguishes an agent that makes steady, valid progress from one that remains stuck and then produces an unverified final burst.

Progress metrics must also resist gaming. If the score counts translated functions, the agent may translate trivial functions first and avoid hard dependencies. Weight milestones by independently estimated difficulty or business importance, keep some checks hidden, and inspect suspicious discontinuities.

Evaluate Real Product Use

Offline tasks are controlled and reproducible, but they cannot fully reproduce how developers respond to suggestions inside an editor.

An in-product evaluation can compare two systems on the same real context:

flowchart TB
    C["Same editor context"] --> A["Candidate from system A"]
    C --> B["Candidate from system B"]
    A --> L["Latency and presentation controls"]
    B --> L
    L --> U["Blinded user comparison"]
    U --> I["Immediate preference"]
    U --> O["Deferred outcome<br/>edited, tested, reverted, or retained"]

Copilot Arena used pairwise comparisons inside an IDE to collect preferences on model-generated completions. This evaluates a realistic task distribution: the code, cursor position, and developer intent come from actual work rather than benchmark authors.

Pairwise evaluation is useful, but only if the experiment controls factors other than model quality:

• randomly assign model pairs to eligible requests
• hide model identity from the user
• balance which candidate appears first or in each position
• give both systems the same available context and tool policy
• record whether either candidate timed out or failed to render
• separate results by language, task type, repository size, and user cohort
• run enough comparisons to report uncertainty, not only a rank

Latency is both quality and a confounder

Developers are less likely to accept a suggestion that arrives after they have continued typing. A faster system can therefore win more comparisons even when its completed answer would be worse.

This creates two legitimate questions:

1. Which output is preferred when response opportunity is comparable?
2. Which product experience is preferred with each system’s real latency?

Do not collapse them into one number.

For a quality-focused comparison, use latency-matched buckets, a common presentation deadline, or another balancing method so both candidates receive a comparable chance to be seen. For a product comparison, preserve real latency but report it explicitly alongside preference and abandonment. Artificially delaying a fast system may improve experimental control while making the test less representative of the deployed experience.

Acceptance is not the same as correctness

Pressing Tab, choosing a patch, or accepting an edit is an immediate preference signal. It can reflect relevance and convenience, but it does not prove that the code is correct or maintainable.

Combine immediate and deferred evidence:

Signal	What it can tell you
Candidate viewed	The response arrived in time to be considered
Candidate selected	The developer preferred it at that moment
Kept without substantial editing	The suggestion was close to usable
Build and tests passed	Objective behavior remained valid
Survived review and merge	Maintainers accepted the change
Not reverted or repaired later	The result remained useful over time
Task completion time	The complete workflow became faster or slower

Be careful with selection bias. Experts and beginners may use agents differently. Developers may decline tasks where a weaker system is expected to struggle, or accept only requests suited to AI assistance. Report the user population, eligible events, missing comparisons, and reasons suggestions were not shown.

Production evaluation should also respect source-code privacy, user consent, and data-retention rules. Store the minimum context needed for analysis, restrict access, and separate telemetry collection from model-training consent.

Offline suites and field evidence should reinforce each other:

1. production failures become reproducible offline tasks
2. offline improvements pass regression and safety gates
3. candidates enter a limited randomized field test
4. preference, latency, correctness, and review burden are measured separately
5. accepted changes feed new failure cases back into the suite

This loop keeps the evaluation aligned with real work without making noisy user preference the only grader.

Capability And Regression Suites

Maintain two kinds of suites.

Capability suite

Question:

What difficult tasks can the agent solve now?

These tasks should challenge the system. A low initial pass rate is acceptable because the suite identifies opportunities for improvement.

Examples:

• unfamiliar multi-module bugs
• migrations with implicit dependencies
• architecture-sensitive fixes
• tasks requiring several tools

Regression suite

Question:

Can the agent still perform behavior we already depend on?

These tasks should pass consistently and run whenever you change:

• model
• prompt
• tool definitions
• permissions
• context or compaction behavior
• agent harness

When a capability task becomes reliable and important, promote it into the regression suite.

Build The Initial Dataset

You do not need hundreds of tasks to start.

Anthropic’s current agent-evaluation guidance recommends beginning with roughly 20 to 50 tasks drawn from real behavior and failures, then expanding as the system matures.

Good sources include:

• bugs users reported
• tasks developers repeatedly try
• pull requests that required substantial correction
• production incidents
• security-review findings
• model-upgrade regressions
• manual checks used before release

For each task, record:

Task ID:
User-visible goal:
Starting repository revision:
Setup:
Allowed tools and permissions:
Required outcomes:
Forbidden outcomes:
Graders:
Limits:
Reference solution:
Why this task matters:

Turn Real Coding Work Into A Reproducible Environment

A real issue, coding session, commit, or pull request is a useful source of evaluation data. It is not yet an evaluation task.

Raw work usually contains information that a clean task should not expose:

• follow-up messages that reveal the answer
• the accepted patch or later commits
• private repository data and credentials
• dependencies installed manually on one developer’s machine
• tests written specifically for the accepted implementation
• unrelated edits accumulated during the session

It may also be missing information the agent needs, such as the exact starting revision, the original failure, or the user’s actual acceptance criteria.

The conversion process must reconstruct a fair problem that another agent can attempt from the same starting state.

flowchart LR
    A["Real coding event"] --> B["Eligibility, consent and provenance"]
    B --> C["Reconstruct user intent"]
    C --> D["Pin the starting state"]
    D --> E["Reproduce the failure"]
    E --> F["Build an independent verifier"]
    F --> G["Create an isolated environment"]
    G --> H["Test valid and invalid solutions"]
    H --> I["Assign to train, development or held-out evaluation"]

Projects such as R2E, SWE-bench, and Harbor use different formats, but share this basic idea: package an instruction, a starting environment, and an independent scoring procedure so attempts can be repeated.

1. Qualify the source event

Start by deciding whether the event may and should become a task.

Check:

• Provenance: Which repository, revision, issue, and user request produced it?
• Consent: Did the developer agree to evaluation collection, model training, both, or neither?
• License: May the code, issue text, and derived environment be redistributed?
• Privacy: Do prompts, files, traces, logs, or test fixtures contain secrets or personal information?
• Completeness: Are both the pre-change state and accepted outcome available?
• Substance: Does the task measure a useful capability rather than a typo or dependency installation accident?
• Reproducibility: Can the original problem be observed from a clean machine?

Consent for product telemetry is not automatically consent for model training or public redistribution. Record those permissions separately.

Do not select only impressive successes or dramatic failures. A collection built only from agent failures measures the weaknesses of that particular model, harness, product, and user population. A collection built only from merged patches hides rejected and abandoned work.

2. Reconstruct the intent

The first prompt may not describe the complete task. During real work, developers clarify requirements, reject partial fixes, and reveal constraints over several messages.

Reconstruct a self-contained task statement from:

• the original request
• relevant clarifications
• the reported or reproduced failure
• issue and pull-request discussion
• review comments
• externally visible behavior of the accepted change

Separate the requirement from the historical implementation:

Requirement:
Refreshing a valid access token must preserve the user's session.

Historical implementation:
The accepted patch moved expiry validation into refreshSession().

The first statement belongs in the task. The second is one possible solution and normally remains hidden.

Ask a maintainer to review the reconstructed instruction. If two informed reviewers infer materially different success criteria, the task is not ready.

3. Pin and reproduce the starting state

Find the repository state immediately before the fix. A commit hash alone may be insufficient.

Also capture:

• submodule revisions
• dependency lockfiles and package registries
• compiler, runtime, operating-system, and system-package versions
• database schemas and seeded data
• feature flags and configuration
• generated files required to build
• services or APIs the task depends on
• time, locale, and randomness where behavior depends on them

Run the original reproduction from a clean environment. The starting state should fail the focused verifier for the expected reason while unrelated baseline tests still behave as documented.

If the problem cannot be reproduced, do not silently convert the accepted patch into the specification. Either repair the environment, redesign the task with an independently observable contract, or exclude it.

4. Isolate and reset the environment

Each trial must start from the same state. A container is common, but a VM, sandbox, or disposable worktree can also work.

The environment should define:

• writable and read-only paths
• network availability
• allowed tools and commands
• CPU, memory, disk, time, turn, and spend limits
• service startup and health checks
• reset behavior between trials
• artifacts retained for grading and diagnosis

Prevent future-state leakage. Do not expose the fixing commit, later branches, hidden tests, cached patches, or build artifacts containing the solution. If Git history is part of the intended product experience, provide only history available at the starting revision rather than deleting all repository metadata without considering the task.

The Harbor task format separates the task instruction, environment, reference solution, and verifier. That separation makes it easier to reset the environment and audit what the agent could observe.

5. Build a verifier around the outcome

The accepted patch proves that at least one solution exists. It does not define the only valid implementation.

A strong verifier checks the externally observable contract:

• the reported behavior is corrected
• previous behavior remains valid
• public interfaces and data contracts are preserved
• security and policy constraints hold
• required artifacts exist
• forbidden side effects did not occur

It should not require the agent to:

• edit the same file as the historical patch
• use the same helper or algorithm
• follow the same sequence of tool calls
• produce an identical diff
• add the same test names

Process checks remain appropriate when the process is part of the contract, such as obtaining approval before deployment or avoiding network access to protected data.

Verifier quality is a major source of benchmark error. OpenAI stopped using SWE-bench Verified for frontier-model evaluation after finding that some tasks had flawed tests that rejected correct alternative solutions. The problem was not merely noisy scoring; the grader encoded an implementation-specific or otherwise incorrect definition of success. See Why We No Longer Evaluate SWE-bench Verified.

Validate every verifier against several classes of solution:

Candidate	Expected result	Purpose
Untouched starting state	Fail	Confirms the task detects the original problem
Historical reference solution	Pass	Confirms the task is solvable
Independently written valid solution	Pass	Detects overfitting to the historical patch
Superficial or test-specific patch	Fail	Detects reward hacking
Fix that breaks existing behavior	Fail	Confirms regression coverage
Attempt to modify or bypass the verifier	Fail	Confirms grader isolation

Run these checks from a clean reset. A verifier that passes only the reference patch is evidence of a narrow test, not a strong task.

6. Decide how the environment will be used

The same environment package can support evaluation or learning, but those uses have different rules.

Use	What happens	Required separation
Held-out benchmark	A frozen agent attempts unseen tasks and receives scores	Task, verifier, and outcomes must not have influenced development
Development eval	Engineers inspect failures while changing prompts, tools, or the harness	Keep a separate final test set
Supervised fine-tuning	Successful actions or trajectories become training examples	Remove those tasks from held-out reporting
Reinforcement learning	Repeated attempts receive rewards that update the policy	Protect hidden checks and watch for reward exploitation

In a benchmark, reward measures a fixed system. In reinforcement learning, reward becomes a training signal used to change the policy. That distinction affects more than the execution command: it changes contamination controls, verifier exposure, versioning, and what results may be reported as held out.

Maintain at least three pools:

1. training, which optimization may use directly
2. development, which engineers may inspect and tune against
3. held-out evaluation, which remains unseen until a release decision

Once a task, solution, verifier, or detailed failure explanation has been used to improve the system, do not continue presenting performance on that task as evidence of unseen generalization.

7. Document selection bias

“Real-world” does not mean representative.

A task collection inherits the distribution of its source:

• product and editor used
• models and harnesses deployed
• developers who opted in
• programming languages and repositories represented
• open-source versus private enterprise work
• tasks users considered suitable for an agent
• failures that caused visible intervention
• successful sessions that were retained or discarded

Report those boundaries with the benchmark. Use stratified results by language, task type, repository size, and source when the sample supports it. Do not generalize from one product’s users to all software engineering.

Cline Bench as an early example

Cline Bench demonstrates the packaging pattern with a task instruction, containerized starting state, reference solution, and tests. Cline says its source events come from opted-in Cline Provider interactions on public open-source repositories. That gives the tasks useful provenance, but also makes the collection Cline-specific and excludes private enterprise work.

When checked on July 27, 2026, the public repository still described itself as early access, contained 12 task directories, and showed no public commit after December 11, 2025. Treat it as a small implementation example, not evidence that its task distribution represents coding work generally. The useful lesson is the reconstruction method, independent of Cline’s claims about the uniqueness or value of its data.

Write Solvable, Unambiguous Tasks

A broken task creates misleading failure data.

Before using a task:

1. ask whether two maintainers would agree on pass or fail
2. run the reference solution through every grader
3. verify setup from a clean environment
4. make every grader requirement visible in the task unless it is intentionally a hidden behavioral test
5. remove assumptions about paths, tools, or package versions

If a task asks for a script but the grader silently expects one exact filename, a correct agent may fail because of the evaluation rather than the implementation.

Include Positive And Negative Cases

One-sided evals produce one-sided agents.

If you test only:

The agent should search when current information is required.

the system may learn to search every time.

Also test:

The agent should not search when the answer is available in the repository.

Other balanced pairs:

• edit when needed / do not edit for an explanation-only request
• ask for approval / proceed without unnecessary approval
• retry a transient failure / do not retry a validation failure
• delegate a large independent task / keep a small task in the main context
• add a regression test / do not create meaningless tests for a typo

Use Several Grader Types

Code-based graders

Examples:

• unit and integration tests
• exact or regex checks
• compiler and type checker
• static security analysis
• repository-state inspection
• tool-call and argument checks

Strengths:

• fast
• reproducible
• inexpensive
• easy to debug

Limitations:

• may accept a narrow test-specific patch
• can reject valid alternative outputs
• cannot judge every maintenance concern

Model-based graders

Examples:

• rubric-based code quality
• pairwise patch comparison
• instruction-following review
• explanation groundedness

Strengths:

• handle open-ended outputs
• capture nuanced criteria
• scale beyond manual review

Limitations:

• nondeterministic
• sensitive to the rubric
• can share blind spots with the agent
• require calibration against human judgment

Human graders

Examples:

• maintainer review
• domain-expert review
• blind patch comparison
• sampled audit

Strengths:

• understand product and architectural context
• expose flaws in automated graders
• provide the quality standard that model graders should approximate

Limitations:

• slow
• expensive
• may disagree

Use deterministic graders for objective conditions, model graders for bounded qualitative criteria, and humans for calibration and high-impact judgment.

Example Evaluation Task

task:
  id: auth-refresh-regression-01
  goal: preserve the session after a valid token refresh
  repository_revision: 4f2a8c1
  limits:
    max_turns: 40
    max_minutes: 25
  permissions:
    network: false
    writable_paths:
      - src/auth/**
      - tests/auth/**
  graders:
    - type: deterministic_test
      command: npm test -- refresh-session.test.ts
    - type: regression_test
      command: npm test -- auth
    - type: typecheck
      command: npm run typecheck
    - type: diff_policy
      forbidden_paths:
        - migrations/**
        - package-lock.json
    - type: human_rubric
      criteria:
        - fixes the root cause
        - preserves expiry validation
        - does not add an unnecessary retry
  metrics:
    - turns
    - tool_calls
    - elapsed_seconds
    - input_tokens
    - output_tokens

The exact schema depends on your harness. The design matters more than the file format.

Evaluate More Than One Trial

Agent behavior is probabilistic.

If an agent succeeds once and fails four times, reporting the successful example is misleading.

Useful views include:

• pass@1: first-attempt success rate
• pass@k: probability that at least one of several attempts succeeds
• pass^k: probability that all of several attempts succeed

The product requirement determines which matters:

Product behavior	Useful view
Developer may choose among several generated patches	pass@k
Unattended action must work every time	pass^k and failure analysis
Interactive assistant gets one normal attempt	pass@1

A single average can hide a task that fails consistently or a model that is highly variable.

Historical Replay And Backtesting

Historical tasks are valuable because they represent real failures.

A replay process can:

1. capture the original task and repository revision
2. remove the original fix
3. run the current agent in an isolated workspace
4. apply current graders
5. compare results with the accepted patch

Replay helps answer:

• did the new model fix cases the old one missed?
• did a prompt change break an existing workflow?
• did a new tool reduce repeated failures?
• did improved success increase cost or review burden?

Keep a held-out set that prompt and tool authors do not repeatedly optimize against. Otherwise the agent may improve on the visible suite without becoming more useful.

Environment Reproducibility

An agent eval is unreliable if the environment changes between trials.

Pin:

• repository commit
• dependency lockfiles
• runtime and tool versions
• environment variables
• available network access
• seeded test data
• time and locale where relevant
• model identifier and settings
• agent harness version

Use a fresh container, VM, worktree, or other isolated workspace for each trial. One trial’s files, caches, or services must not influence another.

Metrics To Track

Quality

• task success rate
• focused and regression-test pass rate
• maintainer acceptance rate
• severity of review findings
• escaped regression rate

Reliability

• success distribution across repeated trials
• timeout rate
• tool failure rate
• permission-denial rate
• incomplete-run rate

Efficiency

• elapsed time
• model turns
• tool calls
• input and output tokens
• cost per accepted task
• repeated reads, edits, or tests

Human effort

• review time
• number of requested changes
• correction time
• percentage of work rewritten by a maintainer

Optimize for accepted outcomes, not the smallest token count or fewest tool calls in isolation.

Common Evaluation Failures

Failure	Consequence	Better approach
Only evaluate the final text	Agent can claim work it did not complete	Inspect repository and environment state
Only run happy-path tests	Regressions and unsafe behavior remain hidden	Add negative and boundary cases
Require one exact trajectory	Valid strategies are rejected	Grade process only when required
Run one trial	Random success looks reliable	Run repeated trials
Let tasks drift	Results cannot be compared	Pin revisions and environments
Use only an LLM judge	Shared blind spots and grader variance	Combine code, model, and human graders
Tune repeatedly on the entire suite	Evaluation becomes training data	Keep a held-out set
Ignore review effort	Fast generation appears productive	Measure accepted change and correction cost
Keep broken tasks	Agent quality is underestimated	Verify with reference solutions
Report only an aggregate	Consistent failures disappear	Review per-task and per-category results

Release Gate Example

Before changing the production model or harness:

1. run the regression suite on the old system
2. run the same tasks and trial count on the candidate
3. compare quality, reliability, cost, and latency
4. inspect changed trajectories for important tasks
5. review failures and unexpectedly large improvements
6. calibrate model graders with sampled human review
7. canary the change on limited real traffic
8. monitor production outcomes and rollback criteria

Do not promote a candidate because one benchmark score improved.

Checklist

Dataset:

• Tasks come from real user behavior and failures.
• Source events have documented provenance, consent, license, and privacy review.
• Each task has clear success and failure criteria.
• A reference solution passes every grader.
• At least one independent valid solution passes the verifier.
• Positive, negative, and boundary cases are represented.
• Training, development, and held-out tasks are separated.

Environment:

• Repository, dependencies, tools, and model are pinned.
• The original failure reproduces from the pinned starting state.
• Every trial starts from a clean isolated state.
• Future commits, solutions, hidden tests, and cached artifacts cannot leak.
• Permissions and network access match the intended product.
• Time, turns, tokens, and spend are bounded.

Graders:

• Outcomes are checked independently of the final response.
• Focused and regression tests are included.
• Qualitative graders have explicit rubrics.
• Model graders are calibrated against humans.
• Process checks represent real requirements.
• Valid alternative implementations are not rejected for differing from the reference patch.
• The agent cannot modify, inspect, or bypass hidden grading logic.
• Long tasks have state-based progress graders as well as final acceptance checks.

Reporting:

• Several trials are run where reliability matters.
• Per-task results remain visible.
• Source-population and task-selection biases are documented.
• Quality, reliability, cost, latency, and review effort are separated.
• Failures include trajectories and environment details.
• Product comparisons control presentation effects and separate preference from correctness.

Key Takeaways

• Evaluate the model, harness, tools, environment, and policy as one system.
• The final repository state matters more than the agent’s claim of completion.
• Combine tool tests, checkpoint tests, end-to-end outcomes, trajectory analysis, and human review.
• Keep capability and regression suites separate.
• Start with real tasks and failures instead of invented benchmark prompts.
• Run multiple trials because agent behavior varies.
• Grade trajectories only when the path itself affects safety, cost, or trust.
• For long tasks, record verified intermediate progress without weakening the final acceptance standard.
• In-product comparisons must control latency and presentation while tracking deferred correctness.
• A real coding session must be reconstructed, isolated, and independently verified before it becomes a trustworthy task.
• Treat the accepted patch as proof of solvability, not as the only correct implementation.
• Separate training, development, and held-out tasks, and disclose the source distribution’s biases.
• Pin environments and keep a held-out suite.
• Measure accepted work and review effort, not generated code volume.

Resources

• Source talk: How Claude Code Works
• Coding Evals: From Code Snippets to Codebases - Naman Jain, Cursor
• Hard Won Lessons from Building Effective AI Coding Agents - Nik Pash, Cline
• Cline Bench repository
• Cline Bench initiative
• R2E: Turning Any GitHub Repository into a Programming Agent Environment
• Harbor task documentation
• Why We No Longer Evaluate SWE-bench Verified
• Syzygy: Dual Code-Test C to Safe Rust Translation using LLMs and Dynamic Analysis
• Copilot Arena: A Platform for Code LLM Evaluation in the Wild
• Demystifying Evals for AI Agents
• Building Effective AI Agents
• Understanding Coding-Agent Benchmarks
• Verification-Driven Agentic Coding
• DSPy Evaluation and Optimization