Understanding Coding-Agent Benchmarks

Coding agents are improving quickly on software benchmarks. That does not automatically mean they can replace the same amount of real developer work.

The important distinction is that benchmark capability, task reliability, code quality, and developer productivity are different measurements. Each answers a different question.

The Apparent Contradiction

Two findings can look contradictory:

coding agents are completing increasingly difficult and longer benchmark tasks
developers do not always finish real work faster when they use those agents

Both can be true. Benchmarks usually test an agent on a clean, self-contained task with a clear success condition. Real software development includes repository history, architectural expectations, ambiguous requirements, review, maintenance, and coordination with other people.

The benchmark measures what the agent can accomplish in a controlled environment. A productivity study measures the complete human-agent system inside a real workflow.

What A Task-Completion Time Horizon Means

A task-completion time horizon uses human completion time as a rough measure of task difficulty.

Suppose an evaluation contains tasks that take skilled humans anywhere from a few minutes to several hours. If an agent has a two-hour 50% time horizon, the evaluation predicts that it will successfully complete about half of the tasks that take a human expert approximately two hours.

It does not mean that the agent:

works continuously for two hours
can perform any two hours of human work
can replace a developer for two hours
is reliable enough for production use

The two hours describe the difficulty of the evaluated task using human effort as the reference. They do not describe the agent’s runtime or general autonomy.

What The Rising Trend Tells Us

METR’s measurements show a strong historical increase in the length of software, machine-learning, and cybersecurity tasks that frontier agents can complete.

This is meaningful evidence that agents are becoming more capable. However, the evaluated tasks are primarily:

self-contained
clearly specified
accessible through code and text tools
checked with explicit success criteria
attempted with little prior organizational context

The result therefore supports a narrow conclusion: agents are improving rapidly on tasks shaped like those in the evaluation suite.

It does not establish that the same improvement applies equally to every domain or every kind of work. Current METR guidance also warns that measurements above 16 hours are unreliable because the latest models are approaching the limits of the available task suite.

Capability Is Uneven

An agent can be excellent at one long coding task and fail at a shorter task involving a browser, undocumented data, organizational knowledge, or communication.

This unevenness is sometimes called a jagged capability profile. Task length is only one dimension. Performance also depends on:

the domain
how clearly the goal is specified
the available tools and interface
how much hidden context is required
the required reliability
how the result is evaluated

A single time-horizon number is useful for tracking a defined task distribution. It is not a universal measure of intelligence or employability.

From Benchmark Success To Useful Work

flowchart LR
    A["Clean benchmark task"] --> B["Automated success"]
    B --> C["Time-horizon estimate"]
    C --> D["Human review<br/>and quality bar"]
    D --> E["Real workflow<br/>and productivity"]

    F["Context / architecture<br/>maintenance expectations"] --> D
    G["Prompting / waiting<br/>verification / correction"] --> E

Moving from left to right adds requirements that a benchmark may not measure.

An automated grader can confirm that tests pass. A maintainer may still reject the change because it:

conflicts with the architecture
introduces unnecessary complexity
ignores repository conventions
creates future maintenance work
solves the immediate test without solving the real problem

In a METR analysis of SWE-bench submissions, many patches that passed the automated grader were not considered mergeable by maintainers. The exact difference depends on the task suite and evaluation method, but the durable lesson is that passing tests and producing an acceptable change are not identical outcomes.

Why Productivity Is A Different Measurement

Developer productivity includes the entire human-agent loop:

explaining the task
supplying missing context
waiting for the agent
reviewing its approach and output
correcting mistakes
integrating the change
taking responsibility for the result

An agent can generate code faster than a human while still increasing total completion time if review and correction cost more than the generated work saves.

The opposite can also happen. On familiar, well-scoped work with strong tests, an agent may save substantial time even if it could not complete the task independently.

That is why autonomous benchmark success and assisted human productivity should be measured separately.

Context Changes The Human Baseline

Human completion time is not a fixed property of a task.

Two equally skilled developers may require very different amounts of time when one already knows:

why the surrounding architecture exists
which earlier approaches failed
where the relevant code and tests live
which behavior customers depend on
which trade-offs maintainers will accept
who needs to review the change

The other developer may need to discover all of that before making progress.

METR’s task-completion benchmarks deliberately use self-contained tasks and skilled human baseliners who have little prior task context. This makes the comparison fairer because the agent and human receive similar instructions and tools.

It also changes what the resulting time means.

Low-context baseline	High-context professional
Learns the repository from the task environment	Has years of repository knowledge
Searches for the likely implementation path	May already know the relevant files
Reconstructs design constraints	Remembers why the constraints exist
Verifies assumptions from available evidence	Carries tacit knowledge from earlier work
Solves a mostly self-contained task	Coordinates the task with ongoing work

An agent matching a low-context contractor on a two-hour task has demonstrated useful capability. It has not necessarily matched a maintainer who can solve the same issue in twenty minutes because the maintainer already knows the territory.

This distinction also explains why an agent may help a developer who is new to a codebase more than it helps the project’s top contributor.

Reliability Creates A Verification Tax

Success probability and productivity are connected through the cost of checking failures.

A simplified calculation is:

net time saved =
  manual completion time
  - delegation time
  - agent waiting time
  - review time
  - correction time
  - integration time

Fast generation improves only one part of this equation.

Suppose a developer can make a change manually in 30 minutes. An agent produces a candidate in five minutes, but the developer spends:

five minutes explaining the task
ten minutes reviewing the patch
ten minutes correcting a subtle mistake
five minutes rerunning checks and integrating it

The agent generated code quickly, but it did not reduce total task time.

The required reliability depends on the cost and observability of failure:

Task shape	Reliability requirement
Reversible change with fast, strong tests	Moderate reliability may still be useful
Draft that a human already intends to rewrite	Imperfect output can save time
Large change requiring careful line-by-line review	Higher reliability is needed
Poorly observable data corruption	Very high reliability and external controls are needed
Security, financial, or production side effect	Reliability alone is insufficient; use policy, approvals, and deterministic checks

METR notes that some reliability-critical or poorly verifiable tasks may require success rates above 98% before automation is worthwhile. That is not a universal threshold. It illustrates why a 50% time horizon is a capability measurement, not a deployment recommendation.

Higher reliability is also harder to measure. Estimating a 99% success rate requires many diverse, low-noise tasks because even a few broken tasks or grading errors can dominate the result.

Task Interdependence Limits Delegation

Real work is rarely a bag of independent tickets.

Consider three connected tasks:

flowchart LR
    A["Task A<br/>Investigate the current architecture"] --> B["Task B<br/>Choose and implement a change"]
    B --> C["Task C<br/>Review downstream consequences"]

    X["Agent completes Task A"] --> S["Summary and artifacts"]
    S --> R["Human reconstructs the relevant context"]
    R --> B

If the agent completes Task A faster, the human may still need to understand:

what evidence was examined
which alternatives were rejected
what assumptions remain uncertain
which architectural constraints shaped the result

That reconstruction cost can reduce or erase the speedup before Task B begins.

Delegation works better when:

the task has a clear input and output contract
downstream work needs the result, not the full reasoning history
correctness can be checked independently
failures remain local and reversible
the agent returns durable evidence rather than only a summary

For tightly coupled tasks, it may be better for the human and agent to work through the sequence together. The agent can search, test, and draft while the human retains the context needed for later decisions.

Task interdependence is therefore different from task difficulty. An agent may be capable of completing each subtask in isolation while the overall workflow still benefits from continuity.

Capability Elicitation Is Part Of The Result

A benchmark does not evaluate a bare model. It evaluates a model inside an agent system.

Evaluators may deliberately improve capability elicitation by:

choosing a suitable agent scaffold
tuning prompts and tool descriptions
providing a generous token and time budget
running several independent trials
fixing environment or inference problems
checking for reward hacking
ensuring the model did not fail only because its budget was too small

These steps are appropriate when the question is:

What can this model-agent system do under a strong evaluation setup?

Everyday developers often face a different optimization problem:

Capability evaluation	Everyday use
Maximize measured task success	Maximize useful work after all costs
Carefully selected scaffold	Product defaults or team configuration
Large, fixed evaluation budget	Limited time, tokens, and attention
Several trials may be acceptable	Usually one or a few attempts
Failed run can be reset	Failure creates review and recovery work
Automated grader defines success	Maintainers and users define acceptance

A strong benchmark score remains real evidence. Its correct unit is the model, scaffold, tools, budget, task distribution, and grader together.

When comparing a benchmark with a deployed product, check whether the product provides similar tools, context, inference budget, and retry behavior. A model can look weaker because the deployed system elicits less of its capability, or look less useful because benchmark-level elicitation is too expensive for routine work.

A Benchmark Has A Useful Life

A benchmark is not permanently informative just because it was carefully designed.

Its signal can weaken in several ways:

training contamination: tasks, tests, or solutions appear in model-training data
runtime contamination: the agent retrieves a later commit, merged pull request, or published solution while being evaluated
saturation: nearly every capable system passes, so the benchmark can no longer separate them
excessive difficulty: nearly every system fails, leaving too little signal about partial capability
task-distribution drift: real usage changes while the benchmark continues measuring an older workflow
grader exposure: repeated optimization against public tests rewards benchmark-specific behavior

flowchart TB
    A["Fresh task pool"] --> B["Useful separation<br/>between systems"]
    B --> C["Tasks and graders<br/>become familiar"]
    C --> D["Contamination, saturation,<br/>or targeted optimization"]
    D --> E["Score becomes a weaker<br/>measure of the intended ability"]
    E --> F["Refresh tasks, difficulty,<br/>and evaluator controls"]
    F --> B

LiveCodeBench addresses part of this problem by continuously collecting recently released competition problems and allowing results to be calculated over release-date windows. This reduces some contamination risk and makes the evaluation easier to refresh. It does not make every coding claim contamination-free: recent contest problems still measure a different capability from maintaining a large repository.

Training contamination and runtime contamination

These are related but different failures.

Training contamination occurs before evaluation. A model may have seen the benchmark problem or solution during training. A high score can then mix memorization with general problem-solving ability.

Runtime contamination occurs during the agent run. A coding agent with shell, Git, and internet access may discover:

the future commit that fixed the issue
the accepted pull request
a patch quoted in an issue discussion
a package release containing the repaired source
cached artifacts left in the evaluation environment

Retrieval is useful behavior in normal engineering. It becomes an evaluation problem when the benchmark claims to measure whether the agent can derive a solution from the supplied repository and issue. The tool use is not inherently wrong; it changes what the successful run proves.

A benchmark must therefore define its intended construct before deciding what to restrict:

Intended measurement	Environment implication
Can the agent independently diagnose and fix the issue?	Remove future Git history, known solutions, leaked caches, and unrestricted answer lookup
Can the agent resolve the issue using normal engineering resources?	Permit relevant search, but report retrieval-assisted success separately
Can the product complete users’ work in its deployed environment?	Match production tools and permissions, then measure accepted outcomes rather than isolated reasoning

Current research from Cursor shows why this distinction matters: increasingly capable agents can retrieve known benchmark fixes when Git history or the public web exposes them. Sealing future history, controlling network access, and auditing trajectories are now part of benchmark design, not merely infrastructure details.

Difficulty must be calibrated

A benchmark with a 99% pass rate may be too easy to distinguish frontier systems. One with a 1% pass rate may show that a problem is unsolved, but provide little evidence about which system is making useful progress.

This creates ceiling and floor effects:

Mostly pass   -> weak separation at the ceiling
Mixed results -> useful comparison and failure analysis
Mostly fail   -> weak separation at the floor

The desired range depends on the purpose:

a regression suite should usually pass at a high rate because it protects behavior already relied upon
a frontier capability suite should contain enough difficult tasks to separate current systems
a release gate should concentrate around the quality threshold required by the product

Do not respond to saturation by silently replacing the entire benchmark. Preserve an anchor set for historical comparison and add a rolling frontier set of newer or harder tasks. Version both the task pool and graders so a score always identifies the exact evaluation window.

Construct validity: does the score support the claim?

Construct validity asks whether an evaluation actually measures the capability named in the conclusion.

A task can be perfectly reproducible and still have weak construct validity. Competitive-programming problems with hidden tests can measure algorithmic code generation well, but they provide limited evidence about repository navigation, architectural judgment, or maintainability. A repository bug benchmark can test multi-file editing, but its claim weakens if the agent can look up the merged fix.

For every benchmark, connect the claim to observable evidence:

Claimed capability	Evidence the evaluation needs
Generate correct algorithms	Recent, well-specified problems; broad hidden tests; calibrated difficulty
Repair repository bugs	Realistic repository state; hidden regression tests; controlled future history; maintainability review
Optimize software	Semantic equivalence; representative workloads; strong baselines; end-to-end performance measurement
Work autonomously for long periods	Realistic dependencies; intermediate progress evidence; final acceptance; bounded human intervention

The GSO benchmark, for example, sources optimization tasks from real commit histories and grades both behavior and runtime improvement. That design supports a claim about repository-scale performance optimization more directly than asking a model to rewrite isolated snippets.

A practical benchmark-maintenance loop is:

state the capability and deployment decision the score is meant to support
preserve a versioned anchor set for trend comparisons
add recent tasks that current systems do not uniformly pass
check release dates and remove access to future solutions where independent problem solving is the target
test reference solutions and grader sensitivity from a clean environment
inspect failures and suspiciously large improvements
report results by task category, difficulty, and evaluation window
refresh the rolling set as models and real usage change

Recent tasks reduce exposure risk; they do not prove absence of contamination. A model’s exact training corpus and cutoff may be uncertain, and public tasks can become searchable quickly. Treat freshness as one control among several.

When The Agent Optimizes The Test

An agent may find a way to improve the benchmark score without improving the intended system. This is often called benchmark gaming, reward hacking, or specification gaming.

The problem is easiest to see in a performance benchmark:

flowchart LR
    A["Intended goal<br/>Implement the operation correctly"] --> B["Visible evaluator<br/>Inputs, checks, and timing"]
    B --> C["Agent searches for<br/>the highest score"]
    C --> D{"Does the evaluator fully<br/>represent the goal?"}
    D -- "Yes" --> E["Useful optimization"]
    D -- "No" --> F["Shortcut that passes<br/>the visible test"]

Suppose the requested operation clamps every input into the range [-1, 1]. If every benchmark input already falls inside that range, returning the input unchanged will pass the visible examples and appear much faster. The implementation has optimized the test distribution, not the general operation.

Similar shortcuts can appear when an agent:

hard-codes a known answer or fixed input shape
skips work whose result is not inspected
calls the reference implementation through an allowed fallback
exploits an overly broad numerical tolerance
produces values that match while violating layout, mutation, or error behavior
specializes to public examples without declaring the restriction

This does not require malicious intent. The agent was given a score and found a path to maximize it. The evaluator failed to encode part of the real contract.

Why performance benchmarks are especially vulnerable

A normal correctness benchmark asks whether an output matches an expected result. An optimization benchmark usually asks for two things simultaneously:

preserve the reference behavior
reduce time, memory, or cost

Weakening the first requirement can create a dramatic improvement in the second. Extraordinary speedups therefore deserve stronger correctness checks, not weaker ones.

A generated GPU kernel can also look successful because the test:

uses only one tensor shape or data type
checks too few randomized inputs
measures asynchronous launch time instead of completed GPU work
includes compilation in one path but not the other
compares against a weak baseline
permits a fallback to the original framework operation
measures an isolated operation whose improvement disappears end to end

See AI-Generated GPU Kernels for the hardware and measurement details behind this example.

Designing a harder-to-game evaluator

A stronger evaluation separates candidate generation from final judgment:

Control	What it prevents
Hidden and randomized tests	Memorizing visible examples
Held-out shapes, values, and task variants	Overfitting to the public distribution
Property and metamorphic tests	Passing examples while violating general behavior
Independent semantic checks	Trading correctness for speed
Fallback detection	Reusing the reference path while claiming generated capability
Strong baseline and end-to-end measurement	Inflating a local or weak comparison
Adversarial cases	Exploiting untested boundaries
Immutable final evaluator	Modifying the test instead of the implementation
Human review of surprising results	Accepting physically or logically implausible claims

No finite benchmark proves general correctness. The practical goal is to make shortcuts less profitable, reserve important cases for final evaluation, and state exactly what the score covers.

A restricted implementation can still be useful. A kernel optimized only for several production shapes is legitimate when the restriction is explicit, dispatch logic enforces it, and unsupported inputs use a correct fallback. Hidden specialization becomes gaming when the reported claim is broader than the implemented contract.

Agent Ability Is A Curve, Not A Point

The same agent can produce different performance at different levels of expenditure.

Here, expenditure means resources consumed to obtain the result, such as:

tokens or inference compute
number of independent attempts
elapsed time
tool and infrastructure cost
human prompting and review time

An agent that solves 70% of tasks with one short attempt may solve more with a larger reasoning budget or several attempts. Another agent may reach the same score more cheaply.

This creates several valid measurements:

Measurement	Question it answers
Score at a fixed budget	Which agent performs better with equal resources?
Cost to reach a fixed score	Which agent reaches the required quality most efficiently?
Score at a practical plateau	What can the agent do when additional spending stops helping much?
Expenditure-adjusted value	Is the additional quality worth its token, time, and monetary cost?
Human-relative savings	How much expenditure is saved compared with having a human achieve the same result?

No measurement is always best. The right one depends on the decision.

For an offline migration, spending more tokens to improve reliability may be sensible. For interactive autocomplete, latency may dominate. For a security-sensitive operation, the relevant target may be a fixed high reliability level rather than the highest average score.

This is why “Which agent has the highest benchmark score?” is incomplete. A practical comparison asks:

At what budget, quality threshold, reliability level, and human-review cost?

Interpreting The Developer Study

In early 2025, METR ran a randomized study involving 16 experienced open-source developers completing 246 real issues in repositories they knew well. Developers expected AI assistance to make them faster, but the measured result showed them taking about 19% longer when AI was allowed.

This was useful evidence about that population, those repositories, and early-2025 tools. It was not evidence that AI slowed down all developers.

METR now treats that result as a historical snapshot rather than an estimate of current productivity. A later experiment suggested that newer agentic tools probably provided more benefit, but participation and selection effects made the size of the current speedup unreliable.

Developers who expected the greatest benefit from AI were increasingly unwilling to accept tasks where AI was disallowed. Developers also avoided submitting some tasks they especially wanted to perform with AI. Parallel agent use made task-time measurement harder. These effects systematically removed some of the people and tasks with potentially high AI benefit from the experiment.

The lasting lesson is methodological:

do not infer current productivity from an old tool generation
do not generalize from one developer population to all software work
do not rely only on self-reported speedups
measure total task completion time and output quality
account for which developers and tasks are missing from the sample
account for parallel agents and work that continues outside the measured task
record the model, tools, tasks, users, and evaluation method

Five Questions, Five Measurements

Question	Useful measurement
Can the agent complete a controlled task?	Benchmark success rate
How difficult are the tasks it completes reliably?	Task-completion time horizon
Is the result acceptable to responsible maintainers?	Human review and merge decision
Does using the agent reduce real effort or elapsed time?	Controlled field study or production telemetry
What does that performance cost?	Score-versus-expenditure curve and human-relative savings

No single row replaces the others.

Practical Interpretation

When reading a new coding-agent result, ask:

What kinds of tasks were included?
How much context did the agent receive?
Which tools and agent scaffold were used?
Was success determined automatically or by humans?
What reliability level is being reported?
Is the claim about autonomous capability or human assistance?
Does the evidence measure generated output or total workflow productivity?
Is the human baseline low-context or already familiar with the work?
What token, time, retry, and tool budget was used?
Are the evaluated tasks independent, or does later work depend on retaining their context?
Could the agent improve the score by exploiting the evaluator instead of satisfying the intended contract?
Were important tests hidden, held out, or independently reviewed?
When were the tasks released relative to the model and evaluation run?
Could the agent retrieve a future commit or published solution at runtime?
Is the task difficulty concentrated at the floor or ceiling?
Does the evaluation design support the capability claimed in the conclusion?

These questions make impressive numbers more useful without dismissing genuine capability improvements.

Key Takeaways

A time horizon measures task difficulty using human completion time; it is not the agent’s runtime.
Rising time horizons show real progress on the evaluated task distribution.
Capability varies significantly across domains, interfaces, and task shapes.
Low-context benchmark baselines are not equivalent to high-context maintainers.
Useful automation may require much higher reliability than a capability benchmark measures.
Task dependencies and context reconstruction can reduce the benefit of delegation.
A benchmark result belongs to the complete model, scaffold, tools, budget, and grader.
Static benchmarks lose signal through contamination, saturation, and task-distribution drift.
Use versioned anchor tasks plus a rolling frontier set instead of treating one public suite as permanent.
Control both training-time exposure and runtime access to known solutions.
Construct validity determines which conclusions a benchmark score can support.
An agent can improve a benchmark score by exploiting gaps between the evaluator and the intended goal.
Agent ability changes with expenditure, so cost and human effort belong in the comparison.
Automated benchmark success does not guarantee maintainable or mergeable code.
Developer productivity includes prompting, review, correction, and integration.
Treat productivity results as snapshots tied to particular tools, tasks, and users.
Use benchmarks, human review, field studies, and production evidence together.