Evaluating and Optimizing DSPy Programs

An LLM answer can sound convincing and still be wrong. For a repeatable application, “this example looks good” is not enough evidence.

DSPy treats evaluation as part of the program:

1. collect representative examples
2. define how one result will be scored
3. measure the unoptimized program
4. let an optimizer search for better instructions or examples
5. test the result on data the optimizer did not see
6. save and deploy the accepted program

This page assumes you understand the basic terms from DSPy: Programming Language Models, but it starts with a plain-language explanation of optimization.

What Optimization Means Here

An optimizer is an automated experiment runner.

Imagine a support-classification program that is correct 72 times out of 100. You could manually rewrite its prompt, run the same few examples again, and decide whether it feels better.

Or you could give DSPy:

• the current program
• a set of examples
• a grading rule
• an experiment budget

The optimizer tries changes, runs the program, grades the results, and returns the highest-scoring version it found.

flowchart TD
    A["Current DSPy program"] --> D["Optimizer runs experiments"]
    B["Training examples"] --> D
    C["Metric<br/>What better means"] --> D
    D --> E["Candidate instructions or demonstrations"]
    E --> F["Validation score"]
    F -->|Budget remains| D
    F -->|Search complete| G["Compiled DSPy program"]
    G --> H["Held-out test evaluation"]
    H -->|Accepted| I["Save and deploy a versioned artifact"]
    H -->|Rejected| A

In DSPy, the word compile does not mean converting Python into machine code. It means producing a new DSPy program whose instructions, demonstrations, or model state have been selected by an optimizer.

Most DSPy optimizers do not change the base model’s weights. They optimize prompt-related parameters such as:

• instructions: natural-language directions attached to Signatures
• demonstrations: selected input/output examples placed in context
• weights: model parameters, only for optimizers that perform fine-tuning

Optimization Starts With A Working Program

Do not start with an optimizer.

Start with the simplest program that expresses the task correctly:

from typing import Literal
import dspy


class ClassifyUrgency(dspy.Signature):
    """Classify how quickly a support message requires human review."""

    message: str = dspy.InputField()
    urgency: Literal["low", "medium", "high"] = dspy.OutputField()


program = dspy.Predict(ClassifyUrgency)

Before optimization, verify that:

• inputs and outputs represent the real task
• types and labels are correct
• deterministic business rules are outside the model
• the program runs without parsing or tool errors
• obvious failure cases are understood
• the model and adapter are explicitly configured

An optimizer can improve the behavior rewarded by a metric. It cannot repair an incoherent task definition or decide what the business actually wants.

Examples: The Data DSPy Learns And Tests From

An example is one realistic program input plus any expected output or metadata needed for scoring.

examples = [
    dspy.Example(
        message="Production checkout rejects every payment.",
        urgency="high",
    ).with_inputs("message"),
    dspy.Example(
        message="Please add dark mode when convenient.",
        urgency="low",
    ).with_inputs("message"),
]

.with_inputs("message") tells DSPy that message is sent into the program. urgency remains available as the expected answer for the metric.

What makes examples useful

Useful data represents the decisions the application must make:

• common cases
• rare but important cases
• ambiguous cases
• malformed and incomplete inputs
• domain terminology
• each supported language or region
• cases where the correct action is to abstain or escalate
• known historical failures

Twenty nearly identical examples are less useful than twenty examples that cover genuinely different situations.

The workshop suggests that tens of examples can sometimes produce meaningful gains. That is an observation, not a universal sample-size rule. The required amount depends on task diversity, label quality, metric stability, model capability, and the cost of an error.

Split Data By Purpose

Do not optimize and report results on the same examples.

Split	Purpose	Can the optimizer use it?
Training set	Supplies examples and feedback during optimization	Yes
Validation set	Compares candidates and guides selection	Yes, depending on optimizer
Test set	Estimates final performance on unseen data	No

A simple split might look like:

trainset = examples[:60]
valset = examples[60:80]
testset = examples[80:100]

Real data should be shuffled or split deliberately so one customer, document, conversation, or duplicated record cannot leak across sets. Time-based applications may need a chronological split to measure behavior on newer data.

The test set should remain untouched until the program and optimizer choices are settled. Repeatedly checking the test score while changing the program turns the test set into another validation set.

Metrics: Define What Better Means

A DSPy metric is a Python callable that receives:

• the original example
• the program’s prediction

and returns a score.

The simplest metric compares the predicted label with the expected label:

def urgency_accuracy(example, prediction) -> float:
    return 1.0 if prediction.urgency == example.urgency else 0.0

This is preferable to an LLM judge when correctness can be checked exactly.

Three metric families

Metric family	Example	Strength	Limitation
Deterministic rule	Exact label, valid range, required citation, executable test	Repeatable and inexpensive	Cannot judge every semantic quality
Reference comparison	Exact match, token F1, semantic similarity	Uses known answers	A single reference may not represent every valid answer
LLM judge	Rubric-based relevance, faithfulness, style, or completeness	Can assess nuanced outputs	Costs more and may be inconsistent, biased, or gameable

Prefer deterministic checks wherever possible. Add model-based judgment only for properties that ordinary code cannot reasonably score.

Composite metrics

Many applications have several requirements. A customer reply may need to be correct, grounded, concise, and policy-compliant.

def reply_metric(example, prediction) -> float:
    grounded = source_supports(prediction.answer, prediction.citations)
    policy_safe = passes_policy_checks(prediction.answer)
    concise = len(prediction.answer) <= 800

    if not grounded or not policy_safe:
        return 0.0

    return 1.0 if concise else 0.8

This example treats grounding and policy as gates rather than allowing a fluent answer to compensate for a dangerous failure.

Metric weighting is a product decision. If false negatives are much more costly than false positives, plain accuracy may be the wrong metric. Use recall, precision, cost-weighted error, or a task-specific scoring rule that reflects the real consequence.

Run A Baseline Evaluation

Measure the original program before optimizing it:

evaluate_validation = dspy.Evaluate(
    devset=valset,
    metric=urgency_accuracy,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

baseline_result = evaluate_validation(program)
print(baseline_result.score)

Record more than the aggregate score:

• program version
• DSPy version
• model and adapter
• dataset revision
• metric revision
• cost and token usage
• latency distribution
• failed or timed-out examples
• score by important category

An average can hide a serious problem. A classifier that scores 95% overall but misses half of urgent incidents is not a successful urgent-incident classifier.

Metrics For GEPA: Score Plus Feedback

Most DSPy optimizers need only a numeric score. GEPA can also consume natural-language feedback about why a particular predictor succeeded or failed.

def urgency_feedback(
    example,
    prediction,
    trace=None,
    pred_name=None,
    pred_trace=None,
):
    if prediction.urgency == example.urgency:
        return dspy.Prediction(
            score=1.0,
            feedback="The urgency label matches the reviewed answer.",
        )

    return dspy.Prediction(
        score=0.0,
        feedback=(
            f"Expected {example.urgency}, but received {prediction.urgency}. "
            "Pay attention to immediate service impact and whether all users are blocked."
        ),
    )

The additional arguments have specific purposes:

Argument	Meaning
example	The original data record and expected fields
prediction	The final program result
trace	Information about the program execution when supplied by an optimizer
pred_name	The particular predictor being reviewed in a multi-module program
pred_trace	Trace information for that predictor

At normal evaluation time, the score is what matters. During GEPA optimization, feedback can help a reflection model propose better instructions for the relevant predictor.

Feedback should identify an observable failure. Vague text such as “think harder” gives the optimizer little useful direction.

How GEPA Works

GEPA is DSPy’s reflection-driven instruction optimizer. The associated research describes the approach as reflective prompt evolution. Its important practical feature is feedback-guided instruction search.

At a high level:

1. run the current program on training examples
2. score the predictions
3. provide failures and textual feedback to a reflection model
4. ask that model to propose revised instructions
5. evaluate the candidates
6. retain strong candidates and continue until the budget is used

flowchart TD
    A["Run candidate program"] --> B["Score each example"]
    B --> C["Collect predictor-specific feedback"]
    C --> D["Reflection model proposes instruction changes"]
    D --> E["Evaluate revised candidate"]
    E --> F{"Improves the search frontier?"}
    F -->|Yes| G["Retain candidate"]
    F -->|No| H["Discard candidate"]
    G --> I{"Budget remains?"}
    H --> I
    I -->|Yes| A
    I -->|No| J["Return compiled program"]

Configure and run GEPA:

reflection_lm = dspy.LM("openai/gpt-5.4")

optimizer = dspy.GEPA(
    metric=urgency_feedback,
    reflection_lm=reflection_lm,
    auto="light",
    num_threads=4,
)

optimized_program = optimizer.compile(
    program,
    trainset=trainset,
    valset=valset,
)

Important details:

• reflection_lm proposes instruction changes; it does not need to be the same model used by the program
• auto="light", "medium", or "heavy" controls the search budget within GEPA
• num_threads controls concurrency and must respect provider rate limits
• compile() returns a new program; it does not mutate the original program
• more budget means more model calls, not guaranteed improvement

Use a capable reflection model when its proposals materially improve a smaller or cheaper task model. Measure the total optimization cost before deciding that the arrangement is economical.

Choosing An Optimizer

DSPy includes several optimizer families. Choose based on what needs to change, the data available, and the experiment budget.

Optimizer	What it changes	Data and signal needed	Good starting use
LabeledFewShot	Selects labeled demonstrations	Labeled examples	Near-zero-cost few-shot baseline
BootstrapFewShot	Generates and keeps successful demonstrations	Examples and usually a reliable metric	First practical optimizer to try
BootstrapFewShotWithRandomSearch / BootstrapRS	Searches across several demonstration sets	Train and validation data	When one bootstrap sample is unstable
KNNFewShot	Chooses similar demonstrations at inference time	Embedded example collection	Inputs that need different local examples
COPRO	Rewrites instructions	Metric and prompt model	Instructions are the likely bottleneck
GEPA	Rewrites instructions using reflective feedback	Feedback-rich metric and reflection model	Complex instructions and interpretable failure feedback
MIPROv2	Jointly searches instructions and demonstrations	Metric, examples, and larger budget	Both wording and examples need improvement
SIMBA	Adds rules or demonstrations targeted at weak minibatch examples	Metric and representative batches	Repeated failure patterns
BootstrapFinetune	Fine-tunes model weights from successful traces	Tunable model and training infrastructure	Prompt-only methods have plateaued
BetterTogether	Composes prompt and weight optimization	Both optimizer types and substantial budget	Prompt-only and weight-only approaches have already been tested

A practical decision flow is:

flowchart TD
    A["Do you have a working program and metric?"] -->|No| B["Build and evaluate those first"]
    A -->|Yes| C["Try a low-cost few-shot baseline"]
    C --> D{"What remains weak?"}
    D -->|Instructions| E["Try COPRO or GEPA"]
    D -->|Examples| F["Try BootstrapRS or KNNFewShot"]
    D -->|Both| G["Try MIPROv2 or GEPA"]
    E --> H{"Prompt-only optimization plateaued?"}
    F --> H
    G --> H
    H -->|No| I["Validate and save the compiled program"]
    H -->|Yes and model is tunable| J["Consider BootstrapFinetune"]

Start with BootstrapFewShot or another inexpensive baseline. A costly optimizer is wasteful if a simple demonstration set already meets the requirement.

GEPA, MIPROv2, And Fine-Tuning Are Different

These techniques are sometimes described as interchangeable improvements, but they search different things.

GEPA

• focuses on instruction evolution
• can use textual feedback for each predictor
• leaves model weights unchanged
• works with closed model APIs

MIPROv2

• searches instructions and few-shot demonstrations together
• uses a metric as a black-box score
• leaves model weights unchanged
• can cost more because the joint search space is larger

Fine-tuning

• changes model weights
• requires a provider or local model that supports training
• adds dataset preparation, training, deployment, and model governance
• may help after prompt-only methods stop improving

The talk references research where GEPA or MIPROv2 performed better than an RL-based method in the reported benchmark. The valid conclusion is narrow: prompt optimization can outperform weight-updating methods on some evaluated tasks and budgets. It does not establish that GEPA universally beats reinforcement learning or fine-tuning.

Protect Against Metric Gaming

An optimizer searches for behavior that scores well. If the metric is incomplete, the optimizer may exploit what it rewards instead of satisfying the real goal.

For example, a summarizer rewarded only for short length may return an empty or unhelpfully vague summary. A support reply judged only for politeness may confidently invent a resolution.

Use several safeguards:

1. Combine deterministic requirements with semantic scoring.
2. Keep a held-out test set the optimizer never sees.
3. Review a sample of outputs, not only aggregate scores.
4. Track important subgroups and high-impact failure classes.
5. Test adversarial and out-of-distribution inputs.
6. Compare the compiled program with the original baseline.
7. Re-evaluate with an independent judge or human reviewers.
8. Version metrics so a score can be reproduced.

LLM judges need their own evaluation

An LLM judge is another model call, not an objective authority.

Check whether the judge:

• agrees with human reviewers on representative examples
• is sensitive to irrelevant style, verbosity, or ordering
• can be influenced by text inside the candidate answer
• treats demographic or language groups differently
• produces stable scores across repeated runs
• shares failure modes with the model being judged

When possible, hide unnecessary candidate metadata, use clear rubrics, randomize comparison order, and combine the judge with code-based checks.

Validate The Compiled Program

After optimization, evaluate both versions on the untouched test set:

evaluate_test = dspy.Evaluate(
    devset=testset,
    metric=urgency_accuracy,
    num_threads=4,
    display_progress=True,
    display_table=5,
)

baseline_test = evaluate_test(program)
optimized_test = evaluate_test(optimized_program)

print("baseline:", baseline_test.score)
print("optimized:", optimized_test.score)

Do not deploy merely because the optimized score is numerically higher. Define acceptance criteria such as:

• minimum overall score
• no regression in critical categories
• maximum allowed false-negative rate
• cost per request
• p95 latency
• parse and tool failure rates
• human-review outcome

A small score gain may not justify a large increase in cost or complexity.

Save The Accepted Program

Optimization can be expensive. The normal production pattern is compile once, save, and reuse.

Save state as JSON:

optimized_program.save("support_urgency_v2.json")

Load it into the same program definition:

program = dspy.Predict(ClassifyUrgency)
program.load("support_urgency_v2.json")

State-only JSON is the preferred default because it is readable, diffable, and does not execute code while loading. It assumes the loading application has the same program definitions.

DSPy can also save a complete program using pickle-based serialization. Only load a pickle from a trusted source. Deserializing a malicious pickle can execute code. Current DSPy requires explicit opt-in for pickle loading and avoids serializing API keys.

Version the saved artifact with:

• source commit
• DSPy and Python versions
• model identifier
• adapter configuration
• dataset and metric versions
• optimizer and budget
• baseline and test results
• approval date

Delayed User Feedback

The workshop Q&A asks whether thumbs-up or thumbs-down feedback arriving minutes or days later can optimize a program online.

The safer default is an offline release loop:

flowchart TD
    A["Production predictions"] --> B["Collect outcomes and user feedback"]
    B --> C["Remove sensitive data and investigate labels"]
    C --> D["Curate a new dataset revision"]
    D --> E["Optimize in an isolated environment"]
    E --> F["Evaluate on held-out data"]
    F --> G["Human or automated release gate"]
    G --> H["Deploy a versioned program"]
    H --> A

Do not feed every user reaction directly into a live optimizer. Feedback may be accidental, manipulated, sparse, delayed, or unrelated to model quality. It also may contain personal or confidential information.

Curate feedback before using it:

• connect it to the exact program and model version
• distinguish answer quality from product dissatisfaction
• resolve contradictory labels
• protect private data
• weight expert review appropriately
• retain examples of both success and failure
• preserve a separate final test set

Online adaptation is possible to engineer, but it introduces model governance, rollback, audit, and safety problems beyond the normal DSPy optimizer workflow.

Cost And Rate Limits

DSPy itself does not make one model call inherently expensive. Optimization becomes expensive because it intentionally runs many candidate programs over many examples.

Estimate cost before running:

candidate programs
x examples per evaluation
x model calls per program execution
x tokens per call
x provider price

Multi-module and agentic programs can multiply calls further because one example may trigger several predictors and tool-loop iterations.

Control cost by:

• measuring a small baseline first
• using auto="light" before larger budgets
• starting with a subset that still represents the task
• limiting tool and agent iterations
• setting conservative thread counts
• using cached results only when they remain valid
• tracking optimization and inference cost separately
• stopping when gains flatten
• saving and reusing accepted compiled programs

Concurrency reduces elapsed time but does not reduce token cost. It can also trigger provider rate limits.

Moving A Task To A Smaller Model

One valuable workshop pattern is model transfer:

1. establish the task and metric
2. evaluate a capable but expensive model
3. evaluate a smaller model with the same program
4. optimize the smaller model
5. compare quality, cost, and latency on the held-out set

If the optimized smaller model meets the acceptance criteria, the organization may reduce inference cost without changing the high-level program.

This is not automatic portability. Different models have different knowledge, context limits, tool support, multimodal capabilities, safety behavior, and structured-output reliability. Re-run the entire evaluation for each model and adapter combination.

What The Workshop Demonstrates

The kmad/aie repository contains useful teaching examples:

• a typed sentiment classifier
• structured information extraction from a PDF
• multimodal image interpretation
• a custom Module that combines an LLM call with deterministic text rules
• tool use with ReAct
• document classification and routing
• recursive summarization
• visual page-boundary detection
• GEPA metrics with predictor-specific feedback

The durable lesson is the separation of concerns:

task contract -> execution strategy -> application control flow -> metric -> optimizer

The repository itself describes the code as workshop material rather than production-ready software. Production implementations need stronger validation, security, error handling, evaluation data, and operational controls.

Practical Checklist

Before optimization:

• The program represents the intended task.
• The baseline runs reliably.
• Examples cover realistic and difficult cases.
• Labels have been reviewed.
• The metric reflects actual business value.
• A held-out test set exists.
• Cost and rate limits are understood.

Before deployment:

• The optimized program beats the baseline on held-out data.
• Critical categories have not regressed.
• Outputs have been manually sampled.
• Adversarial and malformed inputs have been tested.
• Cost and latency meet the requirement.
• The program artifact and dependencies are versioned.
• Rollback to the previous program is possible.

Current Version Context

The workshop repository locks DSPy 3.0.4. The stable DSPy release as of July 19, 2026 is 3.2.1; 3.3.0b1 is a beta.

The core optimizer concepts remain current, but exact optimizer internals and result objects can change. In particular, the 3.3.0b1 release updates the GEPA integration and detailed result shapes. Use the official documentation for the installed DSPy version when inspecting optimizer internals.

Sources And Further Reading

• DSPy official documentation
• DSPy official GitHub repository
• Metrics and evaluation
• Choosing an optimizer
• GEPA optimization
• Saving and loading DSPy programs
• GEPA paper
• MIPRO paper
• Source talk: DSPy - The End of Prompt Engineering
• Workshop code: kmad/aie