Compare AI Prompt Versions With a Python Regression Test

An edited AI prompt routes five of six tickets correctly instead of four. Ship it? Not if the new mistake sends a locked-out customer to the general queue. A regression is a case that previously passed and now fails; an average can hide one.

Build a local Python comparison that lists improvements, exposes regressions, and blocks a chosen critical regression. Synthetic saved predictions need no account, packages, or customer data. These numbers demonstrate the checker, not model improvement.

1. Freeze the cases and the acceptance rule

Use the same inputs and independently reviewed answers for both prompt versions. Here is the support-ticket fixture:

Mark locked_out as critical before viewing results. Our demonstration rule allows no drop in total correct answers and no newly broken critical case. Noncritical regressions remain visible for review. This is a chosen comparison policy, not a universal deployment standard.

Anthropic’s evaluation guidance recommends task-specific tests, edge cases, and automated grading where suitable. Exact label matching fits this narrow task; it does not evaluate free-form answer quality.

For a real comparison, save outputs under stable case IDs and record both prompt texts, model identifier, generation settings, and collection time. Change only the prompt if you want to isolate that change. Review ambiguous labels before scoring; our ticket-routing tutorial explains the routing rules and per-class checks.

2. Compare saved predictions by ID

Save this as compare_prompts.py and run it with Python 3.9 or newer. Replace the prediction dictionaries with your reviewed exports. Exact ID-set matching prevents missing or extra cases from silently changing the comparison.

expected = {
    "reset": "access", "charge": "billing", "refund": "billing",
    "locked_out": "access", "setup": "general", "invoice": "billing",
}
critical = {"locked_out"}
baseline = {
    "reset": "access", "charge": "general", "refund": "general",
    "locked_out": "access", "setup": "general", "invoice": "billing",
}
candidate = {
    "reset": "access", "charge": "billing", "refund": "billing",
    "locked_out": "general", "setup": "general", "invoice": "billing",
}


def compare(before, after):
    labels = {"access", "billing", "general"}
    if not expected or not critical <= expected.keys():
        raise ValueError("invalid reference cases")
    for name, predictions in (("baseline", before), ("candidate", after)):
        if predictions.keys() != expected.keys():
            raise ValueError(f"{name}: case IDs must match exactly")
        if any(not isinstance(v, str) or v not in labels
               for v in predictions.values()):
            raise ValueError(f"{name}: unknown label")

    old = {key: before[key] == label for key, label in expected.items()}
    new = {key: after[key] == label for key, label in expected.items()}
    improved = sorted(key for key in expected if not old[key] and new[key])
    regressed = sorted(key for key in expected if old[key] and not new[key])
    critical_regressions = sorted(critical.intersection(regressed))
    blocked = sum(new.values()) < sum(old.values()) or bool(critical_regressions)
    print(f"baseline={sum(old.values())}/{len(expected)}")
    print(f"candidate={sum(new.values())}/{len(expected)}")
    print(f"improved={improved}")
    print(f"regressed={regressed}")
    print(f"critical_regressions={critical_regressions}")
    print("gate=" + ("BLOCK" if blocked else "PASS"))
    return 1 if blocked else 0


if __name__ == "__main__":
    raise SystemExit(compare(baseline, candidate))

python3 compare_prompts.py

3. Validate the gate, not just the score

The supplied fixture prints:

baseline=4/6
candidate=5/6
improved=['charge', 'refund']
regressed=['locked_out']
critical_regressions=['locked_out']
gate=BLOCK

The process exits with status 1 and no stderr. That failure is intentional: two billing fixes outweigh one access mistake numerically, but the newly broken case is critical. Python’s exit-status documentation explains why a nonzero status lets automation detect failure.

Run these additional checks from the same directory:

python3 - <<'PYTEST'
from compare_prompts import baseline, candidate, compare

fixed = dict(candidate, locked_out="access")
assert compare(baseline, fixed) == 0

missing = dict(fixed)
del missing["invoice"]
try:
    compare(baseline, missing)
except ValueError as error:
    print(error)
else:
    raise AssertionError("missing case was accepted")
PYTEST

The corrected fixture reports candidate=6/6, regressed=[], critical_regressions=[], and gate=PASS. The missing-case check prints candidate: case IDs must match exactly. The combined command exits 0 without stderr. Both fixtures were checked locally with Python 3.11.

4. Troubleshoot misleading comparisons

• ID mismatch: join exports by stable IDs, not row position. Investigate missing cases instead of silently scoring their intersection.
• Unknown label: check spelling and the output contract. The checker rejects unknown labels instead of guessing.
• A passing gate despite existing errors: this rule checks change, not absolute quality. Compare the baseline with itself and it passes at 4/6; add minimum accuracy and per-class requirements before deployment.
• Results vary between runs: saved-output scoring is deterministic; model generation may not be. Repeat collection for both prompts and inspect how often each critical case fails. Do not report this six-case demonstration as statistical evidence.

Keep development cases separate from final evaluation cases. Scikit-learn’s data-leakage guidance explains why test data must not drive model choices; apply that separation to prompt tuning too. Our train/test overlap checker helps find repeated text, though it cannot certify independence.

If outputs are probabilities rather than labels, use an additional Brier score check; a regression list does not measure probability quality.

Practical takeaway

Keep the headline score, but always ask which cases changed. Freeze the rule first, compare identical IDs, and review newly broken critical cases before accepting a prompt revision. A passing comparison is one checkpoint, not permission to deploy.