Evaluate AI Field Extraction With Exact-Match Precision and Recall

An AI assistant converts maintenance requests into asset IDs and due dates. The JSON parses, but one asset is wrong, a date is invented, and another request is unanswered. How much of the extraction is actually correct?

Build a small, offline scorer that separates incorrect predictions from missing facts and counts fully correct records. It uses Python’s standard library, synthetic examples, and no model account. If your input is raw model JSON, first reject ambiguous JSON; valid syntax is not evidence of accurate extraction.

1. Freeze the reference and matching rules

Start with three source messages, kept unchanged as test data:

• r1: “Inspect pump A7 by 2026-11-10.”
• r2: “Inspect fan B2; no due date provided.”
• r3: “Inspect valve C9 by 2026-11-12.”

Ask the extractor for exactly two fields per record: asset, containing only the asset identifier, and due, containing an ISO-format date or JSON null when absent. In Python, represent null with None. A person reviews the sources and writes GOLD independently of the predictions. Do not let the model revise its own answer key.

Use exact, case-sensitive string matching. Spaces and date formatting matter; repeated values in different records remain distinct. This example handles one value per field, not repeated invoice line items or nested entities. Preserve record IDs even when extraction fails, with both fields set to None.

2. Count facts, not empty cells

Represent each nonempty fact as (record_id, field, value):

• True positive (TP): a predicted fact matches the reference.
• False positive (FP): a predicted fact has no reference match.
• False negative (FN): a reference fact has no prediction match.

A wrong value creates both an FP and an FN. Predicting a date where none exists creates only an FP. These distinctions follow Google Document AI’s extraction evaluation definitions; our scorer is a small exact-match implementation, not a reproduction of that service.

Precision is TP / (TP + FP); recall is TP / (TP + FN). F1 is 2*TP / (2*TP + FP + FN). These are the standard precision, recall, and F-score definitions. For more intuition, see recall versus precision.

3. Run the local scorer

Save this as evaluate_extraction.py and use Python 3.10 or later. CANDIDATE contains deliberately flawed, hand-written predictions, not measured model output. The checker rejects missing record IDs, unexpected fields, and invalid value types instead of silently excluding them.

FIELDS = ("asset", "due")
GOLD = {
    "r1": {"asset": "A7", "due": "2026-11-10"},
    "r2": {"asset": "B2", "due": None},
    "r3": {"asset": "C9", "due": "2026-11-12"},
}
CANDIDATE = {
    "r1": {"asset": "A7", "due": "2026-11-10"},
    "r2": {"asset": "B3", "due": "2026-11-11"},
    "r3": {"asset": None, "due": None},
}


def facts(rows):
    if not isinstance(rows, dict) or set(rows) != set(GOLD):
        raise ValueError("record IDs must match gold")
    result = set()
    for record_id, row in rows.items():
        if not isinstance(row, dict) or set(row) != set(FIELDS):
            raise ValueError("each record needs asset and due")
        for field, value in row.items():
            if value is not None:
                if not isinstance(value, str) or not value.strip():
                    raise ValueError("values must be nonempty strings or None")
                result.add((record_id, field, value))
    return result


def score(rows):
    expected, actual = facts(GOLD), facts(rows)
    tp = len(expected & actual)
    fp, fn = len(actual - expected), len(expected - actual)
    precision = tp / (tp + fp) if tp + fp else None
    recall = tp / (tp + fn) if tp + fn else None
    denominator = 2 * tp + fp + fn
    f1 = 2 * tp / denominator if denominator else None
    whole = sum(rows[key] == GOLD[key] for key in GOLD)
    return tp, fp, fn, precision, recall, f1, whole


def report(name, rows):
    tp, fp, fn, precision, recall, f1, whole = score(rows)
    fmt = lambda value: "n/a" if value is None else f"{value:.3f}"
    print(f"{name}: tp={tp} fp={fp} fn={fn}")
    print(f"precision={fmt(precision)} recall={fmt(recall)} f1={fmt(f1)}")
    print(f"whole_records={whole}/{len(GOLD)}")


if __name__ == "__main__":
    report("candidate", CANDIDATE)
    report("perfect control", GOLD)
    empty = {key: dict.fromkeys(FIELDS) for key in GOLD}
    report("empty control", empty)

python3 evaluate_extraction.py

Expected output:

candidate: tp=2 fp=2 fn=3
precision=0.500 recall=0.400 f1=0.444
whole_records=1/3
perfect control: tp=5 fp=0 fn=0
precision=1.000 recall=1.000 f1=1.000
whole_records=3/3
empty control: tp=0 fp=0 fn=5
precision=n/a recall=0.000 f1=0.000
whole_records=0/3

4. Validate the arithmetic and inspect failures

The candidate returns four facts, but only two match. The incorrect B3 contributes one FP and leaves B2 as one FN. Its invented due date contributes another FP. Both facts from r3 are missing, adding two FN. Only r1 is entirely correct.

The perfect control must reach 1.000 for all three metrics. The empty control must have zero recall; its precision is undefined, printed as n/a, because it predicts nothing. Matching absences do not earn TP credit, although whole-record comparison includes every field.

These controls test the scorer, not a model’s production quality. The command exits successfully even when predictions are poor; it reports measurements rather than enforcing a deployment gate.

5. Troubleshoot before comparing models

• Unexpected misses: inspect facts(GOLD) - facts(CANDIDATE). Check the source before changing labels.
• Formatting-only errors: agree on normalization before evaluation. Apply the same documented rule to both sides; do not erase meaningful identifier differences.
• High overall F1, bad dates: score each field separately and report its reference count. Common fields can hide failures in rarer ones.
• Surprisingly perfect results: check for answer leakage. Keep development examples separate from a held-out test set.

For real trials, use approved, redacted requests spanning missing dates, unusual identifiers, and unreadable inputs. Record model and prompt versions. Fix one failure pattern on development data, then use the same frozen test set for a final comparison. Our prompt regression tutorial shows how to track changes across versions.

Practical takeaway

Check exact facts and complete records before automating an extraction workflow. A parseable response can still contain invented values; a high average can still hide unusable records.