Check AI Summary Numbers Against a Source Table With Python

An AI-written weekly report says your team resolved 85% of tickets. The source table says 80%. The sentence sounds reasonable, but publishing it would introduce a false claim. NIST identifies confidently presented false content as a generative-AI risk in its Generative AI Profile.

Build a small offline Python gate that compares structured numerical claims with a trusted report. It flags wrong values, units, periods, and invented metrics. You need Python 3.9 or newer, no packages, and no model account. All data below is synthetic; these are checker tests, not measured model performance.

1. Freeze the reference before asking AI

Use one reviewed support report for ISO week 2026-W43. Here are its already-calculated metrics:

Keep the export, reporting scope, and metric definitions with your run. Do not ask the same model to invent the reference values.

Request a companion claim list alongside the prose: each row must contain metric, period, unit, and a string-valued value. Tell the assistant to copy existing metrics, not calculate new ones. For example, “80 tickets resolved” maps to resolved, 2026-W43, tickets, and "80".

Validate parsing and field types first. Our strict AI JSON tutorial introduces that separate gate; adapt its schema to exactly four string fields: metric, period, unit, and value. The tuples below represent validated rows in that order, not an arbitrary JSON parser.

2. Compare meaning-bearing fields, not isolated digits

Save this as check_summary_numbers.py in a fresh directory. The five claims deliberately contain four errors. Only the first detected problem in each row is reported.

from decimal import Decimal, InvalidOperation

PERIOD = "2026-W43"
source = {
    "resolved": ("tickets", "80"),
    "resolution_rate": ("percent", "80"),
    "median_hours": ("hours", "3.5"),
}
claims = [
    ("resolved", PERIOD, "tickets", "80"),
    ("resolution_rate", PERIOD, "percent", "85"),
    ("median_hours", PERIOD, "minutes", "3.5"),
    ("backlog", PERIOD, "tickets", "12"),
    ("resolved", "2026-W42", "tickets", "80"),
]


def check(metric, period, unit, value):
    if metric not in source:
        return "unknown metric"
    if period != PERIOD:
        return "period mismatch"
    expected_unit, expected_value = source[metric]
    if unit != expected_unit:
        return "unit mismatch"
    if not isinstance(value, str):
        return "invalid number"
    try:
        number = Decimal(value)
    except InvalidOperation:
        return "invalid number"
    if not number.is_finite():
        return "invalid number"
    return "supported" if number == Decimal(expected_value) else "value mismatch"


def main():
    reasons = [check(*row) for row in claims]
    for row, reason in zip(claims, reasons):
        print(f"{row[0]}: {reason}")
    print(f"supported={reasons.count('supported')}/{len(reasons)}")
    return int(not reasons or any(r != "supported" for r in reasons))


if __name__ == "__main__":
    raise SystemExit(main())

The Python Decimal documentation explains exact decimal representation from strings and the is_finite() check. Here, 80 and 80.0 compare equally without converting through binary floating point. Nonfinite values are rejected.

This is an exact-copy policy: 0.8 percent does not mean 80 percent, and 3.5 minutes does not mean 3.5 hours. Conversions and rounded summaries need separate, explicitly tested rules.

3. Validate failure and success paths

Run:

python3 check_summary_numbers.py

Expected stdout:

resolved: supported
resolution_rate: value mismatch
median_hours: unit mismatch
backlog: unknown metric
resolved: period mismatch
supported=1/5

The exit status is 1, with empty stderr. That is an intentional review gate, not a Python crash. Do not publish while any row is unsupported.

Now run these edge checks and replace the in-memory claims with correct reference copies:

python3 - <<'PYTEST'
import check_summary_numbers as demo

assert demo.check("resolved", demo.PERIOD, "tickets", "80.0") == "supported"
assert demo.check("resolution_rate", demo.PERIOD, "percent", "0.8") == "value mismatch"
for value in ("NaN", "Infinity", "unknown", True):
    assert demo.check("resolved", demo.PERIOD, "tickets", value) == "invalid number"

demo.claims = [
    (metric, demo.PERIOD, unit, value)
    for metric, (unit, value) in demo.source.items()
]
raise SystemExit(demo.main())
PYTEST

Expected stdout:

resolved: supported
resolution_rate: supported
median_hours: supported
supported=3/3

This command exits 0, with empty stderr. The assertions also check equivalent decimal spelling, a percent/fraction mistake, invalid text, booleans, and nonfinite numbers. Both commands were tested locally with the supplied fixtures; neither calls a model or modifies the source report.

When comparing prompt changes, save these cases and retain the bad examples. The prompt regression tutorial shows how to detect newly broken cases instead of trusting an improved average.

4. Troubleshoot the boundary

• Right number, wrong week: keep the period check. Never silently substitute the current report.
• Unit mismatch: regenerate the claim or use an approved conversion; do not just relabel it.
• Rounded figure rejected: define precision and tolerance per metric before allowing approximation.
• Unknown metric: locate independent evidence or remove the claim. Another AI answer is not a reference.
• All rows pass, prose is wrong: compare every sentence with its claim row. This gate cannot detect omitted claims, misleading comparisons, wrong entities, or invented explanations.

For required coverage, maintain a separate checklist of metrics the report must mention. A correct subset is not necessarily a complete summary. Route rejected rows to a person rather than treating “unsupported” as proof of fabrication; review-threshold evaluation helps measure that workload.

Practical takeaway

Ask AI for traceable claims, preserve the trusted reference, and check value, unit, and period together. Use a passing gate as evidence of numerical agreement—not proof that the whole summary is true.