Evaluate AI Probability Forecasts With the Brier Score in Python

Two AI classifiers both make three correct decisions out of four. Are their forecasts equally useful? Not if one assigns almost certain probability to its mistake. Accuracy hides that difference; the Brier score exposes it.

This tutorial compares binary probability forecasts with a small Python checker. You will validate the arithmetic, reject malformed inputs, and see why a lower probability loss does not automatically prove a model is ready for deployment. Everything runs locally without packages, API calls, or customer data.

1. Define the event before collecting probabilities

Imagine a support classifier predicting whether a ticket needs specialist review. Let 1 mean specialist review is required and 0 mean it is not. Every probability must describe that same positive event, regardless of the predicted label.

A forecast of 0.2 means a 20% probability of needing specialist review, not 20% confidence in whichever class the model chose. Do not substitute token probabilities or a chatbot’s self-reported certainty without separately validating their relationship to this event.

Keep reviewed outcomes, stable ticket IDs, and model forecasts. Match records by ID before constructing the aligned lists below. Unlike our ticket-routing evaluation, this exercise evaluates probabilities rather than only category decisions.

2. Understand the binary score

For each record, subtract the observed outcome from its forecast and square the difference. Average those squared errors: Brier = mean((probability − outcome)²). We use the binary convention ranging from 0 to 1; lower is better. This matches the binary definition in the scikit-learn metric reference.

Predicting 0.99 when the outcome is 0 contributes 0.9801; predicting 0.6 contributes 0.36. Both decisions are wrong at a 0.5 threshold, but the more confident error costs more.

Do not round probabilities into labels before scoring. Keep threshold-based precision and recall as separate measures of the resulting decisions.

3. Save a dependency-free evaluator

Save this as evaluate_brier.py and use Python 3.10 or later. The forecasts are hand-written fixtures, not measured model performance. The names describe these examples, not general classes of models.

import math


def brier(labels, probabilities):
    if not labels or len(labels) != len(probabilities):
        raise ValueError("Use nonempty, equal-length lists")
    if any(type(y) is not int or y not in (0, 1) for y in labels):
        raise ValueError("Labels must be integers 0 or 1")
    for p in probabilities:
        if type(p) not in (int, float) or not math.isfinite(p) or not 0 <= p <= 1:
            raise ValueError("Probabilities must be finite numbers in [0, 1]")
    return sum((p - y) ** 2 for y, p in zip(labels, probabilities)) / len(labels)


def self_checks():
    assert math.isclose(brier([0, 1], [0.0, 1.0]), 0.0)
    assert math.isclose(brier([0, 1], [1.0, 0.0]), 1.0)
    assert math.isclose(brier([0, 1], [0.5, 0.5]), 0.25)
    invalid = [
        ([], []),
        ([0, 1], [0.2]),
        ([2], [0.5]),
        ([1], [1.2]),
        ([0], [float("nan")]),
        ([1], [float("inf")]),
        ([True], [0.5]),
        ([1], [True]),
    ]
    for labels, probabilities in invalid:
        try:
            brier(labels, probabilities)
        except ValueError:
            continue
        raise AssertionError("Invalid input was accepted")
    print("self-checks: passed")


if __name__ == "__main__":
    self_checks()
    labels = [0, 0, 1, 1]
    forecasts = {
        "cautious": [0.2, 0.6, 0.7, 0.8],
        "bold": [0.01, 0.99, 0.95, 0.99],
        "constant": [0.5, 0.5, 0.5, 0.5],
    }
    for name, probabilities in forecasts.items():
        loss = brier(labels, probabilities)
        accuracy = sum(
            int(p >= 0.5) == y for y, p in zip(labels, probabilities)
        ) / len(labels)
        print(f"{name}: accuracy={accuracy:.2f} brier={loss:.4f}")

4. Run and validate the example

Run in a terminal without Python’s -O option, which disables the assertions:

python3 evaluate_brier.py
echo "exit=$?"

Expected output:

self-checks: passed
cautious: accuracy=0.75 brier=0.1325
bold: accuracy=0.75 brier=0.2457
constant: accuracy=0.50 brier=0.2500
exit=0

The self-checks verify perfect forecasts, completely wrong forecasts, and constant 0.5 forecasts. They also require rejection of empty or unequal-length lists, invalid labels, out-of-range probabilities, nonfinite values, and booleans. Success writes to standard output with exit status 0 and no standard error.

Both cautious and bold get 75% accuracy. The second ticket is negative, yet bold assigns it 0.99. That mistake outweighs its improvements on other tickets. The constant reference scores 0.25; it is an illustrative comparison, not a universal acceptance threshold.

5. Compare real forecasts without leaking answers

Replace the fixtures with forecasts recorded before their outcomes were known to the model. Evaluate competing versions on the same reviewed records and retain per-record squared errors. Check train–test overlap before interpreting improvements.

For a practical reference, predict the training set’s event rate for every test record. Never estimate this reference from test outcomes. Keep the test population representative; artificially balancing rare events changes what the aggregate measures.

A lower Brier score does not isolate calibration. The scikit-learn calibration guide explains that discrimination also influences it. On a larger held-out set, compare average forecasts with observed event frequencies in probability bins, reporting each bin’s count. Fit any recalibration on separate data, then evaluate on untouched records.

6. Troubleshoot suspicious results

• Unexpectedly poor score: verify the positive-class definition and record alignment. Reversed probabilities can invalidate the comparison.
• Excellent accuracy, weak probability forecasts: inspect confident errors and compare with the constant reference.
• Large swings between runs: examine sample size, label disagreements, event rates, and changed populations before blaming the formula.
• Only hard labels available: obtain event probabilities if supported. Scoring only zeros and ones reduces this binary loss to the classification error rate.

Four fixtures validate a calculator, not a deployment decision. Set acceptance criteria using your workflow’s costs, review capacity, and held-out evidence.

Practical takeaway

Save probabilities alongside decisions. Check the scorer with known cases, compare on identical held-out records, and inspect confident mistakes before changing automation thresholds.