Evaluate AI Ticket Routing With a Confusion Matrix

An AI router can be “80% correct” while missing half the account-access requests. Before connecting it to a queue, inspect mistakes behind that score.

Build an offline evaluator for three labels. It checks prediction quality, not export formatting. Python’s standard library suffices; synthetic data requires no model account or customer information.

1. Freeze the routing rules and answer key

Define one destination per ticket:

For mixed requests, agree on precedence with support before labeling. Reviewers must assign reference labels without seeing predictions. Keep IDs stable and redact sensitive details.

Use a development set for prompt changes and a separate, untouched test set for the final comparison. This applies the separation principle in scikit-learn’s data-leakage guidance to prompt-based routing. Do not put the test answer key into the classifier’s input.

If you already have a text-classification workflow, capture its predictions by ticket ID. The fixture below supplies pretend predictions to test the evaluator, not evidence of any model’s performance.

2. Read the matrix and recall

Here, matrix rows are reviewed labels and columns are predictions, matching scikit-learn’s convention. Diagonal cells are correct routes; other cells show mistakes. See our confusion-matrix explainer for background.

Accuracy is correct routes divided by all tickets. Per-class recall is correct routes for that class divided by its actual ticket count. Macro recall averages those class recalls equally, following the official recall documentation. It prevents the largest queue from dominating this summary.

For this exercise, choose a review gate of macro recall at least 0.80 and access recall at least 0.90. These are illustrative workflow choices, not universal release standards.

3. Run a dependency-free evaluator

Save this as evaluate_ticket_routing.py. Use Python 3.10 or later. Keep label values and code unchanged in either language.

LABELS = ("general", "billing", "access")
# ticket_id: (reviewed label, synthetic model prediction)
FIXTURE = {
    "t1": ("general", "general"),
    "t2": ("general", "general"),
    "t3": ("general", "general"),
    "t4": ("general", "general"),
    "t5": ("general", "general"),
    "t6": ("general", "general"),
    "t7": ("billing", "billing"),
    "t8": ("billing", "general"),
    "t9": ("access", "access"),
    "t10": ("access", "general"),
}
GOLD = {key: pair[0] for key, pair in FIXTURE.items()}
PREDICTED = {key: pair[1] for key, pair in FIXTURE.items()}


def evaluate(gold, predicted):
    if set(gold) != set(predicted):
        raise ValueError("missing or extra ticket IDs")
    if any(v not in LABELS for v in [*gold.values(), *predicted.values()]):
        raise ValueError("unknown label")
    matrix = [[0] * len(LABELS) for _ in LABELS]
    for key, actual in gold.items():
        matrix[LABELS.index(actual)][LABELS.index(predicted[key])] += 1
    support = [sum(row) for row in matrix]
    if not all(support):
        raise ValueError("every class needs a gold example")
    recall = {label: matrix[i][i] / support[i]
              for i, label in enumerate(LABELS)}
    accuracy = sum(matrix[i][i] for i in range(len(LABELS))) / len(gold)
    return matrix, accuracy, recall


def report(name, predicted):
    matrix, accuracy, recall = evaluate(GOLD, predicted)
    macro = sum(recall.values()) / len(LABELS)
    print(name)
    print("columns:", ", ".join(LABELS))
    for label, row in zip(LABELS, matrix):
        print(f"{label}: {row} recall={recall[label]:.3f}")
    print(f"accuracy={accuracy:.3f} macro_recall={macro:.3f}")
    ready = macro >= 0.80 and recall["access"] >= 0.90
    print("review_gate=" + ("PASS" if ready else "HOLD"))


if __name__ == "__main__":
    report("candidate", PREDICTED)
    report("constant baseline", {key: "general" for key in GOLD})

python3 evaluate_ticket_routing.py

Expected output:

candidate
columns: general, billing, access
general: [6, 0, 0] recall=1.000
billing: [1, 1, 0] recall=0.500
access: [1, 0, 1] recall=0.500
accuracy=0.800 macro_recall=0.667
review_gate=HOLD
constant baseline
columns: general, billing, access
general: [6, 0, 0] recall=1.000
billing: [2, 0, 0] recall=0.000
access: [2, 0, 0] recall=0.000
accuracy=0.600 macro_recall=0.333
review_gate=HOLD

4. Validate the result before trusting it

The candidate gets eight of ten tickets right, but sends one billing ticket and one access ticket to general. Both smaller classes have recall 0.50, so the gate holds the candidate for review.

The constant baseline sends everything to general: 60% accuracy but zero recall for both smaller classes. Scikit-learn documents constant predictors as basic comparisons. Choose the baseline using development or training data, never the final test set.

Replace PREDICTED with dict(GOLD) to check the scorer: accuracy and macro recall should reach 1.000, with PASS. This checks arithmetic only. Printing HOLD still exits successfully; the program is not a shell-level deployment blocker.

5. Troubleshoot and expand carefully

• Missing or extra IDs: match predictions to the frozen input list; never silently drop unanswered tickets.
• Unknown label: enforce the three-value vocabulary. Do not automatically turn invalid responses into general.
• A class has no examples: add reviewed examples; missing coverage is not perfect recall.
• Unexpectedly perfect results: inspect for leaked labels or repeated tickets across development and test sets.

Two examples per smaller class cannot establish production readiness. Collect more representative tickets, report class counts, and review the missed cases. Change one routing rule or prompt instruction, rerun development checks, then evaluate the untouched test set. For the trade-off with false alarms, see recall versus precision.

Practical takeaway

Compare against a baseline, inspect off-diagonal errors, and set class-specific expectations before automating. The matrix identifies queues needing attention; overall accuracy alone does not.