Find Train/Test Text Overlap Before Evaluating an AI Model

A support-ticket classifier can look accurate because its test set repeats examples used for training. Before trusting a score, check whether the two sets share text—even when capitalization, spacing, or character width differs.

This tutorial builds a local, read-only overlap checker with Python’s standard library. It needs no model account or packages. The fixtures are synthetic: they demonstrate the checker, not a measured improvement in AI accuracy.

1. Define what counts as overlap

For a small ticket dataset, export each split as a JSON array of input strings, without labels or customer details. Use the exact text your classifier receives. Keep originals elsewhere; the checker never edits them.

Our review policy treats texts as matching after Unicode NFKC normalization, case folding, and whitespace collapsing. We do not strip punctuation or accents. Python documents NFKC compatibility normalization and case folding; these operations can merge distinctions your task needs, so approve the policy first.

This checks one leakage route, not every route. Information from test data can make evaluation estimates overly optimistic; learned preprocessing must also stay training-only. Scikit-learn’s leakage guidance explains that broader problem. If splits are unfamiliar, start with our cross-validation introduction.

2. Save the checker

Save this as split_audit.py in an empty working directory. Use Python 3.10 or later.

import json
import sys
import unicodedata


def key(text):
    normalized = unicodedata.normalize("NFKC", text).casefold()
    return " ".join(normalized.split())


def load_texts(path):
    with open(path, encoding="utf-8") as stream:
        rows = json.load(stream)
    if not isinstance(rows, list) or not rows:
        raise ValueError(f"{path}: expected a nonempty list")
    if any(not isinstance(text, str) or not key(text) for text in rows):
        raise ValueError(f"{path}: expected nonempty strings")
    return rows


def main():
    if len(sys.argv) != 3:
        print("Usage: python3 split_audit.py TRAIN.json TEST.json",
              file=sys.stderr)
        return 2
    try:
        train, test = [load_texts(path) for path in sys.argv[1:]]
    except (OSError, ValueError) as error:
        print(f"ERROR: {error}", file=sys.stderr)
        return 2
    first = {}
    for i, text in enumerate(train):
        first.setdefault(key(text), i)
    matches = []
    for j, text in enumerate(test):
        if key(text) in first:
            matches.append((j, first[key(text)]))
    for j, i in matches:
        print(f"MATCH test[{j}] -> train[{i}]")
    print(f"overlap={len(matches)}/{len(test)} test rows")
    return int(bool(matches))


if __name__ == "__main__":
    raise SystemExit(main())

Each reported index starts at zero. The count is matching test rows, not unique texts or all matching pairs. When training rows repeat, the dictionary retains their first index. Only indices are printed, but the process still reads the full text into memory; run it locally on approved data.

3. Create fixtures and run them

In Bash, run the following in that fresh directory. These commands create six small JSON files; do not run them over existing files. Do not enable set -e: nonzero exits are intentional.

printf '%s\n' '["Reset my password", "Track my parcel"]' > train.json
printf '%s\n' '["Change my address"]' > clean.json
printf '%s\n' '["Reset my password"]' > exact.json
printf '%s\n' '["  ＲＥＳＥＴ\tmy  PASSWORD  "]' > formatted.json
printf '%s\n' '["Please reset my password"]' > near.json
printf '%s\n' '["   "]' > invalid.json
for case in clean exact formatted near invalid; do
    python3 split_audit.py train.json "$case.json"
    echo "exit=$?"
done

The formatted fixture includes full-width letters and a JSON tab escape. Both it and the exact fixture should match. The near fixture adds a word, deliberately exposing a limitation.

4. Validate the results

Locally tested with Python 3.11.4:

Stderr is empty except for invalid, which prints ERROR: invalid.json: expected nonempty strings. The shell prints each exit status separately. The checker left every fixture unchanged.

Exit 0 means no normalized overlap found, not “no leakage.” Exit 1 means review matches before scoring. Exit 2 means fix the input or invocation. Keep both matching and nonmatching controls; a checker that always reports zero would otherwise seem successful.

5. Troubleshoot and fix the split

• Wrong format: use a nonempty JSON array of nonblank strings, not JSONL or objects. Check encoding is UTF-8.
• Unexpected match: inspect the original indexed rows. Case, layout, or compatibility characters may matter in your domain; narrow the policy and update fixtures.
• Missed paraphrase: expected. This is equality after normalization, not semantic similarity. Related tickets from the same conversation can also leak context without sharing text.
• Large export: this demonstration loads both arrays into memory. Start with bounded files; it is not a streaming audit.

Do not silently delete flagged test rows and present the resulting score as comparable. Review provenance and labels, group duplicates before assigning splits, then freeze and version the revised dataset. For conversations or repeated customers, consider group-based splitting; for forecasting, respect time order. Our data-quality checklist covers surrounding cleanup work.

Practical takeaway

Run this check before training and before evaluating a new dataset version. Record the policy, file versions, and overlap count. Then use a separate, reviewed test set for your ticket-routing evaluation. Passing this narrow check is a useful gate—not proof of generalization or absence of pretraining contamination.