Test AI-Generated Regex with Small Python Fixtures

An AI assistant can produce a regular expression that accepts your example and still admits the wrong identifiers. Before using it to clean a spreadsheet or validate an import, give it a small set of strings whose answers you already know.

This walkthrough builds an offline acceptance test with Python’s standard library. No model account, package installation, or real customer data is needed. The patterns below are hand-written examples of plausible mistakes, not measured outputs from an AI product.

1. Write the rule before asking AI

Suppose an internal identifier must contain exactly two uppercase ASCII letters, one hyphen, and four ASCII digits. Leading zeros are valid; lowercase letters, spaces, extra characters, and a trailing newline are not. This is an invented identifier format, not an email or international naming standard.

Ask your assistant for a Python re pattern, specifying those rules and requesting both accepted and rejected examples. Do not let the assistant’s examples become your only test data: write your own cases first and keep a few additional cases for later review.

Use whole-string validation. Python’s fullmatch() succeeds only when the entire string matches; search() can find a valid-looking fragment inside invalid input. Python’s regular-expression reference.

2. Create a small, readable fixture

Save the following as regex_check.py in an empty working directory. It contains two valid identifiers and eight invalid inputs, covering length boundaries, case, unwanted prefixes, whitespace, and Unicode digits.

import re
import sys

CASES = [
    ("AB-0000", True),
    ("ZX-9876", True),
    ("AB-123", False),
    ("AB-12345", False),
    ("ab-1234", False),
    ("XXAB-1234", False),
    ("AB-1234\n", False),
    ("AB-\u0661\u0662\u0663\u0664", False),
    ("", False),
    ("AB-12 4", False),
]
PATTERNS = {
    "candidate": r"[A-Z]{2}-\d{4}",
    "fixed": r"[A-Z]{2}-[0-9]{4}",
    "reject_all": r"(?!)",
    "accept_all": r"[\s\S]*",
}


def main(name):
    pattern = re.compile(PATTERNS[name])
    failures = 0
    for text, expected in CASES:
        actual = pattern.fullmatch(text) is not None
        if actual != expected:
            failures += 1
            print(f"FAIL {ascii(text)}: expected={expected}, actual={actual}")
    print(f"{name}: {len(CASES) - failures}/{len(CASES)} passed")
    return int(failures > 0)


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1]))

The candidate pattern looks reasonable, but Python string patterns treat \d as Unicode decimal digits, not just 0 through 9. The Arabic-Indic digits in one case expose that mismatch. The fixed pattern makes the ASCII requirement explicit with [0-9]. Python’s digit-class documentation.

Notice that we have not changed the expected answers to make the repaired pattern pass. The specification, rather than the generated suggestion, decides correctness.

3. Run the candidate and negative controls

Run each command separately from the directory containing the file:

python3 regex_check.py candidate
python3 regex_check.py fixed
python3 regex_check.py reject_all
python3 regex_check.py accept_all

The locally checked results with Python 3.11.4 are:

Failing runs also print each mismatched input with its expected and actual Boolean values. All four commands have empty stderr. Exit status 1 is intentional for the three failing patterns; it lets automation stop an import.

The controls check the checker: a validator that rejects everything must fail the positive cases, while one that accepts everything must fail the negatives. This is the same principle used in testing AI-written SQL against known results.

Require every case to pass, not merely a high percentage. Here, an always-rejecting pattern scores 80% while being useless.

4. Troubleshoot without weakening the rule

• An invalid suffix or prefix passes: check the calling code as well as the pattern. Keep fullmatch() rather than switching to a partial-match operation.
• Unexpected whitespace passes: inspect whether an earlier step calls strip(). Trimming changes the input and should be a separate, documented policy.
• Backslashes behave strangely: retain Python raw strings for patterns. Their r prefix avoids an extra layer of string-escape interpretation. Python’s regex HOWTO.
• A correctly formatted ID does not exist: regex checks shape, not record existence or uniqueness. Enforce those separately, for example with database integrity constraints.

When testing another AI suggestion, replace only the candidate pattern string and rerun the unchanged fixture. Add a new regression case whenever you find a mistake. These ten inputs do not prove correctness for every string or benchmark performance on long inputs.

Practical takeaway

Treat an AI-generated regex as a proposal: define the format, test both valid and invalid inputs, and require negative controls to fail. Keep the fixture alongside the importer and rerun it after pattern or prompt changes, following a continuous-testing workflow.