Reject Ambiguous AI JSON Before Automation With Python

An AI assistant can return something that looks like JSON but leaves your workflow with two conflicting labels or a confidence value that is not a number. A successful parse is not enough: Python’s default decoder accepts repeated keys and keeps the last value. Python documents this behavior.

Build a small offline gate that rejects ambiguity before a response reaches automation. You need Python 3.10 or later and no extra packages or model account. The fixtures below are synthetic parser tests, not measured AI responses.

1. Define the receiving contract

For a ticket-labeling prototype, require one object with exactly two fields:

{"label":"billing","confidence":0.8}

Allowed labels are billing, technical, and other. Confidence must be a finite number from 0 to 1, inclusive; booleans and numeric strings are not accepted. Limit each UTF-8 response to 16,384 bytes. These are our workflow rules, not universal JSON requirements.

RFC 8259 recommends unique object names and warns that duplicate handling varies. Its number grammar excludes NaN and Infinity. Reject those cases instead of guessing the model’s intention. As with SQL integrity constraints, make acceptance rules explicit before using the data.

2. Save the strict checker

Save this as check_ai_json.py in a fresh working directory. It reads one bounded response file and never executes model-written code or routes a ticket.

import json
import math
import sys


def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError(f"duplicate key: {key}")
        result[key] = value
    return result


def reject_constant(token):
    raise ValueError(f"non-JSON constant: {token}")


def load_reply(path):
    with open(path, "rb") as stream:
        payload = stream.read(16_385)
    if len(payload) > 16_384:
        raise ValueError("response exceeds 16384 bytes")
    try:
        data = json.loads(payload.decode("utf-8"),
                          object_pairs_hook=unique_object,
                          parse_constant=reject_constant)
    except json.JSONDecodeError as error:
        raise ValueError("invalid JSON syntax") from error
    if not isinstance(data, dict) or set(data) != {"label", "confidence"}:
        raise ValueError("expected only label and confidence")
    if type(data["label"]) is not str or data["label"] not in {
        "billing", "technical", "other"
    }:
        raise ValueError("unknown label")
    score = data["confidence"]
    if type(score) not in (int, float):
        raise ValueError("confidence must be a number, not bool or string")
    if not 0 <= score <= 1 or not math.isfinite(score):
        raise ValueError("confidence must be finite and between 0 and 1")
    return data


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python3 check_ai_json.py RESPONSE.json")
    try:
        result = load_reply(sys.argv[1])
    except (OSError, ValueError, RecursionError) as error:
        raise SystemExit(f"FAIL: {error}")
    print(f"PASS: structure accepted; label={result['label']}")

The decoder hooks inspect original key pairs and reject non-JSON constants. Duplicate detection also catches escaped spellings of the same key. For numeric validation, Python booleans are integer subtypes, so the exact-type check deliberately excludes them.

The range check comes before isfinite, avoiding conversion of enormous out-of-range integers. The syntactically valid JSON number 1e400 can overflow the decoder’s default float representation; it must still fail our confidence policy.

3. Create positive and negative fixtures

Save this beside the checker as make_json_fixtures.py. Run it in the fresh directory: it creates json-demo and refuses to reuse an existing folder. The adversarial cases follow the same principle as small regex acceptance tests.

from pathlib import Path

root = Path("json-demo")
root.mkdir()
good = '{"label":"billing","confidence":0.8}'
cases = {
    "good": good,
    "duplicate": '{"label":"billing","label":"technical","confidence":0.8}',
    "nan": good.replace("0.8", "NaN"),
    "infinity": good.replace("0.8", "Infinity"),
    "overflow": good.replace("0.8", "1e400"),
    "boolean": good.replace("0.8", "true"),
    "quoted": good.replace("0.8", '"0.8"'),
    "extra": good[:-1] + ',"note":"hello"}',
    "prose": good + " Here is your answer.",
    "array": "[" + good + "]",
    "escaped": r'{"label":"billing","la\u0062el":"other","confidence":0.8}',
}
for name, text in cases.items():
    with (root / f"{name}.json").open("x", encoding="utf-8") as stream:
        stream.write(text)

Run these commands in Bash. Leave set -e disabled because ten failures are intentional.

python3 make_json_fixtures.py
for case in good duplicate nan infinity overflow boolean quoted extra prose array escaped; do
    python3 check_ai_json.py "json-demo/$case.json"
    echo "exit=$?"
done

4. Validate the observed results

Locally tested with Python 3.11.4, good prints this to stdout, leaves stderr empty, and exits 0:

PASS: structure accepted; label=billing

Every other case exits 1 with empty stdout. Its stderr contains one line:

The Bash loop prints each exit status separately. All eleven input files remained unchanged during these checks. Require the valid fixture to pass and every negative fixture to fail before integrating the checker.

Passing says nothing about whether billing is the correct label. Measure that separately with a labeled dataset and a ticket-routing confusion matrix. A model-supplied confidence number is not automatically a calibrated probability.

5. Troubleshoot without silently repairing data

• Markdown wrappers or trailing explanations: request a raw JSON object again. Do not extract a convenient substring and ignore the rest.
• Duplicate keys: regenerate the response. Keeping one value silently discards conflicting evidence.
• Quoted numbers or booleans: fix the output contract rather than coercing them into numbers.
• Unexpected fields: version the receiving contract and update fixtures before allowing new keys.
• Still accepts duplicates with strict=True: that decoder option controls unescaped control characters, not key uniqueness. Use the hooks above.

Keep this gate separate from downstream actions. The byte cap is a tutorial safeguard, not a complete resource-isolation or security boundary. Production ingestion also needs appropriate resource limits and a review path for rejected responses.

Practical takeaway

Treat AI JSON as untrusted input. Reject ambiguous syntax and invalid field values first, then evaluate whether the accepted content is useful and correct. Keep the original response and rejection reason for debugging.