Check AI JSON Edits Against a Field Allowlist With Python

An AI assistant asked to turn off email notifications might also “improve” your retry settings. Both documents can be valid JSON and satisfy the same schema. What changed matters as much as whether the result parses.

Build a read-only gate that compares an original configuration with a proposed replacement and rejects edits outside one approved field. This tutorial uses Python 3.10 or later, standard-library modules, and synthetic local fixtures. It calls no model and changes no files.

1. Define the permitted change

Keep the original document and the permission list in application-controlled state. Ask the assistant for a complete candidate document, preserving every field except notifications.email. Do not let the candidate redefine its own permissions.

Use the exact JSON Pointer /notifications/email, not a prefix such as /notifications. RFC 6901 defines JSON Pointer paths and escaping: a slash inside a key becomes ~1, and a tilde becomes ~0. Therefore a literal key named a/b differs from nested keys a and b.

This gate assumes small, already validated JSON documents with string keys and finite numbers. Before connecting raw model responses, apply strict JSON input checks. Python’s default decoder accepts duplicate keys; its JSON documentation explains the relevant decoder hooks.

2. Save the comparison gate

Save this as audit_json_edits.py. Objects are compared recursively. Added or removed members count as changes, even when a removed value was null. Arrays are indivisible: any changed element or order marks the entire array field.

The sorted serialization comparison ignores object-key order and distinguishes booleans from numbers, including inside arrays. It also treats 1 and 1.0 as different. This is a conservative workflow rule, not a universal definition of JSON equality or cryptographic canonicalization.

import json
from copy import deepcopy

BASELINE = {
    "notifications": {"email": True},
    "retry_limit": 3,
    "channels": ["email", "sms"],
}
ALLOWED = {"/notifications/email"}


def serialized(value):
    return json.dumps(value, sort_keys=True, allow_nan=False)


def changed_paths(before, after, path=""):
    if isinstance(before, dict) and isinstance(after, dict):
        changes = set()
        for key in sorted(before.keys() | after.keys()):
            token = key.replace("~", "~0").replace("/", "~1")
            child = path + "/" + token
            if key not in before or key not in after:
                changes.add(child)
            else:
                changes.update(changed_paths(before[key], after[key], child))
        return changes
    return {path} if serialized(before) != serialized(after) else set()


def decision(candidate):
    changes = changed_paths(BASELINE, candidate)
    blocked = changes - ALLOWED
    if blocked:
        return "REJECT " + ", ".join(p or "(root)" for p in sorted(blocked))
    if type(candidate["notifications"].get("email")) is not bool:
        return "REJECT email must be boolean"
    return "ALLOW " + (", ".join(sorted(changes)) or "(unchanged)")


approved = deepcopy(BASELINE)
approved["notifications"]["email"] = False
overreach = deepcopy(approved)
overreach["retry_limit"] = 9
missing = deepcopy(BASELINE)
del missing["notifications"]["email"]
wrong_type = deepcopy(BASELINE)
wrong_type["notifications"]["email"] = 0

for name, candidate in [
    ("unchanged", BASELINE),
    ("approved", approved),
    ("overreach", overreach),
    ("missing", missing),
    ("wrong_type", wrong_type),
]:
    print(f"{name}: {decision(candidate)}")

assert changed_paths({"a/~b": 1}, {"a/~b": 2}) == {"/a~1~0b"}
assert changed_paths({"x": [True]}, {"x": [1]}) == {"/x"}
assert changed_paths({"x": None}, {}) == {"/x"}
assert changed_paths(BASELINE, []) == {""}
assert changed_paths(BASELINE, deepcopy(BASELINE)) == set()
print("edge checks: PASS")

The permission check and value check serve different purposes. Changing only an approved path still cannot replace its boolean with zero or delete the required field.

3. Run and validate the fixtures

From the directory containing the script, run:

python3 audit_json_edits.py

Expected standard output:

unchanged: ALLOW (unchanged)
approved: ALLOW /notifications/email
overreach: REJECT /retry_limit
missing: REJECT email must be boolean
wrong_type: REJECT email must be boolean
edge checks: PASS

This example was checked locally with Python 3.11.4: exit status 0, no standard error, and exactly the output shown. These results test the gate, not any model’s reliability.

The overreach fixture contains the requested email edit plus an unrelated retry-limit edit. The entire proposal is rejected; do not apply its “good” part automatically. The assertions exercise escaped keys, a boolean-to-number array edit, deletion of null, whole-document replacement, and an unchanged copy. Follow the same negative-fixture approach when testing AI-generated SQL.

4. Troubleshoot before connecting a writer

• A parent path is rejected: replacing the entire notifications object with a scalar changes /notifications. A leaf permission does not authorize that replacement.
• An array edit is rejected: allow the entire array only when that broader permission is intentional; this example does not authorize individual array indexes.
• A no-op is allowed: the gate checks scope, not task completion. Separately require the final email value to be False when the task demands disabling it.
• Input is too large or deeply nested: reject it before comparison. This recursive teaching example is not an unbounded-input security boundary.

In a real integration, validate the complete candidate schema, check authorization, and stop on parsing or comparison errors. Keep the checked candidate immutable and reject stale originals using a version check in the write transaction. Approval must apply to the same document you eventually save. A read-only file-renaming preview uses the same separation between proposal and execution.

Practical takeaway

A schema says what a document may contain; a field allowlist says what this particular operation may change. Check both, review the proposed difference, and keep the writer separate from the model.