Check AI Translations for Broken Placeholders With Python

An AI-translated notification can sound natural while breaking your application. If {order_id} becomes {id_pedido}, the application still supplies the original key. If a placeholder disappears, formatting may succeed but silently omit information.

This tutorial builds a small Python gate for translated templates. It catches structural changes before rendering, without an API key or external packages. It does not grade translation quality. For a generation pipeline, see our LCEL translation workflow.

1. Define the template contract

Keep an approved source template alongside each proposed translation, identified by the same message ID. Ask your translation tool to translate surrounding prose, preserve placeholder names and counts, and return only the template. Use synthetic names and order numbers during testing, not customer data.

Our deliberately strict contract permits simple ASCII named fields such as {name} and {order_id}. Their order may change; their spelling, capitalization, and occurrence counts may not. Repeating {name} twice in the source therefore requires two occurrences in the translation. That repetition rule is a project policy, not a universal linguistic requirement: languages sometimes need different sentence structures.

Named fields let translators rearrange information, as the GNU gettext Python guidance explains. Here, positional fields, attribute/index lookups, conversions, and nonempty formatting specifications are outside scope. So are ICU messages, Jinja templates, and %s strings; those need their own parsers.

2. Parse fields instead of guessing with regex

Save this as check_placeholders.py using UTF-8. The fixtures are handwritten simulations of AI output, not measurements from a live model.

Python’s Formatter parser exposes replacement fields separately from literal text. A Counter preserves repetitions that a set comparison would lose. The checker never evaluates generated code.

from collections import Counter
from string import Formatter


def placeholders(template):
    if not isinstance(template, str):
        raise ValueError("expected text")
    counts = Counter()
    for _, field, spec, conversion in Formatter().parse(template):
        if field is None:
            continue
        if not field.isascii() or not field.isidentifier():
            raise ValueError("only simple named fields allowed")
        if spec or conversion is not None:
            raise ValueError("format options are outside this contract")
        counts[field] += 1
    return counts


def check(source, translated):
    expected = placeholders(source)
    try:
        actual = placeholders(translated)
    except ValueError:
        return "REJECT"
    return "ACCEPT" if actual == expected else "REJECT"


source = "Hi {name}; order {order_id} is ready, {name}."
cases = {
    "good": ("Hola {name}; pedido {order_id} listo, {name}.", "ACCEPT"),
    "reordered": ("Pedido {order_id} listo, {name}. Gracias, {name}.", "ACCEPT"),
    "renamed": ("Hola {nombre}; pedido {order_id} listo, {nombre}.", "REJECT"),
    "missing": ("Hola {name}; tu pedido está listo, {name}.", "REJECT"),
    "duplicated": ("Hola {name}; pedido {order_id} {order_id} listo, {name}.", "REJECT"),
    "broken": ("Hola {name; pedido {order_id} listo, {name}.", "REJECT"),
    "attribute": ("Hola {name.first}; pedido {order_id} listo, {name}.", "REJECT"),
    "conversion": ("Hola {name!r}; pedido {order_id} listo, {name}.", "REJECT"),
    "wrong-meaning": ("Hola {name}; pedido {order_id} cancelado, {name}.", "ACCEPT"),
}
for label, (translated, expected) in cases.items():
    result = check(source, translated)
    if result != expected:
        raise AssertionError(label)
    print(f"{label}: {result}")

approved = cases["good"][0]
if check(source, approved) == "ACCEPT":
    print(approved.format(name="Ana", order_id="A-17"))

The approved source is parsed outside the exception handler intentionally: an invalid baseline stops the run instead of masquerading as a rejected translation. Invalid candidate text returns REJECT.

3. Run the fixtures and inspect the result

With Python 3.10 or later, run:

python3 check_placeholders.py

Expected output:

good: ACCEPT
reordered: ACCEPT
renamed: REJECT
missing: REJECT
duplicated: REJECT
broken: REJECT
attribute: REJECT
conversion: REJECT
wrong-meaning: ACCEPT
Hola Ana; pedido A-17 listo, Ana.

The reordered sentence passes because names and counts match. Renaming a key, omitting it, duplicating it, or breaking brace syntax fails. Attribute access and conversions also fail under this contract. The final line verifies that an accepted template renders with concrete fixture values.

Notice wrong-meaning: ACCEPT: the Spanish sentence says the order was canceled, not ready. All placeholders remain intact, so this structural gate correctly accepts it. ACCEPT means placeholder integrity only. A bilingual reviewer must still check meaning, tone, numbers, and context before release.

The demonstrated run exits successfully because every fixture produced its expected classification, including intentional rejections. A changed classification raises an error. When integrating this checker into a batch job, explicitly block that batch if any actual candidate returns REJECT; this fixture runner alone is not a deployment gate.

4. Troubleshoot before expanding the checker

• A valid translation is rejected: check whether your language requires fewer repetitions. Approve a locale-specific contract rather than silently weakening every check.
• Literal braces cause trouble: Python uses doubled braces, {{ and }}, for literal brace characters. The parser ignores those as fields; this check does not verify their displayed content.
• Formatting options are required: {amount:.2f} is rejected here. Extend the contract and fixtures deliberately, or use the checks provided by your localization system.
• The result passes but looks wrong: inspect the rendered notification. HTML, URLs, plural agreement, and factual meaning are not validated. Keep fixed URLs outside translated prose or supply them through named fields, following GNU gettext’s preparation guidance.

For other input formats, compare our regex fixture tutorial and strict AI JSON checks. A parser suited to one format is not a validator for another.

Practical takeaway

Freeze the source, validate placeholder structure, render with fixture values, then review meaning. Keep rejected candidates out of the application and add each real failure to your regression cases. A small deterministic check complements AI translation; it does not certify it.