Check AI Quote Citations Against Local Sources With Python

An AI assistant cites your returns policy and puts a sentence in quotation marks. Before forwarding the answer, check whether that sentence actually appears in the cited source. A plausible document ID is not enough: the model may have altered the wording or attached the wrong reference.

Build an offline citation gate with Python 3.9 or newer and no extra packages. It checks source IDs, normalized quote presence, and ambiguous matches. All examples use synthetic text; they test the checker, not a model’s accuracy.

1. Freeze the sources and request quote records

Save the reviewed text your assistant received, with versioned IDs such as returns-v1. Keep the original file, retrieval date, and a SHA-256 hash of its bytes in your run record. A changing web page is not a reproducible reference. Restrict this registry to sources the current user is authorized to read.

Ask the assistant for one record per citation: citation_id, source_id, and quote. Require a copied excerpt, not a paraphrase. Keep the answer sentence alongside the record for human review. If you receive JSON, apply strict parsing and field validation before using it; the tuples below are already structured fixtures, not a JSON loader.

This check follows retrieval, rather than replacing it. The RAG retrieval evaluation tutorial helps determine whether the required evidence reached the assistant at all.

2. Locate quotes without erasing meaningful differences

Save this as check_citations.py in a fresh directory:

import re
import unicodedata

sources = {
    "returns-v1": "Returns are accepted within 30 days.\nA receipt is required.",
    "help-v1": "Contact support. Read the guide. Contact support.",
    "cafe-v1": "Café orders close at noon.",
}
citations = [
    ("c1", "returns-v1", "Returns are accepted\nwithin 30 days."),
    ("c2", "returns-v1", "A receipt is not required."),
    ("c3", "missing-v1", "A receipt is required."),
    ("c4", "help-v1", "Contact support."),
    ("c5", "returns-v1", "   "),
]


def normalize(text):
    return " ".join(unicodedata.normalize("NFC", text).split())


def check(source_id, quote):
    if not isinstance(source_id, str) or not isinstance(quote, str):
        return "invalid fields"
    if source_id not in sources:
        return "unknown source"
    needle = normalize(quote)
    if not needle:
        return "empty quote"
    text = normalize(sources[source_id])
    pattern = re.escape(needle)
    if re.match(r"\w", needle[0]):
        pattern = r"(?<!\w)" + pattern
    if re.match(r"\w", needle[-1]):
        pattern += r"(?!\w)"
    hits = sum(1 for _ in re.finditer("(?=" + pattern + ")", text))
    if hits == 0:
        return "quote not found"
    return "located" if hits == 1 else "ambiguous quote"


def main():
    reasons = [check(source_id, quote) for _, source_id, quote in citations]
    for (citation_id, _, _), reason in zip(citations, reasons):
        print(f"{citation_id}: {reason}")
    print(f"located={reasons.count('located')}/{len(reasons)}")
    return int(not reasons or any(reason != "located" for reason in reasons))


if __name__ == "__main__":
    raise SystemExit(main())

The Python Unicode documentation explains NFC normalization, which lets canonically equivalent accent spellings match. Using split() without a separator collapses runs of whitespace. This is an explicit policy for ordinary prose, not byte-for-byte matching; avoid it where layout carries meaning, such as source code or tables.

Case, punctuation, numbers, and negation remain unchanged. The conditional word-edge checks reject 7 days inside 17 days and paid inside unpaid. They are conservative character checks, not universal linguistic segmentation. Lookahead counts overlapping occurrences too. A unique occurrence receives located, deliberately not true or supported.

3. Test rejection and correction

Run:

python3 check_citations.py

Expected stdout:

c1: located
c2: quote not found
c3: unknown source
c4: ambiguous quote
c5: empty quote
located=1/5

Exit status is 1, with empty stderr: four records need review. That is intentional, not a crash. An empty citation list also fails, so silence cannot masquerade as success.

Next, run the edge tests and replace the in-memory records with correct quotes:

python3 - <<'PYTEST'
import check_citations as demo

assert demo.check("cafe-v1", "Cafe\u0301 orders close at noon.") == "located"
assert demo.check("returns-v1", "within\t30 days.") == "located"
assert demo.check("returns-v1", "0 days") == "quote not found"
assert demo.check("returns-v1", "a receipt is required.") == "quote not found"
assert demo.check("returns-v1", "A receipt is required!") == "quote not found"
assert demo.check("returns-v1", None) == "invalid fields"
assert demo.check([], "anything") == "invalid fields"
assert demo.check("returns-v1", "\t\n") == "empty quote"
assert demo.check("help-v1", "Contact support.") == "ambiguous quote"
demo.sources["edge-v1"] = "17 days unpaid"
assert demo.check("edge-v1", "7 days") == "quote not found"
assert demo.check("edge-v1", "paid") == "quote not found"
demo.sources["overlap-v1"] = "!!!"
assert demo.check("overlap-v1", "!!") == "ambiguous quote"
print("edge checks passed")
demo.citations = [
    ("c1", "returns-v1", "Returns are accepted within 30 days."),
    ("c2", "cafe-v1", "Café orders close at noon."),
]
raise SystemExit(demo.main())
PYTEST

Expected stdout:

edge checks passed
c1: located
c2: located
located=2/2

This command exits 0, with empty stderr. Both commands were checked locally with these fixtures. The assertions cover accents, whitespace, token edges, overlapping matches, case, punctuation, and invalid field types. The successful run does not rewrite the saved script or original sources.

4. Troubleshoot the right boundary

• Unknown source: use the exact versioned registry ID; never silently swap in a newer document.
• Quote not found: compare the raw excerpt. PDF extraction, OCR errors, or smart quotes may explain a mismatch. Fix extraction or review manually instead of stripping all punctuation.
• Ambiguous quote: request a longer excerpt. W3C’s Text Quote Selector illustrates using surrounding prefix and suffix text to distinguish repeated passages. This demo rejects ambiguity rather than implementing that model.
• Every record is located, but the answer is wrong: inspect the full paragraph and the claim it supposedly supports. A substring can omit an exception or nearby negation; presence is not entailment, currentness, or truth.

Do not automatically fetch model-supplied URLs in this gate. Resolve approved IDs locally, retain raw text for review, and do not treat normalized character positions as original-file offsets. Add separate checks for uncited claims and required source coverage. For copied figures, the summary-number checker verifies units and reporting periods that quote presence alone cannot assess.

Practical takeaway

Require traceable quote records, freeze the evidence, and reject unknown, missing, or ambiguous excerpts before forwarding an answer. A passing result narrows the review task; it does not remove the need to check what the quotation means in context.