scripts.gen_intent_lexicon

Reference › scripts › gen_intent_lexicon

scripts/gen_intent_lexicon.py generates prolog/intent_lexicon.gen.pl, the lexicon fact clauses the Prolog intent layer consults. The facts are rendered from the Python intent vocabulary in orchestrata_ultimate.orchestrator.intent_vocabulary, orchestrata_ultimate.orchestrator.intent_types and orchestrata_ultimate.orchestrator.pipeline_router, so the Python tables remain the single source. A human runs it after changing the vocabulary; the --check mode is the form a gate uses.

render builds the file content. With no argument main writes the file through dump_text_atomic from orchestrata_ultimate.core.atomic_json, because the Prolog engine consults the file and a truncating rewrite could hand a running query half a file. With --check it writes nothing: it exits 1 with an "intent lexicon drift" message when the file is missing or differs from the rendered content, and exits 0 when it is current.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

API

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/scripts/gen_intent_lexicon.py

#!/usr/bin/env python3
"""Generate ``prolog/intent_lexicon.gen.pl`` — the lexicon FACT clauses
consumed by ``intent_grammar.pl``'s DCG, from the single source of truth:
``orchestrator/intent_vocabulary.py``'s ``IntentVocabulary.default()``.

Why this exists (L-D2, ``reports/research-dsl-benefit.md`` §3.C / §5 D2):
``intent_grammar.pl`` claimed to "mirror" ``intent_vocabulary.py`` since the
day both were created in one commit, but the two lexicons drifted
immediately — 16 keywords across 9/10 ``WorkType`` categories (plus 13
``IntentTarget`` keywords) existed in the Python vocabulary but not the
Prolog lexicon, silently degrading those words from the preferred DCG-grammar
classification tier to the lower-confidence vocabulary fallback tier. The
covering test passed only via a decoy keyword (see
``tests/orchestrator/test_intent_classifier.py``). Decision D2 (generation
over a parity test) makes the Prolog lexicon a build artifact of the Python
vocabulary so the two tiers can never re-diverge.

Scope: this generates ONLY the lexicon FACT clauses (``work_type_word/1``,
``target_word/1``) into ``intent_lexicon.gen.pl``. The DCG grammar RULES
(tokenizer, scanners, ``parse_intent_string/2``) stay hand-authored in
``intent_grammar.pl``, which consults the generated file via
``:- include('intent_lexicon.gen.pl').``.

Usage::

    python scripts/gen_intent_lexicon.py            # regenerate intent_lexicon.gen.pl
    python scripts/gen_intent_lexicon.py --check     # non-zero exit if the committed file is stale

Deterministic and offline: no network, no DB connection. Imports
``intent_vocabulary.py`` directly (pure dataclasses, no heavy deps) rather
than regexing it, so the generator can never itself drift from the real
default-vocabulary construction logic.
"""

from __future__ import annotations

import sys
from pathlib import Path

REPO = Path(__file__).resolve().parent.parent
SRC = REPO / "src"
VOCAB_SOURCE = "orchestrata_ultimate/orchestrator/intent_vocabulary.py"
OUT = REPO / "src" / "orchestrata_ultimate" / "prolog" / "intent_lexicon.gen.pl"

if str(SRC) not in sys.path:
    sys.path.insert(0, str(SRC))


def render() -> str:
    """Render the generated lexicon-facts .pl content."""
    # Imported here (not at module scope) so `--check`/generation stays a
    # pure function of the real vocabulary construction, without forcing
    # every importer of this script to pay the package import cost.
    from orchestrata_ultimate.orchestrator.intent_types import IntentTarget
    from orchestrata_ultimate.orchestrator.intent_vocabulary import IntentVocabulary
    from orchestrata_ultimate.orchestrator.pipeline_router import WorkType

    vocab = IntentVocabulary.default()
    generic = vocab.generic_verbs

    lines = [
        "% ═══════════════════════════════════════════════════════════════════",
        "% intent_lexicon.gen.pl — GENERATED lexicon facts for intent_grammar.pl",
        "%",
        f"% Source of truth: {VOCAB_SOURCE} (IntentVocabulary.default()).",
        "% Generator: scripts/gen_intent_lexicon.py — regenerate with:",
        "%     uv run python scripts/gen_intent_lexicon.py",
        "% Drift check (CI): uv run python scripts/gen_intent_lexicon.py --check",
        "%",
        "% DO NOT HAND-EDIT THIS FILE — add/remove keywords in",
        "% intent_vocabulary.py's default seed dicts and regenerate. See L-D2,",
        "% reports/research-dsl-benefit.md §3.C / §5 D2.",
        "%",
        "% GROUND FACTS ONLY. intent_grammar.pl stems these at load time into",
        "% its match index (REQ-003-US-001-050), so an inflected surface form",
        "% ('crashing', 'tests', 'documentation') still reaches the grammar",
        "% tier instead of dropping to the vocabulary-scoring fallback.",
        "% ═══════════════════════════════════════════════════════════════════",
        "",
        "% ── work_type_word_raw/2 — DECISIVE work-type words: they name the",
        "%    work artifact or the defect. Mirrors",
        "%    IntentVocabulary.default().work_type_keywords MINUS the generic",
        "%    verbs below, which are demoted so 'add tests' is test work, not",
        "%    feature work (REQ-003-US-001-052). ────────────────────────────────",
    ]

    for work_type in WorkType:
        keywords = sorted(vocab.work_type_keywords.get(work_type, set()))
        for keyword in keywords:
            if keyword in generic:
                continue
            lines.append(f"work_type_word_raw({work_type.value}, {keyword}).")

    lines.append("")
    lines.append(
        "% ── generic_verb_raw/2 — a generic change verb only says SOMETHING is"
    )
    lines.append(
        "%    being changed; it supplies a work type only when no decisive word"
    )
    lines.append(
        "%    appears. Mirrors IntentVocabulary.default().generic_verbs. ──────"
    )

    for keyword, work_type in sorted(generic.items()):
        lines.append(f"generic_verb_raw({work_type.value}, {keyword}).")

    lines.append("")
    lines.append(
        "% ── target_word_raw/2 — mirrors IntentVocabulary.default()"
        ".target_keywords ─"
    )

    for target in IntentTarget:
        keywords = sorted(vocab.target_keywords.get(target, set()))
        for keyword in keywords:
            lines.append(f"target_word_raw({target.value}, {keyword}).")

    lines.append("")
    return "\n".join(lines)


def main(argv: list[str]) -> int:
    content = render()
    if "--check" in argv:
        if not OUT.is_file():
            print(
                f"intent lexicon drift: {OUT} does not exist — "
                "run scripts/gen_intent_lexicon.py",
                file=sys.stderr,
            )
            return 1
        current = OUT.read_text(encoding="utf-8")
        if current != content:
            print(
                f"intent lexicon drift: {OUT} is stale — "
                "run scripts/gen_intent_lexicon.py",
                file=sys.stderr,
            )
            return 1
        print("intent_lexicon.gen.pl up to date.")
        return 0
    # Atomic: the Prolog engine consults this .pl file, so a truncating
    # rewrite can hand a running query a half-file. See dump_text_atomic.
    from orchestrata_ultimate.core.atomic_json import dump_text_atomic

    OUT.parent.mkdir(parents=True, exist_ok=True)
    dump_text_atomic(OUT, content)
    print(f"wrote {OUT} ({len(content)} bytes)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:]))