Orchestrata Ultimate › Gates › lexicon-drift

Gate: lexicon-drift

At a glance

What it refuses

A committed src/orchestrata_ultimate/prolog/intent_lexicon.gen.pl that no longer equals what scripts/gen_intent_lexicon.py would generate from IntentVocabulary.default() — i.e. the DCG grammar consulting a lexicon the Python vocabulary has moved past.

Why it exists

From the generator's docstring:

"Why this exists (L-D2, reports/research-dsl-benefit.md §3.C / §5 D2): intent_grammar.pl claimed to 'mirror' intent_vocabulary.py since the day both were created in one commit, but the two lexicons drifted immediately — 16 keywords across 9/10 WorkType categories (plus 13 IntentTarget keywords) existed in the Python vocabulary but not the Prolog lexicon, silently degrading those words from the preferred DCG-grammar classification tier to the lower-confidence vocabulary fallback tier. The covering test passed only via a decoy keyword… Decision D2 (generation over a parity test) makes the Prolog lexicon a build artifact of the Python vocabulary so the two tiers can never re-diverge."

Why the drift matters beyond the parser (registry comment): "the parser matches against the GENERATED intent_lexicon.gen.pl, while lexicon-collision reads IntentVocabulary. If those two drift, the collision gate is checking a vocabulary the parser does not use, and passes while the real lexicon holds a colliding word." The comment also records that this "was a CI-ONLY gate until now — the same hole this file already documents for mypy."

How it runs

• Commands:

  python scripts/gen_intent_lexicon.py            # regenerate intent_lexicon.gen.pl
  python scripts/gen_intent_lexicon.py --check    # non-zero exit if the committed file is stale
  

• When it fires: the scripts/gates.py battery (registered as lexicon-drift, running the --check form). An earlier version of this page also listed CI; the current .gitlab-ci.yml has no job that calls gen_intent_lexicon.py. The registry comment says the gate "was a CI-ONLY gate until now"; whether a CI job existed earlier was not checked in this run.
• What it generates: "ONLY the lexicon FACT clauses (work_type_word/1, target_word/1)… The DCG grammar RULES stay hand-authored in intent_grammar.pl, which consults the generated file via :- include('intent_lexicon.gen.pl')."
• Determinism: "Deterministic and offline: no network, no DB connection. Imports intent_vocabulary.py directly (pure dataclasses…) rather than regexing it, so the generator can never itself drift from the real default-vocabulary construction logic."
• Inputs: fingerprint roots ("src", "scripts") — src carries both the vocabulary and the generated .pl; scripts is "this gate's own .py file, same reasoning as lexicon-collision."
• Caching: the verdict is cached in .cache/gates.json against _fingerprint(("src", "scripts")): a sha256 over the relative path and bytes of every file under those roots whose suffix is .py, .toml, .pl, .json, .md or .sh (_interesting() in scripts/gates.py, read through the AST toolkit on 2026-10-23). Both the vocabulary (.py) and the generated lexicon (.pl) are in that set. There is no per-gate suffix mask: this page previously quoted one, and git log --all -S GATE_MASKS -- scripts/gates.py finds no commit that ever contained a GATE_MASKS table.

Typical failures and how to fix them

• Exit code: --check exits non-zero when the committed file is stale; 0 when byte-identical to what generation would produce.
• What a failure looks like (representative): a staleness report naming the generated path and instructing regeneration.
• Un-blocking: uv run python scripts/gen_intent_lexicon.py and commit the regenerated file — never hand-edit it (its own generated header says so).
• A vocabulary edit committed without regenerating. The usual cause: a keyword added to intent_vocabulary.py alone. Regenerate in the same commit, then run lexicon-collision too, because a new work-type word can also collide with a module name.

Wiring

• Invokers: scripts/gates.py (battery).
• The generated artifact is registered project law under generated-artifacts — belongs in git, header names its generator.
• Tests: tests/scripts/test_gen_intent_lexicon.py.

Related

• Gates master index
• Development workflow
• RUL-030 — Generated artifacts belong in git
• lexicon-collision — the paired vocabulary-vs-codebase gate
• plugin-drift — the same "generation is enforced, not available" doctrine at plugin scale