Orchestrata Ultimate › Plans › intent-dcg-accuracy-design

Intent DCG accuracy — design contract (SPEC-003 US-001, increment 2)

Source: plans/intent-dcg-accuracy-design.md at 8a2cb42ad · last changed 2026-08-19 · first committed 2026-08-08 · 505 lines.

Prose pending.

Plan text

Intent DCG accuracy — design contract (SPEC-003 US-001, increment 2)

Status: DESIGN (2026-08-08). Successor to plans/intent-nl-classifier-design.md, which landed the cascade itself (REQ-003-US-001-040..049). That increment proved the shape: DCG tier first at confidence ~0.9, vocabulary tier as fallback at ~0.5–0.7. This increment is about the hit rate of the preferred tier — every sentence the grammar returns unknown for is a sentence that silently drops to the weaker tier.

Ratified constraint carried forward from plans/prolog-utilization-plan.md §4: the Python vocabulary tier stays as the fallback. Nothing here removes it.

1. Measured baseline — the grammar as committed

Harness: parse_intent_string/2 driven through janus against a 20-sentence corpus, run 2026-08-08 on the live intent_grammar.pl + intent_lexicon.gen.pl (42 work-type words across 10 WorkTypes, 23 target words across 11 IntentTargets).

14 / 20 correct (70%). The six failures, and their root causes:

Cause A — first-match-wins. work_type_scan//1 walks the token list and cuts on the first lexicon hit:

work_type_scan(WT) --> work_type_word(WT), !.
work_type_scan(WT) --> [_], work_type_scan(WT).

In "add tests for the parser" the generic verb add is at position 0 and tests at position 1, so feature wins on position alone. Position is the wrong tiebreak: English puts the verb first and the object is what names the work artifact.

Cause B — no morphology. The lexicon holds crash, the sentence says crashing; the lexicon holds document, the sentence says documentation. Exact-atom terminals cannot see through inflection. Enumerating every surface form by hand is not a fix — it is a permanent maintenance tax with no closure condition.

Cause C — vocabulary gaps. split is not in the refactor set; neither is simplify, decompose, consolidate. Real gap, real fix: add the words.

2. Design

2.1 Cause B — stem-normalized terminals (library(porter_stem))

SWI ships library(porter_stem) (porter_stem/2, tokenize_atom/2, atom_to_stem_list/2). Measured on the failing forms:

crashing      -> crash        documentation -> document
crashes       -> crash        documenting   -> document
tests         -> test         refactoring   -> refactor
splitting     -> split        optimizing    -> optim

Both sides get stemmed — the lexicon at load, the input at parse — so the stemmer's output need not be a real word (optimize → optim matches optimizing → optim). Self-consistency is the only requirement.

Collision check (mandatory, gated): two words in different categories must not share a stem, or the grammar becomes ambiguous. Measured across the current 42-word lexicon: zero cross-category collisions.

The check lives in Prolog, not in the generator, because that is where the stemmer is: lexicon_stem_collision/3 enumerates every colliding stem and lexicon_collision_report/1 marshals the set across the janus boundary as an atom. A test asserts the report is none — the assertion is the mechanism (RUL-014). Putting it in gen_intent_lexicon.py would have forced that generator, which is deliberately offline and dependency-free, to embed a second Porter implementation that would then have to agree with SWI's byte for byte.

library(porter_stem) is a FOREIGN extension, and foreign extensions do not resolve their libswipl symbols when SWI is embedded in Python — Python loads C extensions RTLD_LOCAL. The fix already exists and is already the only sanctioned janus entry point: core/janus_bootstrap.py preloads libswipl with RTLD_GLOBAL before importing janus_swi, for exactly this class of failure (it was built for library(json)). Every production janus consumer already goes through it — verified: janus_bootstrap.py holds the only direct import janus_swi in src/. Anything that imports janus_swi directly will see undefined symbol: PL_unify_atom_chars and must be fixed to use the bootstrap.

Shape: the generated file emits ground FACTS, and the grammar derives the stem index once at load via forall/2 + assertz/1. Ground facts only, derived deterministically from the file — the session-165 rule ("variable-bearing rules live in .pl; only GROUND facts are assertz'd") holds.

Porter is a suffix stripper, not a lemmatizer: broken → broken (not break), slower → slower. Irregular forms still need explicit lexicon entries. That is cause C's job, not cause B's.

2.2 Cause A — decisive words outrank generic verbs

Split the work-type lexicon into two classes:

• decisive — a word that names the work artifact or the defect: test, coverage, spec, docs, refactor, bug, crash, benchmark.
• generic verb — a word that only says something is being changed: add, create, build, implement, new, update, make, write.

Resolution order, replacing the positional cut:

1. If the sentence contains any decisive word, the leftmost decisive word wins.
2. Otherwise the leftmost generic verb supplies its default work type.
3. Otherwise unknown.

Rule 1 keeps the verb-first heuristic within the decisive class, which is what makes "test the fix" → test and "document the test suite" → documentation both come out right. Rule 2 preserves every current generic-verb classification: "build the new dashboard" still lands on feature.

Generic verbs are not removed from _DEFAULT_WORK_TYPE_KEYWORDS — the Python scoring tier keeps scoring them exactly as today. The generic set is an additional, overlapping declaration on IntentVocabulary, consumed only by the grammar generator. One source of truth (intent_vocabulary.py), two consumers.

2.3 Cause C — vocabulary expansion

Add keywords to the _DEFAULT_*_KEYWORDS seed dicts in intent_vocabulary.py and regenerate. intent_lexicon.gen.pl is never hand-edited — gen_intent_lexicon.py --check gates drift and is already wired into the battery.

Expansion covers both registries: work types (the ten WorkType members) and targets (the eleven IntentTarget members). Existing entries are not moved or re-categorised — pr→ARCHITECTURE and toggle→FILE stay where they are; retargeting them is a behaviour change with its own tests, not part of an additive vocabulary pass.

2.4 What this increment does NOT do

• Multi-target extraction. work_intent(WorkType, Target, Rest) returns one target and discards the rest. library(dcg/high_order)'s sequence//4 would return all of them, but the term shape is consumed by intent_classifier.py, so it is a contract change with its own requirement ids. Next increment.
• Semicontext lookahead. The decisive/generic split resolves the measured ambiguities without pushback. Reach for look_ahead//1 when a failure needs it.
• Tokenization inside the grammar. split_string/4 + a pad set is what ships; library(dcg/basics) would fold punctuation handling into the grammar. No measured failure currently traces to tokenization.

2.5 The normalization tier — an LLM in front of the grammar

Origin: user directive — "the NLQ component would have a haiku phase first that corrected the spelling and rephrased according to the last 2 turns in context, so the nlq is partnered with haiku so the language model helps the inference model", and then "we want performance, not some purity you invented — be pragmatic, don't feed prolog spelling mistakes and bad phrases."

The measurement that decides it

The grammar is a keyword parser and people do not type keywords. Against 76 real user turns mined from this project's own conversation history (the clean set — one slice of 16 was excluded for fabricated rewrites, see §4):

The headroom is spelling and phrasing, not vocabulary, and the shipped prompt now reaches that ceiling. (28/76 measured before the grounding check in clean_reply, 29/76 with it — one turn, so read it as "grounding costs nothing", not as a gain.) The 38.2% is itself optimistic because the agent that wrote those rewrites also wrote the labels, so treat 29/76 as a soft ceiling rather than a target to chase past.

The raw and normalizer rows read 19/76 and 26/92 in the first cut of this section. Both moved for the same reason and are only comparable at the same grammar version: stem-normalized matching (§2.1) and majority-vote resolution (REQ-003-US-001-065) both landed in between. The vote is not incidental to the prompt — see the bake-off below, where it inverts which prompt wins.

Examples of what the headroom looks like:

raw  : Why is my test suite runnig in 124v mins, did yiou forget the threads again?
       -> analysis   (the interrogative `why` wins)
clean: run the test suite
       -> test

raw  : check k3 work        -> unknown
clean: review the work      -> review

Design

orchestrator/intent_normalizer.py — IntentNormalizer.normalize(text, context) calls Haiku with the preceding turns and returns one clean sentence. It runs unconditionally, in front of every classification. NLIntentClassifier gains classify_async, IntentEvaluator gains evaluate_async, and IntentBroker.process_text — already async — awaits it with a context argument.

The prompt bake-off — five candidates, one corpus

Every candidate was scored by parsing its own output through the production grammar and comparing against the corpus labels. On the 20-turn stride sample the rivals share:

and on the full corpus, run as two contiguous halves whose raw baselines sum to exactly the pinned floor of 24/76: 29/76.

The terse row is the interesting one, because it refutes a prediction made in this document's own §2.2 reasoning. Short imperatives should win a leftmost-wins scan — fewer tokens, fewer decoys — and they came last. Majority vote changed the objective: compressing a sentence down to one decisive word leaves the vote nothing to count, so the prompt that keeps every mention of the work kind wins, and the winner is the longest output of the five. A grammar change silently invalidated a prompt rule, which is the argument for keeping both measured against one corpus rather than tuning either alone.

Five rules were tried and dropped, each on a number rather than an argument:

1. "Repair tier only" — gate normalization on grammar failure. Dropped: it bought nothing measurable, added a branch, and still fed Prolog raw typing whenever the grammar returned something wrong rather than nothing.
2. Withholding what the output is for — dropped: it produced "the rtm shows ghost rows." from "why is the rtm showing ghost rows", deleting the interrogative that carried the classification.
3. "Keep the grammatical mood — a question stays a question" — added after observing (2), generalised from ONE sentence in the held-out corpus, and wrong. On 92 real turns it capped the gain at 26/92 while more decisive rewrites of the same turns reached 39/92. "Why is my test suite runnig in 124v mins?" is a request to make it faster, not a question about slowness. The prompt now states the work a complaint asks for, and the test that pinned the old rule pins its reversal instead, with the number that overturned it.
4. "Do NOT name a category of work." A blindfold, invented so the model could not quietly become the classifier. It cost the entire gain — 8/20, exactly the raw-text baseline, versus 11/20 without it. The concern it was reaching for is real but narrower: the prompt must not hand the model a TYPE IDENTIFIER, and that is what the test asserts now. WorkType members are named for the English words themselves (FIX, TEST, REVIEW, REFACTOR), so the original rule forbade and required the same tokens.
5. "Do NOT add facts the user did not state." Measured on top of the winning prompt: 10/20 against 11/20. One point on twenty turns is weak evidence, but a rule earns its place or it goes. Containment is structural instead — ClassifiedIntent keeps .normalized_text beside the raw turn, clean_reply rejects an over-long reply, and an invented fact can only move the verdict if it happens to be a word in the bounded lexicon.

Five properties, all because this sits on the hot path:

1. Never blocks. Any failure — no provider, rate limit, timeout, empty reply — returns the original text. An outage degrades accuracy to the 31.6% raw baseline; it never fails a classification.
2. Cached by (text, context), so a replayed turn cannot produce a different verdict and a retry costs nothing.
3. Transparent. The normalized sentence is stored on ClassifiedIntent .normalized_text, so a verdict is reconstructable from what the grammar actually read. The raw turn becomes the intent title.
4. Bounded. One sentence in, one sentence out; a reply over the character ceiling is rejected in favour of the original, because an elaborating model is a worse parser input than a messy user.
5. Grounded. The rewrite must share at least one token of three or more characters with what the user actually wrote — the turn and its context turns, since resolving "it" against a preceding turn legitimately imports every content word from there and none from the turn itself. Added after a live failure, not in anticipation: clean_reply used to accept any non-empty, short-enough string, which means the tier trusted the model to have rewritten the sentence rather than checking. Under the suite's cassette-synth seam the model returns canned text unrelated to the input, and "fix the crash in /tmp/x_widget.py" — unambiguous FIX work on the raw string — came back classifying as feature at 0.30 confidence, under the dispatch threshold. The synth reply was a test artifact; accepting it was not. A refusal, an answer to the wrong question, or a truncated stream fails identically in production and silently, because the raw turn still reads fine in the audit trail.

The prompt IS given plain work-kind words and is NOT given the WorkType/IntentTarget type names — a distinction that took a measurement to find. The original rule was "do not name a category of work at all", and it cost the entire gain (8/20, the raw baseline, against 11/20 without it). The concern behind it survives in the narrower form: a model handed the TYPE NAMES writes them into the sentence, which makes the Prolog verdict a readback of the model's guess instead of a parse of the user's words. Prolog keeps the deterministic, explainable decision the governance layer reasons about; the LLM only cleans the input.

2.7 Confidence carries one bit — the evidence tier (REQ-003-US-001-068/069)

The finding

ClassifiedIntent.confidence was assumed to be a graded score. It is not. Measured over all 185 turns in the three corpora, with a Prolog bridge present, it takes exactly two values, 0.9 and 0.3, and nothing between:

The 0.5–0.7 band in _confidence_from_scores is not merely unobserved, it is unreachable while a bridge is present, and that follows from the cascade rather than from the corpus:

1. Both tiers read the same work-type lexicon — intent_lexicon.gen.pl is generated from IntentVocabulary (gen_intent_lexicon.py), so a token is a decisive hit for the grammar exactly when it scores for the keyword tier.
2. work_type_of/3 succeeds on any decisive hit, and target_of/4 ends in a catch-all clause that cannot fail, so classify_tokens/2 returns a term whenever one decisive hit exists.
3. So the keyword tier is reached only when there were no decisive hits at all — and then scores is empty and the function returns 0.3.

The band is kept rather than deleted because it is live and correct in the bridge-less configuration (prolog_bridge=None), which the sync path uses.

The consequence for dispatch: every IntentEvaluator threshold in (0.30, 0.85] is the same threshold. Sweeping the gate over the raw corpus gives 100% coverage at ≤0.30, 64.5% coverage / 49.0% precision across the entire 0.31–0.85 range, and 0% at ≥0.86. The knob has one position.

Confidence does carry signal — 49% correct at 0.9 against 30% at 0.3 on the raw turns — but exactly one bit of it.

What was built (REQ-003-US-001-068)

parse_intent_evidence/4 returns the vote tally the grammar already computed and previously discarded, and ClassifiedIntent carries it: tier, decisive_hits, winning_hits. Proved term-for-term identical to parse_intent_string/2 over all 185 turns before the production query switched. Evidence is reported, never scored — no verdict moved.

The distribution shows why it is worth having:

23 turns parse from a generic verb with no decisive word anywhere and score 0.9 — indistinguishable, in the confidence value, from a unanimous three-word parse.

What was NOT built, and why (REQ-003-US-001-069)

Grading confidence from that evidence is the obvious next step and it was not taken, because the only candidate rule the corpus offers does not survive validation:

The dev set says a single hit is the worst case; the held-out set says it is the best one. This is the same signal that appeared as p=0.0037 (Fisher exact) under a different framing and reversed there too — one fixed 76-turn set queried repeatedly is the garden-of-forking-paths setup, and this is what it produces.

Grading must therefore be fitted on production telemetry, not on this corpus. The evidence fields exist so that telemetry has something to fit.

2.8 Why the lexicon stops growing here

run is the most common word the grammar does not know — 17 of 76 raw turns. Adding it and three siblings (push, promote, answer) as generic verbs raises raw accuracy 24/76 → 30/76. That change is rejected, and the reason generalizes past these four words.

The +6 is bought by moving 13 turns off unknown, of which 6 land correctly: marginal precision 46%, against the 49% the system already gets on what it dispatches. Coverage 64.5% → 81.6%, precision down. Accuracy rises; the risk-coverage curve gets worse. The held-out corpus cannot referee it — one turn contains run, none contain the others.

What the newly covered turns actually are:

no run allowed I will cancel anything you run, i command you to stop → TEST, confidence 0.9, decisive_hits == 0

A stop order dispatching a test run, at maximum confidence on no decisive evidence. Also newly misclassified: Check all running background work (status ping), numbers every 5 mins while the swarms run (standing instruction), keep working from longest to fastest do not stop.

unknown is the reject option working. Until the taxonomy can express "this is not a work request", every mechanism that raises coverage converts correct rejections into confident errors — so vocabulary growth is blocked on the speech-act dimension, not on more words.

Pinned by tests/orchestrator/test_intent_classifier.py:: test_a_turn_that_is_not_a_work_request_is_rejected_not_guessed.

3. Requirements (add to specs/003-intent/checklists/requirements.md)

• REQ-003-US-001-050: Stem-normalized lexicon matching — intent_grammar.pl stems both lexicon words (at load, via porter_stem/2) and input tokens (at parse), so inflected surface forms (crashing, tests, documentation) reach the grammar tier instead of falling through to the vocabulary tier.
• REQ-003-US-001-051: Stem-collision gate — lexicon_stem_collision/3 enumerates every stem claimed by two different categories (within the work-type lexicon, within the target lexicon, and generic verbs shadowed by a decisive word), and a test asserts the set is empty.
• REQ-003-US-001-052: Decisive-over-generic work-type resolution — a work-type word that names a work artifact outranks a generic change verb regardless of position; ties inside each class are broken leftmost-first.
• REQ-003-US-001-053: IntentVocabulary.generic_verbs declares the generic-verb set (word → default WorkType); the words remain in work_type_keywords so the vocabulary-scoring tier is unchanged.
• REQ-003-US-001-054: Expanded default vocabulary — every WorkType and every IntentTarget carries the keywords the measured corpus exercises.
• REQ-003-US-001-055: A committed accuracy corpus (tests/orchestrator/data/ intent_corpus.json) of labelled sentences, with a test that fails if grammar-tier accuracy drops below the ratified floor — the regression floor is the mechanism that stops accuracy silently eroding.

Normalization tier (§2.5)

• REQ-003-US-001-060: IntentNormalizer.normalize(text, context) rewrites a raw user turn into one clean sentence via a small fast model, using the preceding turns to resolve pronouns and vague references, and runs unconditionally in front of the grammar tier.
• REQ-003-US-001-061: Normalization never blocks classification — any provider failure, timeout, empty reply, or over-length reply returns the original text, and the passthrough is counted in stats().
• REQ-003-US-001-062: The sentence the grammar actually parsed is recorded on ClassifiedIntent.normalized_text, and the raw turn becomes the intent title, so a verdict is reconstructable from the parser's input.
• REQ-003-US-001-063: NLIntentClassifier.classify_async, IntentEvaluator.evaluate_async and IntentBroker.process_text(context=…) form the production cascade; the synchronous classify/evaluate remain for callers with no event loop and skip normalization.
• REQ-003-US-001-064: Identical (text, context) returns the identical normalization from cache, so a replayed turn cannot produce a different verdict.
• REQ-003-US-001-066: The normalizer prompt is chosen by measurement against intent_corpus_raw_turns.json, never by argument. Every rule removed from it is recorded in source with the number that removed it, and the shipped prompt must beat the raw-text baseline on that corpus — a prompt that merely ties it (as the first shipped one did, 8/20) is buying nothing for its latency.

4. Measured result

Three corpora, all scored through the same PrologBridge the production classifier uses. They are listed in increasing order of honesty.

Read the last row, not the first. intent_corpus.json was written in the same sitting as the vocabulary it measures, so 100% proves self-consistency and nothing more. intent_corpus_raw_turns.json is 76 verbatim turns a person actually typed into this project — typos, pronouns, ALL CAPS and all — mined from 14,978 captured turns. 28.9% is what the grammar is worth on real input.

What the corpora caught, that review would not have

• Four keywords added in this increment had to be removed again after the held-out set caught them naming subsystems rather than kinds of work: memory, trace, integration, version (see the DELIBERATELY ABSENT block in intent_vocabulary.py). "wire the memory engine into the gateway" was classifying as performance work.
• dispatch was mined, accepted by consolidation on the evidence "message dispatch", applied — and then rejected by the main corpus, because dispatcher stems to it and "refactor the dispatcher into smaller functions" started resolving to api. kernel/dispatcher.py is code.
• One miner fabricated its output. 15 of the 92 mined rephrasings were templated 2–4 word imperatives ("Delegate task", "Optimize performance"), several duplicated verbatim, none showing the required context resolution. They inflated every number computed over them — the rewrite ceiling in §2.5 was first measured at 42.4% and is 38.2% once that slice is excluded. The exclusion is asserted by a test, not left to a comment.

Where the remaining gap is

The vocabulary swarm (16 miners over the full conversation history, consolidated by a reviewer that rejected 44 of 62 proposed work-type words on the subsystem test) moved raw-turn accuracy 25.0% → 28.9%. The normalization tier in §2.5 moves it another ~3 points independently measured. Neither is close to the 38.2% a clean rewrite of the same turns reaches, so the gap is still phrasing, and closing it is the next increment's work.

Gates

• gen_intent_lexicon.py --check exits 0.
• Collision report: none across 107 decisive + 17 generic + 92 target stems.
• tests/orchestrator/test_intent_grammar_accuracy.py 23 passed, test_intent_classifier.py 28 passed, test_intent_vocabulary.py 13 passed, tests/scripts/test_gen_intent_lexicon.py 10 passed. Then the RUL-008 battery.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks