Orchestrata Ultimate › Plans › intent-nl-classifier-design

Extensible NL intent parser + evaluator — design contract (SPEC-003 US-001 extension)

Source: plans/intent-nl-classifier-design.md at 8a2cb42ad · last changed 2026-08-19 · first committed 2026-07-11 · 86 lines.

Prose pending.

Plan text

Extensible NL intent parser + evaluator — design contract (SPEC-003 US-001 extension)

Origin: user directive — "create an intent parser and evaluator step and make it so we can add vocabulary, keywords and a mapper to categories and workflows." This is the honest NL→structured-Intent bridge that lets the real IntentBroker be fed structured intents instead of the gateway's coarse action="create" heuristic (which has no resource_path and doesn't use the IntentAction/IntentTarget taxonomy).

Reuse (do NOT reinvent)

• orchestrator/nlp.py: Tokenizer, NamedEntityRecognizer (already extracts resource paths, git refs, @agent, commands, URLs, measurements), ContextAwareParser. The classifier consumes these — NER gives the real resource_path (no fabrication).
• Target taxonomies (map TO these — do not invent categories):
  • WorkType (StrEnum, pipeline_router.py) = the workflows: feature/refactor/test/documentation/analysis/fix/review/spec/dependency/performance. Each already maps to a stage/agent chain via WORK_TYPE_CAPABILITIES + PipelineRouter.
  • IntentAction (read/write/delete/execute/query/create/modify/analyze/review/plan), IntentTarget (file/code/database/api/system/directory/command/git/network/impact/ architecture), IntentPriority (CRITICAL..BACKGROUND) — the broker Intent model.

Borrowed architecture — dr-hannah NLQ cascade (~/workspace/dr-hannah/src/dr_hannah/nlq.py + prolog/nlq_grammar.pl)

Three-tier parse cascade with confidence reflecting the backend (borrow this shape):

1. Prolog DCG grammar (PREFERRED, ~0.9–0.95): a .pl DCG (parse_intent_string(String, Term)) produces a compound term like work_intent(fix, code, Path); composable, testable grammar rules
   • a lexicon (action_word/target_word/priority_word). Extensible by ADDING grammar rules. Mirrors dr-hannah's query(Q) --> task_query(Q) | ... + parse_query_string/2 + phrase/2.
2. Vocabulary/keyword scoring (FALLBACK, ~0.5–0.7): the data-driven IntentVocabulary below — runtime-addable keywords. Used when no Prolog bridge is wired OR the grammar returns unknown.
3. Both tiers emit the same ClassifiedIntent; NER (nlp.py) supplies resource_path in both. The classifier mirrors dr-hannah's NaturalLanguageQueryParser(prolog_engine=None) — try Prolog, gracefully fall back — and a _prolog_term_to_classified() (mirrors _prolog_term_to_query).

Components (new)

0. orchestrator/prolog/intent_grammar.pl — DCG intent grammar (borrowed shape)

parse_intent_string(+String, -Term): tokenize (split_string) + phrase(intent(Term), Tokens), Term = unknown on no match. intent(work_intent(WorkType, Target, Rest)) composed from lexicon rules — action/verb words → WorkType (fix/feature/refactor/test/documentation/review/analysis/ spec/dependency/performance), target words → IntentTarget, priority words → priority. Keep it a consulted .pl (variable-bearing rules live in .pl; only GROUND facts ever assertz'd — project rule).

1. orchestrator/intent_vocabulary.py — data-driven, extensible

class IntentVocabulary holds keyword→category registries, all runtime-mutable + config-loadable:

• work_type_keywords: dict[WorkType, set[str]], action_keywords: dict[IntentAction, set[str]], target_keywords: dict[IntentTarget, set[str]], priority_keywords: dict[IntentPriority, set[str]].
• Methods: add_keywords(category, words), remove_keywords, merge(other), from_dict(d) / to_dict() (YAML/JSON-round-trippable), classmethod default() (seeded defaults, e.g. fix←{fix,bug,broken,error,crash,regression}, feature←{add,create,implement, build,feature,new}, refactor←{refactor,restructure,clean up,rename,extract}, documentation← {document,docs,readme,docstring}, test←{test,coverage,unit,e2e}, review←{review,audit,inspect}, analysis←{analyze,investigate,why,explain}, performance←{optimize,slow,perf,latency}, dependency←{dependency,upgrade,bump,pin}, spec←{spec,requirement}; priority urgent/critical/asap→CRITICAL, etc).
• Extensibility is the point: callers add domain vocab without code edits.

2. orchestrator/intent_classifier.py

• @dataclass ClassifiedIntent: work_type, action, target, priority, title, resource_path, assigned_volk, entities, confidence: float, scores: dict, matched_keywords: dict.
• class NLIntentClassifier(vocabulary=None, prolog_bridge=None, tokenizer=None, ner=None):
  • classify(text) -> ClassifiedIntent: cascade — if prolog_bridge is wired, try the DCG grammar (parse_intent_string) and map the compound term via _prolog_term_to_classified() (confidence ~0.9); on unknown/no-bridge fall back to vocabulary keyword scoring (tokenize → score each category by keyword hits → argmax, confidence = margin/total, ~0.5–0.7). NER supplies resource_path/@volk/entities in both tiers. Empty/ambiguous → sensible default + low confidence.
  • to_intent(classified, requester) -> Intent: build the broker Intent.
• class WorkflowMapper(overrides=None): workflow_for(classified) -> WorkType (default: the classified work_type; overridable) + stages_for(work_type) delegating to PipelineRouter. Extensible via register(work_type_or_category, workflow).
• class IntentEvaluator(vocabulary=None, classifier=None, mapper=None, min_confidence=0.34): the evaluator step — evaluate(text, requester) -> EvaluationResult{intent, work_type, stages, confidence, reasons, needs_disambiguation}. Below threshold ⇒ needs_disambiguation=True.

Spec + traces

Add REQ-003-US-001-040..049 to specs/003-intent/checklists/requirements.md (extensible vocabulary, keyword→category mapping, category→workflow mapping, evaluator confidence/disambiguation, NER-sourced resource_path). @traces(spec="003", story="US-001", reqs=[...]) on every new class.

Tests (real, no doubles)

tests/orchestrator/test_intent_vocabulary.py + test_intent_classifier.py: default vocab classifies canonical phrases to the right WorkType/action/target/priority; NER extracts a real path from "fix the bug in src/foo.py"; add-a-keyword changes classification (extensibility); from_dict/to_dict round-trip; WorkflowMapper → real PipelineRouter stages; evaluator confidence

• disambiguation on ambiguous text. Function-scope fixtures; no mocks/skips/monkeypatch.

Out of scope for THIS build (separate follow-up)

Wiring the evaluator into the gateway/kernel ingress + the broker. Build the parser/evaluator as a self-contained, tested unit first; wire second.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks