Orchestrata Ultimate › Plans › nlq-research-2026-08

Natural-language query → work intent: research findings, 2026-08-08

Source: plans/nlq-research-2026-08.md at 8a2cb42ad · last changed 2026-08-19 · first committed 2026-08-08 · 596 lines.

Prose pending.

Plan text

Natural-language query → work intent: research findings, 2026-08-08

Method. Six parallel literature lanes (semantic parsing, grammar-based NLU, conversational query rewriting, intent classification/calibration, evaluation methodology, production architecture) over ~70 sources, 2024–2026 weighted. Every claim below is either (a) cited to a source with its measured number, or (b) measured here, on this repo's own corpora, and marked as such. Claims the literature makes without a number attached are labelled unmeasured.

This document supersedes nothing. It is the evidence base for the next increment of plans/intent-dcg-accuracy-design.md.

0. The headline, before anything else

The 38.2% "ceiling" on real user turns is a property of the taxonomy, not of the parser. Measured here: 24 of 76 turns (32%) are not requests for engineering work at all. They are status queries, process control, display requests, and policy statements — and every one is force-labelled with one of the ten WorkType values, because the schema has no way to say "this turn is not a work request."

No amount of lexicon expansion, stemming work, or prompt tuning can classify "i see no threads" or "can i restart now to rearm?" into {feature, fix, test, …} correctly, because there is no correct answer in the codomain. A human rewriter hits the same wall — which is exactly why the human-rewrite ceiling came out at 29/76 rather than somewhere near 100%.

This reframes every other finding in this document. The highest-value change available is not to the parser. It is to add an intent-kind dimension above WorkType — minimally a NOT_A_WORK_REQUEST class, better a small speech-act enum (work_request | status_query | process_control | display_request | policy_statement) resolved before work-type classification is attempted.

The literature independently supports the shape of this fix: CLINC150 and every serious intent benchmark ship an explicit out-of-scope class scored separately from in-scope accuracy, never a single blended number (CLINC150/OOS); DROID and DETER both report known-intent F1 and unknown-intent F1 as two figures (DROID, DETER).

1. The statistical reckoning

1.1 What n=76 can and cannot support — measured here

raw-text accuracy        24/76 = 31.6%   95% Wilson CI [22.2%, 42.7%]   width 20.5 pts
normalized accuracy      31/76 = 40.8%   (a second run of the same prompt gave 29/76)
majority-class baseline  20/76 = 26.3%   ("always answer feature")
random over 10 classes          ~10%

The CI on the raw baseline is 20.5 points wide, and the 38.2% ceiling sits inside it. Exact McNemar on paired systems needs 6 clean one-directional flips (7.9 points) to reach p<0.05 on this corpus:

Re-judging every decision made in the session that produced this system:

Only the normalization tier itself clears the bar. Everything else shipped in that increment is, on this evidence, indistinguishable from run-to-run variance — and run-to-run variance is real and measured: the same prompt at temperature 0 scored 29/76 on one run and 31/76 on the next. Temperature 0 does not buy determinism; a 235B MoE produced 80 unique completions across 1,000 identical T=0 runs (QAnswer, arXiv 2606.26185).

1.2 The system is not yet separable from a trivial baseline

Measured here, paired McNemar against "always answer feature":

This is the number that should govern how much confidence the dispatch path places in a classification today. The system is close — 0.0895 — but on this corpus it has not yet demonstrated that it beats a constant.

1.3 The corpus is being over-mined

Roughly twelve prompt and rule variants have now been scored against this same 76-item set, with each variant chosen in response to the previous one's score. That is the textbook adaptive-holdout failure (Dwork et al., reusable holdout; Gelman & Loken, garden of forking paths). Sampling SD at p≈0.38, n=76 is ±5.6 points; the expected maximum of twelve draws from pure noise sits 8–11 points above the mean. That is larger than the entire measured span between the raw floor and the current ceiling.

No bad faith is required for this; it is the arithmetic of repeated selection.

1.4 The ceiling is contaminated

The 29/76 ceiling comes from rewrites produced by the same mining agents that produced the labels. Same-source generation and judging is measured at ~20 points of inflation (98–99% self-consistency vs 76–79% for an independent judge) (When LLMs Benchmark Themselves; Self-Preference Bias); annotation artifacts from writer-side tells are a named, measured phenomenon (Gururangan et al.); and professionally curated benchmarks still carry ≥3.3% label error (Northcutt et al.). One mining agent here was caught fabricating 15 of 92 rewrites — a 16% rate, ~5× the benchmark norm. Catching it proves the failure mode occurred; it does not bound the residue in the agents that were not caught.

Treat 29/76 as an upper bound with unknown, plausibly double-digit positive bias — not as a target.

2. What the normalization tier actually does

It was designed as spelling-and-pronoun correction. Measured here, it is doing schema linking.

turns where the rewrite INJECTED a closed-lexicon word absent from the raw turn : 27/76
turns where the lexicon-word set was unchanged                                  : 48/76
turns where the rewrite LOST a lexicon word the raw turn had                    :  4/76

raw turns containing NO work-type lexicon word at all                           : 41/76
   ...of those, the rewrite supplied one                                        : 17/41

Every one of the 8 turns it fixed, it fixed by supplying a missing lexicon word:

This matches the text-to-SQL literature's schema linking — grounding free utterance tokens against a fixed vocabulary before parsing — measured at +1.8 to +2.5 points across five independent systems (MCS-SQL, BASE-SQL, XiYan-SQL, EviLink).

Consequence: a deterministic synonym table mapping out-of-lexicon tokens onto in-lexicon ones would capture part of this effect with no model call, no latency, and full auditability. That is a cheaper and more explainable path to the same mechanism, and it should be measured before any further prompt tuning.

The single turn the normalizer broke confirms §0 from the other direction:

want=analysis   got=refactor
raw : another task answer this question why aren't we loading and syncing the local ast…
norm: Add local AST JSON file loading and syncing to the orchgraph installer instead of…

It converted a question into a command. The speech act was destroyed, and with it the classification. This is the documented "entity substitution" / "query drift" failure of rewriters (Nexumo, RAG rewrite failures), and it is why §0's speech-act dimension has to be resolved before rewriting, not after.

3. Field findings that bear on this system

3.1 Where the ceiling really sits for this task shape

• Closed-domain intent benchmarks reach 96–97% — but only with full supervision (~100 examples/class): RoBERTa-base 96.22% on CLINC150, 93.86% on Banking77 (SOTA tables).
• Few-shot degrades sharply with granularity: SetFit at 8 examples/class scores 91.1% on 8-class MASSIVE but 77.9% on 77-class Banking77 (arXiv 2602.18922).
• The closest published analogue to this exact task — software commit classification into bug/feature/refactor/maintenance — scores 75–90%, not 96%: 90% F-measure for refactoring-commit detection, 75% average accuracy for maintenance-category random forests across 800 Java projects, ~76% accuracy / κ=0.63 for word2vec commit classifiers (ScienceDirect, Sci Reports).

So the realistic target for a well-built version of this system is 75–90% on work-request turns, not 96%, and not the 38.2% blended number currently reported — which mixes work requests with turns that have no correct answer.

3.2 The DCG tier is a defensible choice, with a known coverage ceiling

• PRECISE reached 93.8–94% on ATIS by proving correctness on the "semantically tractable" subset and declining outside it — but only ~80% of GeoQuery questions are tractable at all, versus ~90% for ATIS (AAAI). Closed-grammar coverage falls fast as domain breadth grows. Declining rather than guessing is the design this system already implements via its confidence gate.
• Federative neuro-symbolic architectures (LLM front, symbolic back — this system's exact shape) measure far larger gains than injective ones: Logic-LM +39.2%, DSR-LM +20%, PAL +15%, versus Syntax-BERT's 1–2% (Frontiers 2026 taxonomy).
• A deterministic-core/LLM-as-explainer pipeline measured 100% execution determinism and 100% traceability over 100 runs, against 0% for a pure-LLM pipeline (Springer ICCS 2026).
• Porter is the right stemmer for this system specifically: of the classic three, it has the lowest over-stemming rate (at the cost of the highest under-stemming rate) — and over-stemming, i.e. cross-category stem collision, is precisely what the existing collision gate exists to prevent (Paice 1996). Keep Porter. Keep the gate.

3.3 Confidence: every constant in this system is unmeasured

The tier confidences (0.9 for DCG, 0.5–0.7 for keyword scoring) and the 0.34 dispatch gate were all hand-chosen. The literature is unambiguous that this is not acceptable practice:

• DROID and DETER both fit their thresholds by grid search on held-out in-domain validation data; DETER lands at 0.7, not a round guess (DROID, DETER).
• Rasa's own documentation states there is no universal confidence threshold — it must be tuned per pipeline and corpus (Rasa).
• Heuristic confidence scores are badly calibrated and fixable cheaply: SetFit ECE 0.515 → 0.040 after temperature scaling (13×), on a small held-out set (arXiv 2602.18922). Platt scaling (2 parameters) is the data-efficient choice at n=76; isotonic regression needs more data than exists here.
• Youden's J (sensitivity + specificity − 1), or a cost-weighted variant when false-dispatch costs more than false-reject, is the standard principled way to pick a cutoff from labelled data (Youden's J).

CORRECTION (measured after this section was first written). The original claim here was that "0.34 sits below the keyword tier's own stated floor of 0.5, so the gate cannot reject anything the fallback tier produces." That is wrong, and the measurement is worth more than the guess it replaced: a 0.3 no-hit floor exists below the keyword tier, and the gate does reject — 27 of 76 turns on the raw corpus.

The real defect is sharper than the one I filed. Confidence on this corpus takes exactly two values, 0.3 and 0.9 — there is nothing in between. Sweeping the dispatch gate across its whole range:

Every value from 0.31 to 0.85 gives identical behaviour. The threshold is not a knob, it is a switch with one usable position, and calibrating it (Platt, Youden's J, anything) is meaningless until confidence is graded rather than binary. That reordering is what R3 below now asks for.

3.4 Accuracy is the wrong headline metric for a system that rejects

A classifier with a reject option must be scored by a risk-coverage curve (selective risk vs coverage), summarised by AURC — plain accuracy conflates "good model" with "rejected the hard cases" and is not comparable across systems with different reject rates (Geifman & El-Yaniv; NeurIPS 2024, Common Flaws).

The closest published analogue reports the trade explicitly: a tuned confidence threshold on a semantic parser cut incorrect-program execution by 76% at the cost of coverage falling to 32% (Did You Mean…?). That is the curve this system should be publishing, not 38.2%.

3.5 Acting on a misparse is the real risk, and thresholds do not bound it

• 88% of organisations reported a confirmed or suspected AI-agent security incident in the prior year; nine publicly documented cases in 14 months of coding agents destroying production data (DeepInspect, Adversa).
• The PocketOS incident (April 2026): an agent deleted a production database and all volume backups in 9 seconds via one GraphQL mutation. The lesson stated in the incident write-up is that the agent's own prompt-level safety rules were "advisory, not enforceable" (Zenity).
• Production cascades do not auto-act below 0.5 confidence, and use bands —

  0.9 execute, 0.7–0.9 confirm, 0.5–0.7 offer options, <0.5 fallback (tianpan.co, Hamming AI).

3.6 A genuine gap in the public literature

No published process was found for statically checking new lexicon entries against a live codebase symbol table — the exact defect this project hit when memory, trace, integration, and version were added as work-type signals while each also names a package here. The closest adjacent work is NER false-positive taxonomy. This has to be built, not adopted.

4. Recommendations, ranked by expected value

Each carries how to measure it. Nothing here should be shipped on a 1–2 turn delta ever again (§1.1).

R1 — Add the speech-act dimension above WorkType. (largest effect available) Resolve work_request | status_query | process_control | display_request | policy_statement before attempting work-type classification; report in-scope accuracy and out-of-scope detection separately, as every serious intent benchmark does. Measure: re-score the 76 turns with the 24 non-work-request turns excluded from the work-type denominator; the realistic target on the remaining 52 is the 75–90% band of §3.1, not 38.2%.

R2 — Stop scoring against the 76-turn set; grow and re-label it. Freeze the current corpora as regression floors only. Mine 300–500 turns (the sample is currently 76 of 14,978 captured = 0.5%), and get the labels adjudicated by a party that did not write the rewrites, so a real inter-annotator κ exists. n=300 gives ±5.2 points; n=76 gives ±10.2. Measure: CI width, and κ between the two independent labellings.

R3 — Make confidence graded before trying to calibrate it. (reordered after measurement; see the correction in §3.3. Step 1 DONE — tier, decisive_hits and winning_hits now ride on every ClassifiedIntent, sourced from parse_intent_evidence/4; REQ-003-US-001-068. Steps 2–3 are REQ-003-US-001-069, open.) Calibration is not the first step here — there is nothing to calibrate. Confidence takes two values, 0.3 and 0.9, so every gate setting in (0.30, 0.85] is the same setting, and Platt scaling over a two-point domain fits noise. The order is:

1. Expose the evidence the parse already has — how many decisive words matched, whether the tiers agreed, whether the target resolved — as fields on ClassifiedIntent, so a confidence can be derived rather than stamped.
2. Grade confidence from that evidence, producing a distribution with more than two points.
3. Then fit Platt scaling per tier and pick the gate by cost-weighted Youden's J.

Do step 3 on production telemetry, not on the 76-turn set — see the non-replication note below for why fitting on it is unsafe. Measure: number of distinct confidence values (must exceed 2); then ECE before/after; publish the risk-coverage curve and its AURC as the headline metric in place of accuracy.

A signal that did not replicate, recorded so it is not re-mined. On the labelled corpus, turns with exactly one decisive lexicon hit appeared to be far more often correct than turns with several (p=0.0037, Fisher exact) — an obvious candidate feature for step 2 above. It does not survive: on the held-out corpus the effect runs the opposite direction (66.7% vs 33.3%, p=0.52), and the labelled corpus saturates at 100% in one cell. One fixed 76-turn set queried repeatedly is exactly the garden-of-forking-paths setup described in §1.3, and this is what it produces. No calibration was built on it.

R4 — Raise the dispatch bar until the telemetry earns its lowering. (two of the three mechanisms originally filed here are refuted; see below)

Require dual-tier agreement (DCG and keyword tier concur) plus ≥0.5 confidence to auto-dispatch — both are impossible as stated, and the same measurement kills them:

• Dual-tier agreement cannot exist. The keyword tier is reached only when the grammar found nothing (§3.3 correction), so the two tiers never both produce a verdict on the same turn. There is nothing to agree or disagree.
• ≥0.5 confidence is not a bar. Confidence is 0.9 or 0.3, so 0.5 admits and rejects exactly what the current 0.34 admits and rejects.

What survives, and is the part that was always load-bearing:

• Preview the parse before sending the CXP packet. Independent of confidence, and the only mechanism here that degrades gracefully when the classifier is wrong.
• Gate every irreversible volk tool call on its own confirmation, regardless of upstream confidence — prompt-level rules are advisory and threshold tuning cannot bound blast radius (§3.5). This one is strengthened, not weakened, by the finding: an upstream signal carrying one bit is not something to hang a destructive action on.

A real dispatch bar has to wait for REQ-003-US-001-069 (graded confidence). The evidence it will be derived from now exists on ClassifiedIntent — see R3.

R5 — Build the lexicon/codebase collision gate. (no prior art) DONE — scripts/check_lexicon_collisions.py, wired into scripts/gates.py. Categorical lexicon words (work-type + generic-verb registries; targets exempt by design) are Porter-stemmed and matched against every package directory and module file two levels into src/orchestrata_ultimate. Four discovery strategies were measured against the five words that actually shipped and had to be reverted:

Depth 2 with module files ships: it is the only strategy that catches all five, and the two the others miss (version → cxp/version.py, dispatch → cxp/dispatcher.py) are plain modules, not packages. Depth 3 doubles the exemption burden for nothing. Cost: nine documented exemptions out of 144 categorical words, each naming its colliding path, each required by test to be a live collision so the list cannot accrete dead entries.

Measured: all five known-bad words flag (tests/scripts/test_check_lexicon_collisions.py, 20 tests). Stemming is load-bearing — the tree says tracing/integrations/ dispatcher where the lexicon says trace/integration/dispatch.

R6 — Try the deterministic synonym table before more prompt tuning. MEASURED AND REJECTED. Do not ship this; a regression test now blocks it.

The idea was sound and the measurement killed it. §2 shows the normalizer's mechanism is lexicon injection, so a static stem→lexicon table should capture part of the gain with no model call. It does — and the gain is not worth what it costs.

The candidates come from the data, not from taste: of the 27 raw turns that do not parse, the most common unmatched content words are run (7 turns, 17 occurrences overall), push, promote, answer, show. Added as generic verbs (the conservative tier — they fire only when no decisive word exists):

+6 on the dev corpus looks decisive. It is not, for three separate reasons:

1. The held-out set gives no evidence either way. It contains one turn with run and zero with push/promote/answer, so +0 is silence, not confirmation. The one turn that did move went the wrong way ("nobody knows how to run the harness", documentation → test).

2. The gain is bought by destroying the reject option. The four additions move 13 turns off unknown; 6 land correctly and 7 do not. Marginal precision on the newly covered turns is 46%, below the 49% the system already achieves on what it dispatches. Coverage rises 64.5% → 81.6% while precision falls — a strict move the wrong way down the risk-coverage curve (§3.4), invisible to accuracy as a headline number.

3. The turns it newly misclassifies are the dangerous ones. They are the non-work-requests of §0 — questions, status pings, standing instructions, and a stop order:

   no run allowed I will cancel anything you run, i command you to stop → dispatches as TEST

   at confidence 0.9 with decisive_hits == 0, which is to say: maximally confident, on no decisive evidence at all.

Conclusion, and it is the strongest independent support R1 has. The lexicon cannot be grown until the taxonomy can say "this is not a work request". Every mechanism that increases coverage before that exists converts correct rejections into confident errors. tests/orchestrator/test_intent_classifier.py:: test_a_turn_that_is_not_a_work_request_is_rejected_not_guessed pins the five turns above; it fails if anyone adds these words.

R7 — N-best rewrites selected by the parser. The rewrite literature needs a learned reranker to choose among candidates; this system has something strictly better and free — a deterministic parser whose success/failure is a hard signal. Generate 2 rewrites, take the first that parses. Measure: among currently-unparsed turns, how many does candidate 2 recover?

R8 — Sweep the context window. No paper found ablates turn count for this task shape — it is a genuine literature gap. Two turns was a guess. Measure: 0/1/2/3 preceding turns, parse-success per condition, on the enlarged corpus from R2.

5. What does not apply

• Spider/BIRD-scale machinery (multi-candidate agentic ensembling, 20+ sample voting, enterprise schema linking). Those fight thousand-column schemas; this lexicon is 270 closed words.
• Token-level grammar-constrained decoding. The measured GCD results require logit access; a hosted Haiku endpoint does not expose it. The post-hoc symbolic parser already provides the stronger guarantee.
• Fine-tuning / RL approaches. They need thousands of labelled examples; this corpus has 76.
• Isotonic regression and conformal risk-controlling sets. Both need far more calibration data than n=76 provides. Platt scaling is the n=76-appropriate tool.
• 96%+ CLINC150 numbers as a target. Full supervision, ~100 examples/class. The commit-classification band (75–90%) is the honest comparison.
• Literal self-consistency (N stochastic LLM samples, majority vote). This system's "majority vote" is over decisive words in one deterministic parse — same name, different mechanism. Importing the LLM version would cost N× latency and reintroduce the nondeterminism the architecture exists to avoid.
• Dependency-parse-informed head finding. General parsers score 74–83% LAS off-register, and terse dev commands are exactly off-register.
• EU AI Act Annex III high-risk documentation. Disproportionate for internal agent dispatch — though the "verdict reconstructable from what the parser read" property already satisfies Article 13's spirit.

6. Sources

Semantic parsing: Spider 2.0-AIFunc · Spider2 · BIRD · Text-to-SQL Benchmarks are Broken (CIDR 2026) · CypherBench · MCS-SQL · BASE-SQL · XiYan-SQL · EviLink · Query Plan Generation (VLDB TaDA 2025) · GCD for Structured NLP (EMNLP 2023) · CRANE · JSONSchemaBench · Capacity, Not Format

Grammar / neuro-symbolic: SWI dcg/basics · SWI dcg/high_order · SWI tabling · DCG primer · PRECISE on ATIS · Clark & Curran, wide-coverage CCG · Frontiers neuro-symbolic taxonomy 2026 · Logic-LM · DSR-LM · Deterministic hybrid pipelines (ICCS 2026) · ITLC SemEval-2026 Task 11 · Paice 1996, stemming error counting · RuleChef · Open Intent Extraction (IJCAI 2021)

Query rewriting: CANARD · QReCC · TREC CAsT 2021 · CHIQ · Not All Queries Need Rewriting · Detecting Ambiguities to Guide Query Rewrite · Spelling Correction in Healthcare QA · LLM Query Expansion Fails · When Smart Query Rewriting Breaks Your RAG · RAG faithfulness evaluation

Intent / calibration: CLINC150 OOS · DROID · DETER · Structured Intent Canonicalization · CICC conformal clarification · Selective prediction evaluation (ACL 2023) · Youden's J · Rasa intentless policy · Commit classification · Refactoring-type detection

Evaluation methodology: With Little Power Comes Great Responsibility · Annotation Artifacts in NLI · When LLMs Benchmark Themselves · Self-Preference Bias in LLM-as-a-Judge · Pervasive Label Errors · The Reusable Holdout · Garden of Forking Paths · Selective Classification for DNNs · Common Flaws in Selective Classification (NeurIPS 2024) · Did You Mean…? (semantic parsing trade-offs) · mlxtend McNemar

Production: Intent Classification Layer · Signal-Driven Routing · CARGO · Hamming AI voice-agent scale · PocketOS incident · 9 coding-agent data-loss incidents · Preventing wrong agent actions · Copilot CLI permission model · LLM non-determinism at T=0 · LLM Output Drift (financial workflows) · Redis LLM router best practices · EU AI Act summary

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks