Orchestrata Ultimate / Specs / 106-context-semantic-layer

106-context-semantic-layer: Context Semantic Layer

Context assembly today is all-or-nothing (static map tiers, whole-artifact reads) or silently degraded (six uncoordinated KG writers with divergent merge keys; injector queries that name relationships no writer produces). Agents and volks either drown (context thrashing — 4 agent deaths in one night) or starve (empty context sections that look like "no data").

At a glance

Item	Value
Spec directory	`specs/106-context-semantic-layer/`
Status line in the spec	not stated
Requirement IDs defined	38
With a `@traces` citation in `src/`	36 (94.7%)
With a `@traces` citation in `tests/`	26 (68.4%)
Tracked files	2
RTM page	106-context-semantic-layer

Reading this spec

This spec defines the context semantic layer, which decides what code and knowledge context an agent receives for a request. The problem it states is that context assembly is all-or-nothing (static map tiers, whole-artifact reads) or silently degraded: six uncoordinated knowledge-graph writers use divergent merge keys, and injector queries name relationships that no writer produces. Agents and volks either drown in context (the spec cites four agent deaths in one night) or starve on empty sections that look like "no data".

One ruling constraint, labelled D11 and non-negotiable, shapes the design: the LLM never drives node selection. Selection is deterministic end to end, so the same request and graph state produce a byte-stable pack, apart from timestamps. The LLM consumes packs and may expand them through tools, and humans or orchestrators may trim a pack before injection. No LLM scoring appears in the selection loop.

The six functional requirements are FR-001 canonical graph identity and schema (code nodes labelled Module, Class and Function with id <relpath>::<qualname>, and one canonical DDL source), FR-002 deterministic request analysis (a pure function producing task type, entities, budget and requester with no LLM calls), FR-003 deterministic selection (rank ties break lexicographically on node id), FR-004 budgeted packing and the editable pack (token counting through the token counter protocol, a trimmed pack re-validates and injects without regeneration), FR-005 integrations and measurement, and FR-006 symbolic Prolog relevance refinement (tiers feed the score as a discrete bonus, and ownership recursion uses bounded tabling so cyclic graphs terminate). When the knowledge graph is unavailable the result is an explicit empty pack with a reason, never a silent partial one. Selection and packing at repository scale target a p50 under 500 ms. The spec also lists descoped items for v1. The 38 requirement IDs are numbered REQ-106-NNN.

The code is spread over the context and knowledge_graph packages: selector (9 requirements), canonical schema, architecture writer, context injector and the context assembler. The requirements traceability matrix lists each ID. The spec states no status line and tracks 2 files. Measured coverage is 36 of 38 requirements cited by at least one @traces in src/ (94.7%), spread over 15 modules; the module citing the most of them is orchestrata_ultimate/knowledge_graph/context_pack/selector (9 requirements). In tests/, 26 requirements are cited by a test-side @traces. The 2 uncited requirements are REQ-106-041 (the pack script for agent briefs) and the performance requirement on selection latency, REQ-106-044.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID	Done	Requirement	Traced by
`REQ-106-001`	yes	Code nodes use labels Module/Class/Function with id `<relpath>::<qualname>`; methods are Function nodes carrying class_id	10
`REQ-106-002`	yes	Requirement nodes key on the literal requirement id string; TRACES_TO targets RequirementItem	6
`REQ-106-003`	yes	Volk keys on alias; Session on session_id; Project on slug — one key per label, everywhere	9
`REQ-106-004`	yes	Containment uses DEFINES; test linkage uses TESTED_BY — no writer-specific synonyms	1
`REQ-106-005`	yes	`canonical_schema.get_all_ddl()` is the single constraint source and has an `apply()` CLI (`scripts/kg_schema.py apply`)	7
`REQ-106-006`	yes	A conformance gate flags any MERGE using non-canonical label/key/rel and runs in CI (`scripts/kg_schema.py check` wired into `.github/workflows/ci.yml`; migration run against shared Neo4j)	13
`REQ-106-007`	yes	ContextInjector's label mismatches (TRACES, TESTS, Requirement-vs-RequirementItem) are fixed and regression-tested against real Neo4j	1
`REQ-106-008`	no	Every production writer either conforms to the canon or is retired with traces-union (partial — canonical_schema is the foundation; writer migrations owed)	7
`REQ-106-010`	yes	analyze_request produces {task_type, entities, budget, requester} with no LLM calls	1
`REQ-106-011`	yes	Task-type classification reuses the existing deterministic parsers (intent/work_type/SUBJECT_TASK_TYPE tables)	1
`REQ-106-012`	yes	Entity linking matches exact and normalized node names against the canonical graph	1
`REQ-106-013`	yes	Ambiguous entity matches are retained with recorded ambiguity, never dropped or guessed	1
`REQ-106-014`	yes	Identical request text yields an identical RequestFrame (pure function of input + graph names)	1
`REQ-106-020`	yes	Each task type has a parameterized query template rooted at the linked entities	1
`REQ-106-021`	yes	Expansion is k-hop bounded with a default of 2 and per-template overrides	1
`REQ-106-022`	yes	Ranking uses only graph signals: hop distance, trace degree, TESTED_BY presence, recency, ambiguity penalty	1
`REQ-106-023`	yes	Rank ties break lexicographically on node id for reproducibility	1
`REQ-106-024`	yes	No embedding or LLM scoring participates in selection	1
`REQ-106-025`	yes	The six SPEC-009 task-type handlers are expressed as selection templates without losing their traced REQ coverage	1
`REQ-106-030`	yes	Packing is greedy by rank under an explicit token budget that is never exceeded	4
`REQ-106-031`	yes	Token counting uses the TokenCounterProtocol implementations (tiktoken with word fallback)	1
`REQ-106-032`	yes	A pack is a markdown file with YAML frontmatter recording frame, snapshot id, budget and spend	2
`REQ-106-033`	yes	Every pack section carries provenance: node id, selecting template, rank, token cost	6
`REQ-106-034`	yes	Same frame + same graph state produces a byte-identical pack body	4
`REQ-106-035`	yes	A trimmed pack (sections deleted) re-validates and injects without regeneration	8
`REQ-106-040`	yes	ContextInjector.inject_for_task consumes the selector/packer instead of ad-hoc LIMIT queries	2
`REQ-106-041`	yes	scripts/context_pack.py emits packs for agent briefs (task-type, target, budget CLI)	none
`REQ-106-042`	yes	Each injection records budget, spend and section list for measurement	3
`REQ-106-043`	yes	KG-unavailable degrades to an explicit empty pack with reason, never a silent partial	6
`REQ-106-044`	yes	Selection+packing p50 stays under 500ms at repo scale against live Neo4j (`scripts/benchmark_context_pack.py` + `tests/knowledge_graph/test_context_pack_performance.py`)	none
`REQ-106-060`	yes	The retriever asserts ONLY ground facts derived from the selected subgraph's graph signals (kg_candidate/3, kg_root/1, kg_traces/2, kg_tested/1, kg_defines/2, kg_task/1, kg_ambiguous/1); no variable-bearing rule is asserted at runtime — all rules live in prolog/kg_relevance.pl, consulted from the rules dir	1
`REQ-106-061`	yes	kg_relevant/1 and kg_relevance_tier/2 are deduced by task-type rules — a linked root entity is always core; a candidate that shares_requirement (transitive over kg_traces) with a root is at least related; task-type rules promote containment/test/trace relationships to core per the task's need	1
`REQ-106-062`	yes	Tiers feed FR-003's score as a discrete bonus (core >> related >> peripheral), applied after the linear score, with the FR-003 lexicographic node-id tie-break unchanged — byte-stable given the same subgraph and task type	1
`REQ-106-063`	yes	When no Prolog bridge is wired every node is peripheral (zero bonus) and ranking is byte-identical to the FR-003-only path — FR-006 is strictly opt-in	2
`REQ-106-064`	yes	Projected facts are request-scoped and retracted after each refinement (no cross-request fact leakage); the ruleset is consulted once and reused	1
`REQ-106-065`	yes	Each refinement records the per-tier node counts alongside the FR-005 injection record, so tier distribution is observable	2
`REQ-106-066`	yes	Knowledge-graph ownership recursion (`knowledge.pl:ownership_chain/3`) uses bounded tabling (`max_answers` restraint) so cyclic `owns`/`depends_on` graphs always terminate; answers beyond the cap are undefined and treated as deny+escalate at call sites	1
`REQ-106-067`	yes	Negation-bearing governance predicates with self-referential negation (e.g., `constitutional.pl:assess_harm/3`) remain untabled; tabling them requires converting `\+` to `tnot/1` and explicitly handling `undefined` answers	1

Code that traces this spec

Module	Symbols	Requirements
`orchestrata_ultimate/context/assembler`	1	2
`orchestrata_ultimate/context/budget`	1	1
`orchestrata_ultimate/context/providers/trimming`	1	1
`orchestrata_ultimate/knowledge_graph/architecture_writer`	16	5
`orchestrata_ultimate/knowledge_graph/canonical_schema`	17	5
`orchestrata_ultimate/knowledge_graph/context_injector`	5	6
`orchestrata_ultimate/knowledge_graph/context_pack/cache`	1	1
`orchestrata_ultimate/knowledge_graph/context_pack/frame`	1	8
`orchestrata_ultimate/knowledge_graph/context_pack/packer`	7	7
`orchestrata_ultimate/knowledge_graph/context_pack/relevance`	2	5
`orchestrata_ultimate/knowledge_graph/context_pack/selector`	1	9
`orchestrata_ultimate/knowledge_graph/ontology_mapping`	7	4
`orchestrata_ultimate/knowledge_graph/requirement_resolution`	3	1
`orchestrata_ultimate/knowledge_graph/runtime_writer`	1	1
`orchestrata_ultimate/knowledge_graph/schema`	1	1

Checklists

`checklists/requirements.md`

The requirement list above comes from this file; 37 of 38 checkbox items are ticked.

Full spec text

Verbatim from specs/106-context-semantic-layer/spec.md; headings are pushed down two levels.

SPEC-106: Context Semantic Layer

Version: 0.3.0 (draft) · Date: 2026-07-14 (FR-001/002/003/004/005/006 landed in Phase 2) · Status: Implementation-in-progress — selector/packer/relevance wired into production ContextInjector Depends on: SPEC-008 (knowledge graph), SPEC-009 (volk context injection), SPEC-102 (harness, measurement), D11 (plans/decisions-2026-07-06.md) Evidence base: plans/evidence/w8-context-layer-inventory.md (as-built audit)

Problem

Ruling constraints (D11 — non-negotiable)

The LLM NEVER drives node selection. Selection is deterministic end-to-end: same request + same graph state => byte-stable pack (modulo timestamps). The LLM consumes packs and may EXPAND via tools; humans/orchestrators may TRIM packs before injection. No LLM scoring anywhere in the selection loop.

FR-001: Canonical graph identity & schema

One identity scheme, enforced by one DDL source and one conformance check:

Code nodes: Module/Class/Function (methods are Function with class_id), id = <relpath>::<qualname>.
Spec nodes: Spec {id}, Requirement {id} (id = the literal REQ string), RequirementItem {id}; TRACES_TO targets RequirementItem.
Runtime nodes: Volk {alias}, Session {session_id}, Project {slug}, Intent/PipelineRun/Decision/Artifact {id}.
Containment rel is DEFINES; test linkage is TESTED_BY.
knowledge_graph/schema.py::get_all_ddl() becomes the single constraint source with an apply CLI; a conformance checker flags any writer MERGE that uses a non-canonical label/key/rel (extends architecture_checker or a new scripts/kg_schema_check.py gate).
Existing writers (sync.py, FileSynchronizer, pipeline_kg, session/kg_sync, kg_design) migrate to the canon or are retired; the injector's known label mismatches (TRACES vs DEFINES_REQUIREMENT, TESTS vs TESTED_BY, Requirement vs RequirementItem) are fixed as part of conformance.

FR-002: Deterministic request analysis

analyze_request(text | task payload) -> RequestFrame:

task-type classification via the existing deterministic parsers (orchestrator intent parsing / PipelineRouter work_type / SUBJECT_TASK_TYPE) — table-driven, no model calls;
entity linking: exact + normalized (case/underscore/dash-folded) name match of request tokens against canonical node names (Module/Class/Function/Spec/ Requirement/Volk); ambiguous matches are KEPT with their ambiguity recorded (selection ranks them lower), never guessed away. RequestFrame = {task_type, entities[], budget, requester (volk|agent|human)}.

FR-003: Deterministic selection

select(frame) -> RankedSubgraph:

per-task-type parameterized query templates (generalizing SPEC-009's six handlers: BUILD/REFACTOR/DEBUG/DOCUMENT/TEST/REVIEW) rooted at the linked entities;
k-hop bounded expansion (default k=2, per-template overrides);
deterministic ranking, computed from graph signals only: hop distance from linked entities, trace-link degree, TESTED_BY presence, recency (updated_at), entity-link ambiguity penalty. Ties break on node id (lexicographic) for reproducibility.
NO embedding/vector scoring in v1 (the embedding service is a P1 hash stub — inventory §3); the ranking interface leaves a slot for it later.

FR-004: Budgeted packing & the editable pack

pack(subgraph, budget) -> ContextPack file:

greedy by rank under an explicit token budget (memory/token_counter.py TiktokenCounter with Word fallback); hard ceiling, never exceeded;
pack = markdown with YAML frontmatter: request frame, graph snapshot id, budget/spent, then one section per node with provenance (node id, template that selected it, rank score, token cost);
packs are FILES: trim = delete sections; a trimmed pack re-validates (provenance intact) and injects as-is;
byte-stable: same frame + same graph => identical pack body.

FR-005: Integrations & measurement

Volk path: ContextInjector.inject_for_task consumes select()+pack() instead of its ad-hoc LIMIT-capped queries (SPEC-009 REQs preserved).
Production runner wiring: kernel/pipeline_kg.py::PipelineKGMixin instantiates a UnifiedContextAssembler and routes both the kernel escalation path (_maybe_inject_context) and the orchestrator stage path (orchestrator/pipeline_runner.py::_run_stage) through it. The assembler's output is collapsed into payload["graph_context"] with task_context preserved, so existing volks receive the unified bundle without changing the CXP contract.
Agent briefs: scripts/context_pack.py CLI (context_pack <task-type> <target> --budget N) emits a pack file for swarm-task briefs, replacing hand-written context-safety excerpts.
Measurement (harness/SPEC-102 surface): per-injection record of budget, spent, sections, plus before/after context-size comparison vs the static map tier; acceptance = zero context-thrash agent deaths in a full wave run.

FR-003's ranking is a fixed linear score over independent per-node signals; it cannot express relational relevance ("this node shares a requirement with what you asked for", "this node is a definition contained by a root"). FR-006 adds a deterministic, symbolic refinement layer that CAN — expressed as Prolog rules, never LLM/embedding scoring (D11-compatible: Prolog is symbolic and deterministic; same facts => same deductions).

refine_relevance(RankedSubgraph, task_type) -> {node_id: RelevanceTier} where RelevanceTier ∈ {core, related, peripheral}:

REQ-106-060 — Projection: the retriever asserts ONLY ground facts derived from the already-selected subgraph's graph signals — kg_candidate/3 (NodeId, Label, Hop), kg_root/1, kg_traces/2 (NodeId, ReqId), kg_tested/1, kg_defines/2, kg_task/1, kg_ambiguous/1. No variable-bearing rule is ever asserted at runtime (janus constraint); all rules live in prolog/kg_relevance.pl, consulted from the rules dir.
REQ-106-061 — Deduction: kg_relevant/1 and kg_relevance_tier/2 are deduced by task-type rules. A linked root entity is always core. A candidate that shares_requirement with a root (transitive over kg_traces) is at least related. Task-type rules (BUILD/REFACTOR/DEBUG/DOCUMENT/TEST/REVIEW) promote containment/test/trace relationships to core per the task's need (e.g. DEBUG promotes hop≤1 and failing/tested nodes; TEST promotes tested + trace-shared).
REQ-106-062 — Additive & determinism-preserving: tiers feed FR-003's score as a discrete bonus (core >> related >> peripheral), applied AFTER the linear score, with the FR-003 lexicographic node-id tie-break unchanged. Byte-stable: same subgraph + same task type => same tiers => same order.
REQ-106-063 — Graceful absence: when no Prolog bridge is wired, every node is peripheral (zero bonus) and ranking is byte-identical to the FR-003-only path. FR-006 is strictly opt-in; the default SPEC-106 path is unchanged.
REQ-106-064 — Lifecycle: projected facts are request-scoped and retracted after each refinement (no cross-request fact leakage); the ruleset is consulted once and reused.
REQ-106-065 — Measurement: each refinement records the per-tier node counts alongside the FR-005 injection record, so tier distribution is observable.
REQ-106-066 — Bounded ownership recursion: knowledge.pl:ownership_chain/3 is tabled with a max_answers restraint so cyclic owns/depends_on graphs always terminate. Answers beyond the cap are marked undefined by SWI's tabling restraints and are treated as deny + escalate at call sites.
REQ-106-067 — Tabling safety for negated governance predicates: predicates that contain self-referential negation, such as constitutional.pl:assess_harm/3, remain untabled. Tabling them requires converting \+/1 to tnot/1 and explicitly handling undefined answers under well-founded semantics.

NFRs

Determinism: byte-stable packs (tested by double-run comparison).
Selection+packing p50 < 500ms against the live KG at repo scale.
Degradation: KG unavailable => explicit empty pack with reason, never a silent partial.

Descoped (v1)

LLM/embedding relevance scoring (D11; stub embeddings). (Note: symbolic Prolog relevance refinement is IN scope as FR-006 — it is deterministic and non-LLM, so D11 permits it.)
Automatic pack-usage feedback loop (which sections the consumer actually used) — phase 2, requires provider-side attribution.
vizint-ui/JS traceability (separate W-item).