Orchestrata Ultimate › Plans › thread-ontology-design
Thread ontology + event-backfill design (session-254 Option A)
Source: plans/thread-ontology-design.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-26 · 665 lines.
Prose pending.
Plan text
Thread ontology + event-backfill design (session-254 Option A)
Status: DESIGN — settled, ready to build. No implementation in this document.
Author: analyst (design pass), 2026-07-26.
Governs: src/orchestrata_ultimate/knowledge_graph/canonical_schema.py (SPEC-106 FR-001),
one new backfill script, three test files.
Constitutional rule honoured: ontology before graph import. Every label, edge and
key the backfill writes is declared in canonical_schema.py FIRST; the backfill script
imports its entire vocabulary from that module and hard-codes no graph string.
0. Ground truth — measured on the primary KG, 2026-07-26
All numbers below are live measurements against bolt://10.92.53.11:7687 (read-only),
not estimates. They supersede the session-254 opening figures where they differ (a purge
landed in between).
0.1 Isolated nodes (NOT (n)--()), all labels
| Label | isolated |
|---|---|
| CXPPacket | 6,885 |
| Message | 556 |
| ToolCall | 244 |
| Specification | 178 |
| Artifact | 81 |
| CloudService | 30 |
| Volk | 12 |
| Action | 10 |
| WAFPillar | 5 |
| Requirement | 3 |
| Pattern / File / Task | 2 each |
| Decision / PipelineRun / Test | 1 each |
In scope for this design: CXPPacket, Message, ToolCall, Artifact = 7,766 isolated nodes.
Everything else is out of scope (different root causes; Specification is a non-canonical
legacy label, CloudService/WAFPillar are reference data that is supposed to be
free-standing until an architecture writer links it).
0.2 thread_id coverage — the decisive fact
| Label | total | with thread_id |
isolated | isolated without thread_id |
|---|---|---|---|---|
| CXPPacket | 25,225 | 25,225 | 6,885 | 0 |
| Message | 556 | 556 | 556 | 0 |
| ToolCall | 244 | 244 | 244 | 0 |
| Artifact | 84 | 82 | 81 | 0 |
Verified directly: MATCH (n) WHERE NOT (n)--() AND (n:CXPPacket OR n:Message OR n:ToolCall OR n:Artifact) AND n.thread_id IS NULL returns zero rows. There is no
"no thread_id" residue class at all. The 2 Artifact nodes lacking thread_id are
already connected (one via the GENERATED_BY->Volk precedent edge) and are left untouched.
Distinct thread_id across the four labels: 9,950. Of those, 3,369 (34%) are not
UUIDs — live values include t1, resume-dispatch, pipeline_aae21620,
175098c03b504452b63ac50e35f4e2f4. Zero contain :. Zero are empty strings.
Threads spanning more than one event label: 312 (282 span 2 labels, 30 span 3). Thread is therefore a genuine join anchor, not a per-label synonym.
0.3 The connected CXPPackets are hub-disconnected too
25,225 CXPPackets exist; 18,340 are "connected" — but only to each other, via a
9,355-edge ROUTED_TO packet↔packet web. Sampling 500 connected packets for
EXISTS { MATCH (p:Project)-[*1..4]-(n) } returns 0 reachable. The whole packet
population is orphaned from the Project hub, not just the isolated half.
Design consequence: the backfill is driven by thread_id presence, not by an
isolation predicate. That is simpler, idempotent, and repairs 25,225 packets rather than
6,885. It is also why the IN_THREAD edge count below exceeds the isolated-node count.
0.4 Volk matching
19 :Volk nodes exist; the identifier pool (alias ∪ volk_id ∪ name, non-null) is 25
strings. volk_id is present on exactly one node (v1, test pollution) — so
alias-matching does ~all the work, but volk_id is still honoured per the brief.
Agent fields: CXPPacket carries from_agent/to_agent; Message carries
sender/recipient (not from_agent/to_agent — the session-254 note was wrong here;
verified by key inventory).
| Label | total | sender-side match | recipient-side match | no match either side |
|---|---|---|---|---|
| CXPPacket | 25,225 | 12,195 | 15,171 | 2,677 |
| Message | 556 | 393 | 432 | 0 |
The 2,677 unmatched packets are HUMAN (3,867 endpoint occurrences among isolated),
TEST:Flaky / TEST:DoubleFlaky / TEST:ArtifactSkip, U:Target, U:Missing,
test-<uuid>-U:Cx test pollution — and U:Do (Documentor), a real volk with 43 packet
endpoints and 40 message endpoints that has no :Volk node at all. That is a genuine
data gap; see §7 residue.
0.5 Existing vocabulary already live
REL_IN_THREAD = "IN_THREAD"already exists incanonical_schema.py(Tier H, line 219) with the docstring(signal)-[:IN_THREAD]->(Thread|Session). The Thread target was ratified; only the label was never declared. This design fills exactly that hole and invents no new edge for the event→Thread link.BY_VOLKis live vocabulary: 202(:AuditEntry)-[:BY_VOLK]->(:Volk)edges written bysrc/orchestrata_ultimate/kernel/audit/writers.py:311as a bare string literal. It has noREL_*constant anywhere.(:Project)-[:HAS_SESSION]->(:Session)is canonical and live (180 Sessions, all withsession_id). Tier I mirrored it withHAS_CONVERSATION. Thread mirrors it again.- No
Thread/Workflow/Conflictlabel exists. No constraint name containsthread. - Project hub: exactly one
(:Project {slug:'orchestrata-ultimate', id:'orchestrata-ultimate'}). - Zero
Session.session_idvalues match any packetthread_id— thread and session are disjoint identifier spaces. Do not try to resolve threads to sessions.
1. Canonical schema additions (exact)
All additions go into src/orchestrata_ultimate/knowledge_graph/canonical_schema.py.
Nothing else in that file changes.
1.1 Label — Tier H
Placed in the Tier H block (operational events & messages), immediately after
LABEL_SESSION_END, because a Thread is the correlation anchor for exactly the Tier-H
record types and nothing else.
LABEL_THREAD = "Thread" # correlation anchor for Tier-H operational records
1.2 Merge key — project-qualified id
CANONICAL_MERGE_KEYS.update({LABEL_THREAD: "id"}) # in the Tier H update() block
_CONSTRAINT_NAMES.update({LABEL_THREAD: "thread_id"}) # in the Tier H update() block
Generated DDL (falls out of CONSTRAINTS automatically — no hand-written statement):
CREATE CONSTRAINT thread_id IF NOT EXISTS FOR (n:Thread) REQUIRE n.id IS UNIQUE
Constraint name thread_id follows the Tier-H convention verbatim (LABEL_EVENT →
event_id, LABEL_TOOL_CALL → tool_call_id, both keyed on id). Neo4j constraint names
live in a separate namespace from property names, so the collision with the thread_id
property is cosmetic only, and no live constraint uses that name (verified).
Ruling: Thread is project-qualified, not global
Thread.id = "{project}:{thread_id}", bare thread_id kept as a display property —
the exact Conversation / File / Directory / Package precedent.
Justification against the multi-project KG invariant (plans/kg-project-scoped-identity.md):
- The invariant's global side is spec/requirement ids — deliberately global because
REQ-106-001names one shared artefact that means the same thing viewed from any project. - The invariant's qualified side is everything project-derived whose natural id recurs across projects.
- A thread id is a per-run correlation token, not a shared artefact. 34% of live values
are non-UUID tokens (
t1,resume-dispatch,pipeline_<8hex>) which will unquestionably recur when a second project's kernel emits its ownt1orpipeline_aae21620(an 8-hex namespace has a birthday collision at ~5k threads within one project, let alone across 100). - The nearest precedent is exact: Tier I's
Conversation— also an operational, producer-generated, per-run id — was ratified project-qualified for this identical reason (plans/contracts/spec-021-ontology.contract.jsonD1–D4).
Thread therefore falls on the qualified side. This is not a close call.
Id helper (mirrors conversation_entity_id exactly)
@traces(spec="106", reqs=["REQ-106-003"])
def thread_entity_id(project: str, thread_id: str) -> str:
"""Canonical identity for ``Thread`` nodes: ``<project>:<thread_id>``."""
# raises ValueError on empty part, or ':' in project — same discipline as
# conversation_entity_id / project_qualified_id
Validation contract, copied verbatim from conversation_entity_id: reject empty
project, reject empty thread_id, reject ':' in project only (a ':' inside
thread_id is legal — the first ':' delimits, and zero live thread_ids contain one).
Thread properties — the complete set
| Property | Source | Note |
|---|---|---|
id |
thread_entity_id(project, thread_id) |
canonical key, UNIQUE |
thread_id |
the bare event value | display/lookup, mirrors Conversation.natural_id |
project |
the --project argument |
mirrors Component.project |
No timestamps. The brief allows derived timestamps "if trivially available" — they are
not. Measured valueType() across the four source labels: CXPPacket.timestamp is
STRING, Message.timestamp is ZONED DATETIME, ToolCall.timestamp/Artifact.updated_at
is INTEGER. A min()/max() across three incompatible types needs per-label coercion
rules that nothing in this wave consumes. Omitted as speculative.
No extra indexes. The uniqueness constraint supplies the only index Thread needs.
1.3 Relationships
Three new REL_* constants; one pure reuse. All go in the Tier H relationship block.
| Constant | Value | Shape | New? | Rationale |
|---|---|---|---|---|
REL_IN_THREAD |
IN_THREAD |
(:CXPPacket|:Message|:ToolCall|:Artifact)-[:IN_THREAD]->(:Thread) |
no — already declared | Line 219 already documents (signal)-[:IN_THREAD]->(Thread|Session). Zero new vocabulary for the primary edge. |
REL_HAS_THREAD |
HAS_THREAD |
(:Project)-[:HAS_THREAD]->(:Thread) |
yes | The hub edge. Mirrors the canonical, live (:Project)-[:HAS_SESSION]->(:Session) and Tier I's REL_HAS_CONVERSATION — same direction, same role. This single edge is what makes all 26,107 events hub-reachable. |
REL_BY_VOLK |
BY_VOLK |
(:CXPPacket|:Message)-[:BY_VOLK]->(:Volk) |
constant new, edge already live | 202 live AuditEntry->Volk edges written as a bare literal by kernel/audit/writers.py:311. Formalizing a live writer's string literal is the exact treatment Tier J gave HAS_COMPONENT/MUST_FOLLOW. Extending it to packets/messages reuses the existing "operational record performed by this volk" semantics instead of inventing a parallel SENT_BY. |
REL_TO_VOLK |
TO_VOLK |
(:CXPPacket|:Message)-[:TO_VOLK]->(:Volk) |
yes — the only invented name | No live or canonical name covers the addressee side. TO_VOLK is the exact mirror of BY_VOLK. Rejected DELIVERED_TO: delivery_status is a packet property and many packets were never delivered, so that name would assert a fact the data contradicts. Rejected SENT_TO: asymmetric with BY_VOLK for no gain. |
Declaration block to add (Tier H, after REL_FAILED):
# Thread anchor (session-254 Option A). IN_THREAD above already names Thread as
# its target — this wave declares the label it was always pointing at.
# HAS_THREAD mirrors the canonical Project-[:HAS_SESSION]->Session hub edge.
# BY_VOLK formalizes the bare literal kernel/audit/writers.py:311 already writes
# (AuditEntry->Volk, 202 live edges); TO_VOLK is its addressee-side mirror —
# deliberately NOT "DELIVERED_TO", since delivery_status says many were not.
REL_HAS_THREAD = "HAS_THREAD" # (:Project)-[:HAS_THREAD]->(:Thread)
REL_BY_VOLK = "BY_VOLK" # (:CXPPacket|:Message|:AuditEntry)-[:BY_VOLK]->(:Volk)
REL_TO_VOLK = "TO_VOLK" # (:CXPPacket|:Message)-[:TO_VOLK]->(:Volk)
Explicitly out of scope: adding these four names to
knowledge_graph/types_enums.RelationshipType. Its drift-lock
(TestEnumSchemaDriftLock) only enforces schema.py's REL_* set ⊆ enum, and this wave
adds nothing to schema.py. Leaving the enum alone keeps the change minimal.
1.4 Ontology shape
2. Backfill rules
Deliverable: scripts/backfill_thread_links.py — a new one-shot, idempotent,
dry-run-able CLI modelled structurally on scripts/migrate_neo4j_canonical.py
(plan_* pure-read functions + run_* writer + --dry-run + --uri/--user/--password
python-side _batched(seq, size=500)).
Do not fold this into migrate_neo4j_canonical.py: that script's job is
keyless-node purge/backfill, and it is documented as already-executed. Mixing an edge
backfill into it would break its "re-running is a no-op" contract review.
2.0 CLI surface
python scripts/backfill_thread_links.py [--dry-run] [--project SLUG]
[--batch-size N] [--uri ...] [--user ...] [--password ...]
--project defaults to orchestrata-ultimate. --batch-size defaults to 500.
Loads repo .env with override=False (the migrate_neo4j_canonical.main precedent).
Constants imported from canonical_schema, never literals:
LABEL_THREAD, REL_IN_THREAD, REL_HAS_THREAD, REL_BY_VOLK, REL_TO_VOLK,
thread_entity_id, CANONICAL_MERGE_KEYS, apply.
Source-label table (the only per-label knowledge in the script):
_SOURCE_LABELS: dict[str, tuple[str | None, str | None]] = {
# label : (sender_property, recipient_property)
"CXPPacket" : ("from_agent", "to_agent"),
"Message" : ("sender", "recipient"),
"ToolCall" : (None, None),
"Artifact" : (None, None),
}
CXPPacket is an as-built label with no canonical entry (the canon's
LABEL_MESSAGE = "Message" # CXPPacket comment mis-identifies it — see §8). This wave
neither renames nor re-keys it; it only attaches edges.
2.1 Step 0 — hub guard (read-only, abort on failure)
MATCH (p:Project {slug: $project}) RETURN count(p) AS hubs
Abort with a non-zero exit and zero writes unless hubs == 1. The backfill never
creates a Project — a missing/duplicated hub is a real signal, not something to paper over
(the run_component_id_migration abort precedent).
2.2 Step 1 — plan (pure read, this is the whole dry-run)
-- 1a: distinct threads, per source label
MATCH (n:`<LABEL>`)
WHERE n.thread_id IS NOT NULL AND n.thread_id <> ''
RETURN DISTINCT n.thread_id AS thread_id
-- 1b: the nodes to link, per source label
MATCH (n:`<LABEL>`)
WHERE n.thread_id IS NOT NULL AND n.thread_id <> ''
RETURN elementId(n) AS eid
-- 1c: the volk identifier pool (25 rows — read once, resolved in Python)
MATCH (v:Volk) WHERE v.alias IS NOT NULL
RETURN v.alias AS alias, v.volk_id AS volk_id, v.name AS name
-- 1d: the agent fields to resolve, per label that has them
MATCH (n:`<LABEL>`)
WHERE n.thread_id IS NOT NULL AND n.thread_id <> ''
RETURN elementId(n) AS eid, n.`<SENDER_PROP>` AS sender, n.`<RECIP_PROP>` AS recipient
Volk resolution (Python, deterministic): build identifier -> alias from 1c in this
precedence order — alias → itself, then volk_id → its node's alias, then name → its
node's alias, each added only if the key is not already present. alias therefore
always wins, which is correct: alias is the canonical merge key (REQ-106-003) and
writers.py:311 already learned this lesson (its comment records that MERGEing on
volk_id created divergent nodes). An agent string with no entry produces no volk edge.
Plan output (the dry-run report), all counts derived without a single write:
threads, in_thread_edges per label, by_volk_edges, to_volk_edges,
unmatched_agents (a sorted {agent_string: count} histogram).
2.3 Step 2 — apply the Thread constraint before any MERGE
apply(repo, [stmt for stmt in get_all_ddl() if ":Thread)" in stmt])
Uses the scoped-statement seam canonical_schema.apply(repo, statements) was built for.
MERGEing 9,950 Threads without the uniqueness constraint would race and duplicate. Skipped
entirely when --dry-run.
2.4 Step 3 — MERGE Threads + hub edge (batched over $rows of {id, thread_id})
MATCH (p:Project {slug: $project})
UNWIND $rows AS row
MERGE (t:Thread {id: row.id})
SET t.thread_id = row.thread_id,
t.project = $project
MERGE (p)-[:HAS_THREAD]->(t)
SET is unconditional rather than ON CREATE SET: both values are pure deterministic
functions of the merge key, so unconditional assignment is idempotent and self-heals a
Thread that a future partial run left without its display property. This is the same
reasoning run_migration's backfill SET records.
2.5 Step 4 — MERGE IN_THREAD (batched over $eids, per label)
MATCH (n) WHERE elementId(n) IN $eids
MATCH (t:Thread {id: $project + ':' + n.thread_id})
MERGE (n)-[:IN_THREAD]->(t)
elementId(n) IN $eids is the NodeByElementIdSeek shape already used by
migrate_neo4j_canonical.run_migration's purge step — no label scan. The Thread lookup is
constraint-backed (index seek). Must run after Step 3 (it MATCHes, never MERGEs,
the Thread — so a missing Thread silently drops the row rather than creating an unanchored
one; a post-run assertion in Step 6 catches that).
2.6 Step 5 — MERGE BY_VOLK / TO_VOLK (batched over $rows of {eid, alias})
-- sender side; rows built from CXPPacket.from_agent + Message.sender
UNWIND $rows AS row
MATCH (n) WHERE elementId(n) = row.eid
MATCH (v:Volk {alias: row.alias})
MERGE (n)-[:BY_VOLK]->(v)
-- addressee side; rows built from CXPPacket.to_agent + Message.recipient
UNWIND $rows AS row
MATCH (n) WHERE elementId(n) = row.eid
MATCH (v:Volk {alias: row.alias})
MERGE (n)-[:TO_VOLK]->(v)
Two writes total (the relationship type cannot be parameterized; the source label is
irrelevant once matched by elementId). Volk {alias:} is a constraint-backed seek
(volk_alias). Rows for unresolved agents are simply absent — never a fabricated Volk.
2.7 Idempotency & guard summary
- MERGE only. Zero
CREATE, zeroDELETE, zeroDETACH, zero property removal. - Re-running produces byte-identical counts: node MERGE on the unique
id, edge MERGE on an existing (n, v) pair. --dry-runperforms the plan reads only and asserts a write count of 0.- Every write is batched at 500 rows (
_batchedprecedent). - Nothing is deleted, so a bad run is repaired by editing the vocabulary and re-running;
a full rollback is a single
MATCH (t:Thread) DETACH DELETE t(documented in the script docstring, not implemented as a flag).
3. Expected results (dry-run acceptance numbers)
Measured today; the builder's --dry-run must match these within the drift a live graph
accumulates between now and execution.
| Quantity | Expected |
|---|---|
Thread nodes MERGEd |
9,950 |
HAS_THREAD edges |
9,950 |
IN_THREAD — CXPPacket |
25,225 |
IN_THREAD — Message |
556 |
IN_THREAD — ToolCall |
244 |
IN_THREAD — Artifact |
82 |
IN_THREAD total |
26,107 |
BY_VOLK — CXPPacket / Message |
12,195 / 393 = 12,588 |
TO_VOLK — CXPPacket / Message |
15,171 / 432 = 15,603 |
Residue — what stays isolated, and why
Among the four in-scope labels: ZERO. Every isolated CXPPacket/Message/ToolCall/Artifact
carries a thread_id, so every one of the 7,766 gains an IN_THREAD edge and becomes
Project-hub-reachable in two hops. Additionally 18,340 previously-"connected"-but-orphaned
CXPPackets in the ROUTED_TO web become hub-reachable for the first time.
Nodes that gain a Thread but no volk attribution (not isolated — just unattributed): 2,677 CXPPackets, 0 Messages. Breakdown of the unmatched agent strings:
HUMAN— correct; a human is not a Volk and must not get one.TEST:Flaky,TEST:DoubleFlaky,TEST:ArtifactSkip,U:Target,U:Missing,U:Test,test-<uuid>-U:Cx,test-<uuid>-U:PM— test pollution. Correctly unmatched; do not create Volk nodes for these. Candidates for a futuresweep_test_pollution.pypass, out of scope here.U:Do(Documentor) — 43 packet + 40 message endpoints, and NO:Volknode exists. This is a real gap: Documentor is one of the 11 live volks. Report it; do not fix it inside this backfill (creating a Volk node is a different writer's job and would smuggle node creation into an edge-only migration). File it as a follow-up work item.
Remaining isolated nodes graph-wide after this wave, all out of scope:
Specification 178, CloudService 30, Volk 12, Action 10, WAFPillar 5,
Requirement 3, Pattern/File/Task 2 each, Decision/PipelineRun/Test 1 each.
Deliberately deferred
PipelineRun (2,342) and Intent (2,342) also carry thread_id but are already connected.
Linking them would join event threads to pipeline runs using the identical rule — a real
future win, and a one-entry addition to _SOURCE_LABELS. It is not enabled in this wave
(the ratified Option A scope is the isolated event nodes). Record it as a follow-up.
4. Constraint / migration DDL and its application path
Grounded in how constraints actually get applied in this repo:
- Declaration — adding
LABEL_THREADtoCANONICAL_MERGE_KEYSand_CONSTRAINT_NAMESis sufficient.CONSTRAINTSis a comprehension overCANONICAL_MERGE_KEYS, so theCREATE CONSTRAINT thread_id ...statement is generated, never hand-written, andget_all_ddl()picks it up automatically. Do not add a literal DDL string anywhere. - Preview —
python scripts/kg_schema.py apply --dry-runprints the full statement list including the new one, without connecting to any graph. - Application (primary) —
python scripts/kg_schema.py apply. Idempotent (IF NOT EXISTS), andapply()auto-drops any colliding orphan plain index first. - Application (test fleet) —
kg_schema.py apply --uri bolt://10.92.53.13:7687, and likewise.14and.19. This is mandatory, not optional:kg_schema.py checkis CI-wired (REQ-106-006) and will reportThread.idas a missing constraint on every un-stamped instance the moment the label enters the canon. The Component re-scoping (2026-07-19) set the precedent: "migration EXECUTED on primary + all worker replicas". - Self-application —
backfill_thread_links.pycallsapply(repo, <Thread statement only>)at Step 2, so the script is safe to run standalone on a fresh instance without a priorkg_schema.py apply. - Verification —
python scripts/kg_schema.py checkmust exit 0 afterwards (Threadhas zero nodes lackingid, and its constraint exists).python scripts/bootstrap_graphs.py --neo4j --checkmust also stay at exit 0.
No entry is added to migrate_neo4j_canonical.py: Thread nodes are created by this
backfill already carrying their canonical key, so there is never a keyless Thread for
that script's generic backfill to find.
5. Test plan
5.1 tests/knowledge_graph/test_canonical_schema_thread.py — NEW
Structural twin of test_canonical_schema_conversation.py (the Tier-I precedent). Pure,
no infra.
| Test | Asserts |
|---|---|
test_thread_merge_key_is_id |
CANONICAL_MERGE_KEYS[LABEL_THREAD] == "id" |
test_thread_in_canonical_labels |
LABEL_THREAD in CANONICAL_LABELS |
test_ddl_contains_thread_constraint |
the exact CREATE CONSTRAINT thread_id IF NOT EXISTS FOR (n:Thread) REQUIRE n.id IS UNIQUE string is in get_all_ddl() |
test_thread_entity_id_shape |
thread_entity_id("proj", "t1") == "proj:t1" |
test_thread_entity_id_rejects_empty_parts |
ValueError on ("", "t1") and ("proj", "") |
test_thread_entity_id_rejects_colon_in_project |
ValueError on ("pro:j", "t1") |
test_thread_entity_id_allows_colon_in_thread_id |
"proj:a:b" — the natural id may contain : |
test_thread_vocabulary_is_exposed (the wiring test) |
REL_IN_THREAD == "IN_THREAD", REL_HAS_THREAD == "HAS_THREAD", REL_BY_VOLK == "BY_VOLK", REL_TO_VOLK == "TO_VOLK" all importable from canonical_schema |
5.2 tests/knowledge_graph/test_canonical_schema_completeness.py — EDIT (mandatory)
This file hard-locks the label set and will fail the moment LABEL_THREAD lands. The
builder must, in the same commit:
- add
LABEL_THREADto the import block; - add it to
EXPECTED_CANONICAL_LABELS; - change
test_canonical_label_count_is_70→test_canonical_label_count_is_71(assert len(CANONICAL_LABELS) == 71).
The already-present generic assertions then cover the rest for free:
test_every_merge_key_label_has_a_constraint_name,
test_every_constraint_name_label_has_a_merge_key,
test_constraint_names_are_all_unique (proves thread_id collides with nothing),
test_ddl_has_exactly_one_constraint_per_label,
test_every_constraint_references_its_canonical_key.
5.3 tests/scripts/test_backfill_thread_links.py — NEW
Real Neo4j (no mocks, no skips), scoped to test-only labels (_ThreadTestEvent,
_ThreadTestHub) and a unique per-test project slug, so the shared instance's real
labels are never touched — the exact discipline test_canonical_schema.py's scoped
merge_keys seam and migrate_neo4j_canonical's scoped-DDL tests already use.
Function-scope fixtures; env= injection, no monkeypatch; teardown deletes the test-only
labels.
Survivor cases (the backfill did the right thing):
| Case | Assertion |
|---|---|
single event with thread_id |
one Thread exists with id == f"{project}:t1", thread_id == "t1", project == project; one IN_THREAD edge |
two events sharing a thread_id |
one Thread, two IN_THREAD edges |
| hub edge | (:Project {slug})-[:HAS_THREAD]->(:Thread) exists for every Thread |
sender matching Volk.alias |
one BY_VOLK edge to that Volk |
recipient matching Volk.alias |
one TO_VOLK edge to that Volk |
agent matching only Volk.volk_id |
still resolved (to that node's alias) |
agent matching an alias on node A and a volk_id on node B |
resolves to A — alias precedence |
| an event with no agent fields at all (ToolCall/Artifact shape) | gets IN_THREAD, gets no volk edge, and is not isolated |
Husk cases (the backfill correctly did nothing):
| Case | Assertion |
|---|---|
event with thread_id IS NULL |
untouched, still isolated, no Thread created |
event with thread_id = '' |
untouched, still isolated (the <> '' guard) |
sender = 'HUMAN' (no Volk) |
no BY_VOLK edge — but the node still has IN_THREAD, proving unmatched-agent nodes are not left isolated |
| sender = a nonexistent alias | no edge, no Volk node created (assert :Volk count unchanged) |
Behavioural cases:
| Case | Assertion |
|---|---|
--dry-run |
plan counts equal the real-run counts, and the graph is byte-identical before/after (node + relationship counts unchanged, zero Threads) |
| run twice | second run's node/edge counts are identical to the first's; no duplicate Thread, no duplicate edge |
| missing Project hub | non-zero exit, zero Threads created, zero edges created |
| cross-project isolation | run twice with the same bare thread_id under two different --project values → two distinct Thread nodes. This is the multi-project-invariant proof and the single most important test in the file. |
| vocabulary purity | AST/source scan of scripts/backfill_thread_links.py: no bare "Thread", "IN_THREAD", "HAS_THREAD", "BY_VOLK", "TO_VOLK" string literal — every one comes from a canonical_schema import. This is the ontology-before-import rule made executable. |
6. Builder work breakdown
Ordered. Each step's acceptance criterion must pass before the next begins.
S1 — Declare the vocabulary.
Edit canonical_schema.py: LABEL_THREAD in the Tier-H label block;
CANONICAL_MERGE_KEYS[LABEL_THREAD] = "id" and _CONSTRAINT_NAMES[LABEL_THREAD] = "thread_id"
in the Tier-H update() blocks; REL_HAS_THREAD/REL_BY_VOLK/REL_TO_VOLK in the Tier-H
relationship block with the comment from §1.3; thread_entity_id() next to
conversation_entity_id() with @traces(spec="106", reqs=["REQ-106-003"]). Update the
module docstring's Tier-H sentence to name Thread.
Accept: python scripts/kg_schema.py apply --dry-run | grep "n:Thread" prints exactly one
statement; nothing else in the file changed.
S2 — Fix the completeness drift-lock.
tests/knowledge_graph/test_canonical_schema_completeness.py per §5.2.
Accept: pytest tests/knowledge_graph/test_canonical_schema_completeness.py green (71 labels).
S3 — Write the vocabulary tests (RED first).
tests/knowledge_graph/test_canonical_schema_thread.py per §5.1.
Accept: green against S1's code; would fail without it.
S4 — Stamp the constraint on all four instances.
kg_schema.py apply against primary .11, then --uri for .13, .14, .19.
Accept: python scripts/kg_schema.py check exits 0 on each of the four.
S5 — Write the backfill script.
scripts/backfill_thread_links.py per §2, with @traces(spec="106", reqs=["REQ-106-008"])
on the writer entry point. Vocabulary imported, never literal. plan_* pure-read /
run_* writer / main() split mirroring migrate_neo4j_canonical.py.
Accept: ruff check + black --check clean; scripts/serialize_codebase_ast.py check-deps
adds no undeclared dependency.
S6 — Write the backfill tests (RED first).
tests/scripts/test_backfill_thread_links.py per §5.3, including the cross-project and
vocabulary-purity tests.
Accept: green; every case listed in §5.3 present; no mock/skip/monkeypatch
(scripts/audit_test_purity.py clean).
S7 — Dry-run against the live primary KG.
python scripts/backfill_thread_links.py --dry-run.
Accept: counts match §3 (Threads 9,950; IN_THREAD 26,107; BY_VOLK 12,588; TO_VOLK 15,603;
unmatched-agent histogram containing U:Do). Report the diff against §3 and STOP for
review if any figure moves by more than a few percent — a large move means the graph
changed under us and the design's residue analysis needs re-verification.
S8 — Execute.
python scripts/backfill_thread_links.py.
Accept, all four verified by direct Cypher:
MATCH (n) WHERE NOT (n)--() AND (n:CXPPacket OR n:Message OR n:ToolCall OR n:Artifact) RETURN count(n)→ 0.MATCH (t:Thread) RETURN count(t)→ 9,950, and every Thread has exactly oneHAS_THREADfrom the hub.- A sampled packet is reachable:
MATCH (p:Project {slug:'orchestrata-ultimate'})-[:HAS_THREAD]->(:Thread)<-[:IN_THREAD]-(n:CXPPacket) RETURN count(n)→ 25,225. python scripts/kg_schema.py checkexits 0.
S9 — Re-run for idempotency.
python scripts/backfill_thread_links.py a second time.
Accept: Thread / IN_THREAD / BY_VOLK / TO_VOLK counts are identical to S8.
S10 — Record the residue as follow-up work.
Log, do not fix: (a) U:Do has no :Volk node despite 83 live endpoints; (b) the canon's
LABEL_MESSAGE = "Message" # CXPPacket comment is wrong (§8); (c) PipelineRun/Intent
thread linking deferred (§3); (d) ~2,677 packets attributable only to test-pollution agents.
Accept: four session add-work items filed.
7. STOP conditions for the builder
Stop and report rather than guessing if any of these hold:
- The Project hub query returns 0 or >1 rows.
- Any live
thread_idcontains':'(zero do today) — the id-parsing assumption changes. kg_schema.py checkreports extra constraints onThread(means someone else stamped a different key).- The S7 dry-run's Thread count differs from 9,950 by more than ~5% — re-verify §3's residue analysis before writing.
- Any step needs to CREATE a
:Volk,:Project, or:Sessionnode. This migration creates:Threadnodes and edges. Nothing else. Ever.
8. Ontology drift found while designing (report, do not fix here)
canonical_schema.py Tier H declares LABEL_MESSAGE = "Message" # CXPPacket — inter-agent messages, keyed on message_id. On the live graph Message and CXPPacket are two
distinct labels with different property sets (Message: id/message_id/sender/
recipient/summary; CXPPacket: message_id/from_agent/to_agent/payload/
priority/delivery_status/work_status). CXPPacket — the 25,225-node label, the
largest in the graph — has no canonical entry at all.
This design deliberately does not touch it: giving CXPPacket a canonical identity is its
own ratified migration with its own key/rename decision, and folding it into an edge
backfill would violate the same minimality rule this document is built on. Filed as
follow-up (b) in S10.