Orchestrata Ultimate › Plans › kernel-dispatch-reply-hardening

Kernel Dispatch-Reply HARDENING — Design Addendum (F4 / F5 / F7+F8)

Source: plans/kernel-dispatch-reply-hardening.md at 8a2cb42ad · last changed 2026-08-19 · first committed 2026-08-05 · 539 lines.

Prose pending.

Plan text

Kernel Dispatch-Reply HARDENING — Design Addendum (F4 / F5 / F7+F8)

Status: DESIGN ONLY (RUL-022 — no implementation in this artifact) Parent: plans/kernel-dispatch-reply-design.md (7 packets, all shipped in commit 9914d662 on feat/cxp-direct-transport; app container redeployed on it) Findings: the SET-2 whole-diff reviewer's named follow-ups, filed in the session backlog (context.json deferred_work #393–395; report reports/claude-failure-report-2026-08-05.md "Late addendum 4"):

• F4 — same-thread rounds: refcount memory bind, consume-on-match, bound inbox (absorbs F8 human_inbox growth).
• F5 — zero-terminal-packet paths + background-task shutdown.
• F7 — ReplyRouter eviction + unauthenticated queue minting.

Every as-built claim below was grounded via the AST toolkit (scripts/serialize_codebase_ast.py {symbols,refs,attr-refs,source}), never by hand-reading .py files. Specs are authoritative (RUL-013). Rulings consulted: RUL-002 (no mocks/monkeypatch — injectable seams instead), RUL-003 (no infra-env literal defaults; tunables MAY default), RUL-009 (observed test runs), RUL-012 (done = wired + deployed), RUL-013, RUL-020 (async correctness), RUL-022 (this document), RUL-023 (feature means wired), RUL-024 (pre-existing is a diagnosis, not a disposition).

1. AS-BUILT INVENTORY (post-9914d662, all AST-verified)

1.1 Kernel handler (F4 memory bind, F5 terminal contract)

1.2 Pipeline runner (F4 consume/inbox, F5 produced=False)

1.3 Reply router + servicer (F7)

1.4 Shutdown paths (F5)

1.5 Regression floor (existing tests on these paths — must stay green)

tests/cxp/test_reply_router.py, tests/cxp/test_receive_packets_per_consumer.py, tests/agents/test_kernel_handler_ack_only.py (10), tests/kernel/test_get_last_human_message_thread_filter.py, tests/gateway/test_reply_consumer_addressing.py (3), tests/gateway/test_cxp_send_receipt_check.py (2), tests/gateway/test_kernel_reply_consumer.py (3), tests/gateway/handlers/test_cxp_kernel_dispatch.py, tests/agents/volks/test_memory_context.py, tests/cxp/test_direct_transport*.py.

2. SPEC ANCHORS (RUL-013; ids verbatim from checklists)

• SPEC-001 US-002 (specs/001-kernel/checklists/requirements.md — this story's checklist numbers its reqs REQ-002-NNN): REQ-001-US-002-008 (offline volks/consumers), REQ-001-US-002-019 (delivery acks), REQ-001-US-002-030 (handle queue overflow gracefully), REQ-001-US-002-031..035 (status queued/dispatched/ delivered/acknowledged/failed), REQ-001-US-002-037 (timestamps), REQ-001-US-002-038 (notify sender of status changes), REQ-001-US-002-039 (message tracing across hops).
• SPEC-000 US-004: REQ-000-033..041 — submit_intent/respond contract; this design changes NEITHER signature's public shape (a keyword with a compatible default on get_last_human_message only).
• SPEC-002: REQ-002-US-001-026 — notify sender of permanent failures.
• SPEC-014: REQ-014-001/002 — subscribed clients receive CXP packets as they flow.
• SPEC-032 G243: REQ-032-053/054 — never fabricate a conversational reply; unavailability/failure must be explicit and machine-detectable. Both new terminal packets in this design follow the shipped assistant_status convention and are sent from KERNEL, never a volk alias.
• SPEC-010 (US-010-002 cross-session memory): REQ-010-017 (session/task scope filters on recall) + REQ-010-021 (memories selected by relevance to current context) — the recall path the F4 memory-bind fix keeps alive mid-round.

3. FINDING F4 — same-thread rounds (memory refcount, consume-on-match, bounded inbox)

3.1 Defect statement (three sub-defects, all unlocked by the epic's ack-only change)

Before 9914d662, one thread had at most one in-flight round (the send RPC blocked). Ack-only dispatch made same-thread concurrency the NORMAL case — a user sends a follow-up the moment the ack renders — and three shared structures were never upgraded for it:

F4a — memory bind is last-writer-wins + first-finisher-unbinds. _providers is a plain dict. Round R1: handler binds self._mem (kernel_handler.py:224); submit_intent then OVERWRITES with a fresh MemoryEngine() (pipeline.py:296) that nothing ever unregisters. Round R2 (same thread T, concurrent): binds again. Whichever round finishes FIRST runs unregister_memory(T) (kernel_handler.py:336) and pops the mapping while the other round's volk is mid-LLM — its recall() is best-effort and silently returns "". Failure scenario: M1 arrives (R1, measured 11.5–21.1s round). At t=3s the user sends M2 (R2, response on T). R2 completes at t=8s → unbind. R1's concierge recall at t=12s gets empty context → answers M1 with no memory of the conversation, silently — no error anywhere.

F4b — reply matching is peek-not-consume. get_last_human_message(thread_id=T) (pipeline.py:548-551) returns the newest matching packet WITHOUT removing it, and the respond path reports produced=True unconditionally (kernel_handler.py:234-236) even when it appended nothing (pipeline.py:402-404 unknown-thread early-out; :440-446 reply-to-escalation branch). Failure scenario: R1 emits reply P1 (inbox append at t=10, sunk at t=10). R2's round routes its outcome to a volk escalation — no inbox append — yet produced=True; R2's _build_reply re-matches P1 and re-emits it as the terminal answer to M2: a stale duplicate presented as a fresh reply. The symmetric race (R1 slow to build, R2's P2 lands first → both builds match P2, P2 emitted twice, P1 never) is reachable on the same structures.

F4c (absorbs F8) — human_inbox is unbounded. Three writers (pipeline.py:380, :444, pipeline_session.py:401 — the last also receives terminal qa_complete/compliance_* packets), zero consumers. A long-lived kernel process (the app container runs for weeks) accretes every human-directed packet forever.

FT-correctness hazard named en passant (RUL-024): appends run on asyncio.to_thread/sync-router worker threads while _build_reply's reversed() scan runs on the event loop — mutation during iteration raises RuntimeError; on 3.14t this is a live race today. The fix below removes it (snapshot-scan) rather than stepping over it.

3.2 Chosen design

• D-H1 — Refcounted provider registry (pattern: reference-counted registry). memory/memory_context.py: _providers: dict[str, tuple[Any, int]] — register_memory increments (keeps the NEWEST engine as the provider; the count tracks binders), unregister_memory decrements and pops only at zero (cache purge stays at zero-time). Module function signatures unchanged → zero caller edits; the orchestrator's balanced pair (pipeline_runner.py:756 / :407-409) is naturally a no-op under refcounting.
• D-H2 — Balance the unbalanced binder. submit_intent wraps route_sync in try/finally unregister_memory(thread_id) around its _bind_memory_engine bind — round-scoped, exactly like the orchestrator twin. Safe because _remember uses self._mem directly (kernel_handler.py:171-181); nothing reads the registry after the round. (D-H2 REQUIRES D-H1 first: without refcounting, submit_intent's unbind would strip a concurrent round's binding — the same defect relocated.)
• D-H3 — Consume-on-match (pattern: claim check). get_last_human_message(thread_id: str | None = None, consume: bool = False): when consume=True and a match is found, remove it from human_inbox. Scan over a SNAPSHOT (list(self.human_inbox)), then remove — fixes the FT iteration race for both modes. _build_reply:269 passes consume=True. Each inbox entry now satisfies exactly one wait; a round that matched nothing falls through to F5's no-reply terminal (D-H5) instead of re-emitting history. Default False keeps every other caller byte-identical (REQ-000-039/040 untouched).
• D-H4 — Bounded inbox. human_inbox becomes collections.deque[CXPPacket] with maxlen=human_inbox_max (new PipelineRunner.__init__ tunable, literal default 256 — RUL-003 inverse: a tunable, not an infra endpoint; mirrors ReplyRouter's drop-oldest 256). All verified usage forms (append, reversed, [-1], remove) are deque-native. Annotation twin updated at pipeline_session.py:82. Steady-state size stays near zero once D-H3 consumes matched replies; the cap bounds session-mixin residue (qa_complete etc.). Traces REQ-001-US-002-030.

3.3 Alternatives considered

4. FINDING F5 — zero-terminal-packet paths + background-task shutdown

4.1 Defect statement

The parent design's D3 terminal contract — "one terminal packet per accepted intent" — has three verified holes; each ends an ACKED (accepted: True) round with the sender hearing NOTHING, violating REQ-001-US-002-035/038 and REQ-002-US-001-026:

1. produced=False — submit_intent returns None (Prolog may_execute deny, pipeline.py:349-352) → _process_in_background returns without sinking (kernel_handler.py:322). (The parent design consciously kept no-packet semantics here for the queued-for-offline case; the reviewer's F5 supersedes that call: once the ack promises a background round, silence is indistinguishable from loss. This addendum AMENDS D3.)
2. produced=True, no match — _build_reply finds no inbox packet for the thread → reply_packet={} → if reply_packet: skips the sink (kernel_handler.py:326-328). Reachable today via respond's unknown-thread early-out (pipeline.py:402-404) and its reply-to-escalation branch (:440-446); D-H3 (consume-on-match) makes this path MORE frequent by design — F5's terminal must land before or with F4's consume packet (sequencing constraint, §7).
3. Shutdown cancellation — nothing cancels/drains _bg_tasks (runtime_main.py:86-88 stops only the server; cli_main.py:332 likewise); pending rounds are cancelled by asyncio.run teardown, and CancelledError (a BaseException) escapes the except Exception arm (kernel_handler.py:331) — no failure packet, mid-round work silently lost, "Task was destroyed but it is pending" noise. Failure scenario: this is the DEPLOY-TIME NORM, not an edge — every deploy/orchestrata-test/ kernel restart (RUL-012's own acceptance step) cancels in-flight rounds; the UI thread that was acked waits forever across the restart.

4.2 Chosen design

• D-H5 — Total terminal function (pattern: template method over the round + a shared terminal-packet builder). _process_in_background guarantees exactly one sink() on every exit path:
  • genuine reply (unchanged), or
  • intent_failed on exception (unchanged), or
  • NEW intent_no_reply terminal when produced=False OR no inbox match: built by _build_no_reply_packet, sharing a new private _build_terminal_packet(thread_id, subject, delivery_status, payload, reply_to) core with _build_failure_packet (extraction refactor — one dict shape, two thin wrappers; both keep funneling addressing through _stamp_reply_to). Shape: from_agent="KERNEL", to_agent="HUMAN", subject="intent_no_reply", delivery_status=DeliveryStatus.DELIVERED (the round COMPLETED; there is simply no reply — not a failure), timestamp (REQ-001-US-002-037), payload {"text": "[automated notice] the assistant completed processing without a reply.", "failure": false, "assistant_status": "no_reply"}. G243-honest: machine-detectable, never conversational, never a volk alias (REQ-032-053/054 convention).
• D-H6 — Graceful drain (pattern: explicit lifecycle/disposable — DirectCXPServer.stop:170-196 is the in-repo drain-discipline precedent). _KernelPacketHandler.shutdown(grace: float | None = None): asyncio.wait(self._bg_tasks, timeout=grace); stragglers cancel()ed and awaited. New except asyncio.CancelledError arm in _process_in_background: sink _build_failure_packet(thread_id, "kernel shutting down", reply_to) then RE-RAISE (RUL-020: never swallow cancellation). Grace default resolved at call time: os.environ.get("KERNEL_BG_DRAIN_GRACE_S", "30") — a tunable with a permitted literal default (RUL-003 inverse); 30s covers the measured 11.5–21.1s round and sits inside systemd's stop window.
• D-H7 — Wiring (RUL-023): drain BEFORE transport stop at BOTH boot sites, so still-connected consumers can receive the final terminals:
  • agents/runtime_main.py: signal handler becomes asyncio.ensure_future(_graceful_stop(handler, server)) — a module helper that awaits handler.shutdown() then server.stop(). run_serve_lifecycle's finally remains the idempotent backstop (unchanged).
  • kernel/cli_main.py: _CXPKernelServerLifecycle composes the handler (constructor gains the reference at the :655-662 boot site); its stop() awaits handler.shutdown() then server.stop(grace=1).

4.3 Alternatives considered

5. FINDING F7 — ReplyRouter eviction + unauthenticated queue minting

5.1 Defect statement

ReplyRouter (cxp/reply_router.py:44-138) has exactly five methods and no eviction anywhere; queue_for creates on demand and entries live forever. Three exposures:

1. Rotating consumers (the normal fleet mode). Every gateway construction mints a fresh HUMAN@<12hex> (gateway/server.py:382); every gateway restart and every test-spawned gateway strands its predecessor's queue — bounded per-queue at 256 packets (drop-oldest) but unbounded in KEY COUNT on the long-lived app-container kernel.
2. Unauthenticated drain-side minting. receive_packet:307 and subscribe:330 call queue_for(request.agent_id) for ANY caller on the insecure fleet port — one novel agent_id per call mints one dict entry, an arbitrary-rate growth vector requiring no credentials.
3. Spoofable reply_to / drain-steal. Nothing authenticates a consumer id; receive_packet(agent_id=<victim>) drains the victim's queue. Mitigated in practice by unguessability (48-bit random suffix); the legacy "HUMAN" key is predictable BY DESIGN (back-compat). Failure scenario: weeks of suite runs against 10.93.53.13 leave thousands of dead keyed queues (monotonic RSS growth, growing dict); any buggy/hostile fleet client can accelerate it without bound.

5.2 Chosen design (pattern: idle-TTL cache eviction with drain leases; lazily swept — deliberately NO background task, applying F5's own lesson)

• D-H8 — _QueueEntry value object (<<dataclass>> in cxp/reply_router.py): queue: asyncio.Queue, last_activity: float (monotonic), active_drains: int. Replaces bare Queue values in _queues. Router gains an injectable clock: Callable[[], float] ctor param (default time.monotonic) — the RUL-002-compliant test seam (no monkeypatching time).
• D-H9 — Drain lease (pattern: RAII lease). open_drain(consumer_id) -> asyncio.Queue / close_drain(consumer_id); receive_packet and subscribe wrap their streaming loops in try/finally. active_drains > 0 makes an entry eviction-immune; open/close and every route() refresh last_activity. queue_for remains (route side + back-compat) and also refreshes activity.
• D-H10 — Lazy sweep in route(), at most once per sweep_interval_s (tunable, default 60): evict entries with active_drains == 0 and idleness > idle_ttl_s (tunable, default 900) — traced warning per eviction (key + dropped-packet count). Secondary hard cap max_consumers (tunable, default 1024): if still over cap after the TTL sweep, evict oldest-idle INACTIVE entries until at cap (traced). Safety argument: a live gateway polls in kernel_reply_timeout_s-second windows (default 5, gateway/config.py:173), so its last_activity is always ≤ one window old — TTL cannot bite a live consumer, and under cap pressure a flood's entries are strictly older-idle than any live consumer's. Tunables are ctor params on ReplyRouter (pure DI); the default-construction site (CXPServicer.__init__:159) resolves them at call time via os.environ.get("CXP_REPLY_QUEUE_TTL_S", "900") / ..._SWEEP_S,..._MAX_CONSUMERS — tunables with permitted literal defaults (RUL-003 inverse), env-adjustable for the deploy acceptance probe.
• Minting guard = boundedness, not authn. Drain-side minting REMAINS allowed — the shipped contract explicitly promises connect-before-first- reply (queue_for docstring). The guard is that minted-but-never-routed, undrained queues are precisely the first evicted (inactive + idle + empty) and total keys are hard-capped. Full authentication of the kernel gRPC surface (token handshake) is a SPEC-004/deployment-posture decision — user fork §9.2; unguessable ids are the interim capability.

Sanctioned behavior change (explicit): "a reply routed before a consumer starts draining still arrives once it does" now holds only within idle_ttl_s; a consumer absent longer loses its queue and contents (traced). Everything else — precedence, per-queue drop-oldest 256, legacy "HUMAN" key — is unchanged (§8).

5.3 Alternatives considered

6. CONTRACT DIAGRAM

plans/contracts/kernel-dispatch-reply-hardening.mmd — changed surfaces only, consistent with §§3–5. patternRole map for gen-skeleton: _QueueEntry=dataclass (fully generatable) · open_drain/close_drain/ _sweep=leaf (builder-written policy) · shutdown/_graceful_stop=leaf · _build_terminal_packet/_build_no_reply_packet=leaf · register_memory/unregister_memory refcount + get_last_human_message consume + submit_intent balance = edits to existing code (builder tasks, not generated).

7. BUILD PACKETS (each ≤ one ~5-min builder brief; tests-first inside each)

Dependency spine: P-H1 → P-H2 (refcount before balance — see D-H2) and P-H4 → P-H3 (no-reply terminal before consume-on-match — see §4.1.2). Groups are otherwise independent and individually shippable.

Group H-A — F5 terminal contract (lands first: F4 depends on it)

Group H-B — F4 same-thread rounds

Group H-C — F7 router eviction

Runner discipline (RUL-009, 5-min cap): scripts/run_tests.py --affected / targeted modules only, live-observed (observe_pytest attach), kernel required — never --skip kernel (RUL-010). No mocks/monkeypatch anywhere (RUL-002): slow/failing volks are REAL registered _PluginVolkAdapter handlers (kernel/pipeline.py:50-94, SPEC-015 production feature); time is an injected clock; registries are inspected, never patched. Real infra per env/test.env: Redis 10.93.53.11, graph 10.93.53.10, kernel container 10.93.53.13.

8. COMPAT CONSTRAINTS (the 9914d662 reply contract keeps working)

9. WIRING & DEPLOYMENT (RUL-012 / RUL-023)

Every new element's production importer, named:

Deployment: the test-fleet kernel + gateway run as systemd services in the orchestrata-test app container at 10.93.53.13 (deploy/orchestrata-test/) — restart = deploy; each group ends with a container redeploy. Note: the F5 drain (P-H6) changes what the restart itself does — the redeploy that ships Group H-A is the first restart that drains in-flight rounds instead of killing them.

Acceptance probes (post-redeploy, per group):

• H-A (F5): start a chat round against the deployed kernel, restart the kernel service mid-round → the gateway's reply stream carries an intent_failed ("shutting down") or the genuine reply BEFORE the old process exits; zero "Task was destroyed" lines in the journal.
• H-B (F4): paired same-thread probe — send M1, then M2 two seconds later on the same thread → exactly two terminal packets, distinct, no duplicate of M1's reply; kernel journal shows no empty-recall degradation for the slower round.
• H-C (F7): with CXP_REPLY_QUEUE_TTL_S=30 on the service, mint 50 throwaway ids via receive_packet(agent_id=...), wait TTL+sweep, then confirm via traced eviction log lines that the entries are gone while the live gateway's consumer id survives and its replies still deliver 2/2.

10. NON-GOALS & USER-DECISION FORKS

Non-goals (MVP discipline, unchanged from parent unless noted): ACKNOWLEDGED lifecycle wiring (REQ-001-US-002-034); UI rendering of intent_failed/ intent_no_reply; multi-kernel fan-in; per-round correlation ids (§3.3); proto-level reply_to field mapping (metadata stays the carrier).

Forks needing the user (recommendations attached):

1. Eviction policy constants — idle-TTL 900s + max_consumers 1024 + sweep 60s (RECOMMENDED, §5.2 safety argument) vs a pure LRU cap vs connection-lifetime queues. All are ctor/env tunables, so the choice is revisable without a code change.
2. Kernel gRPC consumer authentication — RECOMMEND deferring: ids are 48-bit capabilities on a fleet-internal insecure port; a token-handshake RPC is a SPEC-004 proto change. Revisit if the port ever leaves the fleet subnet.
3. intent_blocked distinct terminal for Prolog-denied submit_intent — RECOMMEND deferring (needs a REQ-000-033..041 signature change); intent_no_reply covers the terminal contract today.

11. AMENDMENT (2026-08-05) — F-3 accept-during-drain window

11.1 Defect statement (reviewer finding F-3, filed at the F5 landing)

_KernelPacketHandler.shutdown(grace) snapshots self._bg_tasks and drains the snapshot — but the accept path (__call__) keeps spawning NEW rounds for packets arriving during the (default 30s) drain window. Post-snapshot rounds are never awaited by the drain, and their terminal packets sink into a transport that _graceful_stop/_CXPKernelComponent.stop closes the moment the drain returns. Residual of the F5 hole: terminal totality holds per round, but the terminal has no live consumer.

11.2 Chosen design — D-H11: handler-seam _draining flag (pattern: gate

at the single accept seam; transport-agnostic)

• _KernelPacketHandler gains _draining: bool = False. shutdown() sets it True as its FIRST action — before the task snapshot — so accept-reject and drain-snapshot cannot interleave a stray spawn. Both __call__ and shutdown execute on the kernel's event loop, so plain attribute visibility suffices — no lock (document in the docstring).
• Accept path: when _draining and the packet would enqueue background work, DO NOT spawn. Build and return a SYNCHRONOUS rejection receipt via the existing _build_terminal_packet core: failure=true, assistant_status="draining", error code KERNEL_DRAINING, original thread_id/reply_to preserved so the still-open reply stream delivers it. The caller-visible contract: a client that races a deploy gets one explicit, immediate terminal — retry against the restarted kernel is the client's move. Non-spawning subjects (pure acks/status reads) pass through unchanged.
• Ordering note: this AMENDS nothing in F5's drain-then-stop sequence; the alternative (server.stop(grace)-first semantics) was REJECTED because it reverses the ratified ordering, kills in-flight streams mid-drain, and needs per-transport half-open behavior (grpc.aio AND DirectCXPServer), where the flag is one mechanism at the one shared seam.

11.3 Build packet

Executes AFTER the F4+F7 certify+ship cycle — not folded into it.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks