Kernel Dispatch-Reply Hardening (F4 / F5 / F7)

Status: design RATIFIED (commit d47388ed, RUL-022 design-before-build). Group H-A — the F5 terminal-totality group — is SHIPPED: commit e949984a, pushed and deployed to the test-fleet app container; certifying gate11 rc=0, 30,790 / 0 / 0 / 0 at a 265 s test phase (new best). F4 and F7 remain designed-not-built.

Digest of plans/kernel-dispatch-reply-hardening.md (+ contract plans/contracts/kernel-dispatch-reply-hardening.mmd). Parent: plans/kernel-dispatch-reply-design.md — the 7-packet epic shipped in 9914d662 that closed 13.9–21.1 s inline ack latency and the 50/50 reply loss from the single unkeyed reply queue. The adversarial whole-diff review of that epic filed three finding groups; this addendum designs their fixes.

F5 — terminal totality (landed FIRST; F4 depends on it) — SHIPPED e949984a

Contract: every accepted intent sinks exactly one terminal packet — reply | intent_no_reply | intent_failed. This AMENDS the parent design's D3, which consciously allowed no-packet on produced=False; once the ack promises a background round, silence is indistinguishable from loss.

Three verified holes, each ending an ACKED round with the sender hearing nothing:

1. produced=False — Prolog may_execute deny makes submit_intent return None; the background round returned without sinking anything.
2. produced=True but no inbox match — _build_reply found nothing and skipped the sink (reachable via respond's unknown-thread early-out and its reply-to-escalation branch).
3. Shutdown cancellation — nothing drained the handler's _bg_tasks; CancelledError (a BaseException) escaped the except Exception arm, so every kernel restart (the deploy-time NORM) killed in-flight rounds silently.

Design: a total terminal function in _process_in_background (new intent_no_reply terminal via _build_no_reply_packet, sharing a _build_terminal_packet core with _build_failure_packet; G243-honest — from_agent="KERNEL", machine-detectable assistant_status="no_reply", never conversational); handler.shutdown(grace) draining _bg_tasks (KERNEL_BG_DRAIN_GRACE_S, default 30 s, covers the measured 11.5–21.1 s round); an except CancelledError arm that sinks a "kernel shutting down" failure terminal then RE-RAISES; and drain-before-transport-stop wiring at both boot sites (agents/runtime_main.py, kernel/cli_main.py).

All three packets LANDED in e949984a: P-H4 (terminal extraction + intent_no_reply + always-sink; 28/28), P-H5 (shutdown(grace) drain), P-H6 (wiring — runtime_main._graceful_stop on SIGTERM/SIGINT and _CXPKernelComponent.stop in cli_main; drain before transport stop, proven by flip-tested ordering). Adversarial-review must-fixes folded in: defensive KERNEL_BG_DRAIN_GRACE_S parse, try/finally at both drain sites, bounded post-cancel wait (5 s abandon). Named follow-ups filed from that review: F-3 accept-during-drain window, F-4 uncancellable in-thread rounds (plus a reproduced drain-cleanup deadlock when a round swallows cancellation), F-6 assistant_status not surfaced top-level.

F4 — same-thread round safety (designed, not yet built)

Ack-only dispatch made same-thread concurrency the normal case; three shared structures were never upgraded for it:

• F4a — memory bind was last-writer-wins + first-finisher-unbinds: the fastest round's unbind stripped a concurrent round's memory mid-LLM → silent empty recall. Fix: refcounted provider registry in memory/memory_context.py (pop only at zero; module signatures unchanged) + balancing submit_intent's previously unbalanced bind with try/finally unregister (refcount MUST land first).
• F4b — reply matching was peek-not-consume: get_last_human_message could hand the same inbox packet to any number of waits → stale-duplicate replies. Fix: consume-on-match (consume: bool = False keyword; _build_reply passes True; snapshot-scan closes a live 3.14t iterate-while-append race on the inbox). A round that matches nothing now falls through to F5's intent_no_reply — which is why F5 sequences first.
• F4c (absorbs F8) — unbounded human_inbox: three writers, zero consumers, on a kernel that runs for weeks. Fix: deque(maxlen=human_inbox_max) (ctor tunable, default 256, drop-oldest — mirrors the ReplyRouter bound).

Packets: P-H1 (refcount) → P-H2 (balance), P-H3 (consume, after P-H4), P-H9 (bound) — F5 has landed, so this group is now unblocked.

F7 — ReplyRouter hygiene (designed, not yet built)

ReplyRouter had no eviction at all: every gateway restart strands its HUMAN@<12hex> queue forever, and any client on the insecure fleet port mints a queue per novel agent_id (unauthenticated, uncapped key growth).

Design (pattern: idle-TTL cache eviction with drain leases; deliberately NO background sweeper — a sweeper task would recreate exactly the untended-lifecycle class F5 exists to fix):

• _QueueEntry dataclass (queue, last_activity, active_drains) + injectable monotonic clock (the RUL-002 test seam — no time monkeypatching).
• RAII drain leases (open_drain/close_drain wrapped in try/finally by receive_packet/subscribe) — an entry with an open drain is eviction-immune.
• Lazy sweep inside route(), at most once per sweep interval: evict entries idle beyond TTL with zero drains (traced, with dropped-packet count); hard cap on total consumers as flood backstop. A live gateway polls every ~5 s, so TTL can never bite a live consumer.
• Sanctioned behavior change: "route before drain still delivers" now holds only within the idle TTL.

Packets: P-H7 (router) → P-H8 (servicer lease integration, both transports).

Sequencing spine

F5 before F4 (P-H4 → P-H3: consume-on-match makes no-match rounds more frequent, so the no-reply terminal must exist first) and P-H1 before P-H2 (refcount before balancing, else the unbind defect just relocates). Groups H-A/H-B/H-C are otherwise independent and individually shippable; each group ends with an app-container redeploy + a live acceptance probe. The H-A redeploy (e949984a, done) was the first restart that drains in-flight rounds instead of killing them.

As-built: F4 + F7 (groups H-B, H-C) — 2026-08-05

F4 (P-H1/P-H2/P-H3), all landed:

• memory_context.py provider registry is REFCOUNTED (dict[str, tuple[Any, int]]): register increments with newest-engine-wins, unregister decrements and only at zero pops the provider and purges the recall/summary caches. Public signatures frozen — all three production binders untouched.
• submit_intent balances its memory bind in try/finally — mandatory under refcounting (an unbalanced round would leak a refcount per intent).
• Reply claim-check: get_last_human_message(thread_id=None, consume=False) scans a SNAPSHOT of the inbox (free-threaded iterate-while-append race fix); the kernel handler's _build_reply — sole reply-lookup site for sync AND background paths — passes consume=True, so a second same-thread round can never re-emit the first round's reply. human_inbox is a bounded deque (human_inbox_max=256, mirroring the router's drop-oldest). CLI polling readers stay on peek semantics.

F7 (P-H7), landed:

• _QueueEntry (queue, last_activity, active_drains) replaces bare queues; open_drain/close_drain RAII leases in receive_packet/subscribe make live consumers eviction-immune; route() runs a lazy sweep (≤1 per sweep_interval_s): TTL pass, then max_consumers hard-cap pass evicting oldest-idle inactive — every eviction traced with key + dropped-packet count. No background sweeper by design. Injectable clock ctor param is the sanctioned test seam (_ManualClock in tests — a manually-advanced value provider, not an infrastructure double).
• Sanctioned behavior change: "a reply routed before its consumer drains still arrives" now holds only within idle_ttl_s. Precedence, per-queue drop-oldest 256, legacy HUMAN key unchanged. The unauthenticated queue-minting growth vector is closed by boundedness, not authn.
• Coverage at landing: reply_router.py 99 %, grpc_server.py 96 %; flip-proof: an eviction-disabled router (idle_ttl_s=1e12) fails the TTL test.

Rider fixes in the same landing: narrow PluginVolkRegistryProtocol (SPEC-015) replacing the semantically-wrong full-registry annotation on PipelineRunner.plugin_volk_registry (7 pyright errors → 0); cli_main status health URL env-resolved (was literal localhost:8000; health lives on GATEWAY_HTTP_PORT/8766); fleet session.open PermissionError fixed via writable SESSION_ROOT through LAST-listed EnvironmentFile overrides (/etc/orchestrata/*-overrides.env — measured: EnvironmentFile= beats Environment=, the old kernel-unit comment claimed the opposite and its CXP_GRPC_HOST=0.0.0.0 was dead).

Adversarial-review round (2026-08-05, folded in before landing): the consume path's unlocked snapshot was itself the crash site — deque iteration raises RuntimeError on concurrent append and deque.remove raises IndexError mid-__eq__-scan (reachable GIL-on; failed the claim-check open). Fixed with one _human_inbox_lock over scan+match+remove and all three append sites; the forced-ValueError test became a real-threads race proof (2 writers + 2 consumers, exactly-once claims). Router sweep now runs on the mint path too and bypasses its interval under cap pressure (rotating drain-side minting could otherwise flood between ticks). Memory-balance tests observe the registry mid-round (bite even where memory infra is down); _env_float/_env_int gained the environ= seam; the gateway unit's own dead Environment=GATEWAY_HOST fell to the same overrides-file mechanism. Lesson recorded: sweep by seed-vs-assignment counting, not string presence — a reference in one test masked five missing host locks.

Amendment ratified (design §11): F-3 accept-during-drain → D-H11 handler-seam _draining flag + synchronous KERNEL_DRAINING receipt; packet P-H10, next.

New investigation (P1): cxpt native SIGSEGV in Server_dealloc — third py3.14t crash signature, first inside our own extension; repro = batch-under-load; tied to the F-4 cooperative-cancel work.

Recorded decision forks (design §10)

1. Eviction constants — ADOPTED as env-tunables with literal defaults (CXP_REPLY_QUEUE_TTL_S=900, sweep 60 s, max_consumers 1024; RUL-003 inverse: tunables may default) — revisable without a code change.
2. Kernel gRPC consumer authentication — DEFERRED: consumer ids are 48-bit capabilities on a fleet-internal port; a token handshake is SPEC-004 territory. Revisit if the port ever leaves the fleet subnet.
3. Distinct intent_blocked terminal for Prolog-denied intents — DEFERRED: needs a submit_intent outcome-signature change (SPEC-000 REQ-000-033..041); intent_no_reply satisfies the terminal contract today.

Acceptance run (2026-08-05) — H-B/H-C PASS + the env split-brain lesson

• H-B PASS (fleet): two same-thread intents → two DISTINCT non-empty terminal message_ids, exactly one terminal per round, correct reply_to routing. Producer fix: _build_terminal_packet stamped "" — now uuid4 (commit 4eda5f0c).
• H-C PASS (fleet, end-to-end): 50 drain-minted throwaway consumers under CXP_REPLY_QUEUE_TTL_S=30 / SWEEP_S=5 overrides → 51 traced reason=idle-ttl eviction WARNINGs in the kernel journal, arriving in 5 s sweep-cadence batches, dropped=0; a live consumer minted afterwards round-trips untouched. Overrides restored to CXP_GRPC_HOST=0.0.0.0 after.
• Root cause of the day (memorize this): every earlier "fleet" anomaly — blank message_id, "cxpt client hangs", "grpc.aio dies pre-SETTINGS" — was ONE defect: .envrc loaded only .env (the PROD-base layer, CXP_GRPC_HOST=localhost, load-bearing for the prod host user-units), never the env/test.env overlay. Probes therefore talked to the PROD kernel user-unit on localhost:50053 (grpc.aio, pre-fix bytecode from its 08:28 import). cxpt and grpc.aio are structurally non-interoperable (cxpt = raw length-prefixed frames, no HTTP/2) — matched transport pairs only. .envrc now layers env/${ORCHESTRATA_ENV_PROFILE:-test}.env after .env; independent investigation confirmed the fleet's loaded bytecode carried the fix all along. Prod kernel restarted onto the fixed producer 15:37; 29 probe pollution nodes swept from prod orchkg.
• Open decisions surfaced: claude_cli OAuth token revoked (401) on the fleet provider chain (standing item); prod/test port collision — prod kernel wildcard-binds 0.0.0.0:50053 on the host while the test fleet uses the same port, so any unprofiled client reaches PROD (per-env ports vs loopback bind vs probe-side subnet assertion); prod user-units load .env alone instead of declaring ORCHESTRATA_ENV_PROFILE=prod + overlay.
• Fleet units now run PYTHONUNBUFFERED=1 (print diagnostics were block-buffered under systemd → journal-silent failed rounds).

Repo pointers

• Design: plans/kernel-dispatch-reply-hardening.md · contract: plans/contracts/kernel-dispatch-reply-hardening.mmd · design commit d47388ed · F5-group implementation commit e949984a · F4+F7 ship commit 6b6a5f88 · acceptance-fixes commit 4eda5f0c
• Parent epic: plans/kernel-dispatch-reply-design.md · commit 9914d662
• Code under change: src/orchestrata_ultimate/agents/kernel_handler.py, kernel/pipeline.py, memory/memory_context.py, cxp/reply_router.py, cxp/grpc_server.py, agents/runtime_main.py, kernel/cli_main.py