Orchestrata Ultimate › Plans › volk-comms-heartbeat-design

Volk↔Volk Messaging + Orchestrator↔Volk Heartbeat/Status Awareness — Design

Source: plans/volk-comms-heartbeat-design.md at 8a2cb42ad · last changed 2026-08-19 · first committed 2026-08-05 · 443 lines.

Prose pending.

Plan text

Volk↔Volk Messaging + Orchestrator↔Volk Heartbeat/Status Awareness — Design

Status: DESIGN ONLY (RUL-022 — no implementation in this artifact) Branch: feat/cxp-direct-transport · Date: 2026-08-05 · Author: analyst (design phase) Epic: Communication: volk<->volk messaging + orchestrator<->volk with real-time status/heartbeat awareness

Every as-built claim below was grounded via the AST toolkit (scripts/serialize_codebase_ast.py {symbols,refs,source}), never by hand-reading .py files. Specs are authoritative (RUL-013). Rulings consulted: RUL-002 (no mocks), RUL-003 (env from profile, injected seams, no endpoint defaults), RUL-011 (ports from env), RUL-020 (async/sync contract), RUL-022 (design before build), RUL-023 (feature means wired).

1. AS-BUILT INVENTORY

1.1 Volk↔volk messaging path (message_volk → Postman → handler)

1.2 Existing heartbeat (activity-based, dispatch-time)

refs record_heartbeat — exactly two production write sites, both activity-based:

1. orchestrator/intent_router.py:296-312 (inside IntentDispatcher.dispatch): after a successful route_sync delivery, getattr(self._registry, "record_heartbeat", None) → call if callable, swallow all failures. In-code comment states verbatim: "IVolkRegistry does not declare record_heartbeat, so guard for its presence" and "A periodic push-heartbeat from volks themselves is a future extension." @traces(spec="001", reqs=["REQ-001-018"]) + @traces(spec="003", story="US-004", reqs=["REQ-003-US-004-010"]).
2. orchestrator/pipeline_runner.py:1004 (inside PipelineRunner._run_stage): direct self.volk_registry.record_heartbeat(stage_spec.alias) after a successful stage dispatch/response, in a try/except debug-log guard.

Registry side (kernel/volk_registry.py):

• VolkRegistry.record_heartbeat (1021-1053): sets last_heartbeat=now, transitions to VolkStatus.ACTIVE under self._lock, best-effort Redis persist (@traces reqs=["REQ-001-018","REQ-001-017"]).
• VolkRegistry.get_volk_health (968-1019): computes heartbeat_age_seconds; when age > heartbeat_timeout_seconds (ctor default 60, volk_registry.py:364) it lazily flips the volk to INACTIVE and reports is_healthy=False (@traces reqs=["REQ-001-018"]). Returns VolkHealth (volk_registry.py:198).
• Constructor already exposes an observer seam: on_registration: Callable[[VolkRegistration], None] | None (364).
• add_notification_handler (762): zero imports, zero uses in the entire tree (per refs).

Interface landscape (the duck-typing root cause):

• plugin/interfaces.py:21-54 — IVolkRegistry(Protocol): only register_volk + unregister. This is the type annotation on IntentRouter.__init__ and IntentDispatcher.__init__ (intent_router.py:17,74,255).
• core/protocols/decoupling.py:63-103 — VolkRegistryProtocol(Protocol): DOES declare record_heartbeat (91) and get_volk_health (94), but is imported only by core/interfaces, core/protocols/__init__, and kernel/pipeline.py — not by intent_router or tool_registry, which duck-type via getattr (intent_router.py:83-97, 305-307; tool_registry.py:1242-1245).

Health-aware routing consumer: IntentRouter._is_volk_healthy (intent_router.py:82-97)

• _prefer_healthy_candidates (100) deprioritize unhealthy candidates (@traces spec="003" story="US-004" reqs=["REQ-003-US-004-006","REQ-003-US-004-019"]).

Existing test: tests/orchestrator/test_dispatch_heartbeat.py — 3 tests, all @traces(spec="001", story="US-001", reqs=["REQ-001-018"]): test_dispatch_records_heartbeat_for_registered_volk, test_dispatch_to_unhandled_target_fails_without_heartbeat, test_dispatch_heartbeat_reflects_recent_activity.

1.3 Status surfaces (two DISJOINT ones)

Surface A — heartbeat health (VolkRegistry): VolkHealth via get_volk_health; consumed by _is_volk_healthy (routing) and the two tools:

• get_volk_status tool (agents/tool_registry.py:1227-1276, ToolExecutor._get_volk_status): getattr-probes get_volk_health/is_registered on the bound registry, formats status/healthy/heartbeat-age/error. Degrades to "no registry bound to this run" when _volk_registry is None.
• message_volk health pre-warning (1189-1198).

Surface B — roster status (kernel PipelineRunner): kernel/pipeline.py:440-461 PipelineRunner.status_report() (@traces spec="000" story="US-002" reqs=["REQ-000-023"]) reads volk.status.value + current_thread-derived busy off the in-process BaseAgent instances — it never consults VolkRegistry heartbeat health. Flow: gateway VolkHandler._live_status (gateway/handlers/volk.py:219-232, @traces spec="014" reqs=["REQ-014-016","REQ-014-017"]) → kernel client get_status (both transports) → cxp/grpc_server.py:237-276 → kernel packet handler _status_reply (agents/kernel_handler.py:119-142) → status_report().

Push-side surfaces that exist but are not fed by registry transitions:

• VolkHandler.publish_lifecycle_event (gateway/handlers/volk.py:450) — fired only from gateway-initiated handle_register/handle_deregister (volk.py:364, 403).
• Gateway kernel-events forwarder: gateway/server.py:637 _init_kernel_events_forwarder (tested by tests/gateway/test_kernel_event_forwarder.py, tests/kernel/test_cli_main_kernel_events.py).

Production registry constructors (refs VolkRegistry, non-test): agents/runtime.py:71 (VolkRuntime), kernel/cli_main.py:252, 428 (kernel boot), kernel/router.py:188, orchestrator/pipeline_runner.py:190/207, plugin/sdk.py:60.

Unrelated heartbeats (do not conflate): kernel/performance/cluster.py:372 _heartbeat_loop (perf-cluster nodes), gateway/session.py:111 update_heartbeat (WS connection liveness).

2. SPEC CONTRACT REVIEW (specs are authoritative — RUL-013)

SPEC-001 (kernel) — specs/001-kernel/checklists/requirements.md

US-001 "Volk Registration and Discovery":

• REQ-001-009 — Registration notifies other Volks of new agent.
• REQ-001-014 — Volk can subscribe to registry updates.
• REQ-001-015 — Discovery returns Volk metadata.
• REQ-001-017 — Discovery handles partial failures gracefully.
• REQ-001-018 — Discovery provides health status for each Volk. (The anchor requirement of this epic; already traced by the dispatch-heartbeat code+tests.)
• REQ-001-022 — Deregistration notifies dependent Volks.

US-002 "Message Routing and Dispatch" (NOTE: SPEC-001's checklist numbers this story's requirements REQ-002-NNN; they are defined in specs/001-kernel/checklists/requirements.md lines 80-130, distinct from the postman checklist's REQ-002-USn/FRn-NNN series):

• REQ-001-US-002-008 — Handle routing to offline Volks.
• REQ-001-US-002-009 — Handle routing to unknown recipients.
• REQ-001-US-002-019 — Provide delivery acknowledgments.
• REQ-001-US-002-031..035 — Track message status: queued/dispatched/delivered/acknowledged/failed.
• REQ-001-US-002-036 — Provide message status query API.
• REQ-001-US-002-038 — Notify sender of status changes.

SPEC-002 (postman) — specs/002-postman/checklists/requirements.md

• REQ-002-US-001-011 — Support for NOTIFY message type.
• REQ-002-US-001-026 — Notify sender of permanent failures.
• REQ-002-FR9-007 — /health endpoint.
• (Pub/sub + subscription series REQ-002-US2-, REQ-002-FR6- — existing publish_topic/subscribe_topic tools already serve these; not in scope.)

SPEC-005 (multi-agent) — specs/005-multi-agent/checklists/requirements.md

• REQ-005-US-004-001 — Operator can monitor agent spawning metrics and performance to ensure system health.
• REQ-005-FR8-001 — Performance metrics collection for agent spawn, query, escalation operations.
• REQ-005-AC6-001 — Metrics accessible via monitoring interface.
• REQ-005-FR10-001 — Senior→junior delegation uses standard CXP packets routed through Postman, identical to senior-to-senior communication (constrains this design: one delivery path, no side-channel for status).

Neither spec.md nor the checklists of 001/002/005 mandate a specific heartbeat mechanism (no "push every N seconds" requirement exists — grep -i heartbeat over all three specs returns only REQ-001-018's health-status line). The mechanism is therefore a design choice; the CONTRACT is: health status per volk must be available (REQ-001-018), registry updates must be subscribable (REQ-001-014), senders must learn delivery status (REQ-001-US-002-033/035/038, REQ-002-US-001-026), and unknown/offline recipients must be handled (REQ-001-US-002-008/009).

3. GAP LIST (as-built vs spec, each with evidence)

Candidate (b) from the epic brief is confirmed with a correction: the gap is not "no protocol declares record_heartbeat" (VolkRegistryProtocol does, decoupling.py:91) — it is that the seams are typed against the plugin IVolkRegistry and duck-type around it.

4. CONTRACT DESIGN

4.1 Key decisions

1. One narrow Protocol, composition-first (closes G2). Introduce VolkHealthProbe — a thin <<Protocol>> with exactly the three members the routing/tool seams actually use (record_heartbeat, get_volk_health, is_registered). kernel.volk_registry.VolkRegistry already satisfies it structurally (methods exist at 1022, 969, 703) — zero implementation change, no inheritance; VolkRegistryProtocol remains the wide boundary and is a structural superset. Retype IntentDispatcher._registry, IntentRouter health path, and ToolExecutor._volk_registry as VolkHealthProbe | None; delete every getattr probe. (Twin check performed: no existing narrow health protocol exists to reuse; the wide VolkRegistryProtocol stays untouched.)
2. Roster heartbeat pump, threads not asyncio (closes G1). A HeartbeatPump composed of (probe: VolkHealthProbe, roster: Callable[[], list[str]], tracer) beats for every alias in the live roster on a fixed interval using threading.Thread + threading.Event — matching the codebase's existing thread posture (route_sync via asyncio.to_thread, free-threaded 3.14t). Rationale: volks are in-process BaseAgent instances; "the hosting runtime is up AND the handler is registered" IS the correct liveness signal for them — there is no separate volk process to push from. Busy/failed remains BaseAgent.status's job (surface B). Interval from an injected env mapping, env.get("VOLK_HEARTBEAT_INTERVAL_S", "15") — a tunable, NOT an infra endpoint key, so a literal default is permitted under RUL-003's inverse rule. Remote/plugin volks (gRPC-registered) are OUT of MVP scope (recorded below).
3. Status-change observer on the registry (closes G3a/b). VolkRegistry gains an on_status_change: Callable[[str, VolkStatus, VolkStatus], None] | None constructor seam — the exact precedent is the existing on_registration seam (volk_registry.py:364). Both transition sites (record_heartbeat ACTIVE flip, get_volk_health lazy INACTIVE flip) fire it under the same lock discipline, best-effort (REQ-001-017). No new event framework: the pattern already exists in this class.
4. Enrich the EXISTING polled roster instead of a new RPC (closes G3c). kernel/pipeline.py PipelineRunner.status_report() gains an optional injected health_probe: VolkHealthProbe | None; when present each row adds healthy, heartbeat_age_s, last_heartbeat. The enrichment then flows through the entire existing chain with ZERO new wiring: _status_reply → gRPC/cxpt get_status → gateway VolkHandler._live_status → volk.list. Both transports already carry the payload (dict-shaped volks list).
5. Real-time push via the EXISTING kernel-events forwarder (closes G3d). A tiny VolkStatusEventBridge adapter subscribes to decision-3's callback and emits a volk.status_changed kernel event through the already-tested kernel-events → gateway broadcast seam (gateway/server.py:637 _init_kernel_events_forwarder; broadcast, not pubsub.publish, is the delivery mechanism that works — session-193 finding). UI work itself is out of scope; the event on the wire satisfies REQ-001-014's subscription contract.
6. Honest delivery outcome (closes G4). DeliveryOutcome <<dataclass>> + DeliveryFailure <<enum>> (OK / NO_ROUTE / HANDLER_ERROR); new Postman.aroute_outcome(packet) -> DeliveryOutcome; existing aroute becomes a thin delegate (outcome.response if outcome.delivered else None) so all current callers keep exact behavior. message_volk switches to aroute_outcome and reports success=False on NO_ROUTE/HANDLER_ERROR (REQ-001-US-002-009/035/038); on delivered it records the RECIPIENT's heartbeat via the probe (G6, REQ-001-018) — delivery through the recipient's handler is activity evidence, symmetric with the dispatch-time heartbeat.
7. Wire the kernel path (closes G5). Kernel-side PipelineRunner binds the registry into every volk it rosters (same set_volk_registry seam the orchestrator runner already uses at pipeline_runner.py:786), at its volk construction/registration site; kernel boot (kernel/cli_main.py VolkRegistry construction sites at 252/428) owns the HeartbeatPump lifecycle and the VolkStatusEventBridge.

Non-goals (MVP discipline — explicitly deferred): push heartbeats from remote/ plugin volks over CXP; postman offline-queue/health integration (REQ-001-US-002-018 already served by queue manager); VizInt UI rendering of the new event; multicast status digests; any change to IVolkRegistry (plugin SDK surface, SPEC-015-owned).

4.2 Contract diagram

patternRole map for gen-skeleton: VolkHealthProbe=protocol · HeartbeatPump.beat_once=leaf, lifecycle=boilerplate · VolkStatusEventBridge= observer-adapter (leaf body one method) · DeliveryOutcome/DeliveryFailure= dataclass/enum (fully generatable) · Postman.aroute_outcome=leaf · edits to existing classes = builder tasks, not generated.

4.3 Wiring plan (RUL-023 / Hard Rule 8 — named production seams, found via refs, not guessed)

Transport constraint respected: all kernel↔gateway status traffic rides the cxp.transport factory (CXP_TRANSPORT=cxpt in the test profile); both transports already implement get_status, and the enrichment is payload-only — no proto change, no new RPC.

5. TDD TEST PLAN (tests-first; real infra only — Redis 10.93.53.11, graph 10.93.53.10, kernel 10.93.53.13:50053, env profile test; no mocks/monkeypatch per RUL-002; env via injected mappings per RUL-003)

Runner discipline: scripts/run_tests.py --affected / targeted modules only, live-observed (observe_pytest attach), 5-min cap per run; kernel required.

6. BUILD PACKETS (each <5 min for a builder agent; MVP order; tests-first inside each)

P1/P3/P5/P6 are gen-skeleton-friendly (protocol/dataclass/enum/observer shells + traced test stubs); P2/P4/P7/P8 are builder/wirer edits to existing classes.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks