Orchestrata Ultimate › Plans › research › audit-04-tracing-debugging

Audit 04 — Tracing / Debugging / Observability

Source: plans/research/audit-04-tracing-debugging.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-04 · 56 lines.

Prose pending.

Plan text

Audit 04 — Tracing / Debugging / Observability

Headline: core/tracing/__init__.py::IterationTracer is a COMPLETE runtime per-volk tracer with ZERO production callers. Built, fully unwired. Biggest gap + cleanest insertion point.

@traces = static only

core/base/traces.py:28-71 + core/traceability/__init__.py:114-156 — attach __traces__ attr (spec/story/reqs) for the RTM. No timing/IO/runtime wrap. Consumers all AST static-analysis (collect_traces/generate_rtm/validate_traces).

Pipeline observability — two divergent systems

• (a) In-process VizInt orchestrator/pipeline_kernel.py _run_pipeline emits pipeline.start/stage/complete/error → _emit_event → injected broadcast("pipeline.<tid>") or Redis xadd "pipeline:events". Gap A: stage events emitted in a loop AFTER runner.run() returns (burst at end, not live). Gap B: writes "pipeline:events" but gateway reads gw:ringbuf:pipeline.events → Redis fallback never consumed.
• (b) Kernel kernel/pipeline_events.py::PipelineEventBus — richer taxonomy (boot/intent/escalation/swarm/review/test.result/complete); LPUSH orchestrata:pipeline:events
  • xadd per-thread gw:ringbuf:pipeline.<tid> + global gw:ringbuf:pipeline.events. Live + durable, but gRPC-kernel path, NOT the in-process runner VizInt uses.
• Gateway gateway/server.py:460 _consume_pipeline_events reads the KERNEL key.
• Durable reconstruction: Neo4j CXPAuditor (knowledge_graph/cxp_auditor.py:96) MERGEs (:CXPPacket {message_id}) per thread (payload in/out) — best-effort. Recovers inter-volk packets, NOT internal volk execution (prompt/tokens/latency/tool calls).

Logging — good, dormant

core/logging.py StructuredFormatter (JSON, contextvars correlation_id/session_id/volk_id, _EXTRA_FIELDS incl duration_ms/prompt_length/model/gate/cycle...). Gap C: configure_logging() has NO production caller — never installed. Gap D: new_correlation_id() only at gateway/server.py:503 (per WS connection), not per run/volk; contextvars never set around volk execution; thread_id not a log var.

Debug surfaces

• VizInt DebugPanel vizint-ui/gadgets/debug-panel/gadget.js — consumes pipeline.* on ws.message, per-thread stage chips + duration tooltip. Inherits Gap A; shows only alias/status/duration.
• CXPFeed — subscribes cxp.traffic (fed by gRPC reply pump only) + pipeline events.
• audit-trail, volk-status gadgets exist.
• AST debug subcommand (scripts/serialize_codebase_ast.py:5894/6401) = inserts breakpoint() (static pdb), not runtime tracing. add-trace inserts @traces.
• /trace skill (claude/commands/trace.md) = manual pytest/pdb playbook.
• Env-gated llm/client.py:416 _capture_exchange (ORCHESTRATA_CONVERSATION_LOG) — JSONL per call (ts/label/correlation/volk/model/messages/response/usage). Off by default, single flat file, NOT thread_id-keyed, no latency (elapsed_ms only .debug-logged).
• OTel in .venv but imported nowhere in src/.

Gaps + insertion (cleanest end-to-end)

1. orchestrator/pipeline_runner.py:144 run — create IterationTracer(thread_id); set thread_id/tracer contextvar; new_correlation_id().
2. _run_stage (218/280) — live pipeline.stage.start/complete + tracer.trace_event; capture traceback.format_exc() on failure (276 discards traceback — G4).
3. agents/volks/base.py:152 _call_llm + :184 _call_llm_with_tools — tracer.trace_llm_call(...) + record tool calls; set volk_id contextvar.
4. llm/client.py:559-570 / _capture_exchange:416 — feed elapsed_ms + structured usage + thread_id.
5. Startup — call configure_logging(json_format=True).
6. Reconcile the two PIPELINE_STREAM_KEYs (pipeline_kernel.py:42 vs kernel/pipeline_events.py:46).

• Also: pipeline_runner.py dead code (_log_stage_to_session:326, _set_handoff:339 never called) + docstring-promised retry/backoff/checkpoint (MAX_RETRIES/RETRY_BACKOFF/_is_retryable:355) NOT implemented.
• SPEC-100 wants OTel + trace_id/span_id (unimplemented); compliance-audit.md:373 specs exactly the IterationTracer API.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks