Orchestrata Ultimate / RTM / 102-volk-test-harness
RTM: 102-volk-test-harness
Requirement to symbol to test rows for 102-volk-test-harness: Spec 102 — Individual Volk Test Harness.
At a glance
| Item | Value |
|---|---|
| Requirement IDs | 131 |
Traced from src/ |
104 (79.4%) |
| Untraced | 27 |
Cited from tests/ |
0 (0.0%) |
Traced in src/ but not cited from tests/ |
104 |
Reading this matrix
This matrix lists each of the 131 requirement IDs of spec 102-volk-test-harness with the src/ symbols that cite it and the tests that cite it. 104 of them (79.4%) have at least one citing symbol and 27 have none.
0 requirements are also cited from tests/, and 104 are traced in code but never named by a test. A test citation is a @traces on a test function; it shows intent to cover, not that the test runs or reaches the symbol.
Rows showing none in the symbols column are the gaps to close first. The densest untraced groups are REQ-102-* (27).
Requirement to symbols to tests
| Requirement | Symbols in src/ |
Tests citing it |
|---|---|---|
REQ-102-001 Scenarios defined in YAML under harness/scenarios/<volk_name>/ |
orchestrata_ultimate/harness/scenario load_scenarios_for_volk L97 |
none |
REQ-102-002 Scenario schema includes: id, volk, description, mode, input, evaluation |
orchestrata_ultimate/harness/models Scenario L41orchestrata_ultimate/harness/scenario _validate_scenario_fields L43 |
none |
REQ-102-003 Scenario schema optionally includes: setup_hook, llm_model, timeout_seconds |
orchestrata_ultimate/harness/models Scenario L41 |
none |
REQ-102-004 Scenario YAML validated via Pydantic on load |
orchestrata_ultimate/harness/scenario _read_scenario_mapping L20orchestrata_ultimate/harness/scenario load_scenario L75 |
none |
REQ-102-005 Invalid scenario YAML skipped with error logged; harness continues |
none | none |
REQ-102-006 harness list displays all scenarios with volk, mode, description |
orchestrata_ultimate/harness/cli _cmd_list L189orchestrata_ultimate/harness/cli _print_scenarios_table L84orchestrata_ultimate/harness/cli main L343+2 more |
none |
REQ-102-007 harness run <volk> executes all scenarios for the given volk |
orchestrata_ultimate/harness/cli main L343orchestrata_ultimate/harness/runner HarnessRunner L48orchestrata_ultimate/harness/runner run_volk L445 |
none |
REQ-102-008 harness run --scenario <id> executes a single scenario |
orchestrata_ultimate/harness/cli _resolve_scenario_arg L115orchestrata_ultimate/harness/cli _run_selected_scenarios L129 |
none |
REQ-102-009 harness run --trials <n> overrides default trial count |
orchestrata_ultimate/harness/cli _run_selected_scenarios L129 |
none |
REQ-102-010 Each scenario runs exactly 3 independent trials by default |
orchestrata_ultimate/harness/runner run_scenario L78 |
none |
REQ-102-011 Trials use separate git worktrees for isolation |
orchestrata_ultimate/harness/runner run_scenario L78 |
none |
REQ-102-012 Trials use separate Redis DBs per worker (gwN→dbN) |
none | none |
REQ-102-013 Trials use UUID-based test_id for Neo4j resource isolation |
orchestrata_ultimate/harness/runner run_scenario L78 |
none |
REQ-102-014 Scenario timeout defaults to 300s; configurable per scenario |
orchestrata_ultimate/harness/models Scenario L41orchestrata_ultimate/harness/runner _execute_scenario_stage L272 |
none |
REQ-102-015 Scenario passes only if all trials pass |
orchestrata_ultimate/harness/runner HarnessRunner L48orchestrata_ultimate/harness/runner _scenario_passed L433 |
none |
REQ-102-016 Scenario aggregate score is weighted average across evaluators |
orchestrata_ultimate/harness/runner _compute_aggregate_score L409 |
none |
REQ-102-017 Exit code 0 if all scenarios pass; 1 if any fail |
orchestrata_ultimate/harness/cli _cmd_run L161 |
none |
REQ-102-018 Results table printed: scenario, volk, trials, pass/fail, score, time |
orchestrata_ultimate/harness/cli _print_results_table L49 |
none |
REQ-102-019 Results appended to harness/history/runs/<YYYY-MM>.json |
orchestrata_ultimate/harness/history JSONHistoryTracker L111orchestrata_ultimate/harness/history record L106orchestrata_ultimate/harness/runner HarnessRunner L48 |
none |
REQ-102-020 JSON entry includes: scenario_id, volk, trials, scores, timestamp, git_commit |
orchestrata_ultimate/harness/history JSONHistoryTracker L111orchestrata_ultimate/harness/history _result_to_dict L66orchestrata_ultimate/harness/runner _current_git_commit L467 |
none |
REQ-102-021 JSON writes are atomic (temp file + rename) |
orchestrata_ultimate/harness/history JSONHistoryTracker L111 |
none |
REQ-102-022 Corrupt history files handled gracefully (warning + continue) |
orchestrata_ultimate/harness/history record L119orchestrata_ultimate/harness/reporting _load_history_entries L137 |
none |
REQ-102-023 Results synced to Neo4j after each scenario run |
orchestrata_ultimate/harness/history GraphHistoryTracker L209 |
none |
REQ-102-024 Neo4j graph includes :Scenario nodes linked to :Volk nodes |
orchestrata_ultimate/harness/history GraphHistoryTracker L209orchestrata_ultimate/harness/history _canonical_volk_alias L50 |
none |
REQ-102-025 Neo4j graph includes :Run nodes with scores and timestamps |
orchestrata_ultimate/harness/history record L261 |
none |
REQ-102-026 Neo4j graph includes :Regression nodes when snapshot diffs detected |
orchestrata_ultimate/harness/history GraphHistoryTracker L209 |
none |
REQ-102-027 Neo4j sync failures are non-fatal (JSON history still saved) |
orchestrata_ultimate/harness/runner run_scenario L78 |
none |
REQ-102-028 Cypher trend queries available: pass rate, regression count, avg score |
orchestrata_ultimate/harness/cli _cmd_trends L303orchestrata_ultimate/harness/reporting TrendQueries L384orchestrata_ultimate/harness/reporting avg_score_trend L410+2 more |
none |
REQ-102-029 harness report generates HTML from JSON history |
orchestrata_ultimate/harness/cli _cmd_report L284orchestrata_ultimate/harness/reporting _render_report_html L236orchestrata_ultimate/harness/reporting generate_report L260 |
none |
REQ-102-030 HTML report includes: total runs, pass rate, avg score, regression count |
orchestrata_ultimate/harness/reporting _compute_overall_stats L153orchestrata_ultimate/harness/reporting generate_report L260 |
none |
REQ-102-031 HTML report includes per-volk pass-rate bar chart |
orchestrata_ultimate/harness/reporting _compute_volk_stats L164orchestrata_ultimate/harness/reporting generate_report L260 |
none |
REQ-102-032 HTML report includes per-scenario score trends table |
orchestrata_ultimate/harness/reporting _compute_scenario_stats L186orchestrata_ultimate/harness/reporting generate_report L260 |
none |
REQ-102-033 HTML report includes regression diff previews |
orchestrata_ultimate/harness/reporting _compute_regressions L216orchestrata_ultimate/harness/reporting generate_report L260 |
none |
REQ-102-034 Healing triggered only on scenario failure (not timeout) |
orchestrata_ultimate/harness/healing SelfHealingLoop L101 |
none |
REQ-102-035 Healing max depth configurable (default: 3 hops) |
orchestrata_ultimate/harness/healing __init__ L114 |
none |
REQ-102-036 Healing disabled if max_healing_hops set to 0 |
orchestrata_ultimate/harness/runner _maybe_self_heal L349 |
none |
REQ-102-037 Diagnostic packet captures: evaluator results, error, snapshot diff |
orchestrata_ultimate/harness/healing SelfHealingLoop L101orchestrata_ultimate/harness/healing _diagnostic_payload L79 |
none |
REQ-102-038 Diagnostic packet sent as CXP ERROR packet to failing volk |
orchestrata_ultimate/harness/healing _run_hop L210 |
none |
REQ-102-039 Diagnostic packet includes scenario_id and test_id for isolation |
none | none |
REQ-102-040 Repair packet includes previous hop history |
orchestrata_ultimate/harness/healing _run_hop L210 |
none |
REQ-102-041 Repair packet sent as CXP COMMAND packet |
orchestrata_ultimate/harness/healing _run_hop L210 |
none |
REQ-102-042 Repair strategy extracted from response payload ("retry" or "patch") |
orchestrata_ultimate/harness/healing _extract_strategy L58orchestrata_ultimate/harness/healing _has_valid_strategy L305 |
none |
REQ-102-043 Escalation packet includes full diagnostic history |
orchestrata_ultimate/harness/healing _run_hop L210 |
none |
REQ-102-044 Escalation packet references reasoning_llm for complex repairs |
orchestrata_ultimate/harness/healing _run_hop L210 |
none |
REQ-102-045 Escalation packet sent as CXP ERROR packet |
orchestrata_ultimate/harness/healing _run_hop L210 |
none |
REQ-102-046 Success: trial marked passed with healing_hops_used set |
orchestrata_ultimate/harness/healing SelfHealingLoop L101orchestrata_ultimate/harness/healing _repaired_trial L312 |
none |
REQ-102-047 Exhaustion: trial remains failed; diagnostic trace saved to JSON |
orchestrata_ultimate/harness/healing _hop_to_dict L473orchestrata_ultimate/harness/healing _save_diagnostics L385orchestrata_ultimate/harness/healing heal L125 |
none |
REQ-102-048 Healing never modifies main working directory (worktree only) |
orchestrata_ultimate/harness/healing _assert_under_worktree L462orchestrata_ultimate/harness/healing _resolve_safe_path L442 |
none |
REQ-102-049 Fallback mode when dispatcher unavailable: logs + simulated retry |
orchestrata_ultimate/harness/healing SelfHealingLoop L101orchestrata_ultimate/harness/healing _heal_fallback L330 |
none |
REQ-102-050 Scenario output serialized to text snapshot on first run |
orchestrata_ultimate/harness/runner _compare_snapshot L329orchestrata_ultimate/harness/snapshots SnapshotManager L19orchestrata_ultimate/harness/snapshots save L31 |
none |
REQ-102-051 Snapshot stored in tests/volk_harness/snapshots/<scenario_id>.snap |
orchestrata_ultimate/harness/snapshots _snapshot_path L88orchestrata_ultimate/harness/snapshots exists L79 |
none |
REQ-102-052 Snapshot format versioned (# snapshot:v1 header) |
orchestrata_ultimate/harness/snapshots _build_header L94 |
none |
REQ-102-053 Subsequent runs compare output against stored snapshot |
orchestrata_ultimate/harness/runner _compare_snapshot L329orchestrata_ultimate/harness/snapshots SnapshotManager L19orchestrata_ultimate/harness/snapshots compare L43 |
none |
REQ-102-054 Snapshot mismatch flagged as regression (trial fails) |
orchestrata_ultimate/harness/runner _run_single_trial L186orchestrata_ultimate/harness/snapshots _unified_diff L129orchestrata_ultimate/harness/snapshots compare L43 |
none |
REQ-102-055 harness snapshot update refreshes baselines to current output |
orchestrata_ultimate/harness/cli _cmd_snapshot L204orchestrata_ultimate/harness/snapshots SnapshotManager L19orchestrata_ultimate/harness/snapshots update L71 |
none |
REQ-102-056 Snapshot comparison deterministic (sorted keys, ISO datetimes, base64 bytes) |
orchestrata_ultimate/harness/snapshots _serialize L152orchestrata_ultimate/harness/snapshots _transform_value L163 |
none |
REQ-102-057 QA Volk: zero lint/syntax errors; false-positive rate < 5% |
none | none |
REQ-102-058 CodeDirector: proper patterns, no except/pass, functional style |
none | none |
REQ-102-059 Architect: diagrams render, architecture standards respected |
none | none |
REQ-102-060 Composite pass threshold: 0.8 (80%) minimum aggregate score |
orchestrata_ultimate/harness/runner _run_single_trial L186orchestrata_ultimate/harness/runner _scenario_passed L433 |
none |
REQ-102-061 GitLab CI job for nightly batch execution at 02:00 UTC |
none | none |
REQ-102-062 CI pipeline fails if any volk's overall pass rate < 90% |
none | none |
REQ-102-063 CI artifacts: HTML report + JSON history + snapshot diffs |
none | none |
REQ-102-064 syntax_check: parses Python with ast.parse; fails on SyntaxError |
orchestrata_ultimate/harness/evaluators/syntax SyntaxEvaluator L22orchestrata_ultimate/harness/evaluators/syntax _resolve_files L94orchestrata_ultimate/harness/evaluators/syntax evaluate L28+2 more |
none |
REQ-102-065 lint_check: runs ruff check; fails on ERROR severity |
orchestrata_ultimate/harness/evaluators/lint LintEvaluator L22orchestrata_ultimate/harness/evaluators/lint _resolve_files L143orchestrata_ultimate/harness/evaluators/lint evaluate L37+1 more |
none |
REQ-102-066 trace_validator: parses traces for unhandled exceptions |
orchestrata_ultimate/harness/evaluators/trace TraceEvaluator L20orchestrata_ultimate/harness/evaluators/trace _find_unhandled_exceptions L81orchestrata_ultimate/harness/evaluators/trace _is_allowed L115+3 more |
none |
REQ-102-067 file_exists: verifies expected output files created |
orchestrata_ultimate/harness/evaluators/file_exists FileExistsEvaluator L18orchestrata_ultimate/harness/evaluators/file_exists evaluate L24orchestrata_ultimate/harness/models Evaluator L195 |
none |
REQ-102-068 Evaluators run concurrently per trial |
orchestrata_ultimate/harness/runner _run_evaluators_stage L304 |
none |
REQ-102-069 Evaluator crashes don't crash trial (score=0, details=error) |
orchestrata_ultimate/harness/runner _run_evaluator L369 |
none |
REQ-102-070 llm_judge sends output to DeepSeek v4-pro with structured prompt |
orchestrata_ultimate/harness/evaluators/llm_judge LLMJudgeEvaluator L82orchestrata_ultimate/harness/models Evaluator L195 |
none |
REQ-102-071 Judge prompt includes scenario rubric and evaluation criteria |
orchestrata_ultimate/harness/evaluators/llm_judge _build_user_message L29 |
none |
REQ-102-072 Judge temperature=0.0 for consistency |
orchestrata_ultimate/harness/evaluators/llm_judge LLMJudgeEvaluator L82 |
none |
REQ-102-073 Judge returns score (0–100) and categorical feedback |
orchestrata_ultimate/harness/evaluators/llm_judge LLMJudgeEvaluator L82orchestrata_ultimate/harness/evaluators/llm_judge _parse_response L50 |
none |
REQ-102-074 Judge failure (API error) returns score=0, passed=False |
orchestrata_ultimate/harness/evaluators/llm_judge evaluate L88 |
none |
REQ-102-075 Composite score is weighted average of evaluator scores |
orchestrata_ultimate/harness/runner _compute_composite_score L387 |
none |
REQ-102-076 Evaluator weights configurable per scenario |
orchestrata_ultimate/harness/models EvaluatorConfig L25 |
none |
REQ-102-077 Per-evaluator thresholds configurable per scenario |
orchestrata_ultimate/harness/models EvaluatorConfig L25 |
none |
REQ-102-078 Trial passes only if all evaluators pass AND composite ≥ threshold |
orchestrata_ultimate/harness/runner _run_single_trial L186 |
none |
REQ-102-079 Worktree created per trial via git worktree add |
orchestrata_ultimate/harness/isolation WorktreeIsolation L33orchestrata_ultimate/harness/isolation _git_toplevel L107orchestrata_ultimate/harness/isolation _run_git L118+2 more |
none |
REQ-102-080 Worktree path validated to be inside repo |
none | none |
REQ-102-081 Worktree cleaned up after trial (success or failure) |
orchestrata_ultimate/harness/isolation WorktreeIsolation L33orchestrata_ultimate/harness/isolation _run_git L118orchestrata_ultimate/harness/isolation cleanup L28+1 more |
none |
REQ-102-082 Worktree creation failure falls back to temp directory |
none | none |
REQ-102-083 Cleanup failures logged as warnings (non-fatal) |
orchestrata_ultimate/harness/isolation WorktreeIsolation L33 |
none |
REQ-102-084 Incus container mode available for full sandboxing |
orchestrata_ultimate/harness/isolation IncusIsolation L136orchestrata_ultimate/harness/isolation _container_name L203orchestrata_ultimate/harness/isolation _host_dir L216+4 more |
none |
REQ-102-085 Incus fallback to worktree when daemon unavailable |
orchestrata_ultimate/harness/isolation IncusIsolation L136 |
none |
REQ-102-086 Incus containers have no outbound network (except API/Redis/Neo4j) |
orchestrata_ultimate/harness/isolation IncusIsolation L136 |
none |
REQ-102-087 Setup hooks defined in harness/scenarios/<volk>/setup.py |
orchestrata_ultimate/harness/scenario run_setup_hook L180 |
none |
REQ-102-088 Setup hooks are async functions accepting HarnessContext |
orchestrata_ultimate/harness/scenario run_setup_hook L180 |
none |
REQ-102-089 Setup hook timeout: 30s default |
orchestrata_ultimate/harness/runner _run_setup_stage L241 |
none |
REQ-102-090 Setup hook failure marks trial FAILED |
orchestrata_ultimate/harness/runner _run_setup_stage L241 |
none |
REQ-102-091 All hook-created resources cleaned up via isolation mechanism |
orchestrata_ultimate/harness/runner run_scenario L78 |
none |
REQ-102-092 Redis DB isolation: worker gwN maps to dbN |
orchestrata_ultimate/harness/isolation get_redis_db_for_trial L242 |
none |
REQ-102-093 Neo4j resources prefixed with test_id |
orchestrata_ultimate/harness/isolation get_graph_prefix L251 |
none |
REQ-102-094 All Neo4j nodes/relations cleaned up by test_id in teardown |
none | none |
REQ-102-095 cxp_inject mode constructs CXP packet with intent + payload |
orchestrata_ultimate/harness/executor CXPacketExecutor L38orchestrata_ultimate/harness/executor execute L33 |
none |
REQ-102-096 Packet dispatched via kernel dispatcher with timeout |
orchestrata_ultimate/harness/executor CXPacketExecutor L38orchestrata_ultimate/harness/runner dispatcher L71 |
none |
REQ-102-097 Packet recipient built from volk registry definition |
orchestrata_ultimate/harness/executor CXPacketExecutor L38orchestrata_ultimate/harness/healing _make_recipient L37 |
none |
REQ-102-098 CXP response waited for; timeout marks trial FAILED |
orchestrata_ultimate/harness/executor execute L50 |
none |
REQ-102-099 direct_call mode instantiates volk class directly |
orchestrata_ultimate/harness/executor DirectCallExecutor L87orchestrata_ultimate/harness/executor _import_volk_class L126orchestrata_ultimate/harness/executor execute L33 |
none |
REQ-102-100 Method name specified in scenario input (method field) |
orchestrata_ultimate/harness/executor DirectCallExecutor L87 |
none |
REQ-102-101 Method args passed as dict (args field) |
orchestrata_ultimate/harness/executor execute L91 |
none |
REQ-102-102 Sync methods offloaded to thread to avoid blocking event loop |
orchestrata_ultimate/harness/executor DirectCallExecutor L87 |
none |
REQ-102-103 Missing method or import failure marks trial FAILED |
orchestrata_ultimate/harness/executor _import_volk_class L126orchestrata_ultimate/harness/executor execute L91 |
none |
REQ-102-104 Config loaded from harness/config.yaml with env var overlay |
orchestrata_ultimate/harness/cli _cmd_config L318orchestrata_ultimate/harness/config _apply_yaml L65orchestrata_ultimate/harness/config load_config L21 |
none |
REQ-102-105 Env vars prefixed with HARNESS_ override YAML values |
orchestrata_ultimate/harness/config _apply_env L81 |
none |
REQ-102-106 Redis/KG connection params resolved from the active env profile; required endpoints... |
orchestrata_ultimate/harness/config _apply_env L81 |
none |
REQ-102-107 Default LLM: deepseek-v4-flash; reasoning LLM: deepseek-v4-pro |
orchestrata_ultimate/harness/models HarnessConfig L115 |
none |
REQ-102-108 Config directories auto-created on load |
orchestrata_ultimate/harness/config load_config L21 |
none |
REQ-102-109 Scenario YAML schema violation → skip + log error |
none | none |
REQ-102-110 Setup hook import failure → trial FAILED + stack trace |
none | none |
REQ-102-111 LLM API rate limit (429) → exponential backoff (5 retries, base 2s) |
none | none |
REQ-102-112 LLM API auth failure (401/403) → all trials fail immediately |
none | none |
REQ-102-113 Redis connection failure → auto-start attempt once |
none | none |
REQ-102-114 Neo4j connection failure → history sync skipped (non-fatal) |
orchestrata_ultimate/harness/history _connect L239orchestrata_ultimate/harness/history record L261 |
none |
REQ-102-115 Git worktree creation failure → fallback to temp directory |
none | none |
REQ-102-116 Timeout → cancel LLM, kill worktree, mark TIMEOUT, no healing |
orchestrata_ultimate/harness/runner _execute_scenario_stage L272orchestrata_ultimate/harness/runner _run_single_trial L186 |
none |
REQ-102-117 Deterministic evaluator crash → score=ERROR, other evaluators continue |
orchestrata_ultimate/harness/runner _run_evaluator L369 |
none |
REQ-102-118 Snapshot file corruption → skip comparison + warning |
orchestrata_ultimate/harness/snapshots _read_snapshot_body L107 |
none |
REQ-102-119 Parallel resource collision → new test_id + retry |
none | none |
REQ-102-120 Command whitelist enforced: ruff, python, git, orchestrata-tools, black,... |
orchestrata_ultimate/harness/executor SecurityError L24orchestrata_ultimate/harness/executor enforce_whitelist L178orchestrata_ultimate/harness/models HarnessConfig L115 |
none |
REQ-102-121 Whitelist violation → SecurityError + trial FAILED + incident logged |
orchestrata_ultimate/harness/executor SecurityError L24orchestrata_ultimate/harness/executor enforce_whitelist L178 |
none |
REQ-102-122 DeepSeek API key read from .env only; never logged |
none | none |
REQ-102-123 Path traversal blocked; writes restricted to worktree |
orchestrata_ultimate/harness/healing _assert_under_worktree L462 |
none |
REQ-102-124 LLM judge prompts templated with clear delimiters |
orchestrata_ultimate/harness/evaluators/llm_judge _build_user_message L29 |
none |
REQ-102-125 Snapshots scanned for secrets before git commit |
none | none |
REQ-102-126 Harness self-tests cover: parsing, hooks, evaluators, snapshots, healing, whitelist |
none | none |
REQ-102-127 Integration tests use "echo volk" for end-to-end verification |
none | none |
REQ-102-128 All harness tests pass under the threadpool runner python scripts/run_tests.py... |
none | none |
REQ-102-129 No session-scope fixtures; no global mutable state |
none | none |
REQ-102-130 Deterministic evaluators: 100% true-positive, zero false-positive on valid input |
none | none |
REQ-102-131 Snapshot format backward-compatible across versions |
none | none |