Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Tests › Reference › harness

Tests reference — tests/harness

tests/harness — 20 test modules, 297 test functions. Source tree: tests/harness at commit 8a2cb42ad. Generated from the tests AST snapshot by scripts/gen-data-pages.py.

Prose pending.

Modules

Module Tests Covers
conftest.py 0 1
test_cli.py 35 2
test_config.py 12 2
test_cxp_live.py 3 8
test_evaluator_file_exists.py 5 2
test_evaluator_lint.py 11 3
test_evaluator_llm_judge.py 19 5
test_evaluator_syntax.py 7 2
test_evaluator_trace.py 7 2
test_evaluators_registry.py 8 2
test_executor.py 16 2
test_healing.py 19 2
test_history.py 20 2
test_incus_isolation.py 5 1
test_isolation.py 13 1
test_models.py 31 1
test_reporting.py 7 3
test_runner.py 44 2
test_scenario.py 25 2
test_snapshots.py 10 2

Module details

tests/harness/conftest.py

source · 0 tests · 65 lines

Shared fixtures for harness subsystem tests.

Covers: harness.models

tests/harness/test_cli.py

source · 35 tests · 945 lines

Real-infrastructure tests for :mod:orchestrata_ultimate.harness.cli.

Covers: core.env_bootstrap, harness.cli

Kind Name Line Description
function test_main_loads_the_rul019_env_contract 273
function test_config_prints_effective_config_exit_ok 293
function test_config_error_when_yaml_malformed 314
function test_trends_prints_cypher_templates 331
function test_trends_with_volk_includes_pass_rate_query 347
function test_list_against_populated_scenarios_dir 365
function test_list_filtered_by_volk 386
function test_list_empty_when_no_scenarios 405
function test_list_config_error_when_yaml_malformed 417
function test_list_truncates_long_description 429
function test_report_generates_html_from_history 467
function test_report_with_empty_history_still_succeeds 493
function test_report_config_error_when_yaml_malformed 509
function test_report_generation_failure_when_output_dir_missing 521
function test_run_scenario_by_id_success_prints_table_and_exit_ok 540
function test_run_scenario_by_file_path_resolves_directly 562
function test_run_scenario_not_found_by_id_exit_config_error 584
function test_run_all_with_no_scenarios_exit_config_error 596
function test_run_all_scenarios_success_multiple_rows 608
function test_run_by_volk_positional_argument_success 636
function test_run_by_volk_positional_argument_no_scenarios_is_exit_ok_empty 656
function test_run_no_selector_prints_usage_and_config_error 670
function test_run_config_error_when_yaml_malformed 685
function test_run_all_duplicate_scenario_id_is_caught_as_run_failed 697
function test_run_failing_scenario_returns_exit_failure 728
function test_run_keyboard_interrupt_from_setup_hook_is_caught 749
function test_snapshot_update_all_scenarios_success 776
function test_snapshot_update_single_scenario_by_id 808
function test_snapshot_update_scenario_not_found 835
function test_snapshot_update_cxp_inject_failure_aborts_batch 847
function test_snapshot_config_error_when_yaml_malformed 866
function test_no_command_prints_help_and_config_error 893
function test_unknown_subcommand_argparse_errors 905
function test_module_entry_point_help_via_subprocess 918
function test_module_entry_point_no_command_exit_config_error_via_subprocess 936

tests/harness/test_config.py

source · 12 tests · 308 lines

Real-infrastructure tests for orchestrata_ultimate.harness.config.

Covers: harness.config, harness.models

Kind Name Line Description
function test_load_config_parses_yaml_fields 81
function test_load_config_creates_directories 117
function test_load_config_unknown_key_ignored 139
function test_load_config_empty_yaml_uses_defaults 160
function test_load_config_missing_file_uses_defaults 188
function test_env_overrides_take_precedence_over_yaml 209
function test_apply_env_only_sets_present_vars 242
function test_apply_env_changeme_sentinel_treated_as_unset 253 An unsubstituted .env template sentinel (CHANGEME_*) means 'not
function test_apply_env_redis_port_stays_string_when_set 267
function test_apply_yaml_converts_path_fields 283
function test_apply_yaml_path_field_none_left_unchanged 298
function test_apply_yaml_returns_same_instance 305

tests/harness/test_cxp_live.py

source · 3 tests · 267 lines

Test Standards Compliance (see AGENTS.md § Test Standards):

Covers: cxp.proto_codec, cxp.servicer, cxp.transport, harness.executor, harness.healing, harness.models, kernel.dispatcher, core.packet/ (package)

Kind Name Line Description
function test_cxp_inject_executor_against_live_kernel 112 CXPacketExecutor.execute injects a packet that the live kernel receives.
function test_cxp_inject_executor_fails_when_kernel_down 175 With no kernel listening, the executor returns a non-success DispatchResult.
function test_self_healing_loop_against_live_kernel 204 SelfHealingLoop dispatches healing hops to the live kernel via the real,

tests/harness/test_evaluator_file_exists.py

source · 5 tests · 97 lines

Real-infrastructure tests for the file_exists harness evaluator.

Covers: harness.models, harness.evaluators/ (package)

Kind Name Line Description
function test_all_expected_files_present_passes 17 When every expected file exists, the evaluator passes with score 1.0.
function test_one_missing_file_fails 41 A single missing expected file fails the evaluator and is reported.
function test_empty_expected_files_passes 59 No expected files configured is a vacuous pass.
function test_missing_params_key_passes 71 Absent expected_files key defaults to the empty-list (vacuous) pass.
function test_directory_counts_as_existing 82 A directory satisfies existence (evaluator checks .exists(), not is_file()).

tests/harness/test_evaluator_lint.py

source · 11 tests · 295 lines

Real-infrastructure tests for the lint_check harness evaluator.

Covers: harness.evaluators.lint, harness.models, harness.evaluators/ (package)

Kind Name Line Description
function test_clean_file_passes 68 A file ruff accepts passes with score 1.0.
function test_dirty_file_fails 83 A file with an unused import (F401) fails ruff and the evaluator.
function test_severity_param_is_reported 99 The configured severity label is echoed into failure details.
function test_missing_file_is_reported 112 A path that does not exist is flagged as not found and fails.
function test_list_of_files_all_clean_passes 124 Multiple clean files all pass and the count is reported.
function test_no_files_passes_vacuously 140 No resolvable files is a vacuous pass without invoking ruff.
function test_ruff_module_missing_is_reported 152 ADR-0089 (Q2b Phase C): the tool runs as <interpreter> -m ruff, so
function test_interpreter_missing_is_reported 197 A missing INTERPRETER is its own failure mode under the ADR-0089 spawn
function test_ruff_nonzero_exit_with_no_output_is_reported 218 A REAL ruff killed by a signal (OOM/SIGKILL) exits non-zero with no
function test_dict_output_with_list_value_under_files_key 265 A dict output whose files value is a list resolves every entry.
function test_dict_output_with_single_string_value_under_paths_key 283 A dict output whose paths value is a bare string resolves to it.

tests/harness/test_evaluator_llm_judge.py

source · 19 tests · 332 lines

Real-infrastructure tests for the llm_judge harness evaluator.

Covers: harness.evaluators.llm_judge, harness.models, llm.client, core.protocols/ (package), harness.evaluators/ (package)

Kind Name Line Description
function test_parse_response_plain_json 71
function test_parse_response_json_fenced 77
function test_parse_response_bare_fence 84
function test_parse_response_missing_feedback_defaults_empty 91
function test_parse_response_surrounding_whitespace 97
function test_parse_response_non_int_feedback_is_stringified 103
function test_parse_response_invalid_json_raises 110
function test_parse_response_non_object_raises 115
function test_parse_response_missing_score_raises 120
function test_parse_response_non_int_score_raises 125
function test_parse_response_score_out_of_range_raises 130
function test_parse_response_negative_score_raises 135
function test_build_user_message_includes_rubric_and_expected 145
function test_build_user_message_defaults_when_params_absent 165
function test_build_user_message_serializes_non_json_native_output 175
function test_evaluate_returns_structured_result 189 One evaluate() call, replayed from an authored cassette.
function test_evaluate_threshold_gate_on_failure_path 232 A high threshold forces pass=False even when the judge scores a real,
function test_evaluate_call_llm_exception_returns_failure_result 278 call_llm raising is caught and turned into a failure EvaluatorResult.
function test_evaluate_malformed_llm_response_returns_failure_result 306 An unparsable LLM reply is caught by _parse_response's ValueError.

tests/harness/test_evaluator_syntax.py

source · 7 tests · 128 lines

Real-infrastructure tests for the syntax_check harness evaluator.

Covers: harness.models, harness.evaluators/ (package)

Kind Name Line Description
function test_valid_file_via_str_path_passes 31 A syntactically valid .py given as a relative str path passes.
function test_invalid_file_fails 48 A file containing a SyntaxError fails the evaluator.
function test_output_as_list_of_paths 63 output as a list of relative paths is resolved and all parsed.
function test_output_as_dict_with_files_key 77 output as a dict with a files key resolves those paths.
function test_files_from_config_params 92 When output carries no paths, config.params['files'] is used.
function test_no_files_passes_vacuously 106 No resolvable files is a vacuous pass.
function test_non_python_file_is_skipped 118 A non-.py file is skipped (not parsed) and noted in details.

tests/harness/test_evaluator_trace.py

source · 7 tests · 135 lines

Real-infrastructure tests for the trace_validator harness evaluator.

Covers: harness.models, harness.evaluators/ (package)

Kind Name Line Description
function test_traceback_with_value_error_fails 33 A real traceback ending in ValueError is reported as unhandled.
function test_allowed_pattern_makes_traceback_pass 48 An allowed_patterns regex matching the exception line clears the failure.
function test_clean_text_passes 65 Trace text with no exceptions passes.
function test_output_as_existing_file_path_is_read 76 When output is a path to an existing file, its contents are scanned.
function test_output_as_existing_file_path_with_allowed_pattern 92 File-sourced trace also honours allowed_patterns.
function test_standalone_error_line_fails 110 A standalone 'Error:' line (no traceback block) is flagged.
function test_none_output_passes 127 None output resolves to empty trace text and passes.

tests/harness/test_evaluators_registry.py

source · 8 tests · 175 lines

Real-infrastructure tests for the evaluator registry.

Covers: harness.models, harness.evaluators/ (package)

Kind Name Line Description
function test_list_evaluators_contains_all_five 56
function test_list_evaluators_is_sorted 61
function test_get_evaluator_returns_correct_instance 72
function test_get_evaluator_returns_fresh_instance_each_call 79
function test_get_evaluator_unknown_raises_value_error 86
function test_register_evaluator_round_trip 121
function test_registered_evaluator_evaluate_executes 144
function test_register_evaluator_without_name_raises 169

tests/harness/test_executor.py

source · 16 tests · 223 lines

Real-infrastructure tests for orchestrata_ultimate.harness.executor.

Covers: harness.executor, harness.models

Kind Name Line Description
function test_enforce_whitelist_allows_whitelisted_command 65
function test_enforce_whitelist_strips_directory_from_base 71
function test_enforce_whitelist_rejects_unlisted_command 77
function test_enforce_whitelist_rejects_empty_command 82
function test_security_error_is_a_value_error 87
function test_enforce_whitelist_unlisted_caught_as_value_error 94
function test_import_volk_class_imports_real_volk 105
function test_import_volk_class_unknown_module_raises_import_error 114
function test_import_volk_class_empty_stub_module_raises_import_error 119
function test_import_volk_class_imports_architect_volk 127
function test_import_volk_class_imports_qa_volk 138
function test_execute_calls_real_sync_volk_method 154
function test_execute_omitting_args_defaults_to_empty 170
function test_execute_missing_method_raises_value_error 184
function test_execute_unknown_volk_raises_import_error 198
function test_execute_unknown_method_raises_attribute_error 212

tests/harness/test_healing.py

source · 19 tests · 291 lines

Real-infrastructure tests for orchestrata_ultimate.harness.healing.

Covers: harness.healing, harness.models

Kind Name Line Description
function test_heal_fallback_returns_successful_result 62
function test_heal_fallback_repaired_trial_passes 81
function test_heal_respects_custom_max_hops 100
function test_extract_strategy_none_response 116
function test_extract_strategy_from_payload_strategy_field 121
function test_extract_strategy_from_payload_action_field 126
function test_extract_strategy_unknown_payload_strategy_is_none 130
function test_extract_strategy_from_error_message 135
function test_extract_strategy_no_indicators_returns_none 140
function test_diagnostic_payload_includes_only_failed_evaluators 149
function test_diagnostic_payload_empty_when_all_pass 167
function test_repaired_trial_passes_with_hop_count 186
function test_assert_under_worktree_allows_inside_path 205
function test_assert_under_worktree_rejects_escape 215
function test_assert_under_worktree_rejects_dotdot_escape 225
function test_resolve_safe_path_keeps_inside_preferred_dir 235
function test_resolve_safe_path_falls_back_when_outside 247
function test_hop_to_dict_serializes_all_fields 264
function test_heal_diagnostics_are_serializable 282

tests/harness/test_history.py

source · 20 tests · 712 lines

Real-infrastructure tests for harness history tracking.

Covers: harness.history, harness.models

Kind Name Line Description
function test_canonical_volk_alias_maps_known_directory_names 76
function test_canonical_volk_alias_passes_through_already_canonical_alias 82
function test_canonical_volk_alias_passes_through_unknown_name 87
function test_json_record_creates_monthly_file 96
function test_json_record_appends_second_entry 126
function test_json_records_split_by_month 149
function test_json_record_recovers_from_non_list_content 167 A monthly file whose top-level JSON value isn't a list is discarded.
function test_json_record_recovers_from_corrupt_json 190 A monthly file containing invalid JSON text is discarded, not raised.
function test_json_record_atomic_write_failure_cleans_up_tmp_file 213 When the final os.replace fails, the temp file is unlinked and re-raised.
function test_graph_record_creates_nodes_and_regression 242
function test_graph_record_canonicalizes_volk_alias 366 A harness volk_name that is a lowercase package-directory name
function test_graph_connect_reuses_existing_driver 424 A second record call short-circuits _connect (history.py:121-122).
function test_graph_record_swallows_connect_failure 486 record logs and returns (doesn't raise) when _connect fails.
function test_graph_record_swallows_write_failure_after_driver_closed 529 record logs and returns (doesn't raise) when the write itself fails.
function test_read_dotenv_graph_vars_returns_values_for_missing_keys 597
function test_read_dotenv_graph_vars_skips_keys_already_in_environ 628 A key already present in environ wins -- the .env value is ignored.
function test_read_dotenv_graph_vars_ignores_unlisted_keys_and_missing_paths 648
function test_read_dotenv_graph_vars_empty_value_is_ignored 664
function test_read_dotenv_graph_vars_swallows_read_errors 677 A .env path that can't be read as text is skipped, not raised.
function test_graph_tracker_default_construction_resolves_uri_and_no_credentials 696 GraphHistoryTracker() with no explicit args resolves the graph URI

tests/harness/test_incus_isolation.py

source · 5 tests · 59 lines

Tests for IncusIsolation.

Covers: harness.isolation

Kind Name Line Description
function test_container_name_sanitizes_and_prefixes 17
function test_container_name_never_empty_and_cannot_collide_with_infra 26
function test_host_dir_lives_under_harness_incus 34
function test_cleanup_is_idempotent_for_unknown_trial 39
function test_create_then_cleanup_roundtrip 45

tests/harness/test_isolation.py

source · 13 tests · 257 lines

Real-infrastructure tests for orchestrata_ultimate.harness.isolation.

Covers: harness.isolation

Kind Name Line Description
function test_create_returns_existing_directory 50
function test_cleanup_removes_worktree_and_branch 74
function test_cleanup_is_idempotent 91
function test_cleanup_of_never_created_id_is_noop 113
function test_cleanup_outside_git_repo_is_noop_and_warns 124 cleanup must swallow the git failure and warn, not raise, when the
function test_get_redis_db_for_trial_is_identity 148
function test_get_graph_prefix_returns_test_id 152
function test_get_graph_prefix_preserves_arbitrary_string 157
function test_incus_create_requires_incus_binary 167
function test_incus_cleanup_is_idempotent_noop 177
function test_incus_host_dir_outside_git_repo_falls_back_to_cwd 183 _host_dir normally nests under the repo root; outside any git repo
function test_incus_create_and_cleanup_real_roundtrip 207
function test_incus_create_raises_runtime_error_with_daemon_detail 245 A real, failing incus launch (nonexistent image) must surface the

tests/harness/test_models.py

source · 31 tests · 359 lines

Real-infrastructure tests for orchestrata_ultimate.harness.models.

Covers: harness.models

Kind Name Line Description
function test_evaluator_config_defaults 46
function test_evaluator_config_full 54
function test_evaluator_config_params_default_is_isolated 66
function test_evaluator_config_missing_name_raises 73
function test_scenario_minimal_defaults 83
function test_scenario_full_construction 97
function test_scenario_evaluation_accepts_dicts 118
function test_scenario_valid_modes 138
function test_scenario_invalid_mode_raises 146
function test_scenario_missing_required_field_raises 152
function test_evaluator_result_construction 169
function test_evaluator_result_is_frozen 177
function test_trial_result_defaults 183
function test_trial_result_default_list_is_isolated 193
function test_trial_result_is_frozen 202
function test_scenario_result_defaults 208
function test_scenario_result_timestamps_are_distinct_per_instance 219
function test_scenario_result_is_frozen 226
function test_hop_diagnostic_defaults 237
function test_hop_diagnostic_is_frozen 244
function test_healing_result_defaults 250
function test_healing_result_with_payload 256
function test_healing_result_is_frozen 269
function test_harness_config_defaults 280
function test_harness_config_is_mutable 306
function test_harness_config_whitelist_is_isolated 312
function test_harness_config_fixture 319
function test_harness_context_fixture 325
function test_harness_context_is_mutable 336
function test_evaluator_base_name_is_empty 346
function test_evaluator_base_evaluate_raises_not_implemented 350

tests/harness/test_reporting.py

source · 7 tests · 208 lines

Real-infrastructure tests for harness reporting.

Covers: harness.history, harness.models, harness.reporting

Kind Name Line Description
function test_generate_report_from_real_history 69
function test_generate_report_empty_history 102
function test_generate_junit_xml_parses 120
function test_generate_junit_xml_groups_by_volk 157
function test_pass_rate_last_30_days_query 179
function test_regression_count_query 193
function test_avg_score_trend_query 202

tests/harness/test_runner.py

source · 44 tests · 896 lines

Real-infrastructure tests for :mod:orchestrata_ultimate.harness.runner.

Covers: harness.models, harness.runner

Kind Name Line Description
function test_composite_score_is_weighted_average 133
function test_composite_score_equal_weights_is_plain_mean 145
function test_composite_score_uses_config_weight_not_result_order 158
function test_composite_score_missing_result_scores_zero 173
function test_composite_score_empty_results_is_zero 187
function test_composite_score_zero_total_weight_is_zero 192
function test_aggregate_score_averages_trial_means 208
function test_aggregate_score_trial_without_evaluators_contributes_zero 222
function test_aggregate_score_empty_trials_is_zero 229
function test_scenario_passed_all_pass_and_high_score 238
function test_scenario_passed_threshold_boundary_exactly_0_8 243
function test_scenario_failed_when_a_trial_failed 248
function test_scenario_failed_when_score_below_threshold 254
function test_scenario_failed_when_no_trials 260
function test_current_git_commit_returns_40_char_hex 269
function test_runner_construction_wires_managers 281
function test_runner_dispatcher_is_lazy_and_cached 296
function test_list_scenarios_discovers_all_volks 309
function test_list_scenarios_filtered_by_volk 332
function test_list_scenarios_empty_when_no_scenarios_dir 350
function test_init_graph_tracker_construction_failure_is_swallowed 366
function test_run_scenario_end_to_end_passes_and_saves_baseline 387
function test_run_scenario_unexpected_error_in_trial_is_caught 418
function test_run_scenario_isolation_create_failure_recorded_as_trial_error 446
function test_run_scenario_setup_hook_failure_short_circuits_the_trial 467
function test_run_scenario_execution_failure_short_circuits_the_trial 492
function test_run_scenario_failing_evaluator_triggers_self_heal_call 516
function test_run_setup_stage_without_hook_is_a_noop 552
function test_run_setup_stage_hook_succeeds 566
function test_run_setup_stage_hook_raises_is_recorded_as_failure 581
function test_run_setup_stage_hook_timeout_is_recorded_as_failure 599
function test_execute_scenario_stage_cxp_inject_dispatch_timeout 625
function test_execute_scenario_stage_direct_call_unknown_volk_is_caught 651
function test_run_evaluators_stage_no_evaluation_returns_empty 678
function test_run_evaluators_stage_runs_real_evaluator_concurrently 692
function test_compare_snapshot_saves_baseline_when_absent 713
function test_compare_snapshot_detects_regression_on_changed_output 726
function test_run_evaluator_success_returns_real_result 745
function test_run_evaluator_unknown_name_is_caught_as_failure 757
function test_maybe_self_heal_returns_unchanged_when_disabled 775
function test_maybe_self_heal_attempts_real_dispatch_when_enabled 788
function test_run_volk_returns_empty_when_no_scenarios_found 834
function test_run_volk_runs_every_scenario_for_the_volk 856
function test_current_git_commit_returns_empty_string_outside_git_repo 890

tests/harness/test_scenario.py

source · 25 tests · 364 lines

Real-infrastructure tests for orchestrata_ultimate.harness.scenario.

Covers: harness.models, harness.scenario

Kind Name Line Description
function test_load_scenario_valid 55
function test_load_scenario_cxp_inject_mode 69
function test_load_scenario_missing_file 88
function test_load_scenario_invalid_yaml 93
function test_load_scenario_empty_file 99
function test_load_scenario_not_a_mapping 105
function test_load_scenario_missing_required_fields 111
function test_load_scenario_bad_mode 121
function test_load_scenario_empty_evaluation 135
function test_load_scenario_evaluation_not_a_list 148
function test_load_scenario_evaluator_missing_name 162
function test_load_scenarios_for_volk_discovers_multiple 181
function test_load_scenarios_for_volk_missing_dir_returns_empty 200
function test_load_scenarios_for_volk_duplicate_ids 205
function test_list_all_scenarios_across_volks 220
function test_list_all_scenarios_missing_base_returns_empty 240
function test_list_all_scenarios_duplicate_ids_across_volks 245
function test_run_setup_hook_success 306
function test_run_setup_hook_invalid_format 318
function test_run_setup_hook_import_error 325
function test_run_setup_hook_missing_function 332
function test_run_setup_hook_not_async 339
function test_run_setup_hook_no_arguments 346
function test_run_setup_hook_raises_exception 353
function test_run_setup_hook_non_none_return 360

tests/harness/test_snapshots.py

source · 10 tests · 166 lines

Real-infrastructure tests for the harness SnapshotManager.

Covers: harness.models, harness.snapshots

Kind Name Line Description
function test_save_creates_file_with_header_and_body 40
function test_exists_true_and_false 60
function test_compare_returns_none_when_no_baseline 66
function test_compare_returns_none_when_identical 71
function test_compare_returns_diff_when_changed 77
function test_update_overwrites_existing_snapshot 90
function test_roundtrip_datetime_bytes_and_long_string 105
function test_scenario_id_with_slash_produces_sane_filename 140
function test_transform_value_handles_tuples_and_short_strings 155
function test_serialize_is_sorted_and_deterministic 161