Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Tests › Reference › scripts › test_error_report

Tests reference — tests/scripts/test_error_report.py

Module of tests/scripts, split out because the directory page would be too large.

Prose pending.

tests/scripts/test_error_report.py

source · 25 tests · 372 lines

Tests for scripts/error_report.py — the always-on red-list report.

Kind Name Line Description
class TestFailingTestExtraction 63 A module count understates the damage — that was the original error.
method TestFailingTestExtraction.test_pytest_summary_lines_are_read 66
method TestFailingTestExtraction.test_the_batched_runners_own_line_shape_is_read 78 The runner emits module::test — traceback, not pytest's form.
method TestFailingTestExtraction.test_a_test_reported_in_both_shapes_is_counted_once 85
method TestFailingTestExtraction.test_clean_output_yields_nothing 91
class TestTheReportCountsTestsNotModules 95
method TestTheReportCountsTestsNotModules.test_three_modules_holding_eight_tests_report_eight 96 The exact shape of the 2026-08-09 miscount.
method TestTheReportCountsTestsNotModules.test_a_module_with_no_parsable_ids_falls_back_to_its_count 134 Under-reporting is the failure mode; never report zero for a red
method TestTheReportCountsTestsNotModules.test_a_passing_module_is_not_counted 147
class TestPartialRunsAreLabelled 156 A targeted run overwrites the suite's artifact.
method TestPartialRunsAreLabelled.test_a_one_module_artifact_is_called_partial 159
method TestPartialRunsAreLabelled.test_a_green_partial_run_is_still_labelled 175 The dangerous direction: one green module reading as a green suite.
method TestPartialRunsAreLabelled.test_a_full_run_is_not_labelled_partial 192
method TestPartialRunsAreLabelled.test_an_empty_artifact_is_not_labelled_partial 205 No modules at all is an absent run, already reported as such.
class TestStalenessIsStated 216 "No failures recorded 4 hours ago" is not "nothing is red".
method TestStalenessIsStated.test_a_fresh_artifact_carries_no_warning 219
method TestStalenessIsStated.test_an_old_artifact_is_marked_stale 228
method TestStalenessIsStated.test_an_absent_artifact_is_never_reported_as_green 239
method TestStalenessIsStated.test_ages_render_in_a_readable_unit 253
class TestGatesReportCachedVerdicts 261
method TestGatesReportCachedVerdicts.test_a_cached_pass_is_labelled_as_cached 267 A cached PASS says the inputs are unchanged, not that the gate was
method TestGatesReportCachedVerdicts.test_a_failing_gate_is_counted 283
method TestGatesReportCachedVerdicts.test_a_skipped_gate_is_not_a_failure 292
method TestGatesReportCachedVerdicts.test_an_absent_cache_is_reported_not_assumed 298
class TestCorruptArtifactsDoNotCrashTheReport 308 A hook that raises gets disabled, and a disabled reporter reports
method TestCorruptArtifactsDoNotCrashTheReport.test_unparsable_json_is_a_message_not_an_exception 312
method TestCorruptArtifactsDoNotCrashTheReport.test_the_whole_report_builds_against_the_live_repo 325
class TestExitStatus 332
method TestExitStatus.test_it_exits_zero_by_default_so_a_hook_never_blocks 333
class TestItIsWiredIntoSessionStart 337 A reporter nothing invokes is not a delivery (RUL-023).
method TestItIsWiredIntoSessionStart.test_the_hook_script_exists_and_is_executable 340
method TestItIsWiredIntoSessionStart.test_session_start_invokes_the_hook 346
method TestItIsWiredIntoSessionStart.test_the_hook_does_not_pass_exit_code 358 --exit-code would let a red suite block session start, and a hook