orchestrata_ultimate.harness.evaluators

Reference › orchestrata_ultimate › harness › evaluators

Evaluator registry for the Volk Test Harness.

The judging layer of the Volk Test Harness. A scenario's YAML lists evaluators by name; the registry turns each name into an Evaluator instance that returns an EvaluatorResult with a pass flag, a score and details. The deterministic evaluators are file_exists (file_exists), syntax (syntax_check), lint (lint_check, via ruff) and trace (trace_validator, unhandled exceptions in trace text). The subjective one is llm_judge (llm_judge). base re-exports the base class from the harness models.

The runner runs the configured evaluators for each trial concurrently and combines their scores with the weights in the scenario. Importing the package registers all five. To add an evaluator, subclass Evaluator, set name and decorate it with register_evaluator. Start with the registry.

7 modules, 715 lines of code.

Modules

Dependency diagram