orchestrata_ultimate.harness.evaluators

Reference › orchestrata_ultimate › harness › evaluators › init

The evaluator registry of the harness. A module-level dictionary maps a name to an Evaluator subclass. register_evaluator is a class decorator that stores a class under its name class variable and raises ValueError if the class defines none; it is generic over the subclass so decorated classes keep their own type. get_evaluator(name) returns a new instance, or raises ValueError listing the registered names when the name is unknown. The five concrete evaluators, file_exists, syntax, lint, trace and llm_judge, are imported here so that importing the package registers them. The runner looks evaluators up by the names in scenario YAML.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

API

Constants

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/harness/evaluators/__init__.py

"""Evaluator registry for the Volk Test Harness."""

from __future__ import annotations

from typing import TYPE_CHECKING

from orchestrata_ultimate.harness.models import Evaluator

if TYPE_CHECKING:
    pass

# Registry populated by individual evaluator modules at import time.
_EVALUATORS: dict[str, type[Evaluator]] = {}


def register_evaluator[EvaluatorT: Evaluator](
    cls: type[EvaluatorT],
) -> type[EvaluatorT]:
    """Decorator to register an evaluator class by its ``name`` classvar.

    Generic over the concrete subclass (rather than fixed to ``Evaluator``)
    so a decorated class keeps its own static type — e.g. ``LintEvaluator``
    stays ``type[LintEvaluator]`` after decoration instead of widening to
    ``type[Evaluator]``, which would otherwise erase subclass-only members
    (like ``LintEvaluator.evaluate``'s extra ``env`` parameter) for every
    caller of ``get_evaluator()`` that narrows the result via ``cast``.
    """
    if not cls.name:
        raise ValueError(f"Evaluator {cls.__name__} must define a 'name' classvar")
    _EVALUATORS[cls.name] = cls
    return cls


def get_evaluator(name: str) -> Evaluator:
    """Return an instantiated evaluator by name.

    Args:
        name: The evaluator's registered name (e.g. ``"syntax_check"``).

    Raises:
        ValueError: If no evaluator is registered under ``name``.
    """
    if name not in _EVALUATORS:
        available = ", ".join(sorted(_EVALUATORS))
        raise ValueError(f"Unknown evaluator: {name}. Available: [{available}]")
    return _EVALUATORS[name]()


def list_evaluators() -> list[str]:
    """Return a sorted list of registered evaluator names."""
    return sorted(_EVALUATORS)


# isort: off
# These imports MUST stay at the bottom to avoid circular imports.
from orchestrata_ultimate.harness.evaluators.file_exists import (  # noqa: E402
    FileExistsEvaluator,
)
from orchestrata_ultimate.harness.evaluators.lint import LintEvaluator  # noqa: E402
from orchestrata_ultimate.harness.evaluators.llm_judge import (  # noqa: E402
    LLMJudgeEvaluator,
)
from orchestrata_ultimate.harness.evaluators.syntax import SyntaxEvaluator  # noqa: E402
from orchestrata_ultimate.harness.evaluators.trace import TraceEvaluator  # noqa: E402

# isort: on

__all__ = [
    "Evaluator",
    "FileExistsEvaluator",
    "get_evaluator",
    "LintEvaluator",
    "list_evaluators",
    "LLMJudgeEvaluator",
    "register_evaluator",
    "SyntaxEvaluator",
    "TraceEvaluator",
]