orchestrata_ultimate.harness.evaluators.file_exists

Reference › orchestrata_ultimate › harness › evaluators › file_exists

FileExistsEvaluator, registered as file_exists (SPEC-102 US-005, REQ-102-067). It reads expected_files from the evaluator's params and checks that each path exists relative to the trial's worktree. With no files configured it passes with score 1.0; if any file is missing it fails with score 0.0 and lists the missing paths; otherwise it passes with 1.0. Registration happens through the decorator in the evaluator registry; the result type is in models.

At a glance

Class diagram

Dependency diagram

API

Classes

FileExistsEvaluator

Bases: Evaluator · Decorators: register_evaluator, traces(spec='102', story='US-005', reqs=['REQ-102-067']) · Line 19

Verify expected output files exist.

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/harness/evaluators/file_exists.py

"""File existence evaluator for the Volk Test Harness."""

from __future__ import annotations

from typing import Any

from orchestrata_ultimate.core.base.traces import traces
from orchestrata_ultimate.harness.evaluators import register_evaluator
from orchestrata_ultimate.harness.models import (
    Evaluator,
    EvaluatorConfig,
    EvaluatorResult,
    HarnessContext,
)


@register_evaluator
@traces(spec="102", story="US-005", reqs=["REQ-102-067"])
class FileExistsEvaluator(Evaluator):
    """Verify expected output files exist."""

    name = "file_exists"

    @traces(spec="102", story="US-005", reqs=["REQ-102-067"])
    async def evaluate(
        self,
        output: Any,
        context: HarnessContext,
        config: EvaluatorConfig,
    ) -> EvaluatorResult:
        """Check that all expected files exist relative to *worktree_path*.

        Expected files are read from ``config.params["expected_files"]``.
        Each path is resolved relative to ``context.worktree_path``.
        """
        expected_files = config.params.get("expected_files", [])
        if not expected_files:
            return EvaluatorResult(
                name=self.name,
                passed=True,
                score=1.0,
                details="No expected files configured.",
            )

        missing: list[str] = []
        for rel_path in expected_files:
            full_path = context.worktree_path / rel_path
            if not full_path.exists():
                missing.append(rel_path)

        if missing:
            return EvaluatorResult(
                name=self.name,
                passed=False,
                score=0.0,
                details=f"Missing files: {', '.join(missing)}",
            )

        return EvaluatorResult(
            name=self.name,
            passed=True,
            score=1.0,
            details=f"All {len(expected_files)} expected file(s) found.",
        )