scripts.coverage_ratchet

Reference › scripts › coverage_ratchet

A coverage-percentage ratchet: it fails only if the coverage of a scope drops below a given baseline. A gate or a human runs it; no module or test imports it.

Invocation: main requires --scope (a label for the subject, such as claude), --paths (the coverage targets to measure) and --tests (the pytest targets to run to produce the measurement). --min sets the baseline: exit 1 only if the measured percentage drops below it. --epsilon (default 0.05 percentage points) absorbs floating-point jitter, and --timeout (default 300 seconds) bounds the run.

run_coverage runs pytest with coverage over the given tests and targets and returns the parsed coverage report, or None when the measurement could not be made, in which case main returns 2. main prints the measured percentage under the scope label and compares it with the baseline.

At a glance

Class diagram

No classes; see Functions.

Dependency diagram

No internal imports and no importers.

API

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/scripts/coverage_ratchet.py

#!/usr/bin/env python3
"""Coverage-percentage ratchet — fail only if a SCOPE's coverage DROPS below
its recorded baseline.

Mirrors ``scripts/pyright_ratchet.py``'s shape and philosophy (now in codetools:
src/codetools/refactor/pyright/ratchet.py; see its module
docstring for the pattern this gate copies): run the real check, extract one
number, and exit non-zero ONLY when that number gets WORSE than the ``--min``
baseline passed in at the call site — not merely because it is short of some
universal target.

Why a ratchet and not ``fail_under = 90`` directly (user ruling 2026-08-19):
the AGENTS.md "Definition of done" 90% bar stays the AMBITION; this gate is
the ENFORCEMENT MECHANISM that makes ANY floor real without demanding 90% on
day one from code that has never been measured. ``claude/`` scored ~12.8%
covered the day this gate was written (``tests/claude/`` +
``tests/scripts/test_task_context_hook.py`` — the entire test corpus
touching it at the time). A hard ``fail_under = 90`` would have been red on
arrival, and a gate that is red on arrival gets waived, then ignored, then
defanged — this project watched exactly that happen to earlier, unenforced
coverage prose. A ratchet starts GREEN and can only go up: the baseline is a
CLI flag inlined at the call site (mirrors the AST toolkit's ``untyped --max
N`` / ``check-traces --max N`` and ``pyright_ratchet.py --max N`` ratchet
convention, now in codetools: src/codetools/refactor/pyright/ratchet.py), tightened by hand as real tests land, never loosened to make a
red run green.

Invocation is a real ``python -m pytest --cov=<paths> --cov-report=json:...``
subprocess against the given ``--tests`` targets (mirrors
``check_async_await.py`` / ``pyright_ratchet.py``'s subprocess + JSON-parse
shape (now in codetools: src/codetools/refactor/pyright/ratchet.py): the number comes from an actual test run every time, never a stored
number re-asserted. The live-observer plugin
(``orchestrata_ultimate.core.threading.pytest_live_observer``) is always
injected, exactly as ``core.threading.test_executor`` does for every batched
runner subprocess (RUL-009): a gate subprocess is still a test run.

``--min`` is a percentage (0-100). A small epsilon (default 0.05 points,
``--epsilon``) absorbs float rounding, not real regressions — losing even one
small file's coverage on a tree this size moves the total by whole points.

Exit 0 at/above baseline (or when ``--min`` is omitted — advisory-only, same
as the AST/pyright ratchets). Exit 1 when coverage DROPS below ``--min``.
Exit 2 when the coverage run itself failed or its JSON report could not be
produced/parsed.
"""

from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path


def run_coverage(
    paths: list[str], tests: list[str], *, timeout: float = 300.0
) -> dict | None:
    """Run pytest+coverage over *tests*, restricted to *paths*, and return
    the parsed coverage.py JSON report (its ``totals`` section is what the
    ratchet reads).

    Runs with the CALLER's cwd (no ``cwd=`` override): production callers
    (``scripts/gates.py``) already spawn this script with ``cwd=REPO_ROOT``,
    so ``--paths``/``--tests`` resolve against the repo the same way a plain
    ``uv run pytest tests/claude/`` would. Tests of THIS script spawn it with
    a tiny fixture tree as cwd instead, which carries no pyproject.toml of
    its own — a clean sandbox with no inherited addopts/markers.

    ``COVERAGE_FILE`` is pinned to a path INSIDE this call's own temp dir,
    not left at coverage.py's default (``.coverage`` in cwd). The repo's
    ``[tool.coverage.run] parallel = true`` makes pytest-cov auto-COMBINE
    every sibling ``.coverage.*`` file sitting next to the data file at
    teardown — on the shared repo-root default that means silently
    absorbing another concurrent agent's (or a stale prior run's) coverage
    data into THIS scope's number. Measured live 2026-08-19: four
    back-to-back runs at repo-root cwd read 12.96%, 12.96%, 17.46%, 17.44%
    once stray ``.coverage.<host>.<pid>.*`` files were sitting in the repo
    root from earlier invocations — a silent-corruption class, not jitter.
    A private ``COVERAGE_FILE`` per call has nothing to combine with.

    Returns ``None`` if the run produced no JSON report or it could not be
    parsed (covers both "pytest itself failed to start" and "tests failed
    hard enough that coverage never got to write its report").
    """
    with tempfile.TemporaryDirectory() as td:
        report_path = Path(td) / "coverage.json"
        data_file = Path(td) / ".coverage"
        cmd = [
            sys.executable,
            "-m",
            "pytest",
            *tests,
            *(f"--cov={p}" for p in paths),
            f"--cov-report=json:{report_path}",
            "-p",
            "orchestrata_ultimate.core.threading.pytest_live_observer",
            "-q",
            "--no-header",
        ]
        env = dict(os.environ)
        env["COVERAGE_FILE"] = str(data_file)
        proc = subprocess.run(
            cmd, capture_output=True, text=True, timeout=timeout, env=env
        )
        if not report_path.exists():
            print(
                "[coverage-ratchet] ERROR: no coverage report produced.",
                file=sys.stderr,
            )
            print(proc.stdout[-2000:], file=sys.stderr)
            print(proc.stderr[-2000:], file=sys.stderr)
            return None
        try:
            return json.loads(report_path.read_text())
        except json.JSONDecodeError:
            print(
                "[coverage-ratchet] ERROR: could not parse coverage json report.",
                file=sys.stderr,
            )
            return None


def main() -> int:
    parser = argparse.ArgumentParser(
        description=(
            "Ratchet gate: fail only if a scope's coverage DROPS below the "
            "--min baseline."
        )
    )
    parser.add_argument(
        "--scope", required=True, help="label for this ratchet's subject, e.g. 'claude'"
    )
    parser.add_argument(
        "--paths",
        nargs="+",
        required=True,
        help="--cov=<path> target(s) to measure",
    )
    parser.add_argument(
        "--tests",
        nargs="+",
        required=True,
        help="pytest target(s) to run to produce the measurement",
    )
    parser.add_argument(
        "--min",
        type=float,
        default=None,
        dest="min_percent",
        help="Ratchet: exit 1 only if percent_covered DROPS below N (baseline gate)",
    )
    parser.add_argument(
        "--epsilon",
        type=float,
        default=0.05,
        help="tolerance in percentage points absorbing float jitter (default 0.05)",
    )
    parser.add_argument("--timeout", type=float, default=300.0)
    args = parser.parse_args()

    report = run_coverage(args.paths, args.tests, timeout=args.timeout)
    if report is None:
        return 2

    percent = report["totals"]["percent_covered"]
    print(f"[coverage-ratchet] {args.scope}: {percent:.2f}% covered.")

    if args.min_percent is None:
        print("[coverage-ratchet] advisory only (no --min given).")
        return 0

    delta = percent - args.min_percent
    under_min = delta < -args.epsilon
    if under_min:
        print(
            f"[coverage-ratchet] FAIL: {args.scope} dropped to {percent:.2f}%, "
            f"below the --min baseline of {args.min_percent:.2f}% ({delta:+.2f}) "
            "— coverage regressed."
        )
    else:
        print(
            f"[coverage-ratchet] OK: {args.scope} baseline "
            f"{args.min_percent:.2f}%, delta {delta:+.2f}."
        )
    return 1 if under_min else 0


if __name__ == "__main__":
    raise SystemExit(main())