orchestrata_ultimate.gates.differential

Reference › orchestrata_ultimate › gates › differential

The differential between the Prolog gate and the legacy checkers it replaces (plan step P5, before the legacy checkers are deleted in P6). It exists because a replacement that silently finds less than what it replaces is a regression dressed as a cleanup. The module runs both over one corpus and accounts for every disagreement; nothing inside the package imports it.

compare takes the Prolog findings and the legacy findings, plus a LegacyRule description of what each legacy rule can see (files, banned names, keys, substrings), the banned-package registry and indexes of the module or env key at each file and line. It returns a DifferentialReport of agreed findings and Difference records. Every difference is classified by mechanism, never allowlisted: _classify_prolog_only and _classify_legacy_only name why the sides disagree. A Prolog-only finding is a coverage gain once its mechanism is named; a legacy-only finding is a loss, except where the legacy checker used substring matching, which the new layer rejects on purpose because it produces false positives. A difference with no known mechanism is unexplained and fails the differential. equivalent says whether the replacement is safe to land. The docstring records four mechanisms measured against the real checkers on 2026-08-08. The layer it checks is described on the prolog-gate page; banned names come from core.banned_libraries.

At a glance

Class diagram

Dependency diagram

API

Classes

Finding (dataclass)

Decorators: dataclass(frozen=True, order=True) · Line 104

One finding, reduced to what both implementations can be compared on.

LegacyRule (dataclass)

Decorators: dataclass(frozen=True) · Line 121

What one legacy checker's rule can actually see.

Difference (dataclass)

Decorators: dataclass(frozen=True) · Line 138

A finding one side produced and the other did not, with its mechanism.

DifferentialReport (dataclass)

Decorators: dataclass · Line 157

The full accounting: what matched, what did not, and why.

Functions

Source

Permalink: https://gitlab.com/MSamy/orchestrata-ultimate-architecture/-/blob/a4cbc2bdf/src/orchestrata_ultimate/gates/differential.py

"""Differential between the Prolog gate and the checkers it replaces (P5).

Contract: ``plans/contracts/prolog-gates.contract.json``.
Plan: ``plans/prolog-gates-implementation-plan.md`` P5 → P6.

WHY THIS EXISTS. P6 deletes the legacy checkers. A replacement that silently
finds LESS than what it replaces is a regression dressed as a cleanup, and the
only way to know which of the two happened is to run both over one corpus and
account for every disagreement. That accounting is this module.

Every difference is CLASSIFIED, never allowlisted. An allowlist of
"known-different" findings would grow by hand and hide the next real gap in the
noise; a classifier has to name the MECHANISM that produced the difference, and
a difference whose mechanism is not one of the known ones is reported
UNEXPLAINED and fails the differential. That distinction is the whole value:

  - a Prolog-only finding is a coverage GAIN and is fine once its mechanism is
    named (a rule the legacy checker never had; a registry it did not read; a
    key set or file scope narrower than this layer's);
  - a legacy-only finding is a coverage LOSS. Its mechanism is named too, but
    naming it does not make it acceptable — the one exception is a legacy
    finding produced by SUBSTRING matching, which this layer rejects on
    purpose because it manufactures false positives.

Four mechanisms, all measured on 2026-08-08 against the real checkers, not
anticipated:

  1. ``import pyswip`` — found by ``standards``, missed here, because that
     checker held its ban set in a hardcoded ``_FORBIDDEN_IMPORTS`` literal
     never merged into ``.claude/banned-libraries.txt``. A real loss; closed by
     adding the package to the registry.
  2. ``unittest.mock`` — ``_check_no_mocks`` bans it, and ``banned_import``
     compared only TOP-LEVEL packages, so it could not express "unittest.mock
     is banned, unittest is not". Closed by ``ban_target/3``.
  3. ``CXP_GRPC_PORT`` with a default — flagged here, invisible to
     ``_check_env_get``, whose ``ENV_VAR_KEYS`` list covers 6 of the 10
     required infra keys. A gain.
  4. ``from mymocks import helper`` — flagged by ``_check_no_mocks``, whose
     match is the substring "mock". Not a loss: the same rule flags any module
     with "mock" in its name, which is the false-positive class this layer
     refused when it rejected prefix matching for ``docker``/``doc``.
"""

from __future__ import annotations

from dataclasses import dataclass, field

from orchestrata_ultimate.core.banned_libraries import ban_targets

# --- causes -----------------------------------------------------------------
#
# A cause names the MECHANISM behind a difference. Each is a fact about how the
# two implementations are built, not a judgement about a particular finding.

#: A rule the legacy checker does not implement at all. Prolog-only, intended.
RULE_ABSENT_FROM_LEGACY = "rule_absent_from_legacy"

#: The Prolog ban registry (``.claude/banned-libraries.txt``) lists a package
#: the legacy checker's hardcoded set does not. Prolog-only, intended.
BAN_LIST_BROADER_IN_PROLOG = "ban_list_broader_in_prolog"

#: The Prolog infra-key set covers a key the legacy checker's list omits.
#: Prolog-only, intended.
KEY_SET_BROADER_IN_PROLOG = "key_set_broader_in_prolog"

#: The legacy rule never inspected this file — it restricts itself to test
#: files, or skips them. Prolog-only, intended.
FILE_SCOPE_BROADER_IN_PROLOG = "file_scope_broader_in_prolog"

#: The legacy checker's hardcoded set lists a package the registry does not.
#: Legacy-only, and a LOSS: the fix is to add the package to the registry, not
#: to accept the difference.
BAN_LIST_NARROWER_IN_PROLOG = "ban_list_narrower_in_prolog"

#: The legacy finding exists only because that checker matches a SUBSTRING.
#: Legacy-only and intended: exact matching is a deliberate design choice here,
#: made when prefix matching would have flagged ``import doc`` for ``docker``.
LEGACY_SUBSTRING_MATCH = "legacy_substring_match"

#: The file is outside every declared ``project/2`` scope, so the Prolog rules
#: never considered it. Legacy-only, intended: scope is a deliberate input.
OUT_OF_PROJECT_SCOPE = "out_of_project_scope"

#: No known mechanism explains this difference. Never intended.
UNEXPLAINED = "unexplained"

#: Causes that describe a legacy-only finding this layer is right not to have.
#: ``BAN_LIST_NARROWER_IN_PROLOG`` is deliberately NOT here: its mechanism is
#: known and its consequence is still a hole.
KNOWN_LOSS_CAUSES = frozenset({OUT_OF_PROJECT_SCOPE, LEGACY_SUBSTRING_MATCH})

PROLOG = "prolog"
LEGACY = "legacy"

#: The only rule whose differences are decided by comparing an imported module
#: against a ban registry. Asking that question about any other rule produces
#: an answer-shaped lie: a relative-import finding at a line that happens to
#: import ``.relative`` was classified "relative is banned by the legacy
#: checker" on the first run of the full differential.
IMPORT_RULE = "banned_import"


@dataclass(frozen=True, order=True)
class Finding:
    """One finding, reduced to what both implementations can be compared on.

    Rule id, file and line — deliberately NOT the message. Two checkers phrase
    the same defect differently and always will; requiring the prose to match
    would make the differential fail on wording and pass on substance.
    """

    rule: str
    file: str
    line: int

    def __str__(self) -> str:
        return f"{self.file}:{self.line} [{self.rule}]"


@dataclass(frozen=True)
class LegacyRule:
    """What one legacy checker's rule can actually see.

    Every field answers a question the classifier would otherwise have to
    guess. ``files`` is the set the legacy rule INSPECTED, not the set it
    flagged — a rule that only looks at test files is narrower for a reason,
    and a Prolog finding elsewhere is a gain rather than a mystery.
    """

    prolog_rule: str
    files: frozenset[str]
    banned: frozenset[str] = frozenset()
    keys: frozenset[str] | None = None
    substrings: frozenset[str] = frozenset()


@dataclass(frozen=True)
class Difference:
    """A finding one side produced and the other did not, with its mechanism."""

    finding: Finding
    side: str
    cause: str
    detail: str = ""

    @property
    def is_loss(self) -> bool:
        """Does this difference mean the replacement finds LESS?"""
        return self.side == LEGACY

    def __str__(self) -> str:
        suffix = f" — {self.detail}" if self.detail else ""
        return f"{self.side}-only  {self.finding}  [{self.cause}]{suffix}"


@dataclass
class DifferentialReport:
    """The full accounting: what matched, what did not, and why."""

    agreed: list[Finding] = field(default_factory=list)
    differences: list[Difference] = field(default_factory=list)

    @property
    def unexplained(self) -> list[Difference]:
        return [d for d in self.differences if d.cause == UNEXPLAINED]

    @property
    def losses(self) -> list[Difference]:
        """Legacy-only differences that are coverage the replacement lacks.

        A classified cause does not make a hole acceptable — only the causes in
        ``KNOWN_LOSS_CAUSES``, where the finding is one this layer is RIGHT not
        to produce, are excluded.
        """
        return [
            d
            for d in self.differences
            if d.is_loss and d.cause not in KNOWN_LOSS_CAUSES
        ]

    @property
    def gains(self) -> list[Difference]:
        return [d for d in self.differences if d.side == PROLOG]

    @property
    def equivalent(self) -> bool:
        """Is the replacement safe to land?

        Not "are the sets equal" — a gain is not a reason to refuse. The
        question is whether anything is unexplained or lost.
        """
        return not self.unexplained and not self.losses

    def render(self) -> str:
        lines = [
            f"agreed: {len(self.agreed)}",
            f"prolog-only (gain): {len(self.gains)}",
            f"legacy-only (loss): {len(self.losses)}",
            f"unexplained: {len(self.unexplained)}",
        ]
        lines.extend(
            f"  {difference}"
            for difference in sorted(
                self.differences, key=lambda d: (d.side, d.finding)
            )
        )
        return "\n".join(lines)


def compare(
    *,
    prolog: set[Finding],
    legacy: set[Finding],
    legacy_rules: dict[str, LegacyRule],
    banned_registry: set[str],
    module_at: dict[tuple[str, int], str],
    key_at: dict[tuple[str, int], str],
    in_scope: set[str],
) -> DifferentialReport:
    """Account for every finding on both sides.

    ``module_at`` and ``key_at`` map ``(file, line)`` to the imported module and
    the env key read there, so a difference is classified from the two
    REGISTRIES rather than by parsing either side's message. ``in_scope`` is the
    set of files the Prolog rules were allowed to consider.
    """
    report = DifferentialReport()
    report.agreed = sorted(prolog & legacy)

    for finding in sorted(prolog - legacy):
        report.differences.append(
            _classify_prolog_only(
                finding, legacy_rules, banned_registry, module_at, key_at
            )
        )
    for finding in sorted(legacy - prolog):
        report.differences.append(
            _classify_legacy_only(
                finding, legacy_rules, banned_registry, module_at, in_scope
            )
        )
    return report


def _classify_prolog_only(
    finding: Finding,
    legacy_rules: dict[str, LegacyRule],
    banned_registry: set[str],
    module_at: dict[tuple[str, int], str],
    key_at: dict[tuple[str, int], str],
) -> Difference:
    rule = legacy_rules.get(finding.rule)
    if rule is None:
        return Difference(
            finding,
            PROLOG,
            RULE_ABSENT_FROM_LEGACY,
            f"no legacy checker implements {finding.rule}",
        )
    if finding.file not in rule.files:
        return Difference(
            finding,
            PROLOG,
            FILE_SCOPE_BROADER_IN_PROLOG,
            "the legacy rule never inspected this file",
        )

    where = (finding.file, finding.line)
    module = module_at.get(where)
    if finding.rule == IMPORT_RULE and module:
        targets = ban_targets(module)
        matched = [t for t in targets if t in banned_registry]
        if matched and not any(t in rule.banned for t in targets):
            return Difference(
                finding,
                PROLOG,
                BAN_LIST_BROADER_IN_PROLOG,
                f"{matched[0]} is in the registry, not in the legacy set",
            )

    key = key_at.get(where)
    if key and rule.keys is not None and key not in rule.keys:
        return Difference(
            finding,
            PROLOG,
            KEY_SET_BROADER_IN_PROLOG,
            f"{key} is a required infra key the legacy list omits",
        )
    return Difference(finding, PROLOG, UNEXPLAINED)


def _classify_legacy_only(
    finding: Finding,
    legacy_rules: dict[str, LegacyRule],
    banned_registry: set[str],
    module_at: dict[tuple[str, int], str],
    in_scope: set[str],
) -> Difference:
    if finding.file not in in_scope:
        return Difference(
            finding,
            LEGACY,
            OUT_OF_PROJECT_SCOPE,
            "outside every declared project scope",
        )

    rule = legacy_rules.get(finding.rule)
    module = module_at.get((finding.file, finding.line))
    if (
        finding.rule == IMPORT_RULE
        and module
        and not any(t in banned_registry for t in ban_targets(module))
    ):
        if rule and any(part in module for part in rule.substrings):
            return Difference(
                finding,
                LEGACY,
                LEGACY_SUBSTRING_MATCH,
                f"{module} merely contains a banned substring",
            )
        return Difference(
            finding,
            LEGACY,
            BAN_LIST_NARROWER_IN_PROLOG,
            f"{module} is banned by the legacy checker and absent from "
            ".claude/banned-libraries.txt",
        )
    return Difference(finding, LEGACY, UNEXPLAINED)