Orchestrata Ultimate / Specs / 112-scripts-to-modules

112-scripts-to-modules: scripts/ → Modules Consolidation

Consolidate the scripts/ folder so that scripts hold no business logic — every script becomes a thin CLI shim (arg-parse + one call) delegating to a package module under src/orchestrata_ultimate/. Logic that today lives in a script moves into the package where it belongs, with @traces, tests, and the same public behavior.

At a glance

Item	Value
Spec directory	`specs/112-scripts-to-modules/`
Status line in the spec	Draft
Requirement IDs defined	12
With a `@traces` citation in `src/`	0 (0.0%)
With a `@traces` citation in `tests/`	0 (0.0%)
Tracked files	2
RTM page	112-scripts-to-modules

Reading this spec

This spec consolidates the scripts/ folder so that scripts hold no business logic. Every script becomes a thin command-line shim (argument parsing plus one call) that delegates to a module in the src/orchestrata_ultimate/ package, and the logic moves into the package with @traces, tests and the same public behaviour. The reason given is that scripts/ has grown to 137 files, about 96 of which carry real logic. Logic there cannot be imported as a library, escapes the package's structural gates (architecture, imports, standards), is harder to test in isolation and is invisible to the code-intelligence graph. The largest case is the 12,253-line AST toolkit script.

An analysis pass already classified the 137 scripts into three categories. Category A (6 scripts) is the codemod and refactor extraction, done first because it aligns with specs 110 and 111. Category B (52 scripts) holds extraction targets, organized by destination package. Category C (79 scripts) stays in place but is audited for thinness. Extraction proceeds category by category and preserves CLI behaviour exactly, so a shim's help text and outputs stay byte-compatible; a structural gate then keeps scripts/ thin.

The twelve requirement IDs follow the seven functional requirements. REQ-112-FR1 is the thin-shim invariant: scripts contain only CLI wiring, and each extraction leaves a backward-compatible shim at the original path. FR2 is behaviour preservation: the CLI surface and exit codes are byte-compatible, and each script's existing test passes unchanged or only with an updated import path. FR3 requires extracted code to carry @traces and satisfy the structural gates scripts were exempt from. FR4 extracts the six category-A scripts and decomposes the AST toolkit into cohesive modules. FR5 extracts the 52 category-B scripts one package at a time with tests green after each step, creating missing target packages first. FR6 audits the 79 category-C scripts and reclassifies any that hold extractable logic. FR7 adds a gate that fails CI when a script defines domain classes or algorithms beyond a shim threshold.

The spec's status line reads Draft, and no code module cites it yet. The requirements traceability matrix lists every ID. It tracks 2 files. No src/ symbol cites any of the 12 requirements, so measured trace coverage is 0.0%, and in tests/, 0 requirements are cited by a test-side @traces. All 12 requirements have no citing symbol, spread over the groups REQ-112-FR1 (2), REQ-112-FR2 (2) and REQ-112-FR3 (2) among others; the behaviour may still exist in code that was never annotated.

Requirements

Checkbox is the state recorded in the requirements checklist; Traced by is the count of src/ symbols whose @traces names the ID.

ID	Done	Requirement	Traced by
`REQ-112-FR1-001`	no	Scripts under `scripts/` contain only CLI wiring (arg parse + I/O + delegation); no domain classes/algorithms (RUL-scripts-thin, gated by FR-007)	none
`REQ-112-FR1-002`	no	Every extraction leaves a backward-compatible thin shim at the original `scripts/<name>.py` path that imports and calls the extracted module	none
`REQ-112-FR2-001`	no	Post-extraction CLI surface (`--help`, subcommands, flags, exit codes, stdout/stderr) is byte-compatible with pre-extraction behavior	none
`REQ-112-FR2-002`	no	Each script's existing `tests/scripts/test_<name>.py` passes unchanged, or is updated only to the new import path while asserting identical behavior	none
`REQ-112-FR3-001`	no	Extracted module code carries `@traces` to its origin spec/requirement	none
`REQ-112-FR3-002`	no	Extracted modules satisfy the structural gates scripts were exempt from (absolute imports, check-deps, architecture boundaries, standards)	none
`REQ-112-FR4-001`	no	The 6 category-A scripts are extracted into `refactor.codemod`/`refactor`; `serialize_codebase_ast.py` (12,253 L AST toolkit) is decomposed into cohesive modules as it moves	none
`REQ-112-FR4-002`	no	Category-A extraction is sequenced first (the AST toolkit + codemod are the substrate SPEC-110/111 depend on)	none
`REQ-112-FR5-001`	no	The 52 category-B scripts are extracted into their target packages per the mapping, one package at a time, tests green after each	none
`REQ-112-FR5-002`	no	Missing target packages (e.g. `standards` for the 5 static-analysis gates) are created with docstring + `__init__` before extraction	none
`REQ-112-FR6-001`	no	The 79 category-C scripts are audited to confirm genuine thinness; any holding extractable logic are reclassified to A/B	none
`REQ-112-FR7-001`	no	A structural gate (`architecture_checker` extension or new `scripts_are_thin` gate) fails CI when a `scripts/` file defines domain classes/algorithms beyond a shim threshold	none

Code that traces this spec

No src/ symbol cites a requirement of this spec.

Checklists

`checklists/requirements.md`

The requirement list above comes from this file; 0 of 12 checkbox items are ticked.

Full spec text

Verbatim from specs/112-scripts-to-modules/spec.md; headings are pushed down two levels.

SPEC-112: scripts/ → Modules Consolidation

Metadata

Spec ID: 112-scripts-to-modules
Title: scripts/ → Modules Consolidation (thin CLI shims over package logic)
Author: Mohamed (with Claude Opus orchestration)
Status: Draft
Created: 2026-08-03
Priority: P1
Version: 0.1.0
Related: SPEC-110 (codemod), SPEC-111 (live substrate), SPEC-109 (rename), SPEC-103 (refactor)
Components: refactor.codemod, and the target packages (tools, knowledge_graph, verification, factory, standards, …)
Authoritative mapping: plans/scripts-migration-map.json (137 scripts, per-script category + target)

1. Overview

What

Why

scripts/ has grown to 137 files, ~96 of which carry real logic. Logic in scripts/ is: not importable as a library (only runnable), not covered by the package's structural gates (architecture, imports, standards), harder to test in isolation, and invisible to the code-intelligence graph as first-class package API. The single largest offender — serialize_codebase_ast.py (12,253 lines) — is the AST toolkit itself, the very substrate SPEC-110/111 build on, yet it lives as a script. Pulling logic into modules makes it importable, gated, traced, and graph-visible.

How

An analysis pass (already run) classified all 137 scripts into three buckets. Extraction proceeds bucket by bucket, each extraction preserving CLI behavior exactly (the shim's --help and outputs are byte-compatible), with the moved logic tested and traced. A structural gate then keeps scripts/ thin going forward.

2. Categorization (authoritative counts)

Bucket	Count	Meaning
A — extract → codemod/refactor	6	AST-transform / trace-application logic → `refactor.codemod` (SPEC-110) / `refactor`
B — extract → named module	52	real logic belongs in a specific package; script becomes a shim
C — thin / stays	79	already thin, or a genuine dev/CI/build/ops/one-shot runner whose home IS `scripts/`

has_logic=true for 96/137; many category-C runners carry logic but legitimately belong in scripts/ (test runners, wheel builds, one-shot KG migrations, infra/ops).

Category A (6) — the codemod/refactor extraction (do first; aligns with SPEC-110/111)

serialize_codebase_ast.py (12,253 L — the AST toolkit) → refactor.codemod — flagship extraction; it is the substrate's own engine.
codemod.py → refactor.codemod (SPEC-110 CLI)
pyright_autofix.py → refactor.codemod (AST-precise pyright autofix)
add_traces_from_evidence.py → refactor.codemod (writes @traces into source)
attribute_traces_runtime.py → refactor.codemod (runtime-evidence trace attribution)
refactor.py → refactor (SPEC-103 tool-calling refactor CLI)

Category B (52) — extraction targets by package

Target package	Count
`verification` (+ `.review`, `.autofix`)	11
`tools`	10
`knowledge_graph`	8
`factory` (the non-delegating SPEC-033 `factory_*`)	6
`standards` (new package for the static-analysis gates)	5
`orchestrator`	3
`prolog`, `governance`, `integrations`	2 each
`llm`, `memory`, `openspec`	1 each

Category C (79) — stays, but audited for thinness

Genuine runners/ops: test runners (run_tests*), wheel builds (build_ft_*), benchmarks, one-shot KG migrations/repairs, infra readiness (start_test_services, health_check, bootstrap_graphs), git-hook helpers, and scripts that already delegate (e.g. rename_engine.py — 13 package imports; the 10 delegating factory_*; kg_schema, semantic_search, context_pack). No logic to extract; audited only to confirm the shim is genuinely thin.

3. Functional Requirements

FR-001: Thin-shim invariant (the rule)

REQ-112-FR1-001: A script under scripts/ SHALL contain only CLI wiring (arg parsing, I/O, and delegation to a package module) — no domain classes or algorithms. This is a project rule (RUL-scripts-thin), enforced by FR-007.
REQ-112-FR1-002: Every extraction SHALL leave a thin CLI shim at the original scripts/<name>.py path (backward-compatible invocation) that imports and calls the extracted module.

FR-002: Behavior preservation

REQ-112-FR2-001: After extraction, the script's CLI surface (--help, subcommands, flags, exit codes, stdout/stderr contract) SHALL be byte-compatible with pre-extraction behavior.
REQ-112-FR2-002: The existing test for each script (tests/scripts/test_<name>.py) SHALL pass unchanged, or be updated only to import from the new module path while asserting identical behavior.

FR-003: Traceability + gating of extracted code

REQ-112-FR3-001: Extracted module code SHALL carry @traces to the spec/requirement it implements (its origin spec, e.g. SPEC-110 for codemod, SPEC-033 for factory).
REQ-112-FR3-002: Extracted modules SHALL satisfy the package's structural gates that scripts were exempt from: absolute imports, check-deps, architecture boundaries, standards.

FR-004: Category-A extraction (codemod/refactor)

REQ-112-FR4-001: The 6 category-A scripts SHALL be extracted into refactor.codemod / refactor, with serialize_codebase_ast.py (the AST toolkit) decomposed into cohesive modules (not one 12K-line module) as it moves.
REQ-112-FR4-002: Category-A extraction SHALL be sequenced FIRST, because the AST toolkit + codemod are the substrate SPEC-110/111 depend on.

FR-005: Category-B extraction (by package)

REQ-112-FR5-001: The 52 category-B scripts SHALL be extracted into their target packages per scripts_categorized.json, one package at a time, tests green after each package.
REQ-112-FR5-002: Where a target package does not yet exist (e.g. standards for the 5 static-analysis gates), it SHALL be created with a package docstring and __init__ before extraction.

FR-006: Category-C audit

REQ-112-FR6-001: The 79 category-C scripts SHALL be audited to confirm they are genuinely thin/ops; any found to still hold extractable logic SHALL be reclassified to A/B.

FR-007: Enforcement gate

REQ-112-FR7-001: A structural gate (extending architecture_checker or a new scripts_are_thin gate) SHALL fail CI when a file under scripts/ defines domain classes/algorithms beyond a shim threshold, preventing regrowth of logic-in-scripts.

4. Non-Goals

Deleting or renaming category-C runners (they stay where they are).
Changing any script's CLI contract (extraction is behavior-preserving).
Rewriting the extracted logic (this is a move + wire, not a redesign; redesign of the AST toolkit's internal module structure is bounded to cohesive decomposition, not behavior change).

5. Sequencing

A → codemod (6 scripts; flagship: AST toolkit) — unblocks/aligns SPEC-110/111.
B by package, highest-logic-density first (verification, tools, knowledge_graph, factory, standards, …).
C audit + FR-007 gate to lock it in.

6. Traceability

All extracted code SHALL carry @traces to its origin spec. The move itself is tracked against REQ-112-*. The authoritative per-script mapping is scratchpad/scripts_categorized.json, to be committed into plans/ when SPEC-112 execution begins.