Orchestrata Ultimate › Plans › ast-codegen-audit
AST Toolkit Evaluation + Codegen Expansion — audit + roadmap
Source: plans/ast-codegen-audit.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-05 · 60 lines.
Prose pending.
Plan text
AST Toolkit Evaluation + Codegen Expansion — audit + roadmap
Source: ast-codegen-audit swarm (w0kuij9g6, 12 agents, 1.1M tokens). Full results:
workflow journal wf_9d4c9f94-589/journal.jsonl. User directive 2026-07-05.
Toolkit today: scripts/serialize_codebase_ast.py (6568 LOC, 31 subcommands, NO Jinja —
all codegen is hand-rolled f-string line-lists) + orchestrata_tools/ (SourceASTParser,
test_generator, ast_explorer) + scripts/{test-gen,gen_tests,generate_pytest}.py.
Existing generators: interface(Protocol), docs, diagram, gen-tests(per-branch pytest),
gen-dto(dataclass/pydantic from params), pattern(6 GoF).
Phase 1 — FIX the toolkit BEFORE new codegen (broken windows in a CI-wired tool)
Do these first; several are one-file edits in serialize_codebase_ast.py (bundle → one builder to avoid self-conflict). Each needs a regression test + ≥90% coverage on touched code.
| # | Fix | Where | Sev | Eff |
|---|---|---|---|---|
| 1 | Recurse _iter_funcs_with_class into nested funcs + inner classes — the untyped/dead-params/call-graph scan is depth-1, so the CI/pre-commit/pre-push untyped ratchet SILENTLY passes untyped code in nested scopes |
serialize_codebase_ast.py:4825 (also find_untyped 5616, find_dead_params 5559, build_call_graph 5148) | gate-bypass | M |
| 2 | Fix SourceASTParser: populate ParamInfo.default; correct is_abstract/is_pydantic/is_enum (broken → disables code paths on the default parser) |
orchestrata_tools/source_parser.py:247-296,176-185 | high | S |
| 3 | apply_pattern_inplace writes --write with NO ast.parse validation (unlike extract_class/inline) + wrong insert line for multi-line class sigs / single-line bodies → can persist BROKEN source; also delete dead anchor else-branch (4337) |
serialize_codebase_ast.py:4318,4337,4358 | high | S |
| 4 | Parenthesize except (ValueError, SyntaxError): (3.14 portability) |
serialize_codebase_ast.py:2428 | low | S |
| 5 | move_symbol only copies imports when dest is NEW; existing-dest → deps unresolved + emits a FALSE "copies imports" warning |
serialize_codebase_ast.py:4255-4262,4300 | med | S |
| 6 | Generated-enum test synthesis: robust base detection; drop the .value==name.lower() assertion |
(gen-tests enum arm) | med | S |
| 7 | generate_pytest.py: make deterministic + non-destructive (drop datetime stamp; stop clobbering user templates; gate the cross-tree file deletion) |
scripts/generate_pytest.py | med | S |
| 8 | generate_interface ignores decorators (@property/@staticmethod/@classmethod → plain def in Protocol); make decorator-faithful + add --kind abc + implement <Protocol> skeleton |
serialize_codebase_ast.py:2198 | med | M |
| 9 | Configurable generated-import prefix (hardcoded orchestrata_ultimate) |
serialize_codebase_ast.py | low | S |
| 10 | Test the port-critical core (compare_codebases/deep_compare/serialize) + add compile+exec tests for Jinja/Enhanced generators; fix docs "every subcommand" claim | — | med | S/M |
| 11 | gen-stub: emit .pyi from _func_signature/_class_signature (complements untyped) |
new | — | M |
| 12 | (L) Extract a shared codegen/emitter layer (black-clean output, reused by all generators); (L) consolidate the 4 overlapping pytest-from-AST engines behind one model + one entrypoint | — | — | L |
Phase 2 — NEW codegen templates (ranked by instances × determinism × LOC)
Land the shared emitter (12-L) first where it turns M→S; otherwise ship on the existing f-string approach.
| Pri | Generator | Inst | Det | LOC~ | Design |
|---|---|---|---|---|---|
| P1 | gen-traces | 790 | fully | 1000 | Extend add-trace (add_traces@4094, insert-only) into an idempotent stamp/verify codemod driven by the RTM (spec→symbol) map + a class-level @traced decorator (on core/base/traces.py's __traces__) that stamps every method. Retires ~790 hand-pasted @traces placeholders; makes traceability machine-owned + CI-verified. Biggest single lever. |
| P1 | gen-serde | 71 | fully | 600 | gen-serde <Class> emits to_dict/from_dict from dataclass fields (reuse generate_dto field extraction) + a --check drift gate + a SerializableMixin; hoist the 10 duplicate _utc_now copies into one BaseRecord. Removes a real drift hazard. |
| P2 | gen-enum | 55 | fully | 115 | gen-enum <Name> --members a,b,c [--base StrEnum] [--traces ...]; value defaults to member.lower(); auto-stamps @traces. |
| P2 | gen-protocol | 52 | mostly | 240 | = decorator-faithful interface (fix #8) extended: Protocol from a concrete class's public (async) signatures + ABC/impl variants. |
| P2 | gen-facade | 25 | mostly | 1195 | re-export __init__ from siblings' __all__ + compat shims + collapse the interfaces↔protocols twin (one merge ~950 LOC). |
| P2 | gen-volk | 10 | mostly | 450 | gen-volk <alias> --role --escalate --subjects emits volk.py shell + process router + packet builder + prompts.py + init from a per-volk table (sibling to pattern). |
| P3 | gen-wrappers(23), gen-exceptions(21), gen-tool(10), gen-adapter(4) | table-driven thin-wrapper / exception-hierarchy / @tool-schema / provider-adapter generators. |
Execution order (interleave with coverage waves; NEVER edit the toolkit while a coverage wave is using it)
- Phase-1 fixes #1-#7,#9 as one bundled builder (all in serialize_codebase_ast.py + source_parser.py + generate_pytest.py) — highest priority: #1 the gate bypass.
- #8 decorator-faithful interface (enables gen-protocol) + #11 gen-stub.
- gen-traces (P1) — needs the RTM map; verify it reproduces existing @traces byte-for-byte on a sample before mass-stamping.
- gen-serde (P1) + _utc_now → BaseRecord.
- gen-enum, gen-protocol, gen-facade (P2).
- Shared emitter (12-L), then P3 generators on top.
Phase 3 — JSON-spec-driven generators + comprehensive census (Haiku-extracted, verify-gated)
User directive 2026-07-05. Turn every deterministic pattern in the tree into a machine-readable JSON spec + a generator that consumes it, so code becomes generated-from-data not hand-written, with drift caught by a --check gate. Pipeline is propose-then-verify (Haiku proposes, a deterministic gate verifies):
- Comprehensive census — evaluate ALL code patterns across EVERY package (extend the audit's sampling
w0kuij9g6to full coverage). Output: the definitive pattern catalog (one entry per deterministic pattern + every instance). - JSON structure files — one schema-validated spec per pattern under
specs/codegen/(ordata/codegen/): e.g.enums.json(every StrEnum registry + members),adapters.json(provider-wrapper rows),traces.json(the RTM spec→symbol map),serde.json(dataclasses + fields),exceptions.json(hierarchy),packets.json,volks.json,wrappers.json. Each file = the inputs that fully determine the generated code. - Code generators — deterministic generators (
gen-enum/gen-adapter/gen-serde/gen-traces/gen-exceptions/…) that read a JSON spec and emit code, each with a--checkdrift gate (regenerate → diff vs tree → fail on drift). Plug intoserialize_codebase_ast.py/orchestrata_tools. - Haiku does the extraction — Haiku produces the JSON spec files from the code (shallow structured extraction — its sweet spot per the A/B
wme0wptms). This is the ONLY LLM step; generators are deterministic. - Test with Haiku = round-trip verification — Haiku-extracted spec → generator → output must match the existing hand-written code (byte- or AST-equivalent). A match proves BOTH that Haiku's extraction is reliable AND that the generator reproduces the pattern; a mismatch localizes exactly where Haiku (or the generator) is wrong → escalate that slice to Sonnet. The
--checkgate then keeps code and spec in lockstep forever.
Dependencies: gated on the Haiku A/B (wme0wptms) confirming Haiku is reliable for the extraction step; builds on Phase-1 toolkit fixes + Phase-2 generators (many already designed table-driven: gen-serde/gen-enum/gen-adapter/gen-exceptions/gen-wrappers/gen-tool). Sequence: census → JSON specs → one generator+round-trip per pattern (start with the fully-deterministic high-instance ones: traces, serde, enum, adapter).