Orchestrata Ultimate › Plans › jinja-codegen-optimization

Codegen / Jinja optimization plan

Source: plans/jinja-codegen-optimization.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-05 · 62 lines.

Prose pending.

Plan text

Codegen / Jinja optimization plan

Task #23. Grounded in the read-only inventory + Jinja research (agent a7b8…, 2026-07-05). Extends plans/ast-codegen-audit.md item 12-L (shared emitter) + item 10 (compile+exec tests). Execution DEFERRED while any coverage wave is running — the biggest wins edit scripts/serialize_codebase_ast.py, the toolkit the waves use as a gate. Land between waves.

The problem (evidence)

• serialize_codebase_ast.py (6568 LOC) has 6 hand-rolled f-string generators, zero Jinja. Three emit non-black-idempotent output (proven): generate_interface (:2198, two-line stubs), the gen-tests arm (:2606, single-quoted literals), generate_pattern (:2883, trailing blank).
• generate_pytest.py (1513 LOC) is Jinja but writes 3 inline template strings to disk on every init then reloads them; formats via ruff+black subprocess.
• src/orchestrata_tools/test_generator/ = 4+ parallel pytest-from-AST engines (jinja_gen.py Jinja, enhanced_generator.py f-string, combined_generator.py, + stub/interface/ branch/behavioral) — mostly unwired, each with its OWN type-map, import-dedup, branch-detector.
• Duplication census: 4× type→fixture maps · 4× import-block/dedup · 3× branch detector · 2× signature renderer · 2× @traces extractor. This is the core waste.
• Jinja 3.1.6 is the current latest (already pinned) — no upgrade needed.

Jinja features to use (and to avoid)

• Use: template inheritance + {% macro %} + {% import %} (dedup the shared fragments); custom filters — pyrepr (black-idempotent literals, fixes the single-quote defect), fixture_for(type) (unify the 4 type maps), signature/call_args/dedupe_imports/traces_deco; optional FileSystemBytecodeCache + line-statement syntax (readability). Keep autoescape=False (escaping Python source corrupts it).
• Avoid: NativeEnvironment (these emit source text, not native objects — wrong fit); SandboxedEnvironment (inputs are the repo's own trusted AST, not untrusted data); ahead-of-time compile_templates (not warranted at this scale).

Execution order (ranked by impact × effort)

1. [High / Low] Central black-clean post-render pass — one format_src() (in-process black.format_str, fallback to the existing subprocess) wired into each generator. Immediately fixes the 3 proven non-idempotent outputs. Highest ROI; makes generated code trustworthy. Ship first.
2. [High / Low-Med] generate_pattern + generate_interface → Jinja macros — the largest literal-text blocks in serialize_codebase_ast.py; collapses the interface-vs-pattern signature/stub inconsistency onto one macro. Good first demonstration of the shared layer.
3. [High / Med] The shared emitter (audit 12-L core) — a codegen/emitter module with ONE Environment factory (autoescape off, trim/lstrip, optional bytecode cache, the filters above)
   • a _macros.j2 holding the single implementation of: imports() (replaces 4), signature()/ stub() (replaces 2), test_class()/test_method()/aaa_body() (replaces the scaffolds), assertion(). Removes the A5 duplication. Per the audit this turns downstream generators (P1 gen-serde/gen-enum, P3) from M→S.
4. [High / Med-High] Consolidate the 4 test engines behind jinja_gen.py:JinjaTestGenerator (break its 108-line monolith into the shared macros) + one entrypoint; retire enhanced_generator.py + stub/interface/branch/behavioral duplicates; point generate_pytest.py at real .j2 files from the shared env (keep its black/ruff step). Largest LOC reduction. After #3.
5. [Med / Low] compile+exec tests for the Jinja/Enhanced engines (audit item 10) — today only the serialize_codebase_ast generators are gated (tests/scripts/test_ast_{dto,patterns,branches}.py, the last even runs the generated pytest). Add per-engine compile(out,…,"exec") + run-the-tests.
6. [Low / Low] FileSystemBytecodeCache + optional line-statement syntax — minor polish once #3 exists.

Do NOT touch: generate_dto (:3212, already black-clean), generate_docs (:2231), generate_mermaid (:2267) — small / clean / non-Python; templating them is negative ROI.

Anchors

serialize_codebase_ast.py:2198/2231/2606/2883/3212 (shared _func_signature:1631); generate_pytest.py:970/987/1255/1317/1376/1435; orchestrata_tools/test_generator/{jinja_gen.py:170/197/42, enhanced_generator.py:35/82/234, stub_generator.py:21, interface_tests.py:34, branch_tests.py:28, behavioral_tests.py}; gates tests/scripts/test_ast_{dto.py:71,patterns.py:88,branches.py:112/118}. Feeds the P1/P3 generators in [[ast-codegen-audit]] (gen-serde/gen-enum/gen-traces reuse the shared emitter).

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks