Orchestrata Ultimate › Plans › briefs › 2026-10-22-maps-one-load-builder

Brief: maps increment 1 of 4. gen_codebase_map.py --with-full writes both maps in one process (Sonnet builder)

Source: plans/briefs/2026-10-22-maps-one-load-builder.md at 8a2cb42ad · last changed 2026-10-22 · first committed 2026-10-22 · 121 lines.

Prose pending.

Plan text

Brief: maps increment 1 of 4. gen_codebase_map.py --with-full writes both maps in one process (Sonnet builder)

Executor: Sonnet builder. cwd /home/mohamed/workspace/wt-MAP-PRECOMMIT. Create branch feat/maps-one-load from HEAD b5468cb1c (item 2: the pre-commit regenerates the maps).

DESIGN_WAIVER: RUL-041

GOAL: python3 scripts/gen_codebase_map.py --with-full writes both maps in one process, byte-identical to today's two steps, in under 40 s instead of 128 s.

Ruling (Mohamed 2026-10-22): "maps should use delta". It ships in small increments:

1. one process (this brief: the Python change and its proof, no hook edit, no tests, no commit);
2. the hook calls it;
3. tests;
4. commit.

The true per-file delta is a later packet.

Measured (orchestrator, idle machine; do not re-measure)

• bash .claude/hooks/inject-codebase-map.sh: 128.4 s.
• Step 1, python3 scripts/gen_codebase_map.py: 9.8 s.
• Step 2, ten CLI calls python3 scripts/serialize_codebase_ast.py symbols --source <tree>: 9.8–12.2 s each, whatever the tree's size.
• The CLI cost:
  • every CLI process enables the warm cache (_dispatch calls configure_parse_cache);
  • so _cached_extract calls _load_sidecar(), which unpickles the 200 MB plans/ast-ultimate.json.warmcache;
  • and main()'s finally: _save_sidecar() may write it back.
• The toolkit imported in-process, as _class_map already does, behaves differently:
  • _CACHE_STATE["enabled"] stays False and _DATA_ROOT stays None, so there is no pickle;
  • a relative source string comes back as the same relative file paths the CLI prints.
• So one process with the cache left off pays for no pickle at all.

Code facts (read with the toolkit, verified)

scripts/gen_codebase_map.py:

• module constants ROOT, SRC, SCRIPTS;
• _load_ast_toolkit() (lines 57–90):
  • loads serialize_codebase_ast.py by file path as module _gen_map_ast_toolkit and registers it in sys.modules;
  • every call re-executes the module and returns a fresh module object.
• _class_map(add) (125–208) calls it, then toolkit.list_symbols([str(src_root)], kind="class");
• build_map() (211–257);
• main() (260–271): builds, writes with dump_text_atomic(ROOT / "docs" / "CODEBASE_MAP.md", out) (from orchestrata_ultimate.core.atomic_json), writes out to stdout, returns 0. There is no argparse.

Toolkit list_symbols(sources, pattern=None, kind=None) -> list[dict]:

• returns rows with file, line and signature, sorted by (file, line);
• the CLI prints each row as print(f"{r['file']}:{r['line']}: {r['signature']}"), then print(f"\n{len(rows)} symbol(s).").

Change (exact)

1. _load_ast_toolkit(): as the first statement of the body, return sys.modules["_gen_map_ast_toolkit"] when it is already present. _class_map and the new builder then share one module object. Leave the rest unchanged.
2. Module constant, in the hook's order: FULL_TREES = ("src/orchestrata_ultimate", "scripts", "rtm", "end2end", "tools", "harness", "architecture", "features", "specs", "claude").
3. build_full_map(toolkit) -> str: lines joined with "\n", plus a trailing "\n".
   • Six header lines, byte-exact. Copy them from .claude/hooks/inject-codebase-map.sh lines 86–91; they contain em dashes:
     • # Full symbol skeleton — src/ + scripts/ (class / function / method signatures)
     • (empty line)
     • > ON-DEMAND tier (auto-generated with docs/CODEBASE_MAP.md; NOT @imported).
     • > Read in ranges (grep -n first) — never whole.
     • (empty line)
     • a line of three backticks
   • Then, for each tree in FULL_TREES where (ROOT / tree).is_dir():
     • ### symbols: {tree}/;
     • one line per row of toolkit.list_symbols([tree]). Pass the relative string unchanged, in {file}:{line}: {signature} form;
     • an empty line;
     • {len(rows)} symbol(s).
   • Then a closing line of three backticks.
4. main() with "--with-full" in sys.argv[1:]:
   1. First os.chdir(ROOT), because the relative tree strings resolve against cwd.
   2. Write the header map exactly as today.
   3. Get the shared toolkit from _load_ast_toolkit(). If that returns None, treat the count as 0.
   4. Build the full text and count the lines containing : def or : class .
   5. If the count is at least 1000, write it with dump_text_atomic(ROOT / "docs" / "CODEBASE_MAP_FULL.md", text).
   6. Otherwise write nothing, and print to stderr the hook's exact text:
      • when the file exists and is non-empty: [codebase-map] WARNING: symbol generation produced {count} signatures; KEPT the existing docs/CODEBASE_MAP_FULL.md. Re-run: bash .claude/hooks/inject-codebase-map.sh
      • otherwise: [codebase-map] WARNING: symbol generation produced {count} signatures and there is no existing docs/CODEBASE_MAP_FULL.md. The on-demand tier is UNAVAILABLE — use scripts/serialize_codebase_ast.py directly.
   • Without the flag, main() behaves exactly as today.
5. Leave the cache off. Do NOT call configure_parse_cache or configure_repo_root.

FILES:

scripts/gen_codebase_map.py
plans/briefs/2026-10-22-maps-one-load-builder.md

CONSTRAINTS:

• Read Python only through the deployed toolkit, run from /tmp/claude-1001/-home-mohamed/829ea13c-8ceb-4f51-9b28-672fef74daca/scratchpad:
  • /opt/orchestrata-tools/bin/ast symbols --source <abs path>
  • /opt/orchestrata-tools/bin/ast source <name> --source <abs path>
  • Never use sed, cat, grep or Read on a .py file. Shell and markdown can be read directly.
• RUL-037: write Python only with Write or Edit. No python -c, no heredoc into an interpreter, no sed -i.
• Change only scripts/gen_codebase_map.py. Do not edit the hooks; that is increment 2.
• git switch -c feat/maps-one-load first. Stage nothing, commit nothing.
• At most 2 VERIFY runs.

DONE WHEN: VERIFY's log shows all of these:

• HEADER-SAME;
• FULL-SAME;
• NEW rc=0;
• signatures: at least 1000;
• a NEW elapsed time under 40 s.

Your final message is the log's lines verbatim, plus git diff --stat, and nothing else.

VERIFY: bash /tmp/claude-1001/-home-mohamed/829ea13c-8ceb-4f51-9b28-672fef74daca/scratchpad/prove-full.sh /home/mohamed/workspace/wt-MAP-PRECOMMIT > /tmp/claude-maps-prove.log 2>&1; echo "PROVE-EXIT $?" >> /tmp/claude-maps-prove.log

• What the script does, in about 140 s: it rebuilds the reference with the old step 2 on the edited tree (~110 s), runs the header map without the flag, runs --with-full, and cmps both maps.
• Start it with run_in_background: true.
• Wait with P=$(pgrep -f '[p]rove-full.sh' | head -1); ORCH_ALLOW_CONTENDING=1 timeout 300 tail --pid=$P -f /dev/null.
• Then read /tmp/claude-maps-prove.log.
• If it shows FULL-DIFF or HEADER-DIFF, fix the cause from the printed diff lines and run it once more. If it still differs, STOP and report the diff.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks