Orchestrata Ultimate › Plans › briefs › 2026-10-22-maps-delta-builder

Maps step B1 — per-file cache for gen_codebase_map.py (builder brief, 2026-10-22)

Source: plans/briefs/2026-10-22-maps-delta-builder.md at 8a2cb42ad · last changed 2026-10-22 · first committed 2026-10-22 · 161 lines.

Prose pending.

Plan text

Maps step B1 — per-file cache for gen_codebase_map.py (builder brief, 2026-10-22)

GOAL: make scripts/gen_codebase_map.py --with-full re-parse only the files whose content changed since its last run, by keeping a per-file cache of symbol rows and module docstring lines, so a run with no source changes takes about 3 s or less instead of 11.7–13.9 s, and both maps stay byte-identical. This is step B of Mohamed's ruling "maps should use delta" (step A, commit e1f3a5047, put both maps in one process).

FILES: scripts/gen_codebase_map.py only. Nothing else changes in this increment (no tests yet: they are the next increment). Worktree /home/mohamed/workspace/wt-MAP-PRECOMMIT, branch feat/maps-delta (already created from e1f3a5047). Do not stage or commit.

Measured state (do not re-measure)

cProfile of one --with-full run on this tree (21.8 s under the profiler; 11.7–13.9 s unprofiled):

So nearly all the time is parsing and walking unchanged files. Nothing is cached across runs today: the generator loads the toolkit in-process, where _CACHE_STATE["enabled"] is False, so _cached_extract just calls compute() (deliberately: the toolkit's own warm cache is a 200 MB pickle that costs ~9 s to load, which was step A's problem).

Code facts (read with the AST toolkit; line numbers are in the current tree)

scripts/gen_codebase_map.py:

• first_doc_line(path) -> str (29-38): ast.parse(path.read_text(encoding="utf-8")); returns "" on OSError, SyntaxError, ValueError or no docstring; else doc.strip().splitlines()[0].strip() of ast.get_docstring(mod).
• rel(p) (41-42): str(p.relative_to(ROOT)).
• _section(add, title, paths, *, with_counts=False) (45-54): per path, first_doc_line(p if p.is_file() else p / "__init__.py") or "—"; with counts it also does len(list(p.rglob("*.py"))) (no parse).
• _load_ast_toolkit() (57-93): loads SCRIPTS / "serialize_codebase_ast.py" by file path as module _gen_map_ast_toolkit, cached in sys.modules keyed on __file__; returns None (and prints why) on failure.
• _class_map(add) (128-211): toolkit = _load_ast_toolkit(); None → the "UNAVAILABLE" heading. Else rows = toolkit.list_symbols([str(ROOT / "src")], kind="class") (absolute source → absolute file strings), then counts subclasses by bare base name and groups by package; it already normalises relative or absolute file values against ROOT.
• build_full_map(toolkit) -> str (228-251): toolkit.configure_repo_root(ROOT), header lines, then for each tree in FULL_TREES that is a dir: rows = toolkit.list_symbols([tree]) (relative source → root-relative file strings), emitting ### symbols: {tree}/, one f"{r['file']}:{r['line']}: {r['signature']}" per row, "", f"{len(rows)} symbol(s)."; then a closing fence line.
• build_map() -> str (254-300): header lines, _section over scripts/*.py, src entrypoints, packages, then _class_map(add), then _section over tests/**/test_*.py.
• main() (303-317): with_full = "--with-full" in sys.argv[1:]; writes docs/CODEBASE_MAP.md with dump_text_atomic (from orchestrata_ultimate.core.atomic_json, signature (path, text)), echoes it to stdout, then _write_full_map(dump_text_atomic) when with_full.
• _write_full_map (320-326) and _guard_and_write(text, dest, dump_text_atomic) -> int (329-): the 1000-signature guard. Unchanged by this increment.

scripts/serialize_codebase_ast.py (do not edit it):

• list_symbols(sources, pattern=None, kind=None) (1807-1827): rows from _iter_py_files(sources) via _cached_extract(path, text, "symbols", lambda: _extract_symbols(tree, path)), filtered by kind/pattern, then sorted(out, key=lambda r: (r["file"], r["line"])).
• _iter_py_files(sources) (1709-1739), per source: src = _locate(source); rel_base = _DATA_ROOT if the source is relative else None; a .py file → [src], a dir → sorted(src.rglob("*.py")); skips a path when any(skip in path.parts for skip in _DEP_SKIP_DIRS); de-dupes on path.resolve() within the call; skips the file when _parse_cached(path) is None; yields path.relative_to(rel_base) (or path), text, tree.
• _parse_cached(path) (1646-1706): returns None on OSError, UnicodeDecodeError reading (path.read_text(encoding="utf-8")) and on SyntaxError, UnicodeDecodeError from ast.parse(text).
• _extract_symbols(tree, path) -> list[dict] (1768-1804): rows {"kind","name","signature","file": str(path), "line"} in AST visit order.
• configure_repo_root(path) (193-201) sets _DATA_ROOT to the resolved path.

The design (decided — implement exactly this)

1. A cache class in gen_codebase_map.py, e.g. _MapCache, persisted as JSON (stdlib json, hashlib): {"version": <str>, "files": {<absolute path str>: {"sha1": <hex>, "symbols": [rows] | null, "doc": <str>}}}.
   • version = sha1 over Path(__file__).read_bytes() + the toolkit file's bytes + sys.version. A mismatch discards every entry (a toolkit or generator edit means one cold run).
   • Load: a missing file, unreadable file, bad JSON or wrong shape → start empty. A damaged cache degrades to a cold build; it never breaks the map.
   • Per file, read path.read_text(encoding="utf-8") and sha1 its UTF-8 bytes. An entry is a hit when its sha1 matches. A missing field on a hit entry is computed and stored. A file is parsed at most once per run: one ast.parse serves both the symbols and the doc fields.
   • symbols: the rows of toolkit._extract_symbols(tree, rel), where rel = path.relative_to(ROOT), so the stored file is root-relative. Use null when the toolkit would skip the file: reading fails with OSError/UnicodeDecodeError, or the parse fails with SyntaxError/UnicodeDecodeError.
   • doc: exactly first_doc_line's result, including its "" cases. Note its broader catch, OSError, SyntaxError, ValueError.
   • Save only when something changed: a parse, a new field, a version reset, or an entry pruned. Keep only the entries used this run, so deleted files drop out. Write with dump_text_atomic(cache_path, json.dumps(...)) after cache_path.parent.mkdir(parents=True, exist_ok=True).
   • Counters: hits and parsed.
2. A cached list_symbols twin in the generator, used only when a cache is given. It enumerates exactly as _iter_py_files does for one source:
   • src = ROOT / tree for the full map, or ROOT / "src" for the class map;
   • a file gives [src], a dir gives sorted(src.rglob("*.py"));
   • skip on toolkit._DEP_SKIP_DIRS and de-dupe on resolve();
   • the rows are cache.symbols(...) for each file, skipping None;
   • filter by kind, then apply the same sorted(key=(file, line)). For the class map, the stored root-relative file values sort in the same order as the toolkit's absolute ones, because they all share the ROOT prefix, and _class_map already normalises them.
3. Wiring.
   • build_map(cache=None), _class_map(add, cache=None), _section(..., cache=None) and build_full_map(toolkit, cache=None) take an optional cache.
   • None must run today's code unchanged: toolkit.list_symbols and first_doc_line. Existing tests call these with no cache and must still pass, and --no-cache is the reference the proof compares against.
   • main() builds the cache from flags:
     • --cache PATH overrides the default ROOT / ".cache" / "codebase-map.json" (.cache is gitignored, .gitignore:75);
     • --no-cache passes None;
     • --cache-stats prints one stderr line in exactly this format: gen_codebase_map: cache: {hits} hit, {parsed} parsed, {n} files, {path}.
   • One cache object serves the whole run: header map, class map and full map. Save it once at the end of main().
   • When the toolkit fails to load, keep today's behaviour. The class map shows UNAVAILABLE, and the full map goes through the guard with "".
   • Without --cache-stats, print nothing new. The SessionStart hook now shows the generator's stderr.

Traps

• Do not hand-read or grep .py files. Read code with the deployed toolkit, run from the scratch dir: cd /tmp/claude-1001/-home-mohamed/829ea13c-8ceb-4f51-9b28-672fef74daca/scratchpad && env PYTHONPATH=/opt/orchestrata-tools/lib /opt/orchestrata-tools/.venv/bin/python /opt/orchestrata-tools/lib/scripts/serialize_codebase_ast.py source <name> --source /home/mohamed/workspace/wt-MAP-PRECOMMIT/scripts/gen_codebase_map.py The other subcommands are symbols and refs. Use Edit to change the file.
• The generator rewrites docs/CODEBASE_MAP.md without the hook's pointer block. The proof script restores both maps with git checkout HEAD -- docs/.... Never commit maps written by a bare generator run.
• Bash guards block rm, sed -i, a redirect into a .py file, and any command whose text contains a .claude path. Do not try to route around them.
• Do not run run_tests.py or pytest in this increment. Every source edit triggers a 180–234 s trace_intel preflight, and the tests are the next increment.
• Expected and never staged: plans/ast-*.json and reports/run-tests-latest.json show as modified. The hooks regenerate them.
• Output order must match exactly: the rglob order, the sort key, "—" for an empty doc, and the {n} symbol(s). lines. The byte comparison is the judge.

Run budget

Run the proof script at most 3 times. If it is still not green after the third run, STOP and report the failing lines and your diagnosis from /tmp/claude-delta-proof/*.err. If the warm run is green but slower than 3.0 s, STOP and report the timings rather than optimising further.

CONSTRAINTS:

• Edit only scripts/gen_codebase_map.py.
• Stdlib only, no new dependency. No pickle: JSON only.
• No change to serialize_codebase_ast.py or to either hook copy.
• Keep the file's existing comment density and style, and its WHY-comments.
• No mocks, no skips.
• Do not stage, commit, push or stash.

DONE WHEN: one run of bash /tmp/claude-1001/-home-mohamed/829ea13c-8ceb-4f51-9b28-672fef74daca/scratchpad/prove-delta.sh /home/mohamed/workspace/wt-MAP-PRECOMMIT prints all of the following:

• COLD-SAME, WARM-SAME, DELTA-SAME, DELTA-SEEN and BACK-SAME;
• every rc=0;
• cold stats showing 0 hit;
• warm stats showing 0 parsed with a wall time ≤ 3.0 s;
• delta and back stats each showing 1 parsed.

Separately, ruff check scripts/gen_codebase_map.py and ruff format --check scripts/gen_codebase_map.py must be clean, using the worktree's .venv/bin/ruff or uv run ruff.

VERIFY: bash /tmp/claude-1001/-home-mohamed/829ea13c-8ceb-4f51-9b28-672fef74daca/scratchpad/prove-delta.sh /home/mohamed/workspace/wt-MAP-PRECOMMIT > /tmp/claude-delta-proof.log 2>&1; echo "PROOF-EXIT $?" >> /tmp/claude-delta-proof.log

• It takes about 60 s: two no-cache builds at about 13 s each, one cold build at about 14 s, and three warm builds.
• Run it in the background and read /tmp/claude-delta-proof.log once it prints PROOF-EXIT.
• Report the log verbatim, plus git diff --stat.

DESIGN_WAIVER: none — the design above is decided by the orchestrator under Mohamed's ruling "maps should use delta".

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks