Orchestrata Ultimate › Workflow › Improvements

Workflow improvements — dated log

This is the log of changes to how work gets done on orchestrata-ultimate: briefs, hooks, gates, the test runner, the map generators and the deploy launchers. Each entry says what changed, why, the measured before and after, and where it lives. Under RUL-041 a workflow change counts as an improvement only if the measurements improve, so every entry cites its numbers, or says "not measured".

Sources. Commit messages in the orchestrata repo (~/workspace/wt-MAP-PRECOMMIT, origin/master = a4cbc2bdf on 2026-10-23) and the codetools repo (~/workspace/orchestrata.codetools); the dispatch ledger (~/.local/state/codetools/dispatch-ledger.ndjson) for which worker built what and how long it ran; the session's IMPROVEMENT-LOG.md and TEMPLATE-sonnet-agent.md drafts and notes.md; the memory files named in each entry; and the deploy logs. Worker wall times are the ledger's wall_s. "Stopped" in the ledger means the worker's transcript went idle, not that it failed. Clock times are the commits' local time (EDT, UTC−4).

The current workflow these changes add up to is on Workflow; the bugs found on the way are on Bug fixes.

The 2026-10-22 numbers in one table

The figure "7.1 s down to 0.41 s warm" quoted for the ast map fast path in the task brief for this page was not found in any commit, log or ledger row read for it; the measured numbers above are the commit's.

How the brief-improvement loop works

Introduced with the template on 2026-10-22 (d9bcbb7cc, plans/briefs/IMPROVEMENT-LOG.md and TEMPLATE-sonnet-agent.md):

1. Before dispatch, run the template's pre-dispatch check. A failing check is a brief defect caught early; fix it and log it anyway.
2. After dispatch, check the fence, re-run VERIFY, check authorship. Every difference between the brief, the report and the re-run is a defect.
3. Classify: inaccurate (the brief stated something false), incomplete (it left out a guard, stage, stop condition or report field), deviation, agent-error, environment.
4. Fold back the same session: change a template line or add a pre-dispatch check; tighten wording or move a rule into a staged check; add an environment trap to the guard corpus.
5. Mechanize on the second sighting: a defect class seen twice becomes a mechanism candidate (brief_lint.py), not another line of prose.

Measures: first-pass rate, follow-ups per dispatch, fence violations, report fidelity.

The defect log on 2026-10-22 held D1–D22 (D9 out of order) and findings F1–F5. The mechanism queue at the end of the day (second sightings):

2026-10-23

Hooks run uv run --no-sync

• What: start-architecture-sync.sh (two calls, repo and plugin copies) and the ruff step of scripts/hooks/pre-commit (line 166) use uv run --no-sync.
• Why: a plain uv run re-syncs the venv as an editable install in a checkout without UV_NO_EDITABLE=1, and the pre-push source-provenance gate then refuses ("A hook observes; it does not change the environment").
• Before → after: three refused pushes on 2026-10-22 (logs push-item3*.log) → not measured: the commit is not pushed, and the plugin copy reaches ~/.claude only after a push and a config deploy.
• Where: 8a2cb42ad on feat/maps-via-codetools. Worker: Sonnet builder, 49.0 s. Memory uv-run-flips-venv-editable holds the interim workaround.

codetools no longer requires orchestrata-ultimate

• What: read_requirements() takes the OpenSpec provider as a parameter and resolves the first installed one; orchestrata-ultimate moves from a hard dependency to the openspec extra and the dev group.
• Why: a plain codetools install pulled orchestrata-ultimate and its janus-swi source build.
• Before → after: hard dependency → "codetools --version and ast map both run with 0 orchestrata packages installed" (commit message). Deployed as codetools install 75: WALL 43s, INSTALL-EXIT 0.
• Where: codetools a5c419f5. Worker: Sonnet builder, 105.0 s.

2026-10-22

The map hook creates docs/ (23:52)

• What / why: the codetools path moved the maps into docs/ without creating it, and || exit 0 swallowed the mv failure. Caught by the push gate (attempt 3 of the item-3 push).
• Where: a4cbc2bdf. Worker: Sonnet builder, 17.8 s. See the worked example.

Maps through the codetools verb (22:00)

• What: inject-codebase-map.sh runs /opt/codetools/bin/codetools ast map <repo> --with-full into a temp dir and moves the maps into docs/; the full map moves only when it holds 1,000 or more signatures. scripts/gen_codebase_map.py stays as the fallback (CODETOOLS_BIN overrides the path); [tool.codetools.map] in pyproject.toml declares the layout.
• Before → after: the script path of c995f5b88, 1.0 s warm → 0.5 s warm, 10.9 s cold. No data line differs from the previous hook's output; only header and footer blockquotes do (18 and 8 diff lines).
• Where: d4ae44819. Worker: Sonnet wirer, 76.5 s.

codetools ast map fast start path (21:54)

• What: codetools.main routes ast map ... straight to codetools.ast.map_cli.run; every other argv goes to the full CLI unchanged. A successful map run loads no MCP, deploy, gate or CLI modules.
• Why: hooks run codetools ast map at every session and subagent start.
• Before → after: the full CLI spent about 1–1.4 s importing before the verb ran; a tiny repo with --no-cache took 2.4–3.1 s; the codetools import took 118 ms → 0.07–0.08 s, a warm orchestrata probe 0.39 s, import 17.6 ms. 45 tests passed.
• Where: codetools b518f88a; deployed as install 74 (WALL 27s). Worker: Sonnet builder, 402.6 s.

codetools ast map verb with a per-file delta cache

• What: the two-tier codebase map becomes a codetools verb with a JSON cache outside the repo (~/.cache/codetools/map/), keyed by content sha1.
• Before → after: 33–39 s cold → 0.8 s for a no-change run in process; byte-identical in every data line to gen_codebase_map.py (cold, warm, one-file delta, restore). 40 tests in 1.2 s, 98% coverage of codemap.py.
• Where: codetools 25bcbf4f. Workers: Sonnet builder (stopped, 199.5 s) and Sonnet test-writer (504.8 s).

Per-file delta cache in gen_codebase_map.py (21:43)

• What: _MapCache keeps each file's symbol rows and docstring line in .cache/codebase-map.json, keyed by content sha1 and versioned by the script, the toolkit file and the interpreter.
• Why: Mohamed's ruling of 2026-10-22, "maps should use delta": re-parse and re-render only changed modules, byte-identical to a full regeneration.
• Before → after: 15.4 s with --no-cache → 1.0 s warm, 1.3 s with one changed file; proven byte-identical with prove-delta.sh. 8 TestMapCache tests.
• Where: c995f5b88. Workers: Sonnet builder (540.7 s), Sonnet test-writer (490.0 s), Haiku lint-fixer for two T4 markers (51.5 s).

Codebase maps in one process (20:38)

• What: gen_codebase_map.py --with-full builds the full skeleton in the same process from the toolkit it already loaded; the hook calls it once.
• Why: the hook ran the generator and then ten serialize_codebase_ast.py symbols CLI calls, each unpickling the 200 MB warm cache.
• Before → after: 128 s on an idle machine → about 15 s; both maps byte-identical (cmp).
• Where: e1f3a5047. Workers: five Sonnet increments (289.3 s, 62.3 s, 427.8 s stopped, 159.8 s, 23.1 s), split after Mohamed's "no more long waits".

Pre-commit regenerates the maps (19:35)

• What: a commit that stages Python regenerates and stages docs/CODEBASE_MAP.md and docs/CODEBASE_MAP_FULL.md; a failure warns and never blocks. worktrees.py stops copying the primary's full map.
• Why: the maps are tracked (RUL-030, Mohamed's 2026-08-13 ruling) but only SessionStart rewrote them, so every new checkout showed them modified. Ruling "A" of 2026-10-22.
• Before → after: maps dirty in every new worktree → regenerated per commit; the regeneration measured 13.5 s at the time (notes), which drove the four entries above.
• Where: b5468cb1c. Worker: Sonnet builder, stopped at about 25 min on Mohamed's word (3,057.9 s in the ledger).

Tools-installer tests share one read-only install (19:52)

• What: the six tests in tests/deploy/test_install_tools.py that only read an install share one session-scoped install made read-only with chmod -R a-w; each installer ==> phase prints its elapsed seconds.
• Why: every test ran the full installer, so the module outran the 300 s cap. Mohamed 2026-10-22, "1,2 and 3": an explicit exception to the function-scope fixture rule.
• Before → after: the installed fixture ran the installer for each of 8 tests (280 s timeout each); with warm caches a module run finished 9 of 11 tests before the 300 s cap → one shared install. Phases measured: venv and dependencies about 3 s, native build 5–8 s, verify 12.6 s, about 22 s in all, so the venv is not reused.
• Where: 7309a3488, deployed to /opt/orchestrata-tools (VERSION 19:55:40). Workers: two Sonnet builders stopped (616.8 s, then a restart with a better brief at 135.8 s), then two increments (62.5 s, 45.7 s).

The deployed AST toolkit gets bin/ast (19:25)

• What: the tools installer generates /opt/orchestrata-tools/bin/ast, so the toolkit runs from any directory "without littering foreign trees".
• Why: the coordinator had been running the toolkit from another session's checkout, which writes its warm cache to plans/ under the cwd and most likely rewrote that checkout's 349 MB plans/ast-ultimate.json.warmcache (memory use-deployed-tools).
• Where: 2c29cd9d3 (brief b68276bfd), deployed with tools-deploy (DEPLOYED 2c29cd9d3 -> /opt/orchestrata-tools). Worker: Sonnet builder, 1,230.9 s.

Deploy launchers: ct-install, tools-deploy, hermes-deploy

• What: three user-level launchers in ~/bin, each pre-approved by an allow rule in ~/.claude/settings.local.json, each refusing unsafe states and verifying the result:
  • hermes-deploy <commit>: refuses unless the commit exists, master is clean and can fast-forward, no lock is held and the DeepSeek key is set; exports only two provider keys; deploy/install.sh verifies and rolls back. First use 8e62d6f, about 2 min, DEPLOY-EXIT 0 (notes); b1b66a6 later, DEPLOY-EXIT 0 (state log).
  • tools-deploy [--check] <commit>: runs that commit's deploy/install-tools.sh from its own sparse worktree (deploy/ and env/, 5.6 MB) and fails unless VERSION records the commit.
  • ct-install <label> [ref] (Hermes model, 2026-10-22, "make the installer like hermes"): build and promote with no suite gate, then verify-identity and ct-smoke, rollback and exit 12 on failure. It unsets PYTHONPATH and PYTEST_ADDOPTS, because install 68 was refused by codetools' import-shadow guard (codetools 6979a4ad).
• Why: Claude Code's auto-mode classifier blocks each raw installer as "Production Deploy", so every deploy needed a paste; running an installer from a worker's tree also risked bash reading a script while it was being edited.
• Before → after (codetools): install 72 (e1ffbc3) spent 60 s on selection and 3+ min in 8 shards and was stopped → install 73 (same commit) WALL 26s, against 25 s for Hermes. Installs 74 and 75: 27 s and 43 s.

The worker's stop gate: precommit-check + SubagentStop

• What: ~/bin/precommit-check <repo> [file…] runs the exact commit gate on a temporary index; ~/.claude/hooks/precommit_gate_subagent.sh blocks a worker's stop (at most 3 times) until its .py edits pass. → Gate: commit-precheck
• Why: Mohamed, "you keep being blocked on lint every time ... when will you ever fix that". Workers' DONE WHEN ran only ruff and py_compile.
• Before → after: findings first met at git commit → met at SubagentStop; 0.4 s with nothing to review, about 5 s for one test file; the pipe test caught a T4 violation and passed a clean file and a read-only worker (memory). Not in any repo (user level, RUL-035).

The dispatch ledger records itself

• What: ~/.claude/hooks/dispatch_ledger.py (SubagentStop) appends every finished worker to ~/.local/state/codetools/dispatch-ledger.ndjson in the codetools ledger format, with model, wall time, tool calls, tokens and peak context.
• Why: Mohamed: "make the ledger build automatic", "codetools has a ledger, use that"; he cancelled the spreadsheet.
• Before → after: the improvement log's dispatch table had one hand-written row → 41 rows by 2026-10-23 04:44 UTC; backfilled with the session's 21 earlier workers. Rendered at Dispatch ledger.

Briefs: the template, the improvement log, and "better briefs"

• What: plans/briefs/TEMPLATE-sonnet-agent.md (skeleton, pre-dispatch and post-dispatch checks, a guard corpus every brief restates) and plans/briefs/IMPROVEMENT-LOG.md (d9bcbb7cc, 13:55). Later the same day, after an installer worker died at its VERIFY cap, the "better briefs" rules: state already measured, the decided design, a VERIFY that has already run green, the traps, a run budget, exact staging paths, and a self-contained prompt (memory better-briefs).
• Why: the defects in the log: a wrong PYTHONPATH instruction, a worktree without a venv, a VERIFY that could not finish in 300 s because a 180–234 s trace_intel refresh runs after any source edit (notes), labels the dispatch hook rejected.
• Before → after: not measured as a rate. The ledger shows the effect on one packet: the installer brief's first worker stopped after 616.8 s; the rewritten brief's increments finished in 62.5 s and 45.7 s.

Let the tooling run

• What: when a worker or a deploy path has its own verify and rollback, run it; do not stack measurement proposals and options in front of it. A failure is a reason to read the output and fix, not to come back for a decision (memory let-the-tooling-run).
• Measured: the Hermes RUL-037 guard was built in about 9 minutes with 121 tests green (ledger: 521.4 s); install.sh rolled back safely when DeepSeek credit ran out, and the redeploy after top-up passed its live test (notes).

The runner keeps the full failure (13:54)

• What: each batch gets --tb=long, --showlocals and JUnit logging of the failing tests' output; a module rescued from a capped or killed batch keeps its failure text; a hung module gets the killed batch's faulthandler dump. Everything passes through redact_secrets first, because reports/run-tests-latest.json is committed by ship.
• Before → after: one line per frame and an empty stdout for rescued modules → full traceback, locals and output, capped at 60,000 characters per module.
• Where: a934ef33b. Worker: Sonnet builder, 693.9 s ("Commit, smoke-install, ship runner change").

Ship's plain-pytest gates run past the observer (10:58)

• What: deploy/ship.sh's knowledge-wiki and dr-hannah gate arms run through gate_pytest, which stamps ORCHESTRATA_SANCTIONED_RUNNER=ship.sh.
• Why: a Claude session loads the RUL-038 observer into every pytest, and it refused the wiki's 13-file gate; the gate failed from any Claude session and passed from a terminal.
• Where: 7b859bb4a.

The Stop hook blocks only for this session's branches (00:01)

• What: unshipped_guard.py blocks a turn only for a watched repo whose branch moved during this session (reflog or committer date at or after the transcript's first timestamp).
• Why: on 2026-10-21 eight-day-old unpushed commits in another repo blocked every turn of an unrelated session.
• Where: 32b8eb6e8; 15 new tests, 95% line coverage. Open: D22, the guard cannot see a ship in flight. → Gate: unshipped-guard

2026-10-21

• Affected classification before infra (GATE-AFFECTED). run_tests.py resolves --affected before the infrastructure preflight and passes a push whose changed files are all certified inert without booting the services. Before: the infra preflight (kernel, graph, gateway and all worker DBs) ran first on every push, "~3 min" (brief plans/briefs/gate-affected-classify-before-infra.md, "measured 2026-10-21"). After: not measured in this run. 68024751d.
• The push range is the pushed commit. The pre-push hook reads the ref update on stdin, so a chunked push verifies each chunk. e27c12b50.
• The pre-push hook runs the venv python, not uv run, which had re-synced the package editable and tripped the provenance gate. 949c1628c.
• No boolean escalation. affected_tests.py emits per-file verdict rows (bounded / named_full / refused); run_tests.py --affected refuses naming the missing fact instead of running the full suite; the pre-push hook's own "base not found → full suite" arm aborts. 2669f4882 (RUL-040). Follow-up 306e9eb5e makes non-code agent files, empty __init__.py markers and transferred scripts inert instead of refused.
• T21 reaches staged scripts. commit_review.pl routes staged scripts/*.py into the test review's T21 pass, with a 54-row, 36-file worklist baseline so named debt does not block existing files. 0d329660d, 0ffb2d1ec (ADR-0116).

2026-10-20

• The test phase is selected, counted and capped. 300 s hard cap, selected-count and progress lines, the sanctioned-runner stamp, the multi-file refusal. Adopted from a prior session's partial work and completed: 146/146 tests. 4a470482c. → Gate: test-phase-cap
• T21, env-key drift. A literal similar to but not equal to a declared *_ENV constant is an ERROR. Minted after the cap work stamped fabricated env keys. 25f06c3f6.
• Proper install. The wheel ships all three top-level packages, the PYTHONPATH shadow is dropped, and the suite's provenance guard is re-based on PEP 610 (dd919d55c); every env-creating site carries --no-editable, with a state canary (9aa8a72f6).
• The lessons ledger. L11–L25 with evidence and the mechanism that owns each (b113647a5); summary on Lessons — supervisor session 2026-10-19/20.

2026-10-11 to 2026-10-19

• 2026-10-17, code_map.py. Prints a file's top-level names with line spans, or one definition's exact source, in under a second; the brief template gained a Code map section. 4e897a6d9.
• 2026-10-16, dispatch commit. Commits only the brief's files, resets index entries the ruff pre-commit step re-staged, flags hook-modified files. b498ece2c.
• 2026-10-14, codetools shared green ledger. The ADR-0088 shared green ledger is the one per-test proof record (codetools ADR-0044, 44b7283b).
• 2026-10-13, stage-walls ledger. The pre-commit hook records the wall of each stage to reports/precommit-walls.ndjson (ADR-0101 rule 2). a8b3c15f5. It now holds 484 lines.
• 2026-10-13, codetools: one live-systemd test at a time. A per-user flock serializes every test that drives the live systemd --user manager, after an 8-worker run logged the X session out. Verified: a live module blocked 6.8 s behind the lock while a non-live module ran in 0.2 s. codetools b52409af.
• 2026-10-13, codetools batch gate. Environment contract and readiness (codetools ADR-0039), per-merge selected batch under 300 s with a scheduled full batch (ADR-0040), scope against the shared record, a full batch never a prerequisite (ADR-0042). → Gate: codetools-gates
• 2026-10-11, the commit verdict is derived in Prolog. commit_review.pl decides pass, fail or invalid from what each reviewer reported; before, the shell review passed every commit three ways. 8e0df6ff3. → Gate: review-staged

Earlier milestones

• 2026-09-22, RUL-032 proven-content ledger. A test module whose content key matches its last green run is reused. Measured the same day: the selector escalated to 1,608 modules while 197 modules (3,977 tests) had already passed on identical code. c822ec2e6.
• 2026-08-07, one entry point for the review battery. scripts/gates.py: parallel, deduplicated by input fingerprint, self-healing (ruff and black before the gates, AST refresh). Measured: about 50 s serial → about 25 s parallel. 07e5c2559.
• 2026-08-07, solo-retry cap 5 → 25. "the cost premise was wrong (9 modules re-ran solo in 5s against a 300-500s batch phase) and the cap ran backwards" (run_tests.py help).
• 2026-07-27, tracked git hooks. The hooks moved from one untracked .git/hooks shared by every worktree to tracked scripts/hooks/ with a relative core.hooksPath, and pre-push refuses a protected-branch worktree. 854bc6d97.

Still open (2026-10-23)

Related

• Workflow
• Gates — master index
• Bug fixes, Bug-fix log
• Dispatch ledger, Briefs
• Green ledger, Tests
• Deployment, codetools
• Lessons — supervisor session 2026-10-19/20
• RUL-041, RUL-032, RUL-040