Orchestrata Ultimate › Plans › briefs › IMPROVEMENT-LOG
Brief and agent improvement log
Source: plans/briefs/IMPROVEMENT-LOG.md at 8a2cb42ad · last changed 2026-10-22 · first committed 2026-10-22 · 70 lines.
Prose pending.
Plan text
Brief and agent improvement log
The companion to plans/briefs/TEMPLATE-sonnet-agent.md. Every Sonnet dispatch gets one row in
the dispatch ledger. Every defect gets one row in the defect log and a fold-back the same session.
How the loop works
- Before dispatch. Run the template's pre-dispatch check. A check that fails is a brief defect
caught early: fix the brief and log it anyway (class
caught). - After dispatch. Run the post-dispatch check (fence, re-run VERIFY, authorship). Each difference between the brief, the report and the re-run is a defect.
- Classify each defect:
inaccurate: the brief stated something false (a sha, a path, a command, a fact).incomplete: the brief left out a guard, a stage, a stop condition or a report field.deviation: the agent did something the brief did not ask for, or skipped something it did.agent-error: the agent made a mistake the brief had covered.environment: a fleet, hook or tool behaved unexpectedly; no brief line could have helped.
- Fold back the same session:
inaccurateorincomplete: change the template line or add a pre-dispatch check.deviationoragent-error: tighten the wording, move the rule into the STAGES as a check, or add an observable to DONE WHEN.environment: add it to the guard corpus if it will recur.
- Mechanize on the second sighting. A defect class seen twice becomes a mechanism candidate
(a check in
brief_lint.py, M2 indocs/supervisor-workflow.md§9) instead of another line of prose. Record the candidate in the mechanism queue below.
Measures (updated after each dispatch)
- First-pass rate: dispatches that reached DONE WHEN with no coordinator follow-up message.
- Follow-ups per dispatch: SendMessage corrections the coordinator had to send.
- Fence violations: dirty or committed paths outside FILES.
- Report fidelity: reported VERIFY output that the coordinator's re-run did not reproduce.
Dispatch ledger
| # | Date | Brief | Agent | Result | Follow-ups | Fence | Defects |
|---|---|---|---|---|---|---|---|
| 1 | 2026-10-22 | 2026-10-22-runner-failure-detail-ship.md | Sonnet builder | Attempt 1: STOPPED correctly at stage 4 (secrets scanner); accurate report with cause, four options, recommendation; no bypass. | 1 (amendment 1) | clean | D10, D11 |
Defect log
Seeded with this session's coordinator mistakes (2026-10-22). A worker given a brief without these lines would have made each of them.
| # | Date | Class | What happened | Evidence | Fold-back |
|---|---|---|---|---|---|
| D1 | 2026-10-22 | incomplete | A get_project_root call that raises was deployed into the tools tree, which vendors core/ with no pyproject above it. install-tools verification failed and was rolled back. |
6534b35a3, rollback --ref 44265c096, fix 3d77084d0 |
Guard corpus: smoke-install into a scratch PREFIX before any tools deploy. Pre-dispatch: list every deployed consumer of the changed module. |
| D2 | 2026-10-22 | environment | The secrets pre-commit hook flagged ROOT_ENV_KEY = "ORCHESTRATA_ROOT". |
commit 3d77084d0 (renamed ROOT_ENV_VAR) |
Guard corpus: names ending _KEY are flagged; rename, never bypass. |
| D3 | 2026-10-22 | environment | The commit review's mypy step timed out at 30 s on a cold .mypy_cache. |
commit review MYPY_TIMEOUT before 6534b35a3 |
Guard corpus: warm mypy on the changed files before committing. |
| D4 | 2026-10-22 | incomplete | The commit review flagged T21 on pre-existing env literals in touched test files. | tests/scripts/test_promote_env.py, tests/core/test_paths.py |
Guard corpus: T21 covers pre-existing lines in any touched test file. |
| D5 | 2026-10-22 | environment | The AST live-sync hook rewrites plans/ast-*.json and docs/CODEBASE_MAP*.md on every edit. |
git status after every edit |
Guard corpus: restore generated files before staging; stage by name. |
| D6 | 2026-10-22 | environment | Foreground test runs are blocked by a hook. | hook denial, 2026-10-22 | Guard corpus: start every test run in the background and observe it. |
| D7 | 2026-10-22 | incomplete | A full-suite pass failed three modules only because the dev fleet and claude-orchestrata were stopped. |
ship #14 promote log orch_ship14_promote.log |
Pre-dispatch: probe every fleet the step needs; a full-suite brief starts with a fleet pre-flight stage. |
| D8 | 2026-10-22 | inaccurate | The coordinator stated that no deployed venv carries the package, without checking the vendored lib tree. | D1 | Pre-dispatch: every VERIFIED line carries its evidence; a claim without a measurement is not in the brief. |
| D10 | 2026-10-22 | incomplete | Brief 1 warned only about _KEY constant names. The secrets scanner also refuses credential-named assignments in test fixtures (api_key = "sk-or-v1-" + ..., secret = ...), so commit 1 was refused. The coordinator never ran the staged scanners in the pre-dispatch check. |
brief 1 attempt 1, stage 4 hook output | Template: the scanner's real rule plus a pre-commit check command; pre-dispatch: stage the files and run the staged scanners. Second sighting of the secrets class (D2): mechanism candidate. |
| D11 | 2026-10-22 | incomplete | The pre-commit hook rewrites reports/precommit-walls.ndjson; the brief's restore list did not name it. |
brief 1 attempt 1, agent report | Template guard corpus and the brief's Generated list name it. |
| D9 | 2026-10-22 | environment | A foreground ls that named scripts/run_tests.py was refused as an unobserved test run. |
hook denial during the brief-1 pre-dispatch check | Guard corpus: the test hooks match command text; name the log, not the runner. |
Open findings (not brief defects; triage separately)
| # | Date | Finding | Evidence |
|---|---|---|---|
| F2 | 2026-10-22 | At the 13:41 session start the worktree venv was reinstalled editable (_editable_impl_orchestrata_ultimate.pth -> src, no site-packages copy), against the proper-install rule. A start hook's session import in the same second died with No module named 'orchestrata_ultimate.core.graph.config' from site-packages, consistent with importing mid-reinstall. Which hook does the reinstall is not yet identified. |
.pth mtime 13:41; SessionStart:compact traceback |
| F1 | 2026-10-22 | The run_tests progress ticker reported up to failed 14 and restarted its count mid-run, while the authoritative summary said 447 passed, 0 failed, over 368 selected tests. Most likely nested child runs from the test_run_tests* and test_failure_detail modules feed the live counter. Not verified. |
scratchpad runner_deps.log lines 25-44 |
Mechanism queue (second sightings)
| Defect class | Sightings | Candidate check | Status |
|---|---|---|---|
| Staged-content gate refuses a commit the brief did not foresee | D2, D10 | brief_lint.py: for a commit packet, stage the FILES in a scratch index and run every pre-commit scanner before dispatch |
queued |