Orchestrata Ultimate › Plans › ORCHESTRATOR-CHECKLIST

Orchestrator Pipeline Checklist — Universal

Source: plans/ORCHESTRATOR-CHECKLIST.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-04-12 · 183 lines.

Prose pending.

Plan text

Orchestrator Pipeline Checklist — Universal

Mandatory for every subplan. Every step must be marked DONE with evidence before proceeding. Skipping steps is a pipeline violation. If a step fails, stop and investigate — do not proceed.

This checklist is the single source of truth for agent dispatch. Subplan-specific checklists inherit from this one and may add steps, but never remove them.

Phase 0: Automated Preparation (run first)

prepare.py automates Phases A4-A5, B, C, and D below, plus dependency analysis and gap review. Run it before doing anything else:

python scripts/orchestrator/prepare.py --config plans/plan_<name>/agent_config.yaml

This produces:

Fresh AST JSON (plans/ast-ultimate.json)
Preflight check results
Sandbox with work/, context/, verify/
Generated prompt at <sandbox>/verify/prompt.md
Gap review report at <sandbox>/verify/review.md

Read verify/review.md. If any FAIL items exist, fix them before proceeding. WARN items are advisory but must be reviewed.

Exit codes: 0=ready, 1=blocking gaps, 2=warnings only.

Phase A: Context Gathering (manual reading — not automated)

After prepare.py runs, you still must read and understand the artifacts yourself.

[ ] A1. Read the subplan. Read the full subplan document (plans/plan_<name>/subplan-*.md). Understand every requirement, every file involved, every coverage target. Do not skim.
[ ] A2. Read the agent config. Read plans/plan_<name>/agent_config.yaml. Understand writable_files, context_files, verify_commands, requirements.
[ ] A3. Read the execution plan context. Identify which steps in plans/execution-plan.md this subplan corresponds to.
[ ] A4. Confirm AST is fresh. prepare.py re-serialized the AST. Verify it ran successfully from the review report. If you ran --skip-ast, confirm ast-ultimate.json is newer than all source files.
[ ] A5. Read the gap review report. Read <sandbox>/verify/review.md. Understand every FAIL and WARN item. Fix all FAILs.
[ ] A6. Read all source files listed in context_files. Every source file the agent will receive as context — read it, understand its imports, its classes, its methods. Do not dispatch without understanding.
[ ] A7. Read all existing test files listed in writable_files. Understand what tests already exist before extending them.
[ ] A8. Review dependency gaps. The review report flags internal imports not covered by context_files. For each gap, decide: add the dependency to context_files (and re-run prepare.py), or confirm the agent does not need it.

Phase B: Preflight (automated by prepare.py)

[ ] B1. Preflight ran. Confirm from prepare.py output or review report.
[ ] B2. All PASS. Document any WARN/FAIL. If any check is FAIL, stop and fix before proceeding. WARN is acceptable only if documented with reason.
[ ] B3. Git working tree is clean. git status shows no uncommitted changes. If dirty, commit or stash first.
[ ] B4. On correct branch. Either master (for new branch creation) or the target branch from agent_config.yaml.

Phase C: Create Sandbox (automated by prepare.py)

[ ] C1. Sandbox created. Confirm from prepare.py output.
[ ] C2. Verify sandbox structure. List the sandbox directory:
```
ls -R <sandbox_base>/<agent_name>/
```
Confirm: work/ contains writable files, context/ contains read-only context.
[ ] C3. Verify context/ contents. Confirm these exist:
- context/spec.md — the specification extracted from the subplan
- context/standards.md — coding/testing standards
- context/traceability.md — requirements traceability
- All context_files from the agent config
[ ] C4. Verify work/ contents. Confirm writable files were copied correctly.

Phase D: Generate Prompt (automated by prepare.py)

[ ] D1. Prompt generated. Confirm from prepare.py output. Prompt is at <sandbox>/verify/prompt.md.
[ ] D2. Read the generated prompt. Verify it contains: sandbox path, writable files list, context files list, requirements, verify commands, rules, standards, traceability.
[ ] D3. Verify the prompt is self-contained. The agent receives ONLY this prompt and the sandbox files. If critical information is missing from the prompt, fix the template or config and re-run prepare.py.

Phase E: Dispatch Agent

[ ] E1. Dispatch agent with the generated prompt. The agent works in the sandbox — work/ is writable, context/ is read-only.
[ ] E2. Agent produces code in work/. All modified files are in work/.
[ ] E3. Agent produces work/AUDIT.md. The audit document with all 6 required sections and 16 self-audit rows.
[ ] E4. Agent signals READY TO VERIFY.

Phase F: Verify Agent (Automated Gate)

This is the hard gate. No self-reporting. The orchestrator runs the verify commands and the result is pass/fail.

[ ] F1. Run verify_agent.py.

python scripts/orchestrator/verify_agent.py --config plans/plan_<name>/agent_config.yaml

Evidence: full stdout/stderr captured.

[ ] F2. Examine results. If PASS: proceed to Phase G. If FAIL: continue to F3.
[ ] F3. On FAIL: feed test_output.txt back to agent. The agent reads verify/test_output.txt, fixes code in work/, signals READY TO VERIFY again.
[ ] F4. Re-run verify_agent.py. Maximum 3 iterations. If still failing after 3, escalate to human with full diagnostic.
[ ] F5. Record iteration count. How many verify iterations were needed.

Phase G: Validate Audit

[ ] G1. Run validate_audit.py.

python scripts/orchestrator/validate_audit.py --config plans/plan_<name>/agent_config.yaml

Evidence: exit code 0.

[ ] G2. All checks pass. 6 required sections present, 16 self-audit rows, no FAIL rows, all REQ IDs traced, no speculative language, changeset matches work/.
[ ] G3. On FAIL: agent fixes AUDIT.md. Re-run validation.

Phase H: Full Results Report

This is non-negotiable. Every subplan must produce a complete, unfiltered report.

[ ] H1. Run full test suite with coverage.
```
uv run pytest tests/ --cov=<cov_targets> --cov-report=term-missing -v --tb=short
```
Evidence: COMPLETE output — no filtering, no truncation, no hiding.
[ ] H2. Print coverage per file. Every file listed in the subplan must appear with its coverage percentage.
[ ] H3. Verify coverage targets met. Every file in the subplan meets its target (99% unless specified otherwise).
[ ] H4. If ANY file is below target: Stop. Create a remediation subplan for that file. New worktree, repeat the full pipeline from Phase A. Do not merge partial coverage.
[ ] H5. Print full diff. git diff of all changes the agent made.

Phase I: Human Review

[ ] I1. Present to user:
- Full diff
- Full coverage report (with term-missing output)
- AUDIT.md summary
- Verify iteration count
- Any WARN items from preflight
[ ] I2. User approves. Wait for explicit approval. Do not proceed without it.
[ ] I3. If user requests changes: Feed back to agent, re-run from Phase F.

Phase J: Merge

[ ] J1. Run merge_agent.py.

python scripts/orchestrator/merge_agent.py --config plans/plan_<name>/agent_config.yaml

Evidence: exit code 0.

[ ] J2. Verify all tests pass on the branch. Run the full test suite one final time.
[ ] J3. Report final state. Branch name, commit hash, test results, coverage numbers.
[ ] J4. Never auto-push. Print reminder: "Push manually after review: git push origin <branch>"

Anti-Patterns (Things That Must Not Happen)

Violation	Why It's Wrong
Not running `prepare.py` first	Skips AST, preflight, sandbox, prompt, and gap review all at once
Ignoring the gap review report	Dispatches with known missing dependencies or broken config
Skipping Phase A (context gathering)	Agent dispatched blind, writes wrong tests, wastes iterations
Skipping AST serialization (A4)	Agent works with stale understanding of codebase
Not reading source files (A6)	Agent doesn't understand what it's testing
Dispatching without generate_prompt.py (Phase D)	Agent gets ad-hoc prompt instead of structured, complete one
Self-reporting coverage instead of verify gate (Phase F)	Agent claims 99% without automated verification
Truncating or filtering coverage output (Phase H)	Hides failures, defeats the purpose of the gate
Proceeding after verify FAIL without fix loop	Merges broken code
Merging without human approval (Phase I)	Circumvents review
Auto-pushing to remote (Phase J)	Irreversible without review
Dispatching agent without AST context	Agent can't understand module dependencies, imports, signatures
Using a vague prompt instead of generated prompt	Missing standards, traceability, requirements

Checklist Versioning

Version	Date	Change
1.0	2026-04-12	Initial universal checklist, extracted from subphase 0.1 coverage subplan
1.1	2026-04-12	Added Phase 0 (prepare.py), A5 gap review, A8 dependency review

Orchestrator Pipeline Checklist — Universal

Plan text

Orchestrator Pipeline Checklist — Universal

Phase 0: Automated Preparation (run first)

Phase A: Context Gathering (manual reading — not automated)

Phase B: Preflight (automated by prepare.py)

Phase C: Create Sandbox (automated by prepare.py)

Phase D: Generate Prompt (automated by prepare.py)

Phase E: Dispatch Agent

Phase F: Verify Agent (Automated Gate)

Phase G: Validate Audit

Phase H: Full Results Report

Phase I: Human Review

Phase J: Merge

Anti-Patterns (Things That Must Not Happen)

Checklist Versioning

Related