Orchestrata Ultimate › Plans › workflow-improvements-plan

Workflow Improvements Plan — From Memory Export 2026-07-26

Source: plans/workflow-improvements-plan.md at 8a2cb42ad · last changed 2026-08-13 · first committed 2026-07-26 · 369 lines.

Prose pending.

Plan text

Workflow Improvements Plan — From Memory Export 2026-07-26

Source: memory export of 2026-07-26 (global + 4 project memory stores, SESSION_START.md, global CLAUDE.md, 645-prompt Kimi user-history). The prompt history shows the same rules being re-issued verbally for months before they were mechanically enforced. This plan converts the recurring failure patterns into permanent, machine-checked workflow gates.

Conventions: every step has a deterministic verification. No step is "done" until its verification passes and its wiring site is named (Hard Rule 8: built is not done — wired is done). Execute in phase order; steps within a phase are parallelizable where marked.

Phase 0 — Canonical rulings store (foundation; everything else references it)

Problem (evidence): rulings live in 5+ places (global CLAUDE.md, project MEMORY.md ×4, SESSION_START.md, AGENTS.md) and were re-taught per agent tool. The prompt history shows no-xdist, no-mocks, Incus-not-Docker, and .env-only each re-issued >10 times before enforcement.

Decision: the wiki (Wiki.js 172.16.0.26:3000, wiki CLI on PATH) is the single source of truth for rulings. Tool-specific memory files keep tool-specific lessons only.

Read path rule: gates and hooks read rulings from the git-backed markdown files (~/workspace/knowledge-wiki/rulings/*.md — the wiki's disk storage writes into that repo), NEVER via wiki get. A live Wiki.js server must never be a dependency of run_tests.py, health_check.py, or session start — a wiki outage breaking the test gates is a false-red class. wiki get is for humans and liveness checks only.

Step 0.1 — Rulings schema + seed

• Create wiki page rulings/index — table of all rulings: id | date | ruling (verbatim user words where known) | enforcement mechanism | status.
• One child page per ruling: rulings/RUL-NNN-<slug> with sections: Ruling, Evidence, Enforcement, Verification.
• Seed from the 2026-07-26 export. Initial set (ids reserved): RUL-001 no-xdist, RUL-002 no-mocks/stubs/fakes/skips/monkeypatch, RUL-003 no-literal-default-for-required-infra-endpoints, RUL-004 incus-not-docker, RUL-005 threadpool-only, RUL-006 fix-on-sight, RUL-007 no-AI-attribution, RUL-008 review-before-commit, RUL-009 force-test-observation, RUL-010 never-skip-infra-verification, RUL-011 ports-from-env-no-defaults, RUL-012 done-means-wired-and-deployed, RUL-013 specs-authoritative, RUL-014 rulings-are-binding, RUL-015 plan-fidelity (Article XII), RUL-016 ontology-before-graph-import, RUL-017 no-weasel-words, RUL-018 kimi-runs-at-agent-level (IRC nick discipline), RUL-019 pydantic-dotenv-config-pattern, RUL-020 async-by-default.
• Rulings ratified into the constitution link both ways: the ruling page's enforcement: field may be constitution <article> (RUL-015 = Article XII in constitutional.pl), and Step 2.1's lint verifies the article actually exists in the .pl file — otherwise the canon and the constitution drift apart, recreating the exact problem this phase solves.
• Verification: wiki get rulings/index exits 0 and lists ≥20 rulings; each rulings/RUL-* page exists (wiki list | grep -c rulings/RUL- ≥ 20); the files are present in the git-backed store (ls ~/workspace/knowledge-wiki/rulings/RUL-*.md | wc -l ≥ 20).

Step 0.2 — Point every bootstrap at the canon

• Edit AGENTS.md, SESSION_START.md, ~/.claude/CLAUDE.md: replace duplicated ruling text with "Rulings canon: wiki get rulings/index — read at session start; tool-specific notes below."
• Keep the enforcement summaries (hooks table) in AGENTS.md — those are mechanism docs, not rulings.
• Verification: grep -n "rulings/index" AGENTS.md SESSION_START.md ~/.claude/CLAUDE.md — 3 hits. No ruling text remains duplicated: grep -c "NEVER use pytest-xdist" AGENTS.md stays (it's the enforcement doc), but preference prose is gone from all but one location.

Step 0.3 — Ruling-registration protocol

• Add to AGENTS.md (Agent Discipline section): "New user ruling → same session: (1) add wiki page rulings/RUL-NNN, (2) add/extend a mechanical gate or record enforcement: none (judgment), (3) log via session choice. A ruling that exists only in prose is incomplete work."
• Verification: protocol text present in AGENTS.md; next ruling issued in a session lands in the wiki within that session (checked at session close by Step 0.4's lint).

Phase 1 — session prime: one bootstrap command

Problem (evidence): startup context is fragmented across AGENTS.md's manual startup protocol, SESSION_START.md, context.md, wiki, AST refresh, infra check. Prompt history: "refresh yourself", "why do I have to remind you", "when I start kimi in this folder I want you to know what to do".

Step 1.1 — Implement session prime

• Extend the session CLI (src/orchestrata_ultimate/session/, session command) with a prime subcommand that runs, in parallel where independent, and prints one digest:
• Boundary rule: prime lives in the product package, so every input must be repo-relative or env-driven — no hardcoded host paths (~/.claude/..., ~/workspace/knowledge-wiki/...). The rulings-store path and hook locations come from env/config with the digest section degrading to "unavailable" when unset. Host-workflow specifics stay in the hooks that CALL prime (Step 1.2), never inside src/orchestrata_ultimate/session/.
  1. ORCHESTRATA_ENV_PROFILE=test python scripts/start_test_services.py --status (infra: redis/graph/gateway/kernel on the isolated test fleet)
  2. session status + open carryover tasks (session task list)
  3. Last test verdict: latest entry of .flake_recoveries.ndjson + newest reports/*review* mtime
  4. Rulings digest: read rulings/index.md from the git-backed wiki store (id + title lines only) — filesystem, not wiki get (Phase 0 read-path rule)
  5. Stale-tracking warning: flag if plans/execution-plan.md or context.md is >7 days older than the newest git tag
  6. Git state: branch, ahead/behind origin, dirty file count
• Exit 0 always (digest, not a gate); non-zero only on internal error.
• Verification: session prime exits 0 and its output contains all 6 section headers; runtime < 30s with infra up.

Step 1.2 — Wire into session start (both agent tools)

• .kimi/hooks/session-start.sh: run session prime and inject output into context.
• .claude/ SessionStart hook equivalent (alongside session_resume_context.py): same call.
• Replace AGENTS.md's 4-command "Automatic Startup Protocol" with "run session prime".
• Verification: start a fresh kimi session in this repo → prime output appears in first-turn context; grep -n "session prime" .kimi/hooks/session-start.sh AGENTS.md — 2+ hits.

Phase 2 — Rulings enforcement lint (close the prose-only loophole)

Problem: a ruling with no mechanism gets violated again (proven repeatedly in the history).

Step 2.1 — scripts/rulings_lint.py

• Reads rulings/index.md + rulings/RUL-*.md from the git-backed wiki store on the FILESYSTEM (path from env, default ~/workspace/knowledge-wiki/rulings/) — never wiki get, so the gate works with Wiki.js down (Phase 0 read-path rule). Extracts each ruling's declared enforcement: field.
• For enforcement: hook <path> → file must exist and be executable.
• For enforcement: gate <script> <rule-id> → script exists and rule-id appears in it.
• For enforcement: banned-deps <name> → entry in .claude/banned-libraries.txt.
• For enforcement: constitution <article> → the article exists in the governance .pl rulebase (grep the constitution source, e.g. Article XII for RUL-015).
• For enforcement: none (judgment) → pass, but listed in output as judgment-only.
• Exit 1 on any missing mechanism.
• Verification: python scripts/rulings_lint.py exits 0 today (all seeded rulings mapped: RUL-001→.claude/hooks/no-xdist.sh, RUL-002→scripts/test-review.py T-rules, RUL-003/011→architecture_checker/grep gates, RUL-009→.claude/hooks/force-test-observation.sh, …); deliberate breakage test — point one ruling at a nonexistent hook → exits 1.

Step 2.2 — Wire the lint

• Add rulings_lint.py to the safety gates in scripts/run_tests.py (alongside existing gates) and to scripts/health_check.py.
• Verification: python scripts/run_tests.py --skip-infra gate section shows rulings-lint PASS; python scripts/health_check.py includes it.

Phase 3 — Affected-test selection (kill the 30-minute spin)

Problem (evidence): prompt history is full of "too long, max wait 2 minutes", "spinning for 30 mins not an option", "run only the part that fails". The AST toolkit already has who-imports, refs, call-graph, and serialize — the dependency data exists, it's just not wired to the runner.

Step 3.1 — scripts/affected_tests.py

• Build reverse dependency map: for each src/orchestrata_ultimate/** module, the test modules that import it (directly or transitively, depth ≤ 3) — from plans/ast-ultimate.json + plans/ast-tests.json (regenerate via serialize if >1 day stale).
• CLI: python scripts/affected_tests.py [--since HEAD~1 | --files f1 f2 ...] → prints test paths, one per line. Always includes tests/infrastructure smoke set as a floor.
• Non-import dependencies (this suite is full of them): the import closure alone will silently miss prolog/*.pl rule files, .claude/hooks/, prompt templates, the .env contract, and scripts invoked via subprocess. Two hard rules:
  1. Any changed file outside src/ and tests/ → escalate to FULL suite (print why).
  2. A small explicit manifest (plans/affected-tests-manifest.json) maps known non-import edges (e.g. prolog/*.pl → tests/governance/, tests/kernel/test_pam_*, tests/prolog/; scripts/X.py → tests/scripts/test_X.py). The manifest is consulted in addition to the import map, never instead of rule 1.
• Verification: touch src/orchestrata_ultimate/gateway/audit.py → output contains tests/gateway/test_audit_behavioral.py; touch a file with no tests → output is the smoke floor only; map build < 60s warm.

Step 3.2 — Wire into the runner

• scripts/run_tests.py --affected [REV]: resolve test paths via affected_tests.py, run the threadpool executor on just that subset, then print "subset run — non-authoritative, full suite still owed" (mirrors the --skip kernel honesty rule).
• Verification: python scripts/run_tests.py --affected --skip-infra --skip-gates after a one-file src change completes in < 2 min and runs only the affected modules; full python scripts/run_tests.py unchanged (27307 collected).

Step 3.3 — Per-test duration tracking

• Source of truth: pytest_live_observer ALREADY emits per-test duration in its event stream (core/threading/pytest_live_observer.py:466) and is mandated on every run (RUL-009 force-test-observation) — derive from that, do NOT build a second JUnit-parsing path that only covers runner-invoked runs.
• Add scripts/test_durations.py to consume the observer events and append {test, duration, ts} to reports/test-durations.ndjson, with a --slow 20 report flag.
• Wiring (the collection must be automatic, not run-by-hand): the append happens inside the observer plugin's session-finish hook itself (pytest_live_observer already sees every run per RUL-009) OR as an unconditional post-run step in run_tests.py — pick one at build time, name it in the commit. test_durations.py --slow is then a pure reader.
• Slow-test budget: any test > 60s is listed in the run summary as a defect candidate — printed by run_tests.py's summary, not a separate command.
• Verification: run ANY test module the normal way (observer mandate) → reports/test-durations.ndjson grows with zero manual steps; python scripts/test_durations.py --slow 5 prints the 5 slowest with numbers; a run containing a >60s test shows the defect-candidate line in the runner summary.

Phase 4 — Status from artifacts, not hand-maintained files

Problem (evidence): memory entry "Tracking files are stale & contradictory" — plan files, session tasks, context.md, wiki all hold overlapping status.

Step 4.1 — scripts/project_status.py

• Generates one status report purely from artifacts: latest git tag + session number, test verdict (last run_tests JSON report), RTM evidence summary (scripts/generate_rtm_evidence.py → reports/rtm-evidence.json), coverage floor from [tool.coverage], infra status.
• Drift warnings: any plans/*.md checkbox marked done whose linked verification command fails; any session task list item open > 30 days.
• Verification: python scripts/project_status.py exits 0, prints artifact-sourced numbers only (no hand-maintained fields); deliberate contradiction (mark a plan item done with failing verification) → drift warning printed.

Step 4.2 — Convention: intent vs state

• Add to AGENTS.md: "Plan files and session task hold INTENT (what's next) only. STATE (what's done/verified) comes from scripts/project_status.py. Never record state by hand."
• Verification: convention text in AGENTS.md; session prime (Phase 1) embeds project_status.py output instead of reading stale fields.

Phase 5 — Prompt-history mining (quarterly drift catch)

Problem: corrections repeat across sessions; the export showed the pattern. Catch new ones early.

Step 5.1 — scripts/mine_prompt_history.py

• Primary input: the KG's SPEC-021 conversation capture (Turn/conversation nodes in Neo4j) — the Claude producer is deployed user-level and kimi/IRC has its own producer, so the KG already holds structured, multi-platform turns. Mining it avoids re-parsing raw formats and covers all agent tools uniformly.
• Fallback inputs for pre-capture history only: ~/.kimi/user-history/*.jsonl, ~/.claude/history.jsonl (prompts only — no correction context; grade findings accordingly).
• Pass 1: extract prompts containing ruling markers (case-insensitive: "never", "forbidden", "banned", "i commanded", "i want", "do not", "no mock", "no xdist", "hardcod").
• Pass 2: cluster by normalized keyword signature; rank by recurrence count.
• Pass 3: diff clusters against rulings/index (via wiki get) → report: KNOWN+ENFORCED, KNOWN+JUDGMENT-ONLY, UNTRACKED (candidate new rulings).
• Output: reports/ruling-drift-YYYY-MM-DD.md. Read-only; never auto-creates rulings.
• Verification: run on current history → report exists, the known set (xdist, mocks, incus, env, threadpool) grades KNOWN+ENFORCED, and any UNTRACKED clusters are listed with verbatim example prompts.

Step 5.2 — Cadence (mechanical, not prose)

• A checklist line is prose enforcement — the exact loophole Phase 2 closes. Wire it instead:
  1. systemd user timer orchestrata-history-mining.timer (OnCalendar=quarterly, Persistent=true) running the miner and writing reports/ruling-drift-YYYY-MM-DD.md (same pattern as the deployed orchestrata-kg-retention.timer).
  2. session prime staleness check: newest reports/ruling-drift-*.md older than 100 days → digest line "ruling-drift report stale — miner timer broken?". The timer does the work; prime surfaces silent timer failure.
• UNTRACKED clusters still get filed as session task items by whoever reads the report — that judgment step stays human, but the report's existence is machine-guaranteed.
• Verification: systemctl --user list-timers | grep history-mining shows the next fire; systemctl --user start orchestrata-history-mining.service produces the report; with all reports deleted, session prime prints the staleness warning.

Phase 6 — Pre-merge conformance gate for delegated work

Problem (evidence): the Claude-delegation loop failed on scope violations and forbidden patterns in returned code ("you must let claude know which files it can change, and check for those only"). The scripts/factory_* pipeline (dispatch/enforce/conformance/dor_procedure) is the current delegation machinery — audit it first, close gaps second.

Step 6.1 — Audit the factory pipeline

• Read scripts/factory_dispatch.py, factory_enforce.py, factory_conformance.py, factory_dor_procedure.py. Answer: does returned code get (a) diffed against the assigned file scope automatically, (b) run through code-review.py + test-review.py before integration, (c) auto-rejected back to the builder on failure without user involvement?
• Output: gap list appended to this plan.
• Verification: gap list written; each existing capability cited by file:line.

Step 6.2 — Close the gaps

• For each gap from 6.1, extend the factory script that owns that stage (minimal change, matching existing style). Scope check = git diff --name-only ⊆ assigned list. Review gate = both scripts exit 0. Auto-reject = non-zero pipeline stage + builder brief with the violation list.
• Verification: synthetic dispatch with an out-of-scope edit → pipeline rejects, names the violating file, no human step; synthetic dispatch with a mock in a test file → rejected by test-review; conforming dispatch → passes to integration.

Step 6.3 — Inject rulings into delegation briefs (prevent, don't just reject)

• Steps 6.1/6.2 gate returned work post-hoc; it is cheaper to stop violations at the brief. Two injection sites:
  1. .claude/agents/*.md subagent definitions: reference the rulings digest (filesystem read, Phase 0 path) so builder/wirer/test-writer briefs always carry the canon.
  2. The factory DoR procedure (B23, factory/dor_procedure/): a DoR artifact is incomplete without the rulings checklist section — enforced by its existing validate stage.
• Verification: grep -ln "rulings" .claude/agents/*.md covers builder, wirer, test-writer, refactorer; a synthetic DoR without the rulings section fails factory_dor_procedure.py validate.

Execution order & effort

Phases 0–2 first (one session), then 3, then 6; 4–5 fill gaps.

Per-phase checkpoint rule: every phase ends COMMITTED (scoped commit naming the phase) with its verification output shown, before the next phase starts. A mid-plan interruption must never leave half-wired gates — Hard Rule 8 applied to the plan itself.

Definition of done (whole plan)

1. Every step's verification command passes — shown, not asserted.
2. python scripts/run_tests.py full suite green (authoritative, kernel verified).
3. python scripts/rulings_lint.py exits 0; every RUL has a mechanism or explicit judgment mark.
4. AGENTS.md updated for every convention change (per repo rule: keep AGENTS.md current).
5. session end with artifacts, choices, handoff; CHANGELOG + tag per session protocol.

Phase 6 audit findings (2026-07-26)

Audited: scripts/factory_dispatch.py, scripts/factory_enforce.py, scripts/factory_conformance.py, scripts/factory_dor_procedure.py, and the factory/ package modules they front (factory/dispatch/gates.py, factory/enforcement/enforce.py, factory/conformance/engine.py, factory/scheduler/{dag,queue,scheduler,worktrees}.py).

(a) Is returned code diffed against the assigned file scope automatically? No, before this change. factory/dispatch/gates.py:224-258 (staleness_gate) runs git diff --name-only base..HEAD and rejects on overlap with declared paths, but it diffs trunk against the base commit to detect concurrent edits (ADR-0033 §7 dispatch staleness) — it never looks at the worker's own worktree changes. Nothing in factory/ compared a returned diff to TaskSpec.declared_paths (factory/scheduler/dag.py:53-57, the brief's own file-scope declaration). The pipeline's only per-entry admission hook, regate: Callable[[MergeEntry], RegateVerdict] (factory/scheduler/queue.py:92-99, invoked from MergeQueue.drain, queue.py:107-147), was wired in scripts/factory_pipeline.py:140-144 (pre-change) to _default_regate, which checked only store.facts_in(entry.lease.namespace) — the CLI's own docstring called this "intentionally MINIMAL" and said to "swap _default_regate for a real gate once that lane lands" (pre-change factory_pipeline.py:41-47). GAP.

(b) Does it run code-review.py + test-review.py before integration? No. grep -rl "code-review\|test-review" src/orchestrata_ultimate/factory/ scripts/factory_*.py returned zero hits before this change. The nearest neighbor, factory/conformance/engine.py (check_conformance, engine.py:485-533, fronted by scripts/factory_conformance.py), is a separate, unwired structural AST≡design checker — module/symbol/edge existence only, explicitly not lint/type/mock discipline (ADR-0008 boundary, engine.py:1-13) — and nothing in factory/scheduler/ or factory/dispatch/ calls it. GAP.

(c) Is failure auto-rejected back to the builder with the violation list, no human step? The ROUTING mechanism already existed and worked: RegateVerdict(passed=False, reason=...) (queue.py:39-58) → MergeQueue.drain records MergeOutcome(promoted=False, ...) and leaves the namespace untouched as a GC candidate (queue.py:124-130) → run_pipeline turns that into TaskReport(outcome="regate_rejected", ..., detail=outcome.verdict.reason) (factory/scheduler/scheduler.py:253-258) — already proven by tests/factory/test_merge_queue.py:: test_regate_fail_rejects_and_leaves_namespace_intact and tests/factory/test_scheduler_pipeline.py:: test_pipeline_regate_rejection_is_recorded_and_namespace_intact. Not a gap in the mechanism — the gap was that _default_regate (per (a)/(b)) never had a real violation to feed it.

Gaps closed (6.2): src/orchestrata_ultimate/factory/dispatch/ conformance_gate.py (new) adds scope_gate (worktree git diff + git ls-files --others vs declared_paths, reusing gates._path_overlaps) and review_gate (real code-review.py/test-review.py subprocesses on every changed .py file), composed by conformance_regate — a Callable[[MergeEntry], RegateVerdict] dropped straight into the EXISTING (c) routing, no parallel mechanism. RegateVerdict gained one additive, backward-compatible field, violations: tuple[str, ...] = () (factory/scheduler/queue.py), for machine-readable detail alongside the existing reason string. scripts/factory_pipeline.py's _default_regate now calls conformance_regate (closing the CLI's own "swap for a real gate" TODO). Verified end-to-end via three synthetic-dispatch tests in tests/factory/test_conformance_gate.py (real git worktrees, real Redis, real Neo4j, run through the real run_pipeline): an out-of-scope file is rejected naming it, zero human step; a unittest.mock import in a test_*.py file is rejected by the test-review gate; a conforming change promotes. factory/conformance/engine.py (B2 AST≡design) remains a deliberately separate, not-yet-wired lane per ADR-0008 — this phase did not wire it in, since (b)'s brief scoped the gap to the two named review tools.

Related

• Plans
• Orchestrata Ultimate — project index
• Workflow
• Tasks