Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › briefs › 2026-10-22-runner-failure-detail-ship

2026-10-22-runner-failure-detail-ship

Source: plans/briefs/2026-10-22-runner-failure-detail-ship.md at 8a2cb42ad · last changed 2026-10-22 · first committed 2026-10-22 · 72 lines.

Prose pending.

Plan text

DESIGN_WAIVER: Mohamed 2026-10-22, "change the runner, improve it, get full error and debug and trace info"; the change is written and tested by the coordinator, and this packet only commits, smoke-installs and ships it (plan step 1, approved 2026-10-22). Date: 2026-10-22 · Origin: the approved plan, step 1: "commit the runner change and ship it once (affected tests + deploy), after a scratch-prefix smoke install" · Executor: Sonnet builder in worktree /home/mohamed/workspace/orchestrata-wt-unshipped-scope on branch fix/ship-sanctioned-gate-pytest tracking origin/master at 796c4f0b5 · Design settled below; if a fork appears, STOP and report.

GOAL: the runner's full-failure-detail change is on origin/master, and ~/.claude, /opt/orchestrata-tools and /opt/orchestrata-gitlab-runner are deployed at that commit, through ship --no-promote. Prod /opt/orchestrata is NOT touched in this packet.

VERIFIED (coordinator, 2026-10-22 ~13:55; do not re-derive, but STOP if you observe otherwise):

Paths used below. SCRATCH = /tmp/claude-1001/-home-mohamed/829ea13c-8ceb-4f51-9b28-672fef74daca/scratchpad. WT = /home/mohamed/workspace/orchestrata-wt-unshipped-scope. Every command starts with cd $WT && (written out in full; the shell resets its cwd), and every path is absolute.

FILES (repo-relative): Commit 1 (the runner change), exactly these five:

STAGES (in order; each ends in a check; a failed check is a STOP):

  1. Pre-flight. cd WT && git fetch origin && git rev-parse --short HEAD && git rev-parse --short origin/master && git status --short. Expect: 796c4f0b5 twice and exactly the status in VERIFIED. On a different sha or any other dirty path: STOP, report the output.
  2. Restore generated files. cd WT && git checkout -- plans/ast-deep-comparison.json plans/ast-tests.json plans/ast-ultimate.json docs/CODEBASE_MAP.md docs/CODEBASE_MAP_FULL.md reports/run-tests-latest.json && git status --short. Expect: only the three modified src files and the five untracked files.
  3. Lint and warm mypy (the commit review's mypy step times out at 30 s on a cold cache). With the five commit-1 .py files as FILES1: cd WT && bash SCRATCH/envrc_exec.sh ruff check FILES1 && bash SCRATCH/envrc_exec.sh black --check FILES1 && bash SCRATCH/envrc_exec.sh python -m mypy --no-error-summary FILES1 (Bash timeout 600000). Expect: ruff "All checks passed!", black "would be left unchanged", mypy no output or notes only. Any error: STOP, report it in full. 3a. AMENDMENT 1 (coordinator, after the first attempt stopped here): the secrets pre-commit scanner (scripts/check_staged_secrets.py) refuses any assignment whose NAME is credential-shaped and whose value is not a placeholder, test fixtures included. It flagged tests/core/threading/test_failure_detail.py:53 api_key = ... and :209 secret = .... The redaction tests depend on the token shape and the env-var name, not on the local's name, so rename both locals to planted. This is the only file edit in this packet. Print each span first with the toolkit (cd WT && .venv/bin/python scripts/serialize_codebase_ast.py source test_failure_detail --residue --source tests/core/threading/test_failure_detail.py --no-daemon for _FAILING_MODULE, and ... source test_a_tail_cut_through_a_secret_keeps_none_of_it --source tests/core/threading/test_failure_detail.py --no-daemon), then Edit with exactly these bytes:
    • api_key = "sk-or-v1-" + "f" * 40 -> planted = "sk-or-v1-" + "f" * 40
    • the three lines secret = "S3CRET-" + "9" * 30 / environ = {"A_SECRET": secret} / short = f"line one\n{secret}\ntail" -> the same three lines with secret (the local, not A_SECRET) replaced by planted. Then: cd WT && git checkout -- reports/precommit-walls.ndjson; re-run stage 3's three checks on the test file; run the file's 7 tests in the background, observed: cd WT && bash SCRATCH/envrc_exec.sh timeout 300 .venv/bin/python -m pytest tests/core/threading/test_failure_detail.py -q -p no:cacheprovider > SCRATCH/failure_detail_tests2.log 2>&1; echo "EXIT=$?" >> SCRATCH/failure_detail_tests2.log; expect 7 passed and EXIT=0. Then cd WT && git add tests/core/threading/test_failure_detail.py && bash SCRATCH/envrc_exec.sh python scripts/check_staged_secrets.py; echo "secrets EXIT=$?"; expect secrets EXIT=0. Any other result: STOP and report.
  4. Commit 1. cd WT && git add <the five files by name> && git commit -F SCRATCH/commit_failure_detail.txt. The pre-commit hooks run the commit review. If it fails with MYPY_TIMEOUT only, re-run stage 3's mypy command once and retry the commit once. Any other failure: STOP and report the complete hook output. Never --no-verify.
  5. Commit 2. cd WT && git add plans/briefs/TEMPLATE-sonnet-agent.md plans/briefs/IMPROVEMENT-LOG.md plans/briefs/2026-10-22-runner-failure-detail-ship.md && git commit -F SCRATCH/commit_brief_template.txt. Same failure rule. Then record HEAD_SHA=$(git rev-parse --short HEAD).
  6. Smoke install the tools tree from HEAD into a fresh scratch prefix (this is what caught nothing last time and broke prod tools: lesson D1 in IMPROVEMENT-LOG.md). Run with run_in_background: true: cd WT && mkdir -p SCRATCH/tools-smoke-fd && PREFIX=SCRATCH/tools-smoke-fd bash SCRATCH/envrc_exec.sh bash deploy/install-tools.sh --ref <HEAD_SHA> > SCRATCH/tools_smoke_fd.log 2>&1; echo "EXIT=$?" >> SCRATCH/tools_smoke_fd.log Wait for the EXIT= line (Monitor with an until-loop on grep -q '^EXIT=' SCRATCH/tools_smoke_fd.log, or check tail -5 of the log between other reads). Expect: EXIT=0, a line DEPLOYED <HEAD_SHA> -> SCRATCH/tools-smoke-fd, and grep -nE '^\s*FAIL' SCRATCH/tools_smoke_fd.log prints nothing. Otherwise: STOP, report the FAIL lines with 20 lines of context each. Nothing has been pushed yet at this point; leave the commits in place.
  7. Ship without promote. Run with run_in_background: true: cd WT && bash SCRATCH/ship_envrc.sh --no-promote > SCRATCH/ship15_nopromote.log 2>&1; echo "EXIT=$?" >> SCRATCH/ship15_nopromote.log It takes 8-15 minutes. Wait for the EXIT= line the same way. Do not start any other command that runs tests, pytest, ship, uv sync or make while it runs (a hook refuses them); tail, grep, cat, ls on the log are fine. Expect: EXIT=0, ==> shipped, and orchestrata-ultimate-architecture <sha> pushed to origin/fix/ship-sanctioned-gate-pytest. On failure, find the stage from the last ==> line, then:
    • in ==> verify (lint, gates or affected tests): STOP. Report the FAIL: line and the failing modules. For failing tests, report each failing module's stdout/stderr from WT/reports/run-tests-latest.json (that detail is what this change adds; quote it, cut at 80 lines per module). Do not fix anything.
    • in ==> push or ==> merge to master: STOP, report the 40 lines before FAIL:.
    • in ==> deploy or ==> verify deployment: report the 40 lines before FAIL:, then read /opt/orchestrata-tools/VERSION. If it does not record the new sha, or install-tools printed FAIL, roll the tools tree back to the last good commit: cd WT && bash SCRATCH/envrc_exec.sh bash deploy/install-tools.sh --ref 796c4f0b5 > SCRATCH/tools_rollback_fd.log 2>&1; echo "EXIT=$?" >> SCRATCH/tools_rollback_fd.log (background, observed; expect EXIT=0 and DEPLOYED 796c4f0b5 -> /opt/orchestrata-tools). Then STOP.
  8. Confirm the deploy. cd WT && git fetch origin && git rev-parse --short origin/master && head -1 /opt/orchestrata-tools/VERSION ~/.claude/VERSION /opt/orchestrata-gitlab-runner/VERSION && git status --short && git log --oneline -4 && git log -3 --format='%an <%ae>%n%B---'. Expect: origin/master equals the sha ship printed; all three VERSION files record it; status clean apart from generated files; Mohamed as author with no Co-Authored-By, Claude-Session or other attribution lines.

CONSTRAINTS:

DONE WHEN:

  1. Two new commits by Mohamed (the runner change, the docs) are on origin/master, plus ship's run-evidence commit if it made one.
  2. SCRATCH/tools_smoke_fd.log ends EXIT=0 with DEPLOYED <HEAD_SHA> -> SCRATCH/tools-smoke-fd and no FAIL line.
  3. SCRATCH/ship15_nopromote.log ends EXIT=0 with ==> shipped.
  4. /opt/orchestrata-tools/VERSION, ~/.claude/VERSION and /opt/orchestrata-gitlab-runner/VERSION all record origin/master's sha.

VERIFY: stage 1 and 2 outputs; stage 3's three result lines; both git commit summaries; tail -15 SCRATCH/tools_smoke_fd.log; grep -nE '^==> |FAIL|RUL-032|Selected:|tests passed' SCRATCH/ship15_nopromote.log; tail -12 SCRATCH/ship15_nopromote.log; stage 8's full output.

REPORT: for each stage, the command, its exit status and its key output lines (quoted, not paraphrased); every deviation from this brief and why; for each problem, the verified cause, at least two options and a recommendation. Say what you ran and what it printed; never "should work".