Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › briefs › p6b-promoter-verify-settle-hold-builder

P6b — the promoter's verify settles before it judges: builder brief (orchestrator's picks, 2026-10-07)

Source: plans/briefs/p6b-promoter-verify-settle-hold-builder.md at 8a2cb42ad · last changed 2026-10-07 · first committed 2026-10-07 · 236 lines.

Prose pending.

Plan text

P6b — the promoter's verify settles before it judges: builder brief (orchestrator's picks, 2026-10-07)

DESIGN_WAIVER: plan item P6b of plans/ast-daemon-deployment-plan.md § "P6 — enable the primary instances and prove identity end to end" (bullet "P6b — promoter verify settles before it judges"): a correctness fix inside the promote's existing verify stage, no new structure and no new decision; the stage's purpose (ADR-0062: verify refuses false success, or the caller rolls back) is unchanged and every rule below is stated in the plan. A fork not settled here is a STOP-and-report.

GOAL: promotion.stages_verify.verify must refuse a unit that is crash-looping at the moment it polls active. Measured (plan § 3b, row 1fed577b4): orchestrata-conversation-consumer.service starts, exits 1 after ~1 s on the ADR-0072 RuntimeError, is restarted by systemd every RestartSec=10, and the promote's verify caught it inside one of its 1-second active windows and passed. Today the same unit reads ActiveState=failed NRestarts=5 (start limit hit).

1. Exact source at f21ab3af8

Read with .venv/bin/python scripts/serialize_codebase_ast.py source <qname> --no-daemon (qualified names omit orchestrata_ultimate.; module residue with --residue).

2. The change (plan P6b, three rules)

Rule 1 — a timespan parser. Add to promotion/stages_units.py: parse_systemd_timespan(text: str) -> float returning seconds. Accepts the forms systemd prints: an optional sign-free decimal number followed by one of the unit suffixes y (31557600 s), month (2629800 s), w, d, h, min, m (minutes, as systemd), s, ms, us, µs, ns, with any number of whitespace-separated terms ("1min 30s" → 90.0); bare "0" → 0.0; "infinity" → float("inf"); anything else (empty string, "abc", "10 s s", a bare number other than 0) raises ValueError naming the text. No regex from the stdlib beyond re; no new dependency.

Rule 2 — the settle hold. In verify, after the poll loop breaks with not inactive and not shut and BEFORE the problems block, add a hold over the non-empty units:

  1. For every unit read RestartUSec, NRestarts, ActiveState with ONE call per unit: _run([*_systemctl(_scope(unit_file)), "show", "-p", "RestartUSec", "-p", "NRestarts", "-p", "ActiveState", unit], timeout=30), parsed as key=value lines (a helper _unit_properties(ctx, unit) -> dict[str, str]). A unit whose show returns non-zero or lacks any of the three keys is a FAIL "settle hold: cannot read RestartUSec/NRestarts/ActiveState for <unit>".
  2. hold = max(parse_systemd_timespan(p["RestartUSec"]) for every unit) + 1.0; if the maximum is inf, use 1.0 and say so in the OK detail (an un-restartable unit cannot crash-loop). time.sleep(hold) once (not per unit).
  3. Re-read the three properties for every unit. Refuse with "after the settle hold (<hold:.1f>s): <unit> restarted <N> time(s)" for any unit whose NRestarts is not "0", and "after the settle hold (<hold:.1f>s): not active: <unit> (<ActiveState>)" for any whose ActiveState is not active; join all such lines with "; " into the FAIL detail. Order of refusals: restarted first, then not-active, units in units order.
  4. On success, the OK detail gains "; settled {hold:.1f}s (NRestarts 0)" after the existing "identity OK …" text. The zero-units FAIL and the identity check stay exactly where they are and run after the hold.

The hold is NOT bounded by settle (that is the poll's patience, measured on the kernel's 9-s bind); keep settle's default 90.0 and deploy's None. Do not add a constant default for the hold: the plan's rule is "read from the unit properties, not a constant".

Rule 3 — the proof tests. New module tests/promotion/test_verify_settle_hold.py (RUL-021 file budget: do not grow test_stages_coverage.py), real units only, no doubles, no monkeypatch, no skip, test_id fixture for unit names (orchestrata-p6b{test_id.lower()}….service):

@traces: the new function and the hold carry the plan's requirement citation the way the rest of promotion/ does — check promotion.stages_verify and promotion.identity for the exact decorator/comment form used (check_spec_authority.py must stay rc 0; if no requirement id exists for the promote's verify stage, cite none and say so in the report rather than inventing one).

Docs: docs/ship-and-deploy.md — in the promote section, one sentence after the verify description: the verify stage holds for the longest RestartSec + 1 s after the poll succeeds and refuses any restarted unit (NRestarts), so a crash-looping unit cannot pass on a lucky sample.

3. Gates and done-when

4. Fix round 1 (reviewer conditional GO on 7ac5e7f79, 2026-10-07; commits on top of 7ac5e7f79, no rebase or amend)

Reviewer probes (scratchpad p6b_rev_q1.log, p6b_rev_f1.log, p6b_rev_slow.log, p6b_rev_probe.py): with sleep 1; exit 1, RestartSec=10, NRestarts rises in the same 0.2-s sample as the switch back to active (0.01–0.92 s 0 active; 1.15–10.79 s 0 activating; 11.02 s 1 active), so the real verify returned FAIL after the settle hold (11.0s): <unit> restarted 1 time(s) in 11.0 s: the incident shape is refused by the NRestarts branch, every time. Every RestartUSec rendering probed (0, 100ms, 1s, 1.500000s, 1min 30s, 2h, infinity, 1d 2h, 1us, 59min, 1w, 1d 1h, 1y 2month) parses. daemon_units(/opt/orchestrata) returns only the four long-running services (oneshots have sibling timers and are excluded), so the hold never judges a oneshot. The hold is read from the unit (the 4.0-s and 1.0-s assertions prove it).

MAJOR-1 — the restarted N time(s) branch has no test that reliably reaches it. test_a_crash_looping_unit_is_refused_after_the_hold (sleep 2; exit 1, RestartSec=1, hold 2 s) re-reads ~0.05 s after the exit, inside activating with NRestarts=0, so only the not active branch fires and the assertion ("after the settle hold") cannot tell the branches apart; dropping the NRestarts check keeps the suite green. Fix: (a) rename that test test_a_unit_crashing_inside_the_hold_is_refused_as_not_active and give it a 1-s margin on each side: ExecStart=/bin/sh -c 'sleep 2.5; exit 1', Restart=always, RestartSec=5, StartLimitIntervalSec=0 (hold 6.0 s; exit at 2.5 s, restart fires at 7.5 s; the re-read at ~6 s is activating, NRestarts=0); assert "after the settle hold (6.0s): not active: <unit> (activating)" in result.detail and "restarted" not in result.detail. (b) add test_a_crash_looping_unit_is_refused_as_restarted with the reviewer-measured stable shape (3 of 3 runs): ExecStart=/bin/sh -c 'sleep 0.8; exit 1', Restart=always, RestartSec=2, StartLimitIntervalSec=0 (hold 3.0 s; the re-read lands in the second active window, 2.8–3.6 s); assert "after the settle hold (3.0s): <unit> restarted 1 time(s)" in result.detail, "not active" not in result.detail, "identity" not in result.detail. Both units reclaimed in finally.

MAJOR-2 — the texts claim more than the code delivers. A unit whose first crash comes later than poll time + RestartSec + 1 s passes the hold every time (probes: sleep 15; exit 1 with RestartSec=10 and sleep 3; exit 1 with RestartSec=1 both reached the identity stage). The guarantee the code gives is bounded: a unit that crashes within the longest RestartSec + 1 s after the poll finds it active is refused; one that crashes later than that is not seen by this stage. Fix: correct the _settle_hold docstring (the sentence "surviving a full restart interval plus a second is what cannot be sampled by luck") and the docs/ship-and-deploy.md § 5 sentence ("a crash-looping unit cannot pass on a lucky sample") to state exactly that bounded guarantee and name what it does not cover. No minimum-uptime floor is added in this round; the plan records it as a P6 follow-up for the orchestrator to size (it changes how long every promote waits).

MINOR-1 — unit file leak on a failed start. In test_verify_settle_hold.py the _start_unit(...) call and kernel.bind run before try: in the unit tests; a failing systemctl restart (check=True) leaves the file under ~/.config/systemd/user/. Fix: create the unit file and start it inside the try, and make the reclaim helper tolerate a missing file and an unknown unit (unlink(missing_ok=True); stop/reset-failed without check).

MINOR-2 — the hold has no upper bound. RestartSec=1d 2h renders 1d 2h and the stage would sleep 93601 s. Fix: if longest + 1.0 > settle, refuse WITHOUT sleeping: "settle hold: <unit> RestartUSec <rendered> exceeds the verify patience (<settle:.0f>s)", naming the unit with the longest value. settle is the stage's existing patience parameter; no new constant. Proof test test_a_restart_interval_longer_than_the_patience_is_refused_without_waiting: a steady unit with RestartSec=200, verify(ctx, settle=10.0), assert FAIL with that string and elapsed < 5.0.

Gates and commit rules as § 3. Modules to run: tests/promotion/test_verify_settle_hold.py, tests/promotion/test_stages_coverage.py, tests/promotion/test_promotion_prod_lessons.py, tests/promotion/test_promotion_units.py. Report the exact detail strings the two crash tests observed and the elapsed of the patience test.