Knowledge Wiki Orchestrata Ultimate

Orchestrata Ultimate › Plans › pytest-native-audit

Pytest-native features audit — what to adopt, what stays custom

Source: plans/pytest-native-audit.md at 8a2cb42ad · last changed 2026-07-27 · first committed 2026-07-09 · 205 lines.

Prose pending.

Plan text

Pytest-native features audit — what to adopt, what stays custom

Synthesis of 6 research lanes (core config/cache/faulthandler, xdist deep-dive, duration splitting, rerun/flake tooling, timeout/hang tooling, free-threaded parallelism) mapped against src/orchestrata_ultimate/core/threading/test_executor.py, src/orchestrata_ultimate/core/threading/pytest_live_observer.py, scripts/run_tests.py. Environment: pytest 9.0.3, pytest-timeout 2.4.0, pytest-xdist 3.8.0 (installed, banned), Python 3.14t, ~22,700 tests / 1,158 modules, real remote infra (Redis/Neo4j/Prolog on Incus).

Contradictions between lanes were re-verified against the installed packages before writing: faulthandler_timeout + faulthandler_exit_on_timeout confirmed present in .venv/.../_pytest/faulthandler.py:27-29; session_timeout confirmed in .venv/.../pytest_timeout.py:93-108; --report-log confirmed absent from pytest core (it is the separate, uninstalled pytest-reportlog package — one lane misattributed it); the faulthandler timeout value dispute (100s vs 150s) resolved to 150s — it must sit above the per-test --timeout=120 so it fires only when SIGALRM already failed (the uninterruptible C-hang class), not on every slow-but-killable test.

1. Executive verdict table (ADOPT / ADOPT-PARTIAL, ranked by risk-adjusted payoff)

# Feature Concrete flag / change What it retires or hardens Effort Risk
1 pytest core faulthandler_timeout append -o faulthandler_timeout=150 in _batch_pytest_args (test_executor.py:226) The one genuine gap in the stack: names WHERE a hang is (all-thread C-level stack dump), complementing the live observer which names WHICH nodeid. Watchdog is a native C thread writing to the stderr fd → fires even when every Python thread is wedged in C (neo4j bolt read, janus-swi FFI — the exact class --timeout=120's SIGALRM provably can't interrupt). Dump lands in the batch's PIPE'd stderr, drained by communicate(timeout=30) after the killpg SIGKILL, printed by run_tests.py:347-356. Zero deps, never alters outcomes. 1 line none
2 Pin pytest-timeout>=2.4.0 pyproject.toml:52 and :227 (currently >=2.1.0) --session-timeout — which _batch_pytest_args already injects at 0.65×budget and which the whole graceful-JUnit-flush contract depends on — only exists in 2.4.0+. A resolver downgrade would silently pass an unknown flag to every batch. Also corrects the framing: the session budget is not hand-rolled; only the 0.65 ratio policy + graceful/hard pairing are ours. 2 lines none
3 -p no:cacheprovider + -p no:xdist append both in _batch_pytest_args (next to the existing -p ...pytest_live_observer) (a) 8+ concurrent batch subprocesses currently race on one unlocked .pytest_cache (lastfailed written at sessionfinish, last-writer-wins; SIGKILLed batches write nothing) — disabling it keeps the cache meaningful for interactive runs; (b) no:xdist converts the xdist ban from a command-lint hook into a runtime guarantee (xdist 3.8.0 IS installed; its plugin registers in every batch today). Does NOT disturb PYTEST_XDIST_WORKER isolation — that env var is read by our conftest, not the plugin. 2 args none
4 --co -q cold-cache seeding one-shot single-process collect-only pass when .cache/test_durations.json lacks a module; per-module test count replaces the uniform 5.0s/median fallback in _estimate_durations Attacks the diagnosed 2026-07-08 batch-assignment defect class on first-run/new-module packing. Collection imports modules but does not execute fixtures → no remote-infra touch, cheap. ~30 lines low
5 pytest --lf / --sw for the dev loop (docs only) document in AGENTS.md test section; never wire into run_tests.py Nothing custom exists for the interactive fix-verify loop (developers re-type failing paths). Valid only single-process: under the executor, lastfailed is partial (graceful path) or absent (SIGKILL path) and races across batches. doc none
6 SIGINT → SIGABRT → SIGKILL escalation in _run_subprocess (test_executor.py:178-192): killpg(SIGINT), wait ~20s, kill(SIGABRT), then killpg(SIGKILL) as today SIGINT lets pytest's wrap_session run sessionfinish → JUnit survives for signal-interruptible overruns (fewer did_not_run → smaller Phase-2B rounds); SIGABRT forces a faulthandler postmortem dump (pytest enables faulthandler at configure, covering SIGABRT). Final SIGKILL invariant unchanged. Cost: ~20-30s per killed batch. ~25 lines low
7 pytest-run-parallel stress lane (narrow) new opt-in lane in run_tests.py: --parallel-threads=8 over pure in-process modules only (core/threading/, core/lru_cache.py, core/patterns/observer.py) with --skip-thread-unsafe Adds dynamic 3.14t race detection that the static-only scripts/gate_parallel_safety.py cannot do. Must never run over real-infra tests (N threads share one fixture instance → same Redis DB/Neo4j session → mass false failures). Ecosystem standard (NumPy/SciPy ft CI). new dep + lane medium
8 PYTEST_XDIST_WORKER=gwN contract already adopted (test_executor.py _run_subprocess, ~line 138) — keep forever Makes the entire per-worker isolation layer (Redis DB, NEO4J_URI_GW<N>) runner-agnostic and swappable if upstream ever ships a viable distributor. Recorded so nobody deletes it as "dead xdist residue". none —
9 GIL re-enable visibility (3.14t) one-off: python -c "import orchestrata_ultimate, sys; print(sys._is_gil_enabled())" + per-driver imports (janus_swi, grpc, neo4j); optionally scan batch stderr for the re-enable RuntimeWarning in run_tests.py Currently throughput-neutral (parallelism is process-level) but a dependency upgrade that silently re-GILs workers should at least be visible. Do not set PYTHON_GIL=0 (trades a warning for memory-unsafety). ~10 lines none

INVESTIGATE (backlog, not adoptions): faulthandler_exit_on_timeout (per-test hard exit for C-hangs — saves hard_budget − 150 s per hang but needs the early-nonzero-exit orphan-cleanup path in _run_subprocess audited first, since it exits pytest, not the test's grandchildren); nodeid/class-granularity packing for over-target singleton modules (test_sync_queries.py @ 285s nearly consumes a 273s graceful budget — implementable against the existing JUnit harvest, zero new deps, but requires slice-aware attribution keys and an intra-module shared-state audit; measure from .cache/test_durations.json whether singletons actually bound the makespan first); --deselect of the named hung nodeid in Phase-2B solo reruns (resolution polish only); xdist thread-mode proposal (xdist#904 / pytest#13768 — unimplemented; revisit only if it ships a per-thread worker-id fixture seam and a hang-escape story).

2. The xdist question, answered definitively

Verdict: the ban is correct, permanent under current upstream, and no configuration fixes the ~78% deadlock. All six lanes converged; the mechanism decomposes into three structural failure classes, each independently fatal for this suite:

  1. Hung worker, no watchdog. xdist ships no worker heartbeat, no per-worker/session wall-clock timeout, no hard kill — its 3.8.0 option surface contains no timeout option of any kind. A test blocked inside a C extension (neo4j bolt socket read, janus-swi engine call, grpc core) never returns to the eval loop, so pytest-timeout's SIGALRM handler never runs (measured: our --timeout=120 does not fire on these), and the controller waits on the execnet channel forever. This suite measurably produces such waits (285s Neo4j error-path outlier; the 2026-07-09 test_ast_cycles kill). The only hang→crash bridge, --timeout-method=thread, calls os._exit(1) — losing the worker's entire report with no NDJSON rescue.
  2. Crashed worker, broken recovery. --max-worker-restart triggers only on channel EOF (process death), and when it does trigger, rescheduling is broken upstream: pytest-dev/pytest-xdist #658 (open since 2021 — replacement spawns, run stalls, others idle) and #1094; #813's fix in 3.0.2 did not close it. run_tests.py documents a real 3.14t interpreter SIGSEGV (match_dealloc, 2026-07-06) — under xdist that lands on the #658 stall; under our executor it is a contained batch kill + attribution + rerun.
  3. Orphaned grandchildren. This suite's tests spawn real gateway/kernel/CLI subprocesses that inherit the execnet channel FDs; a leaked grandchild prevents channel EOF and wedges the controller even after worker exit (xdist #60/#110/#298) — the same FD-inheritance wedge test_executor.py documents and defeats with start_new_session + os.killpg(SIGKILL).

The reproducible ~78% mark is deterministic (sorted) collection order delivering the hang-prone real-infra region to a worker at the same fraction of every run — evidence for a structural cause, not flakiness. Every candidate "fix" is either a scheduling knob (--dist load/worksteal/loadfile/loadscope/loadgroup, --maxschedchunk — change who runs what, not what happens on a hang; worksteal itself had a hang bug, #884) or the broken crash path. Free-threading changes nothing in xdist's favor: workers are processes (execnet), pytest-xdist and execnet changelogs contain zero free-threading entries, and 3.14t raises the C-extension segfault rate — enriching exactly the failure mode xdist cannot recover from. Two further disqualifiers even for a hypothetically deadlock-free run: -s/live output is unsupported ("no plans to support") and controller-side JUnit is written only at clean session end — a killed run is amnesiac, regressing the entire SIGKILL-durable attribution pipeline. Additionally, the xdist+pytest-cov combination tripped its own deadlock class (execnet main_thread_only, #1071) — this suite runs coverage.

Standing actions: keep the ban + PreToolUse hook + rule T15 verbatim; add -p no:xdist to batch args (row 3) so the plugin cannot register inside executor subprocesses; keep writing isolation against the PYTEST_XDIST_WORKER contract; track #658 as the bellwether for any future re-evaluation.

3. Justified custom machinery — what NOTHING native does

Ecosystem-validated (pytest-split independently converged on LPT; pytest-reportlog and pytest-replay independently converged on flushed-NDJSON durability), but no tool or combination replaces:

  1. External per-batch watchdog with process-group SIGKILL (_run_subprocess: start_new_session + os.killpg). xdist has no watchdog; pytest-timeout's thread method nukes the whole process's report; pytest-forked is fork-with-live-threads (unsafe on 3.14t, unmaintained). The graceful(0.65×)/hard(1.0×) two-layer budget pairing has no native expression — --session-timeout provides only the graceful half's mechanism.
  2. SIGKILL-durable per-test NDJSON with hang naming + session broadcast (pytest_live_observer.py). pytest-reportlog matches the flush-per-line durability but has no logstart event — a test hung in setup (the dominant class with real remote fixtures) cannot be named; our dangling-start detection is what feeds suspected_hang. Neither reportlog nor pytest-replay has the SessionRegistry broadcast that observe_pytest.py attach/list (S201 ruling) depends on. pytest-replay is a strict subset (record-only).
  3. Phase-2B: module-level, orchestrator-driven, post-mortem rerun (_rerun_dropped_modules / _prepare_rerun_batches, test_executor.py:909-1011). The entire rerun ecosystem is in-session and failure-triggered: pytest-rerunfailures/flaky state dies with a SIGKILLed process and a hanging test never emits the failure report that would trigger them; --lf is written at sessionfinish (stale/absent after a kill) and cannot represent "did not run". Nothing retries at module granularity from external artifacts (JUnit + live log), and nothing has hang-aware solo bins with their own kill budget.
  4. Budget-aware LPT packing with adaptive bin growth (_lpt_pack_modules, :598-653). pytest-split's least_duration is the same LPT heuristic but balances relative loads for a fixed group count — it has no per-bin time-budget ceiling, and the 0.5×budget target with bin-count growth is precisely the fix for the 2026-07-08 graceful-timeout dropouts. pytest-test-groups/pytest-shard are duration-blind (the diagnosed defect class, by construction).
  5. Concurrent-writer-safe, self-refreshing duration persistence (_module_durations_from_junit + atomic os.replace into .cache/test_durations.json). pytest-split's .test_durations is a non-atomic, unlocked sessionfinish write requiring explicit --store-durations runs — 8-48 concurrent batches would clobber it.
  6. JUnit-attribution did_not_run semantics (test_executor.py:396-435): "no recorded testcase ⇒ did not run" is strictly more correct than any native failure memory, and it works regardless of how a batch died.
  7. Deliberate non-adoption to preserve: the rejected error::pytest.PytestUnraisableExceptionWarning gate stays rejected (GC-time mis-attribution across batched modules on 3.14t); scripts/check_async_await.py is the documented substitute.

Config surface ([tool.pytest.ini_options], addopts, --strict-config, --strict-markers, -m deselection, --ignore) is already at the idiomatic ceiling — nothing to change.

4. Migration plan — top 3 adoptions

Step 1 — timeout-stack hardening (rows 1+2, one commit).

Step 2 — plugin lockdown in batch subprocesses (row 3).

Step 3 — cold-cache LPT seeding (row 4).

Each step lands as its own reviewed commit (review-before-commit ruling), full batched suite green (~13 min) as the final gate for each.

5. REJECT list

Feature One-line reason
pytest-xdist -n / --dist load No hang watchdog + broken crash recovery (#658/#1094) + grandchild-FD wedge = the measured ~78% deadlock; no config fixes it (§2).
--dist worksteal Scheduling-only; shares all three deadlock classes; its own scheduler had a hang bug (#884).
--dist loadfile/loadscope/loadgroup Reproduces our module affinity but keeps everlasting unwatched workers; one wedged worker still deadlocks the run.
--maxschedchunk Pure granularity knob; zero bearing on hung/crashed/orphaned deadlocks.
--max-worker-restart Triggers only on crash (never hangs), and its rescheduling stalls upstream (#658, open since 2021).
xdist on 3.14t Zero declared free-threading support (xdist+execnet changelogs); workers are processes anyway; 3.14t raises the segfault rate xdist can't recover from.
execnet main_thread_only We get main-thread execution free (plain subprocess); introduced its own deadlock class (#1071, with pytest-cov).
pytest-reportlog --report-log Not pytest core, not installed; equal flush durability but no logstart event (can't name setup hangs) and no session broadcast — replaces nothing, loses two load-bearing capabilities.
cacheprovider inside the executor (--lf state) lastfailed written at sessionfinish → stale/absent after SIGKILL; "did not run" unrepresentable; concurrent batches race one cache dir.
--durations=N (as machinery) Terminal-text-only, per-test, lost on SIGKILL; JUnit time harvest is strictly better. Keep as ad-hoc human diagnostic.
pytest-durations plugin Session-end reporting only, lost on hard kill; live observer already gives durable per-test timing.
pytest-split (--splits/--group) Same LPT we have, minus the load-bearing parts: no per-bin budget/adaptive growth, nodeid splits break module-granularity isolation, non-atomic shared durations file, 48× full collections.
pytest-test-groups / pytest-shard Duration-blind chunking — a regression to the diagnosed 2026-07-08 defect by construction.
pytest-rerunfailures In-session, per-test, failure-triggered: dies with SIGKILL, blind to hangs, blanket reruns violate "fix the code, not the test".
rerunfailures + xdist crash recovery The only ecosystem process-death retry rides the banned, deadlocking xdist scheduler.
flaky (box/flaky) Same model as rerunfailures, unmaintained since 2024-03, no pytest 9/3.14(t) path.
pytest-replay SIGKILL-durable but record/replay-only — a strict subset of pytest_live_observer; validates our design, adds nothing.
--timeout-method=thread (as default) Its os._exit(1) kills the whole batch's JUnit — converts one hang into a whole-batch did_not_run; fires more on 3.14t, making it worse.
pytest-forked fork()-with-live-threads (neo4j/grpc/janus threads present) = child deadlocks, amplified on 3.14t; effectively unmaintained; per-test forking re-pays the startup tax the batch design removed.
pytest-threadpool (thread runner) Isolation seam is process-global env (one os.environ per process → all threads share one Redis DB/Neo4j); threads non-cancellable (no hang containment); pytest-timeout incompatible; beta on private _pytest internals.
pytest-parallel Unmaintained, pre-free-threading, inherits every thread-runner blocker.
pytest-freethreaded Superseded by pytest-run-parallel; 200× reruns of real-infra tests infeasible.
pytest-run-parallel as suite runner Explicitly not a speedup tool; shared fixtures across N threads → mass false failures on real infra (narrow stress lane only — row 7).
tox -p / nox / make -j Static duration-blind partitioning (the diagnosed defect class) with none of the watchdog/attribution/rerun layers; make -j per module = the measured ~24-min startup-tax config.
PYTHON_GIL=0 forced Trades the re-enable RuntimeWarning for memory-unsafety under non-ft-safe extensions; we already do SIGSEGV forensics without self-inflicted ones.
PR_SET_PDEATHSIG preexec_fn Documented thread-unsafe in multithreaded parents — disqualified on the free-threaded executor; use a pidfile/watchdog sweep if orphans ever bite.