Knowledge Wiki Orchestrata Ultimate

Runbook — ENOSPC Gate Cascade from Stale Pytest Basetemps

Closed 2026-08-05 in orchestrata-ultimate (fix shipped in commit bf8cf695), but the failure mode applies to ANY project running large pytest suites on a tmpfs /tmp.

Symptom

A previously green suite suddenly fails en masse mid-run — ~135 test failures in one gate, all OSError: [Errno 28] No space left on device (ENOSPC) from tmp-file writes. Failures start partway through the run and cascade; earlier batches in the same run were green.

Root cause

3,129 stale /tmp/pytest-of-<user>/pytest-* basetemp directories, 16 GB, accumulated across historical suite runs. pytest's basetemp garbage collection only keeps the last 3 per session lineage — thousands of distinct invocations (batched gates spawn many pytest sessions) each leave residue. /tmp is tmpfs (RAM-backed, fixed size); once it hit 100% mid-gate, every tmp write in every concurrently running worker failed.

The trap: after the run, /tmp can read nearly empty again (the failing run's own temp dirs are cleaned on teardown), which mimics a transient spike and sends you hunting for a runaway writer that does not exist.

Diagnosis steps (in the order used)

  1. df -h /tmp — check tmpfs utilization; run it DURING a gate (a 5 s watcher loop was armed alongside gates until the culprit was named).
  2. sudo btrfs filesystem df / — rule out Btrfs metadata exhaustion (was 3.55/9 GiB, healthy) so you do not chase the wrong ENOSPC class.
  3. df -i /tmp — rule out inode exhaustion (was 1%).
  4. journalctl -k | grep -i enospc — rule out kernel/filesystem-level errors.
  5. Count and size the residue: ls -d /tmp/pytest-of-$USER/pytest-* | wc -l and du -sh /tmp/pytest-of-$USER. Thousands of entries / multi-GB = this runbook's culprit.

Permanent fix (as shipped)

prune_stale_basetemps() in scripts/run_tests.py (defined at :81, invoked as preflight step 0.5 at :858): before infra checks, delete basetemp dirs under /tmp/pytest-of-<user> older than 24 hours (max_age_hours=24.0 default), reporting count pruned + MB reclaimed. Covered by 5 real-filesystem tests (90% file coverage). Every gate now self-heals the accumulation instead of trusting pytest's per-lineage GC.

Manual remediation (one-shot)

# prune basetemps older than 24h (safe while no suite is running)
find /tmp/pytest-of-$USER -maxdepth 1 -type d -name 'pytest-*' -mmin +1440 \
  -exec rm -rf {} +
df -h /tmp   # verify headroom restored

In orchestrata-ultimate the next python scripts/run_tests.py does this automatically at preflight.

Lesson

When ENOSPC appears mid-suite on tmpfs, check for cross-run accumulation first — not just live writers. Absence of a persistent fill after the run is not evidence of a transient spike; the failing run's own teardown hides the baseline.