Orchestrata Ultimate › Gates › ci-pipeline

Gate: ci-pipeline

At a glance

Jobs

What the test job really talks to

The job declares redis:7-alpine and neo4j:5-community services with endpoint and credential variables (values not reproduced here). The file's own warning, measured 2026-08-08 inside a live job: tests/conftest.py loads env/test.env with override=True, so the suite reaches the Incus test fleet on 10.53.2.x instead, and "a CI job and a local run_tests.py collide on the same Redis DBs and orchkg_w* worker databases." Making CI hermetic was left as a separate decision.

Because the job calls run_tests.py tests/, the runner's gate-mode behaviour applies in CI too: the freshness guard runs (no --skip-freshness), and the rehearsal exclusion applies (paths are exactly tests/; reports/rehearsal-class-modules.json is tracked in git).

CI and ship check different things

State on 2026-10-23 (measured, read-only)

• Pipelines. The last ten (iid 1169–1178, 2026-10-22T15:09Z to 2026-10-23T03:58Z; branches master and fix/ship-sanctioned-gate-pytest; source push) are all failed, each within about 1–2 s of creation (GitLab API via MCP). Pipeline 2979406170 (commit a4cbc2bdf) has 8 jobs (lint, architecture-check, architecture-ratchet, vizint-compliance, test, vizint-provenance-e2e, coverage-gate, build), all failed; the lint job's trace is 0 bytes, so no script line ran.
• Runners. A system gitlab-runner.service and a user gitlab-runner.service (/usr/local/bin/gitlab-runner run --config ~/.gitlab-runner/config.toml, started 2026-10-22 14:01 EDT) are both active. Neither journal has a single job= line since 2026-10-22 11:00 EDT (15:00Z); the system runner's last job line is 2026-10-22 05:28:55 EDT, job 17376209475, job-status=failed. So no job of those ten pipelines reached a runner on this host.
• Cause: not measured. The signature (every job failed at about 0 s, nothing ran) matches pipeline #2742244804 on 2026-08-07, which the runner README attributes to ci_quota_exceeded on shared runners. The API calls used did not expose a failure reason, so whether today's cause is the same was not established. An open P1 backlog task (session-start report) records a separate, earlier failure: "CI jobs on the private registry image fail: incusd runs skopeo as ROOT and root has no registry.gitlab.com log[in]". That failure happens after a runner takes the job, so it does not explain jobs that no runner received.

The GitHub workflow

.github/workflows/ci.yml (142 lines, last changed d6ce4ce81, 2026-09-10): one job quality on push and PR to main/master, with redis:7-alpine and memgraph/memgraph:3.11.0 services. Steps: uv python install 3.14, uv sync --dev, ruff, code-review.py src/ --min-severity ERROR, architecture_checker.py --verbose, kg_schema.py apply and check, gen_level0.py --check, gen_intent_lexicon.py --check, check-deps --include-tests, gate.py run, untyped --max 0, pyright_ratchet.py --max 0 and --source tests --max 219, check-traces --max 2219, check_async_await.py, audit_test_purity.py, ast_report.py (advisory), validate_ci.py, then run_tests_threadpool.py tests/ -j 4 with coverage and a codecov upload.

• The repo's only remote is GitLab origin; with no GitHub remote, whether this workflow runs anywhere was not measured.
• scripts/pyright_ratchet.py is absent from the tree: it moved to codetools in 71b4cd46f (2026-10-20, ADR-0118), so those two steps would fail as written.
• It calls run_tests_threadpool.py directly, which the GitLab file says validate_ci.py gate 9 forbids for CI. Whether validate_ci.py reads the GitHub file was not measured.

CI contends with the ship (measured 2026-11-06, ship-speed S17)

The runner (orchestrata-incus, id 58422150, concurrent = 2) runs on the same host as the ship, the Hermes workers and the test fleet. Measured with the GitLab API, the job traces and the cgroup tree:

• Two pipelines per ship. deploy/ship.sh pushes the feature branch (step 2) and then fast-forwards master (step 2b): the same sha twice, so GitLab ran two full pipelines. Ship 1106-d (7d3502e1f): master 30 min 39 s, branch 16 min 23 s. Ship N's pipelines were still running when ship N+1 started.
• Jobs run one after another, with 50–100 s between them. The runner journal shows it idle (builds=0 of max_builds=2) for the whole gap, so the delay is GitLab.com's scheduling, not the host. Inside a job the executor costs little: prepare 2–3 s, sources 5–7 s.
• One CI job container weighs as much as the whole ship. Incus puts every container in a root-level cgroup (/sys/fs/cgroup/lxc.payload.<project>_<name>) at cpu.weight 100, a sibling of user.slice, which is also 100 and holds everything run from the desktop session, the ship included. cpu.weight only divides CPU between siblings, so under contention one job got as much CPU as the entire ship. Ship 1106-b's promote prod (05:59:44–06:01:44Z) overlapped a lint job (06:00:27–06:02:12Z) and took 120 s against about 90 s.
• limits.cpu.priority is not the knob. Incus turns it into cpu.shares moved by at most 10 out of 1024. An explicit raw.lxc line (lxc.cgroup2.cpu.weight = 10) is appended last to the LXC config and wins. Incus 6.15 accepts it on an OCI image launch (verified: an ephemeral redis container with it reached RUNNING).
• The uv cache never hit. cache_dir = "/cache" is a path inside the throwaway job container. Every job logged "Cache file does not exist", then spent 16–17 s archiving 32,147 .uv-cache files that no later job could see. A runner's "local cache" only persists if that path is mounted from the host.
• build ran uv sync --dev for nothing. It inherited .uv-setup's before_script, while uv build makes its own isolated build environment.
• Per-job CPU cannot be recovered after the fact: sysstat samples every 10 minutes and the job containers are deleted. Step timings come from the trace's section_start/section_end markers.

The fix (c0ea6bd9b, brief plans/briefs/ci-yields-to-the-ship.md): the ship pushes the feature branch with -o ci.skip when origin is gitlab.com, so only the master push runs a pipeline (-o fails against a remote that does not advertise push options, such as a local bare repo). The job and service containers launch at cpu.weight 10, so CI keeps every idle cycle and about a tenth of a contended one, and nothing is stopped or killed. The cache block is gone, and build has before_script: [].

Typical failures and how to fix them

Related

• Gates index
• Workflow
• validate-ci
• Ship verify
• Pre-push guards
• Architecture · mypy · ruff
• Freshness guard · Rehearsal exclusion
• Running the test suite