Knowledge Wiki Orchestrata Ultimate

CachyOS Host Runbooks

AUR malware defenses — components and verification

Three standing guards, all deployed 2026-07-29, all user-level except the pacman hook:

1. Blocklist enforcement (pacman hook)

2. Daily blocklist sync

3. Weekly maintainer watch

4. Session-start IOC tripwire (inside the boot-health SessionStart hook)

Manual IOC sweep (run if a tripwire ever fires)

ls /sys/fs/bpf/hidden_* 2>/dev/null        # eBPF rootkit maps — must be empty
ls -la ~/.local/bin/sudo 2>/dev/null       # fake sudo grabber — must not exist
ls -d /tmp/.cache 2>/dev/null              # dropper dir — must not exist
pacman -Qm                                  # foreign packages — compare to the 7-package baseline

If INFECTED: step one is credential rotation (GitHub/npm/SSH/API keys — the payload is an infostealer), then restore from a pre-infection snapper snapshot.

Upgrade workflow

Known-benign boot noise (do not chase)

bluetoothd "Host is down" flood — FIXED 2026-08-03

An earlier revision of this page called bluetoothd's Unable to get Hands-Free Voice gateway SDP record: Host is down benign noise. WRONG: it was a 1-per-minute error loop (1,137 lines in one 19-hour boot) that tripped the boot-health FAIL.

doxygen 1.16.1 SIGABRT on CUDD sources (known, app-level)

4 identical coredumps 2026-08-02 came from doxygen Doxyfile inside the SWI-Prolog bddem pack build (~/.local/share/swi-prolog/pack/bddem/cudd-3.0.0/). Deterministic upstream bug: std::string_view::operator[] assertion (__pos < _M_len) in the comment scanner, triggered by CUDD's malformed nested comment (cudd/cuddCompose.c:1704), surfaced by Arch's _GLIBCXX_ASSERTIONS hardening. No functional impact — the pack's bddem.so built and loads. Do not chase new coredumps that match doxygen + this stack; docs for that pack simply don't build.

Package hygiene

orchestrata boot race — FIXED 2026-07-29, regressed, re-fixed 2026-08-11

The three orchestrata user units used to log Failed to start on every cold boot. Root cause was double: cachyos-settings ships DefaultTimeoutStartSec=15s for the user manager (/usr/lib/systemd/user.conf.d/00-timeout.conf), which killed the units' 180s ExecStartPre wait loop at 15s; and the wait loop only probed the bolt TCP port, which opens before the Memgraph cluster can actually route reads — so the kernel could start degraded (no audit trail / no KG injection, one-shot connect with no retry).

Fix (in the unit files + ~/.local/bin/orchgraph-ready): each unit sets TimeoutStartSec=240 (> the 180s wait budget), StartLimitIntervalSec/Burst moved to [Unit] where systemd ≥230 requires them, and ExecStartPre now calls orchgraph-ready, which performs a routed RETURN 1 via the neo4j driver — the unit starts only when the cluster can serve reads. A Failed to start from these units after a boot is now a REAL failure, not noise. The remaining app-level gap (no reconnect after a post-start blip) is in the orchestrata backlog.

2026-08-11 recurrence — the TimeoutStartSec was lost in a unit rewrite

All five orchestrata units failed again on the 2026-08-11 boot (start-pre operation timed out, recovered via Restart=on-failure on the second attempt). Cause: a later rewrite of the unit files renamed the probe orchgraph-ready → orchestrata-kg-ready and dropped the explicit TimeoutStartSec in the process, silently re-exposing the 15s cachyos default. This page kept claiming the 240s fix was in place — stale.

Re-fix: explicit TimeoutStartSec restored on every unit that runs the probe, sized to the probe budget (gateway gets more because it has a second start-pre):

unit TimeoutStartSec start-pre budget
orchestrata-bot, orchestrata-kernel, orchestrata-ast-live-sync, orchestrata-conversation-consumer 200 kg-ready 180s
orchestrata-gateway 260 kg-ready 180s + CXP-port wait 60s
orchgraph-backup (timer) 200 kg-ready 180s

Each unit file now carries a comment stating WHY the value exists and that it regressed once — do not remove it when rewriting a unit. Regression check after any boot: journalctl -b -p err | grep -c 'Failed to start orchestrata' must be 0.

gitlab-runner glr-* container orphans — FIXED 2026-08-11

Sibling failure class, found the same day: when the runner is SIGTERMed during host shutdown, cleanup.sh's incus delete -f races incusd going down and loses (user units cannot order against the system incus.service). The 2026-08-10 shutdown orphaned five glr-*/glrsvc-* containers, which auto-started next boot as zombies — one (glr-15815547501) became permanently unstartable because /run wiped its /etc/hosts bind-mount backing file. Fix: deploy/gitlab-runner/incus-executor/sweep-stale.sh runs as the runner unit's ExecStartPre and deletes any glr-*/glrsvc-* container plus stale glr-hosts-* dirs — safe because custom-executor jobs cannot survive a runner restart. Committed in orchestrata-ultimate-architecture 51c4e536.