Runbook — T9 Incus btrfs pool: the full-qgroup stall and its fix

Closed 2026-10-21 by switching the pool from full btrfs qgroups to simple quotas (squota). This page is the postmortem, the rules that keep the pool healthy, and the runbook for recognizing the same failure again. The T9 hardware findings at the end are still open.

The system

Symptoms (2026-10-19 → 2026-10-21)

• Memgraph slow. The prod orchgraph cluster and the test fleet run on this pool, and every commit waited on the stalled filesystem.
• Syncs that took tens of seconds. sync on the pool took 36–62 s even with Incus stopped. ext4 on the other partition of the same disk synced in 0.02 s, so the disk was not the bottleneck.
• Boot hangs. upower and power-profiles-daemon sat in D state at boot, and incusd came up 14 minutes late.
• Shutdown sync timeouts. Every shutdown's final sync timed out. The T9's SMART log shows 103 unsafe shutdowns in 108 power cycles.
• Test-run stalls. Measured 2026-10-19 (RUL-038 evidence): deleting test instances blocked test-Redis writes for 99,256 ms and 16,922 ms.

Root cause

Full qgroups were enabled on the pool. Full qgroup accounting has to resolve every shared extent's backrefs when a subvolume is dropped. The test fleet deletes clones that share 23–31 GiB each with their base, so each drop made btrfs-cleaner walk backrefs for hours. Transaction commits queued behind it, which is what every sync, fsync and journal commit on the pool waited for.

At diagnosis: 21 deleted subvolumes were still queued for cleaning, and metadata was 19.8 GiB for 80.7 GiB of data.

Why quotas could not simply be turned off

Incus's btrfs driver re-enables quotas on its own (read from the Incus source, SetVolumeQuota):

• When it sets a volume size and btrfs qgroup show -e -f --raw fails (which it does when quotas are off), it runs plain btrfs quota enable followed by btrfs quota rescan -w. That is a full-qgroup enable plus a rescan of the whole pool.
• Limits are set with btrfs qgroup limit <bytes> <qgroup>, and deleting a volume runs qgroup destroy.

The three profiles above set root size=20GB, so the next capped instance would have put full quotas straight back. Quotas had to stay on in a mode whose cost does not grow with sharing.

The fix (2026-10-21)

Simple quotas (kernel 6.7+, btrfs quota enable --simple) account each extent to the subvolume that allocated it, at allocation time. Dropping a subvolume does no backref walk.

It was rehearsed first on a loop-mounted btrfs image:

• Incus's own commands work against squota: qgroup show -e -f --raw, qgroup limit and qgroup destroy.
• A 20 GB-style cap is enforced; writes fail with EDQUOT.
• Plain btrfs quota enable on a squota filesystem is a no-op. It does not switch the filesystem back to full qgroups.

Mohamed ran the three sudo commands; Claude Code's auto-mode classifier blocks it from running btrfs quota on the pool.

1. Disable full qgroups. sudo btrfs quota disable ~/t9-pool hung. The cleaner holds cleaner_mutex for the whole drop of one subvolume, and the drop in progress (subvolume 1048) was estimated at 20–30 hours.

2. Unblock the cleaner. Setting the subtree-drop threshold to 0 tells the kernel to skip qgroup accounting on subtree drops:

   echo 0 | sudo tee /sys/fs/btrfs/aa55c400-b8fc-4998-8a64-fd876ab92c83/qgroups/drop_subtree_threshold
   

   The disable then completed, and all 21 queued deletions drained in about 15 s.

3. Enable simple quotas:

   sudo btrfs quota enable --simple ~/t9-pool
   

   This sets the SIMPLE_QUOTA incompat feature; sysfs then shows qgroups/mode = squota.

Results

Incus then started in about 1 s. The prod orchgraph HA cluster came back with orchgraph-data-2 as main and orchgraph-data-1 as a sync replica, 0 behind.

State on 2026-10-22

• Quotas: qgroups/mode=squota, inconsistent=0, features/simple_quota=1.
• Space: data 79.25 GiB, metadata 19.59 GiB (DUP), 118.9 GiB used of 1.04 TiB.
• Health: no queued deletions; scrub finished 2026-10-21 23:39 with no errors; error counters 0.
• qgroups/drop_subtree_threshold is back at its default of 3, because enabling quotas resets it. Under squota the kernel does no subtree tracing on drop, so the value only matters if full qgroups ever come back (see the rules below).

What simple quotas do NOT give you

• Extents written before squota was enabled are not accounted to anyone. On the instances that existed on 2026-10-21, the 20 GB caps and the usage numbers Incus reports cover only data written since then.
• Usage belongs to the subvolume that allocated the extent. A clone's usage does not include what it shares with its image or base. That is more permissive than full qgroups' referenced/exclusive numbers, and it is exactly what makes drops cheap.

Rules

• Never disable quotas on the pool. With quotas off, Incus's next size operation runs a plain quota enable plus a full rescan, and the stall returns.
• Never run btrfs quota rescan on the pool.
• If quotas ever do end up off, re-enable them with --simple before Incus starts.
• If full qgroups are ever on again and you need to leave them, set qgroups/drop_subtree_threshold to 0 first, or the disable sits behind the cleaner for hours.
• Keep databases on the T9, not the root NVMe. The NVMe flushes at 5.7 ms per sync write and is 41% worn. A 2026-10-21 move of orchgraph-data-1 to /opt/orchgraph-nvme was reverted on 2026-10-22.

Recognizing it again

cat /sys/fs/btrfs/aa55c400-b8fc-4998-8a64-fd876ab92c83/qgroups/mode   # must be: squota
sudo btrfs subvolume list -d ~/t9-pool | wc -l                        # queued deletions; should drain fast
time sync                                                              # milliseconds, not seconds
ps -eo pid,stat,comm | awk '$2 ~ /D/'                                  # btrfs-cleaner / btrfs-transaction stuck in D?

• A mode of qgroup (full) or disabled is the cause by itself.
• A growing deleted-subvolume count, sync taking seconds, and btrfs-cleaner in D state together are the signature of this failure.

T9 hardware findings (open)

Maintenance

• Scrub monthly: btrfs-scrub-t9-pool.timer (installed 2026-10-22 in /etc/systemd/system/) runs btrfs scrub start -B -c 3 /home/mohamed/t9-pool on the first Sunday of each month at 04:00 (+ up to 30 min), and runs at the next boot if the machine was off. It uses the idle I/O class and IOSchedulingClass=idle, and skips when the T9 is not mounted (ConditionPathIsMountPoint). Check the last result with systemctl status btrfs-scrub-t9-pool.service or sudo btrfs scrub status ~/t9-pool.
• TRIM: none possible until the provisioning question above is decided. fstrim.timer is enabled weekly and covers the root NVMe.
• Health check after any incident: run btrfs device stats and btrfs scrub status. Error counters other than 0 mean investigate before anything else.

Open items

• [ ] Swap the T9 cable or port and confirm speed=20000 with 2 lanes.
• [ ] Decide on TRIM (the udev provisioning_mode=unmap rule).
• [x] Schedule the monthly scrub: btrfs-scrub-t9-pool.timer, 2026-10-22.
• [ ] Delete the unused NVMe copies (/opt/orchgraph-nvme, about 2.5 GB) and the pre-move backup rootfs/var/lib/orchgraph.pre-nvme-2026-10-21 in orchgraph-data-1 (657 MB), once Mohamed confirms.

Related: RUL-038 (its 2026-10-19 evidence is this stall), Incus containers.