i've been using linux in some form as my main system for a few years now. most of that time it's been with arch (btw) which i opted to install btrfs because i wanted the snapshot feature (spoiler: i've never taken a snapshot). little did i know that btrfs has a very particular problem that'll come to bite me in the ass. i had known that there were some complaints about btrfs, but i assumed these were niche problems that only powerusers would run into. how naive of me.

so there i was. just got done with work, poured myself a gin and tonic, and sat down to watch some movies with friends on discord. but then god btrfs said no. i was hit with the dreaded 'out of space' error that, apparently, btrfs is well known for. i was doing something when my system slowed to a crawl and then eventually crashed hard. the only thing i could do was access a TTY. you know where this is going.

the reason i got into this state, i surmise, is that i installed/uninstalled a lot of games on this drive until i switched them to a secondary drive (which doesn't use btrfs). since i had offloaded all of my games, my main drive should've had (which it did) plenty of space, yet is was being reported that it was full. why was my system reporting a full drive? the solutions i found weren't exactly straightforward if you're not familiar with btrfs.

my initial debugging had me using the typical tools like df. however, btrfs just has to be special and it has its own set of tools to inspect the file system. traditional tools will misreport the space that's available, so the btrfs commands will be the source of truth. commands like btrfs fi show, btrfs fi usage /, and btrfs fi df / should give a good idea of what btrfs sees. these commands were basically sanity checks to make sure the drive is "correctly" reported as full, so it's definitely a btrfs problem and not some other obscure linux problem.

i drove myself crazy looking for ways to free up space and trying to identify the problematic files. however, no matter what i did, no space was being freed. turns out that btrfs has this concept of metadata and data blocks. i'm still not an expert on btrfs (or file systems in general) so i don't know the technical context behind these blocks, but the point is there's a quirk (i'm being very generous here) with btrfs where these blocks do not get cleaned up automatically. even if you're diligent about keeping your drive clean, these blocks will not get updated to say the space has been freed until you tell it to.

to resolve this issue, i had to run btrfs balance to rebalance the data and metadata blocks which will "free" up space so the correct amount of space available will be reported. however, since the system is saying "no free space", i couldn't run the balance command as it stood. in this case, i needed to use an external drive as a sort of staging area to assist in the rebalancing. if you need to attach a drive, i highly recommend using the fastest drive/connection you have. the rebalancing process is extremely slow even when the target drive is NVMe, so adding a drive with USB overhead is an exercise in pain if using a plain USB stick. get an external SSD over USB 3 or higher. the drive doesn't have to have a ton of space on it though i used an external 1 TB portable drive. it was working with my 128 GB USB stick (albeit slow) but it could probably work with something smaller. i probably wouldn't go smaller than 16 GB.

because it's so slow, start with a small usage number and increase as needed. the balancing can be done as many times as you want. check the state of the system with the commands above after each successful balance to see if more can be balanced. i'd recommend balancing both metadata and data blocks, so you'll use -musage and -dusage.

sudo btrfs balance start -dusage=55 /mnt/external &

sudo btrfs balance start -musage=55 /mnt/external &

depending on size and speed of your drives, this can take a very long time, so sit back and relax while the system is rebalanced. i ultimately repeated the above commands a few times while decreasing the usage numbers until i was satisfied. the more times you do it, the faster it'll go.

lemme crash out for a second. are you fucking kidding me? there is no other file system on the planet that i'm aware of where you have to manually tell the fucker "oh hey btw. those blocks? yeah they're free. plz give me my space back." this is literally like if someone were to gaslight you into thinking they emptied the trash while you're looking right at it and can see it overflowing. if i wasn't lazy, i'd've wiped my drive and just reinstalled with ext4. all told, it probably would've been faster to wipe and reinstall but i was being stubborn. if you don't need btrfs features on your desktop system, and you probably don't, just use ext4 and save yourself the brain aneurysm.

some references:

• file system full
• kernel faq
• arch wiki on btrfs