Autarchy of the Private Cave

I’m moving from (a kind of…) a dedicated server to a VPS, to decrease my frightful anticipation of hardware failures.
Honestly though, that server had been freezing up and restarting spontaneously for several months now, causing sometimes really long down-times…
That server is now about 6-7 years old, built with off-the-shelf components, some of which (the HDD ) had weird noises from the very start.
Definitely time to move!

I’ve purchased a fairly cheap VPS with an easy, one-click upgrade option for after I’m done configuring it.
It comes with a wide selection of OSes to pre-install; I’ve chosen Debian Jessie, version 8.3 as of this writing.

I wanted to use btrfs from the beginning, so could have installed Debian myself, but… VPS provider does some initial configuration (like their Debian mirror and some other things), so I’ve felt that converting to btrfs after the fact would be easier. Now that I’ve done this – I guess it was fairly easy, although preparation did take some time.

Below, I’m providing step-by-step instructions on how to convert your root filesystem from (most likely) ext4 to btrfs.

I’ll be using my provider’s rescue boot mode: this is a live Debian system which is network-booted on my own VPS, and thus has access to the SSD/HDD of my VPS. Hopefully, your provider has a similar feature.

Preparing for this conversion, I had mostly used two sources:

• an older howtoforge conversion tutorial, and
• Sysadmin Guide

Before we begin with actual conversion, here is a

List of hints and useful practices/commands for a future btrfs user

• Changing the default subvolume with btrfs subvolume set-default will make the top level of the filesystem inaccessible, except by use of the subvol=/ or subvolid=5 mount options.
• If top level is no longer your default subvolume, it is useful to have an fstab entry for the top level (note the noauto option!):
  LABEL=toplevel /root/btrfs-top-lvl btrfs subvol=/,defaults,noauto,noatime 0 0
• scurbbing and repairing btrfs; most of the hints below originate from Marc’s 2014 btrfs presentation
• scrub script: http://marc.merlins.org/linux/scripts/btrfs-scrub
• mount -o compress=lzo is fast and best for SSDs; mount -o compress=zlib compresses better but is slower
• you can turn off COW (copy-on-write) for specific files and directories with chattr ­-C /path (new files will inherit this)
• for a highly fragmented VM, btrfs filesystem defragment vbox.vdi could take hours; cp -reflink=never vbox.vdi vbox.vdi.new; rm vbox.vdi is much faster
• cp -reflink=always copies within and between subvolumes without duplicating data
• run scrub nightly or weekly on all btrfs filesystems
• noatime is the best fstab mount option for btrfs IF USING SNAPSHOTS, because relatime still causes at least 1 write every 24h
• if you have errors=remount-ro in your fstab for extX-filesystem, you should remove it after converting to btrfs as it does not understand the option.

We are almost ready to start! Just one last thing: partitioning!

On my previous server, I did have a certain partition scheme, with separate /boot, /home, /var, /usr, and everything else under / partition.
It worked very well for me, never ran into problems (although this is more like a Debian’s feature, not that of my partition scheme ).
However… With the current VPS, it started initially with just everything under /, in a single partition.
I could have re-partitioned everything from the rescue system while converting to btrfs, but what for? What would be the benefits?

Let’s have a look and evaluate

Advantages/disadvantages of keeping specific partitions separate.

• /boot comes the first. Normally I format it with ext2, and set aside just 100-200 megabytes for this small partition.
  This improves boot-reliability of the system. With btrfs, it was also the case that grub could not use /boot on a btrfs volume.
  However,

  Newer GRUBs can handle a /boot partition which is btrfs, so you need not have a separate /boot partition formatted as ext3/4.

  So this time I’m going without a separate /boot.
  I will not even create a separate subvolume for it: for snapshotting purposes, /boot fits perfectly under the / snapshots.

• Next comes /home.
  The reason to separate it is to safeguard user data when upgrading/reinstalling the OS.
  This, however, can be achieved with btrfs subvolumes – no need for partitions!
  I’m going with a “subvolume for home” here, then.
• Next comes /var, or more precisely /var/log.
  One of the reasons to keep /var/log separate is to prevent filesystem overfill if logs suddenly start growing too fast.
  This, however, can be achieved with subvolume quotas. No need for a partition!
• There is also /var/lib/ (and also /var/lib/lxc/…), which hold fairly big data files, often with random write access patterns.
  There are two reasons to make them subvolumes:
  1. for data-only snapshots
  2. to enable nocow/nodatacow at mount time, to avoid high fragmentation with random writes

  However, this can be done later, when these databases/LXC get installed – in just a few commands, without rebooting.

• Finally, / itself should be a separate subvolume – mostly for snapshot and reinstall/upgrade reasons.

Ok, now we are really going to

Convert the filesystem

1. Enable rescue system, write down access credentials. You may need to restart your VPS to boot into rescue system, but check with your VPS provider first.
2. Connect to and log in to the rescue system.
3. Install btrfs-tools in the rescue system, if not there yet. We’ll need btrfs-convert, which should be included in btrfs-tools.
4. I’m assuming that your SSD/HDD is device /dev/sda, and that the only (root) partition there is /dev/sda1; adjust the next steps if your setup is different. Only proceed with commands if you understand what they are doing!
5. It never hurts to fsck /dev/sda1 before doing filesystem conversion You should see something like /dev/sda1: clean, 27282/1564672 files, 351853/6249472 blocks.
6. btrfs-convert -l toplevel /dev/sda1; note that I’m adding a “toplevel” LABEL to the top level of btrfs, for easier mounting it later. The output should look like this:
   

   creating btrfs metadata.
   creating ext2fs image file.
   cleaning up system chunk.
   conversion complete.

   At this point the system is already btrfs!

7. If you forgot to set btrfs label, or decide to do it later: list the filesystems with btrfs filesystem show, or just look for the label with btrfs filesystem label /dev/sda1 (which will produce an empty line if there is no label), then simply btrfs filesystem label /dev/sda1 toplevel; it will produce no output on success.
8. Let’s change into the new system, and start configuring it:
   mount /dev/sda1 /mnt
for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done
chroot /mnt
9. ls now should show your root filesystem files and directories, plus the ext2_saved “directory” (which is a subvolume, actually):
   

   bin dev ext2_saved initrd.img installimage.debug lib64 media opt root sbin sys usr vmlinuz
   boot etc home installimage.conf lib lost+found mnt proc run srv tmp var

10. Run blkid /dev/sda1 to find out the UUID of the new btrfs top-level volume. You can also see it with btrfs filesystem show:
    

    /dev/sda1: LABEL=”toplevel” UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ UUID_SUB=”ffad5668-2ac7-4ea4-83ad-e6ccba7ccf96″ TYPE=”btrfs” PARTUUID=”750b31a7-01″

11. Edit /etc/fstab (with e.g. nano /etc/fstab):
    • change UUID of / to the new one
    • change the filesystem type from ext3/ext4 to btrfs
    • change the options to subvol=root,defaults,noatime,compress=lzo,ssd,discard (yes, subvol=root does not exist yet… also, ssd,discard are for SSDs only, and instead of adding discard here it might be better to setup an fstrim cronjob. Finally, noatime can be left out if you are not going to make snapshots.)
    • change the last number in the line from 1 to 0 (the fsck passes – btrfs does not want boot-time filesystem checking)
12. Some guides mention editing /etc/grub.d/00_header in a specific way, running grub-mkconfig | grep ” ro “ to see if it adds the proper rootflags=subvol=root, then even run update-grub and grub-install /dev/sda here… But all of this seems redundant or plain unnecessary. First of all, the root subvolume simply does not exist yet, so grub-mkconfig will not produce correct rootflags. Secondly, we are not yet done with setting up subvolumes, and we are not going to reboot right now, so running update-grub/grub-install is a bit premature at this stage
13. Now we are going to create some subvolumes. As I want to name the root subvolume simply root, I should first move the root user’s home files somewhere else (otherwise btrfs snapshot complains “ERROR: incorrect snapshot name ‘/’”):
    mv /root /root_orig
btrfs subvolume snapshot / /root
14. The same problem with /home, need to move it as well: mv /home /home_orig
15. I can NOT snapshot /home like I did with /, because it is just a directory, not a volume like /, so:
    btrfs subvolume create /home
    Note: my /home (/home_orig) is still empty, as this is a brand-new installation with only the root user, so double-check steps below, as I haven’t really run them.
    rsync --progress -aHAX /home_orig/* /home
ls -la /home/
ls -la /home_orig/
rm -fr /home_orig/*
ls -la /root/home/
rm -fr /root/home/* # be sure to leave the empty /root/home/ in place: it will be the mount point.
16. nano /etc/fstab to add the /home mount line, using the same UUID as earlier for the / partition, but a different subvol option:
    UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /home btrfs subvol=home,defaults 0 0
    Feel free to add more mount options, similar to those for root subvolume. While we are editing /etc/fstab, it also makes sense to add a non-automount /toplevel entry (UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /toplevel btrfs subvol=/,defaults,noauto 0 0) and create mountpoint for it: mkdir /root/toplevel
17. Follow the same procedure to create the /var/log subvolume. I’m using flat subvolumes layout here:
    btrfs subvolume create /log
rsync --progress -aHAX /var/log/* /log
ls -la /log/
ls -la /var/log/
rm -fr /var/log/*
ls -la /root/var/log/
rm -fr /root/var/log/*

18. Add /var/log entry to fstab:
    UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ /var/log btrfs subvol=log,defaults,ssd,discard,compress=lzo,noatime 0 0
19. /var/lib/ – databases, LXC, nodatacow/nocow – can be created later, not doing it here and now.
20. cp /etc/fstab /root/etc/fstab
21. exit # chroot
22. for fs in proc sys dev/pts dev; do umount /mnt/$fs; done
umount /mnt
23. Let’s enter the new root subvolume!
    mount -o subvol=root /dev/sda1 /mnt
for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done
chroot /mnt
24. Oh, we forgot to move back root_orig into /root! That’s easy: mv root_orig root. If you want your root’s aliases/profile to load, then just exit and chroot /mnt again (I had to do that for the nice pre-configured prompt colours )
25. btrfs subvolume list / to see our new and shiny subvolumes:
    

    ID 256 gen 8 top level 5 path ext2_saved
    ID 260 gen 49 top level 5 path root
    ID 261 gen 35 top level 5 path home
    ID 262 gen 42 top level 5 path log

26. Let’s make root the default subvolume! If suddenly rootflags get lost during grub configuration, then the system should still be able to boot. Subvol ID comes from the output of the command above (260 in my case):
    btrfs subvolume set-default 260 / # no output
    btrfs subvolume get-default / # to verify

    ID 260 gen 49 top level 5 path root

27. Now let’s check grub-mkconfig output: grub-mkconfig | grep " ro ". Now that the subvolume exists, it does show all the desired rootflags=subvol=root.
28. Let’s make sure our system will be bootable:
    update-grub
grub-install /dev/sda
29. Leave the chroot:
    exit
for fs in proc sys dev/pts dev; do umount /mnt/$fs; done
umount /mnt
30. Now let’s mount top-level again, for cleanup. If you remember, we had all the / files there – which are now in the root subvolume:
    mount -o subvol=/ /dev/sda1 /mnt
31. Navigate to /mnt (cd /mnt, yes ) and delete from here everything that is NOT a subvolume.
    The command which supposedly does this is find /mnt -xdev -delete (-xdev is supposed to not descend into mounts of other filesystems), but I haven’t used it – was a bit tired and did not want to screw up the so-far-successful conversion, so I just did rm -rf bin boot dev etc initrd.img lib lib64 lost+found media mnt opt proc run sbin srv sys tmp usr vmlinuz var root_orig – basically, deleted everything except for ext2_saved, home, root, and log – which are all subvolumes.
32. Confirm that only subvolumes are left now:
    ls -la:

    drwxr-xr-x 1 root root 10 Feb 13 21:30 ext2_saved
    drwxr-xr-x 1 root root 0 Feb 13 22:04 home
    drwxr-xr-x 1 root root 510 Feb 13 22:13 log
    drwxr-xr-x 1 root root 318 Feb 13 22:18 root

    btrfs subvolume list /mnt:

    ID 256 gen 8 top level 5 path ext2_saved
    ID 260 gen 52 top level 5 path root
    ID 261 gen 35 top level 5 path home
    ID 262 gen 42 top level 5 path log

    btrfs subvolume get-default /mnt:

    ID 260 gen 52 top level 5 path root

33. We can leave now: cd / && umount /mnt
34. Reboot into the new system. If this fails: can boot rescue again, chroot and check what went wrong…
35. In my case, conversion seemed to have succeeded, BUT: a) for some reason /toplevel was mounted at boot and full of root-like files which should not have been there anymore; b) I could not delete ext2_saved, at all, neither from / nor from /toplevel. The problem was with the toplevel mount line in /etc/fstab (initially, I forgot to specify the filesystem type). After fixing this and rebooting once more, everything was perfect: /toplevel not mounted, and when mounted it only had subvolumes in it. And of course the system booted without any problems. Now I could delete the previous filesystem snapshot: mount /toplevel; btrfs subvolume delete /toplevel/ext2_saved; umount /toplevel. Note that there was also another, different ext2_saved in the root subvolume: it is NOT a subvolume, which was not carried over when we created the / snapshot – only the empty directory was left in place. (Yes, snapshots do not descend into subvolumes.)

The last 2 commands to issue (after mount /toplevel) are btrfs filesystem defragment -r /toplevel and btrfs balance start /toplevel.
I’m not including them in the instructions above: they are not strictly required, and if your filesystem had some heavy use before conversion, you may want to run these 2 commands more intelligently, in phases (metadata first, then big files, then smaller, etc). Another reason is that it may produce significant system load. On my fresh system both finished very quickly.

…And we are done!