I’ve purchased a fairly cheap VPS with an easy, one-click upgrade option for after I’m done configuring it.
It comes with a wide selection of OSes to pre-install; I’ve chosen Debian Jessie, version 8.3 as of this writing.

I wanted to use btrfs from the beginning, so could have installed Debian myself, but… VPS provider does some initial configuration (like their Debian mirror and some other things), so I’ve felt that converting to btrfs after the fact would be easier. Now that I’ve done this – I guess it was fairly easy, although preparation did take some time.

Below, I’m providing step-by-step instructions on how to convert your root filesystem from (most likely) ext4 to btrfs.

I’ll be using my provider’s rescue boot mode: this is a live Debian system which is network-booted on my own VPS, and thus has access to the SSD/HDD of my VPS. Hopefully, your provider has a similar feature.

Preparing for this conversion, I had mostly used two sources:

• an older howtoforge conversion tutorial, and
• Sysadmin Guide

Before we begin with actual conversion, here is a

List of hints and useful practices/commands for a future btrfs user

• Changing the default subvolume with btrfs subvolume set-default will make the top level of the filesystem inaccessible, except by use of the subvol=/ or subvolid=5 mount options.
• If top level is no longer your default subvolume, it is useful to have an fstab entry for the top level (note the noauto option!):
  LABEL=toplevel /root/btrfs-top-lvl btrfs subvol=/,defaults,noauto,noatime 0 0
• scurbbing and repairing btrfs; most of the hints below originate from Marc’s 2014 btrfs presentation
• scrub script: http://marc.merlins.org/linux/scripts/btrfs-scrub
• mount -o compress=lzo is fast and best for SSDs; mount -o compress=zlib compresses better but is slower
• you can turn off COW (copy-on-write) for specific files and directories with chattr ­-C /path (new files will inherit this)
• for a highly fragmented VM, btrfs filesystem defragment vbox.vdi could take hours; cp -reflink=never vbox.vdi vbox.vdi.new; rm vbox.vdi is much faster
• cp -reflink=always copies within and between subvolumes without duplicating data
• run scrub nightly or weekly on all btrfs filesystems
• noatime is the best fstab mount option for btrfs IF USING SNAPSHOTS, because relatime still causes at least 1 write every 24h
• if you have errors=remount-ro in your fstab for extX-filesystem, you should remove it after converting to btrfs as it does not understand the option.

We are almost ready to start! Just one last thing: partitioning!

On my previous server, I did have a certain partition scheme, with separate /boot, /home, /var, /usr, and everything else under / partition.
It worked very well for me, never ran into problems (although this is more like a Debian’s feature, not that of my partition scheme ).
However… With the current VPS, it started initially with just everything under /, in a single partition.
I could have re-partitioned everything from the rescue system while converting to btrfs, but what for? What would be the benefits?

Let’s have a look and evaluate

Advantages/disadvantages of keeping specific partitions separate.

• /boot comes the first. Normally I format it with ext2, and set aside just 100-200 megabytes for this small partition.
  This improves boot-reliability of the system. With btrfs, it was also the case that grub could not use /boot on a btrfs volume.
  However,

  Newer GRUBs can handle a /boot partition which is btrfs, so you need not have a separate /boot partition formatted as ext3/4.

  So this time I’m going without a separate /boot.
  I will not even create a separate subvolume for it: for snapshotting purposes, /boot fits perfectly under the / snapshots.

• Next comes /home.
  The reason to separate it is to safeguard user data when upgrading/reinstalling the OS.
  This, however, can be achieved with btrfs subvolumes – no need for partitions!
  I’m going with a “subvolume for home” here, then.
• Next comes /var, or more precisely /var/log.
  One of the reasons to keep /var/log separate is to prevent filesystem overfill if logs suddenly start growing too fast.
  This, however, can be achieved with subvolume quotas. No need for a partition!
• There is also /var/lib/ (and also /var/lib/lxc/…), which hold fairly big data files, often with random write access patterns.
  There are two reasons to make them subvolumes:
  1. for data-only snapshots
  2. to enable nocow/nodatacow at mount time, to avoid high fragmentation with random writes

  However, this can be done later, when these databases/LXC get installed – in just a few commands, without rebooting.

• Finally, / itself should be a separate subvolume – mostly for snapshot and reinstall/upgrade reasons.

Ok, now we are really going to

Convert the filesystem

1. Enable rescue system, write down access credentials. You may need to restart your VPS to boot into rescue system, but check with your VPS provider first.
2. Connect to and log in to the rescue system.
3. Install btrfs-tools in the rescue system, if not there yet. We’ll need btrfs-convert, which should be included in btrfs-tools.
4. I’m assuming that your SSD/HDD is device /dev/sda, and that the only (root) partition there is /dev/sda1; adjust the next steps if your setup is different. Only proceed with commands if you understand what they are doing!
5. It never hurts to fsck /dev/sda1 before doing filesystem conversion You should see something like /dev/sda1: clean, 27282/1564672 files, 351853/6249472 blocks.
6. btrfs-convert -l toplevel /dev/sda1; note that I’m adding a “toplevel” LABEL to the top level of btrfs, for easier mounting it later. The output should look like this:
   

   creating btrfs metadata.
   creating ext2fs image file.
   cleaning up system chunk.
   conversion complete.

   At this point the system is already btrfs!

7. If you forgot to set btrfs label, or decide to do it later: list the filesystems with btrfs filesystem show, or just look for the label with btrfs filesystem label /dev/sda1 (which will produce an empty line if there is no label), then simply btrfs filesystem label /dev/sda1 toplevel; it will produce no output on success.
8. Let’s change into the new system, and start configuring it:
   mount /dev/sda1 /mnt
for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done
chroot /mnt
9. ls now should show your root filesystem files and directories, plus the ext2_saved “directory” (which is a subvolume, actually):
   

   bin dev ext2_saved initrd.img installimage.debug lib64 media opt root sbin sys usr vmlinuz
   boot etc home installimage.conf lib lost+found mnt proc run srv tmp var

10. Run blkid /dev/sda1 to find out the UUID of the new btrfs top-level volume. You can also see it with btrfs filesystem show:
    

    /dev/sda1: LABEL=”toplevel” UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ UUID_SUB=”ffad5668-2ac7-4ea4-83ad-e6ccba7ccf96″ TYPE=”btrfs” PARTUUID=”750b31a7-01″

11. Edit /etc/fstab (with e.g. nano /etc/fstab):
    • change UUID of / to the new one
    • change the filesystem type from ext3/ext4 to btrfs
    • change the options to subvol=root,defaults,noatime,compress=lzo,ssd,discard (yes, subvol=root does not exist yet… also, ssd,discard are for SSDs only, and instead of adding discard here it might be better to setup an fstrim cronjob. Finally, noatime can be left out if you are not going to make snapshots.)
    • change the last number in the line from 1 to 0 (the fsck passes – btrfs does not want boot-time filesystem checking)
12. Some guides mention editing /etc/grub.d/00_header in a specific way, running grub-mkconfig | grep ” ro “ to see if it adds the proper rootflags=subvol=root, then even run update-grub and grub-install /dev/sda here… But all of this seems redundant or plain unnecessary. First of all, the root subvolume simply does not exist yet, so grub-mkconfig will not produce correct rootflags. Secondly, we are not yet done with setting up subvolumes, and we are not going to reboot right now, so running update-grub/grub-install is a bit premature at this stage
13. Now we are going to create some subvolumes. As I want to name the root subvolume simply root, I should first move the root user’s home files somewhere else (otherwise btrfs snapshot complains “ERROR: incorrect snapshot name ‘/’”):
    mv /root /root_orig
btrfs subvolume snapshot / /root
14. The same problem with /home, need to move it as well: mv /home /home_orig
15. I can NOT snapshot /home like I did with /, because it is just a directory, not a volume like /, so:
    btrfs subvolume create /home
    Note: my /home (/home_orig) is still empty, as this is a brand-new installation with only the root user, so double-check steps below, as I haven’t really run them.
    rsync --progress -aHAX /home_orig/* /home
ls -la /home/
ls -la /home_orig/
rm -fr /home_orig/*
ls -la /root/home/
rm -fr /root/home/* # be sure to leave the empty /root/home/ in place: it will be the mount point.
16. nano /etc/fstab to add the /home mount line, using the same UUID as earlier for the / partition, but a different subvol option:
    UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /home btrfs subvol=home,defaults 0 0
    Feel free to add more mount options, similar to those for root subvolume. While we are editing /etc/fstab, it also makes sense to add a non-automount /toplevel entry (UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /toplevel btrfs subvol=/,defaults,noauto 0 0) and create mountpoint for it: mkdir /root/toplevel
17. Follow the same procedure to create the /var/log subvolume. I’m using flat subvolumes layout here:
    btrfs subvolume create /log
rsync --progress -aHAX /var/log/* /log
ls -la /log/
ls -la /var/log/
rm -fr /var/log/*
ls -la /root/var/log/
rm -fr /root/var/log/*

18. Add /var/log entry to fstab:
    UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ /var/log btrfs subvol=log,defaults,ssd,discard,compress=lzo,noatime 0 0
19. /var/lib/ – databases, LXC, nodatacow/nocow – can be created later, not doing it here and now.
20. cp /etc/fstab /root/etc/fstab
21. exit # chroot
22. for fs in proc sys dev/pts dev; do umount /mnt/$fs; done
umount /mnt
23. Let’s enter the new root subvolume!
    mount -o subvol=root /dev/sda1 /mnt
for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done
chroot /mnt
24. Oh, we forgot to move back root_orig into /root! That’s easy: mv root_orig root. If you want your root’s aliases/profile to load, then just exit and chroot /mnt again (I had to do that for the nice pre-configured prompt colours )
25. btrfs subvolume list / to see our new and shiny subvolumes:
    

    ID 256 gen 8 top level 5 path ext2_saved
    ID 260 gen 49 top level 5 path root
    ID 261 gen 35 top level 5 path home
    ID 262 gen 42 top level 5 path log

26. Let’s make root the default subvolume! If suddenly rootflags get lost during grub configuration, then the system should still be able to boot. Subvol ID comes from the output of the command above (260 in my case):
    btrfs subvolume set-default 260 / # no output
    btrfs subvolume get-default / # to verify

    ID 260 gen 49 top level 5 path root

27. Now let’s check grub-mkconfig output: grub-mkconfig | grep " ro ". Now that the subvolume exists, it does show all the desired rootflags=subvol=root.
28. Let’s make sure our system will be bootable:
    update-grub
grub-install /dev/sda
29. Leave the chroot:
    exit
for fs in proc sys dev/pts dev; do umount /mnt/$fs; done
umount /mnt
30. Now let’s mount top-level again, for cleanup. If you remember, we had all the / files there – which are now in the root subvolume:
    mount -o subvol=/ /dev/sda1 /mnt
31. Navigate to /mnt (cd /mnt, yes ) and delete from here everything that is NOT a subvolume.
    The command which supposedly does this is find /mnt -xdev -delete (-xdev is supposed to not descend into mounts of other filesystems), but I haven’t used it – was a bit tired and did not want to screw up the so-far-successful conversion, so I just did rm -rf bin boot dev etc initrd.img lib lib64 lost+found media mnt opt proc run sbin srv sys tmp usr vmlinuz var root_orig – basically, deleted everything except for ext2_saved, home, root, and log – which are all subvolumes.
32. Confirm that only subvolumes are left now:
    ls -la:

    drwxr-xr-x 1 root root 10 Feb 13 21:30 ext2_saved
    drwxr-xr-x 1 root root 0 Feb 13 22:04 home
    drwxr-xr-x 1 root root 510 Feb 13 22:13 log
    drwxr-xr-x 1 root root 318 Feb 13 22:18 root

    btrfs subvolume list /mnt:

    ID 256 gen 8 top level 5 path ext2_saved
    ID 260 gen 52 top level 5 path root
    ID 261 gen 35 top level 5 path home
    ID 262 gen 42 top level 5 path log

    btrfs subvolume get-default /mnt:

    ID 260 gen 52 top level 5 path root

33. We can leave now: cd / && umount /mnt
34. Reboot into the new system. If this fails: can boot rescue again, chroot and check what went wrong…
35. In my case, conversion seemed to have succeeded, BUT: a) for some reason /toplevel was mounted at boot and full of root-like files which should not have been there anymore; b) I could not delete ext2_saved, at all, neither from / nor from /toplevel. The problem was with the toplevel mount line in /etc/fstab (initially, I forgot to specify the filesystem type). After fixing this and rebooting once more, everything was perfect: /toplevel not mounted, and when mounted it only had subvolumes in it. And of course the system booted without any problems. Now I could delete the previous filesystem snapshot: mount /toplevel; btrfs subvolume delete /toplevel/ext2_saved; umount /toplevel. Note that there was also another, different ext2_saved in the root subvolume: it is NOT a subvolume, which was not carried over when we created the / snapshot – only the empty directory was left in place. (Yes, snapshots do not descend into subvolumes.)

The last 2 commands to issue (after mount /toplevel) are btrfs filesystem defragment -r /toplevel and btrfs balance start /toplevel.
I’m not including them in the instructions above: they are not strictly required, and if your filesystem had some heavy use before conversion, you may want to run these 2 commands more intelligently, in phases (metadata first, then big files, then smaller, etc). Another reason is that it may produce significant system load. On my fresh system both finished very quickly.

…And we are done!

The simplest solution to forward requests from the web-frontend to your API is by using mod_proxy.
If you want to forward any requests to /api/* to your custom back-end server on port 8000, you just add the following lines to your VirtualHost configuration:

ProxyPass “/api” “http://localhost:8000″
ProxyPassReverse “/api” “http://localhost:8000″

I’d suggest not wrapping this fragment with the classical IfModule: as your application will not really work without its API back-end, you actually want Apache to fail as soon as possible if mod_proxy is missing.

That was easy, right? What, it doesn’t work? Can’t be! It’s dead simple! No way you could make a mistake in 2 lines of configuration!!! :mad_rage:

Oh wait… I remember I had this problem before…

Let’s check:

1. Step 1. Did you disable (using a2dissite default or a2dissite 000-default, depending on your Debian-based GNU/Linux) the default website? If your application and the default website are configured in a similar way, then it might be the default site which is serving your app’s pages. The most sure way is to just disable it.
2. Step 2. Did you enable also the proxy_http sub-module? (Using a2enmod proxy_http, followed by service apache2 restart) mod_proxy is only the core module, actual per-protocol work is done by these sub-modules.

Your requests to /api should now be passed on to your API server. If not – please write in the comments what was the problem in your case and how you solved it. HTH!

More details about these tools and runlevels (in German).

As VihangD points out in his serverfault answer, the same can be achieved with aptitude, while also excluding dependent, automatically installed packages (which are included by the classical method). To create the list of packages, run aptitude search -F '%p' '~i!~M' > package_list. Here, -F '%p' asks aptitude to only print package names (instead of the default output, which also contains package state and description); search term ‘~i!~M’ asks for all non-automatically installed packages.

To install packages using the created list, run xargs aptitude --schedule-only install < package_list; aptitude install. The first of these two commands instructs aptitude to mark all the packages from the list as scheduled for installation. The second command actually performs the installation.

Hamish Downer suggests an alternative way of getting the initial package_list: using the deborphan utility, deborphan -a --no-show-section > package_list. This command asks deborphan to show a list of packages, which have no dependencies on them. Sounds very similar to what we did with aptitude above, but using deborphan will most likely result in a much shorter list of packages (on my system, deborphan printed 291 package names, aptitude printed 847, and dpkg printed 3650 package names). One more potentially important difference between aptitude- and deborphan-produced package lists is that aptitude only specifies package architecture when it is different from native (e.g. 'googleearth:i386' on a 64-bit system), while deborphan specifies architectures for all the packages (resulting in e.g. 'google-talkplugin:amd64' and 'googleearth-package:all' on a 64-bit system).

Current setup:

• MX records point to the spam protection mail servers, which then
• connect to my server and deliver (hopefully spam-free) mail.

Problem: some senders (like last.fm) do have proper, strict SPF records. Tumgreyspf on my server then rejects emails relayed through the spam-protection service.

If these spam protection relay servers are the only which send mail to your server, then it makes sense to fully disable/uninstall tumgreyspf. Putting tumgreyspf into the permanent “learning mode” (set defaultSeedOnly = 1 in /etc/tumgreyspf/tumgreyspf.conf) may not fix the SPF problem described above, as SeedOnly seems to only affect greylisting, and not rejecting unauthorized senders.

Solution: whitelist relay server IPs.

I will use MXGuardDog spam blocker as an example. This solution is a slightly extended version of this one, and used tumgreyspf README as the reference.

• For each of the IPs you want to whitelist, create a directory tree under /var/lib/tumgreyspf/config/client_address. Here is a copy-pasteable example for MXGuardDog, based on their list of server IPs, valid as of August 2013:
  
mkdir -p /var/lib/tumgreyspf/config/client_address/108/166/117
mkdir -p /var/lib/tumgreyspf/config/client_address/174/129/28
mkdir -p /var/lib/tumgreyspf/config/client_address/216/58/39
mkdir -p /var/lib/tumgreyspf/config/client_address/222/229/219
mkdir -p /var/lib/tumgreyspf/config/client_address/64/15/147
mkdir -p /var/lib/tumgreyspf/config/client_address/66/85/178

• Into each of these IP range-specific directories, put a config file, which disables checks (or symlink one). First, create /etc/tumgreyspf/disable.conf with the following lines in it:
  
SPFSEEDONLY = 0
GREYLISTTIME = 600
CHECKERS =
OTHERCONFIGS =

  It is just like the default.conf, but has empty CHECKERS and OTHERCONFIGS lines.
  Now, symlink it into each of the IP range directories:
  
ln -s /etc/tumgreyspf/disable.conf /var/lib/tumgreyspf/config/client_address/108/166/117/__default__
ln -s /etc/tumgreyspf/disable.conf /var/lib/tumgreyspf/config/client_address/174/129/28/__default__
ln -s /etc/tumgreyspf/disable.conf /var/lib/tumgreyspf/config/client_address/216/58/39/__default__
ln -s /etc/tumgreyspf/disable.conf /var/lib/tumgreyspf/config/client_address/222/229/219/__default__
ln -s /etc/tumgreyspf/disable.conf /var/lib/tumgreyspf/config/client_address/64/15/147/__default__
ln -s /etc/tumgreyspf/disable.conf /var/lib/tumgreyspf/config/client_address/66/85/178/__default__


Note the double-underscores to the left and right of default.

That’s it.

• current HDD is /dev/sda, it has a GPT (with bios_grub being /dev/sda1), separate /boot partition (/dev/sda2), and a physical LVM volume (/dev/sda3), where LVM holds all the remaining partitions (root, /home, /srv, …); LVM is properly configured, and system reboots with no problems
• your new drive is /dev/sdb, it is identical to /dev/sda, and it comes empty from the manufacturer (this is important! wipe the drive if it is not empty, especially if it used to be a part of another RAID)
• your system is Debian or Debian-based; in this exact example I’ve been using Ubuntu Server 10.04
• your LVM volume group is named vg0
• make sure you understand what each command does before executing it
• you do have an external backup of all your important data, and you do understand that the following operations are potentially dangerous to your data integrity

Inspired by: Debian Etch RAID guide, serverfault question.

1. Create the GPT on the new drive:
   parted /dev/sdb mklabel gpt
2. Get the list of partitions on /dev/sda:
   parted -m /dev/sda print
3. Create /dev/sdb partitions similarly to what you have on /dev/sda (my example numbers follow, use your numbers here):
   parted /dev/sdb mkpart bios_grub 1049kB 2097kB
   parted /dev/sdb mkpart boot 2097kB 258MB
   parted /dev/sdb mkpart lvm 258MB 2000GB
4. Set proper flags on partitions:
   parted /dev/sdb set 1 bios_grub on (GPT doesn’t have MBR, so you create a 1-MB partition instead to hold grub2′s boot code)
   (possibly optional) parted /dev/sdb set 2 raid on
   (possibly optional) parted /dev/sdb set 3 raid on
5. (possibly optional) To make sure /dev/sdb1 (the bios_grub) indeed contains grub’s boot code, I did dd if=/dev/sda1 of=/dev/sdb1
6. apt-get install mdadm
7. Note: at this point, older tutorials suggest adding a bunch of raid* kernel modules to /etc/modules and to grub’s list of modules to load. I’m not sure this is really necessary, but do see the tutorials mentioned at the top for more information. If you do modify the lists of modules – don’t forget to run update-initramfs -u.
8. Create two initially-degraded RAID1 devices (one for /boot, another for LVM):
   mdadm ––create /dev/md0 ––level=1 ––raid-devices=2 /dev/sdb2 missing
   mdadm ––create /dev/md1 ––level=1 ––raid-devices=2 /dev/sdb3 missing
9. Store the configuration of your RAID1 to the mdadm.conf file (important! this is not done automatically!)
   mdadm -Es >> /etc/mdadm/mdadm.conf
10. Verify the contents of your mdadm.conf:
    cat /etc/mdadm/mdadm.conf
    dpkg-reconfigure mdadm, and enable booting in degraded mode
11. Copy your current /boot (/dev/sda2) to the new /dev/md0 /boot partition:
    (one can use something like dd -if /dev/sda2 -of /dev/md0 here as well, but for some reason my attempt at dd failed writing 1 last byte of data)
    mkdir /mnt/md0
    mkfs.ext4 /dev/md0 (one can also use other filesystems here, e.g. mkfs.ext3 or even mkfs.ext2)
    mount /dev/md0 /mnt/md0
    cp -a /boot/* /mnt/md0/
    umount /dev/md0
    rmdir /mnt/md0
12. Now extend your existing volume group to include the newly-created /dev/md1:
    pvcreate /dev/md1
    vgextend vg0 /dev/md1
13. Verify the list of logical volumes you curently have: enter lvm shell, and type lvs. Here’s what I had:
    LV VG Attr LSize Origin Snap% Move Log Copy% Convert
    home vg0 -wi-ao 1.70t
    logs vg0 -wi-ao 4.66g
    root vg0 -wi-ao 10.24g
    srv vg0 -wc-ao 100.00g
    swap vg0 -wi-ao 1.86g
    tmp vg0 -wi-ao 4.66g
14. Now, you can move all the logical volumes to new physical volume in one command: pvmove /dev/sda3 /dev/md1. Personally, remembering the problem I had with dd from /dev/sda2 to /dev/md0, I decided to move all logical volumes one-by-one; as this takes time, you may consider joining these operations with ; or &&, and putting the /tmp last (as the easiest one to re-create if it fails to move):
    pvmove ––name home /dev/sda3 /dev/md1
    pvmove ––name srv /dev/sda3 /dev/md1
    pvmove ––name logs /dev/sda3 /dev/md1
    pvmove ––name swap /dev/sda3 /dev/md1
    pvmove ––name root /dev/sda3 /dev/md1
    pvmove ––name tmp /dev/sda3 /dev/md1
15. To be safer, I ran FS check on a few volumes I could umount:
    umount /dev/mapper/vg0-srv
    fsck -f /dev/mapper/vg0-srv
    mount /dev/mapper/vg0-srv
    umount /dev/mapper/vg0-tmp
    fsck -f /dev/mapper/vg0-tmp
    mount /dev/mapper/vg0-tmp
16. Remove /dev/sda3 from the physical space available to your volume group:
    vgreduce vg0 /dev/sda3
17. Install grub2 to both drives, so as to make them both bootable in case of failure:
    grub-install ‘(hd0)’
    grub-install ‘(hd1)’
18. Edit /etc/fstab, pointing /boot to /dev/md0. You may use UUIDs here, but please do not use UUIDs from mdadm.conf – those are different from FS-UUIDs, instead do ls -l /dev/disk/by-uuid to find the UUID of /dev/md0. Personally, I had no problems just using /dev/md0.
19. Now is the time to add your original /dev/sda to the RAID1; be absolutely sure you have moved all the data off that drive, because these commands will destroy it:
    mdadm ––manage ––add /dev/md0 /dev/sda2
    mdadm ––manage ––add /dev/md1 /dev/sda3
    Re-syncing array will take some time.
20. To be on the safe side, you may want to run again update-initramfs -u and update-grub; I have also edited /etc/grub.d/40_custom, adding there 2 more boot options: from /dev/sda2 and /dev/sdb2 (/boot on both drives) – have no idea if that will work, but having more boot options didn’t hurt
21. Reboot into your new system. Actually, at this point reboot is only necessary to verify that your system is bootable – you may delay this reboot as long as you want to.
22. Many tutorials also suggest testing your RAID1 by manually “degrading” it, trying to boot, and then rebuilding it back. I haven’t done that, but you may want to.

Improvement suggestions, criticism and thank-you are welcome in the comments.

There are a few highly-popular patches, which haven’t yet made it into the Squeeze’s rtorrent version 0.8.6.

Note: ArchLinux already has an rtorrent-extended package, so you may want to look at that one (instead of my Debian package below). I might use Arch’s patchset next time to generate a truly extended rtorrent package.

I’ve incorporated 3 of such patches (#239 ipfilter, #2064 GeoIP support, and #2171 colours/eyecandy), and built a squeeze amd64 rtorrent-enhanced package.

A few excerpts to explain how these extra features are to be configured and used (all come from the above-mentioned trac tickets, some were edited/extended).

ipfilter

ipfilter allows to selectively blacklist/whitelist peers – based on IP address range files (so-called ipfilter files). The ipfilter files may, for example, come from bluetack.co.uk. Each line of the file contains a record in this format: range_description:start_IP-end_IP, where start_IP should be less than or equal to end_IP. Multiple files can be used. Overlapping ranges are merged automatically. Both incoming and outgoing connections are checked against the filter. Exclusions are not supported, so connection to/from IPs in all the loaded ranges will be disallowed and dropped.

Include “ip_filter=” directive in .rtorrent.rc. For example (paths are specified relative to user’s home directory):
ip_filter=ipfilter/level1,ipfilter/level2

It is probably a good idea to reload ipfilter files once in a while, so you can also include “reload_ip_filter” directive on schedule to refresh the filter. The same files named in “ip_filter” will be reloaded.

schedule = filter,18:30:00,24:00:00,reload_ip_filter=

To support the feature you may want to setup cron job to reload and unzip files from bluetack. Examples here and here.

GeoIP

In the Peers view of each torrent a new column – CC, country code – is added, which shows peer’s country code. When examining each peer, you will see more data from the GeoIP database (if you have the relevant files installed): AS num and city. At the very least, you need to have geoip-database installed. You can get free “lite” versions of city/AS-databases, rename them (removing “lite”) and put into an appropriate location to make rtorrent+geoip use them as well. Geop-isp data support isn’t integrated into this rtorrent package.

Colours

This patch somewhat changes the way rtorrent shows torrents. Seeding torrents have bold titles, there are no half-displayed torrents at the top/bottom when scrolling, and you can configure colours for active/done torrents (ticket libtorrent.rakshasa.no/ticket/1382, which seems to be gone now). Source code and testing hint at these configurable colours: done_fg_color, done_bg_color, active_fg_color, active_bg_color, and at these possible values: 1 (red), 2 (green), 3 (yellow).

Geek’s cellar
A related (though not used in any way for the preparation of the package) resource is rtorrent mods page.

Relatively schematically, applying patches and building the package was performed in these steps (starting within some newly-created directory):

1. sudo aptitude install cdbs devscripts [and whatever else you find you're missing]
2. apt-get source rtorrent
3. dpkg-source -x rtorrent_0.8.6-1.dsc
4. cd rtorrent-0.8.6
5. cdbs-edit-patch 01-ipfilter.patch
6. patch -p1 < /path/to/patch/239
7. exit 0
8. cdbs-edit-patch 02-geoip.patch
9. patch -p1 < /path/to/patch/2064-after-ipfilter
10. exit 0
11. cdbs-edit-patch 03-eyecandy.patch
12. patch -p1 < /path/to/patch/2171-mod-with-canvas
13. exit 0 [you could do all 3 patches together, but I prefer cleaner and manageable approaches]
14. [some weird operation to be explained below]
15. dpkg-buildpackage

Weird operation: I didn’t know how to make dpkg-buildpackage run autorun.sh (which is required for one of the patches I’ve used) before configure (could someone enlighten me how to do that?), so I started a new patch with cdbs-edit-patch, then ran one by one `autoreconf -if`, `./autorun.sh`, `./configure`, exited with zero status and was done with that problem. The resulting “patch” was 2MB large

• if you don’t have the python-stdeb package: sudo aptitude install python-stdeb
• pypi-install pycassa

That’s it.

Refer to stdeb readme for more information. You will need that if there are dependencies – which might not be resolved automatically by stdeb.

Before stdeb, it wasn’t exactly trivial to make a .deb from python module.

That’s my first encounter of Shapado, so it was interesting to read Shapado authors’ justification and a related question on meta.SO.

Debian has a number of packages enabling remote (over the network) backup: backupPC, backupninja, backup-manager, dirvish, duplicity, luckybackup, rdiff-backup, and some others.

In my setup I wanted backups to be initiated by the external backup server (as opposed to internal, production server initiated backup). Also, I was going to backup linux server to another linux box. Thus, it would be preferable to have support for rsync on both boxes and hardlinks on the backup box. I needed backup to be run regularly, I didn’t need any kind of GUI, and I would prefer to keep a bunch of older backups around (in case the most recent backup is of a compromised system, for example). Also, having an incremental backup of SQL databases was yet another requirement.

A few tools more or less matched my requirements: backupPC, duplicity, rdiff-backup, and dirvish.

backupPC, judging from its description and a few tutorial articles, is a cool piece of software. I think I might use it next time. But this time I really had no need for over-SMB backups (intended for Windows boxes), and no need for a nice administrative web-interface to backupPC. If not these extra features (which I don’t currently need) I’d go for backupPC. Here’s a guide for Ubuntu.

Duplicity claims to still be in beta, thus I haven’t given it a try. rdiff-backup seemed a bit too simple (and with no ready templates to get working regular backups faster), so I skipped that one as well.

Dirvish really caught my attention, partially thanks to a good and relatively short quick-start tutorial. That tutorial also mentions automysqlbackup – a nice, almost no-configuration tool to take regular dumps of MySQL databases. In combination, these two now regularly backup up my whole server to another linux box.

I would definitely recommend using dirvish+automysqlbackup for remote/off-site backup needs. However, other solutions might be as good or better for your use case.

1. Back up important data from partition just in case
2. Find out size of ‘$Bad’ attribute in $Badclus using ntfsinfo -i 8 partition (partition is for example /dev/sda1). It will be the “Allocated size” value in the “Dumping attribute $DATA (0x80)” (there will be two 0x80 attributes. Only one has an “Allocated size” line). Let us write down (remember) this size as ntfs_size.
3. Use ntfstruncate partition 8 0x80 ‘$Bad’ 0 to set $Bad’s attribute length to zero.
4. Use ntfstruncate partition 8 0x80 ‘$Bad’ ntfs_size to set $Bad’s attribute length back to proper value ntfs_size which was recorded in step 2.
5. Boot into Windows and run chkdsk -f diskname. It will find errors and should fix them.

However, Debian’s ntfsprogs package does not have the ntfstruncate binary.

Here’s how you can easily build one yourself (you may need a few extra packages with build tools for that):

1. apt-get source ntfsprogs
2. cd linux-ntfs-2.0.0
3. ./configure && make extra
4. cd ntfsprogs && ./ntfstruncate

You can also install ntfstruncate system-wide, but I have no idea if that will interfere with the already-installed ntfsprogs package, so I just used this binary directly from linux-ntfs-2.0.0/ntfsprogs.

Bad clusters happen not only when cloning an NTFS partition between 2 physical hard disk drives. Once, after a brief blackout (with a turned on PC, and not on a UPS), the SMART status of my new HDD started showing 1 ‘Pending sector’. Correspondingly, all the SMART tests (performed with a smartctl utility) reported read errors. To help my HDD examine and fix the problem itself, I performed a write operation to that single unreadable sector, which restored it (and reset the ‘Pending sector’ counter back to zero). It was possible, because not the physical sector itself, but the hidden CRC field had errors after power failure; modern hard drives can detect and fix this kind of errors upon write operations (read operations only mark the sector as ‘pending’, because data read is not consistent with the stored CRC).

However, NTFS partition still had 4096 bytes in ‘bad clusters’. A little research revealed the presence of internal NTFS bad clusters counter, which can be reset following the procedures above.

The project does provide a 64-bit package for Ubuntu 8.10 “Intrepid Ibex”, but it has a dependency on python-zope-interface – which in Debian testing is a virtual package, currently provided by python-zope.interface. That single dependency made my aptitude complain all the time, so I edited the .deb-file, renaming depends/python-zope-interface to depends/python-zope.interface (following nice and simple instructions). After that – no more aptitude complaints, and Chandler works fine.

Download the modified Chandler 1.0.3 Debian testing amd64 package (md5: 239071715977bea2eb75f3bb54491a02).

Error in openPDF(vif) :
getOption(‘pdfviewer’) is ”; please use ‘options(pdfviewer=…)’

and you are tired of running this command every time:

> options(pdfviewer=”okular”)

then you should check if your system-wide Renviron file has proper PDF viewer set:

grep -i pdf /etc/R/Renviron

R_PDFLATEXCMD=${R_PDFLATEXCMD-${PDFLATEX-’/usr/bin/pdflatex’}}
R_RD4PDF=${R_RD4PDF-’times,hyper’}
## Default PDF viewer
R_PDFVIEWER=${R_PDFVIEWER-”}

It wasn’t in my case.

To set one, either edit the system-wide Renviron (e.g. by editing the R_PDFVIEWER=${R_PDFVIEWER-”} line to look like R_PDFVIEWER=${R_PDFVIEWER-’/usr/bin/xdg-open’} – this will use MIME types to open your preferred PDF viewer), or one of the per-user/per-directory Renviron files to fix this minor annoyance (format is the same, e.g. R_PDFVIEWER=okular ).

At least two convenient utilities are available in Debian Lenny for non-X Samba browsing.

smbtree (part of smbclient package) will list all visible workgroups, their servers, and share names of those servers – including “hidden” shares like C$, IPC$, ADMIN$, print$. Very handy and greppable!

samba-commander (smbc package) is a ncurses samba browser with “find file” functionality.

First of all, sudo aptitude install mesa-utils. Then run in a Terminal/Konsole glxgears, and wait for ~15 seconds; if your FPS is ~400 or less, then you do have sluggish video performance (usually manifesting itself as slow scrolling in Firefox/Iceweasel, slow window switching/minimziing/maximizing etc).

After reading through several forums and bug reports and blog posts, I’ve ended with the following modifications to my /etc/X11/xorg.conf:

1. backup your current xorg.conf: sudo cp /etc/X11/xorg.conf /etc/X11/xorg.conf.before-exa
2. open your /etc/X11/xorg.conf with a favourite editor; I’ve added these lines to section “module”:
   

   Load “dbe”
   Load “xtrap”
   Load “record”
   Load “GLcore”

   These are not really performance-related and could be skipped; I’m giving them here, because they were suggested as default entries by dexconf or some other xorg.conf-generator utility. Again – they are not supposed to improve performance, so you can skip these.

3. my “Device” section is now:
   

   Section “Device”
   Identifier “Intel Corporation 82915G/GV/910GL Integrated Graphics Controller”
   BoardName “82915G/GV/910GL Integrated Graphics Controller”
   Vendorname “Intel Corporation”
   #Driver “i810″
   Driver “intel”
   BusID “PCI:0:2:0″
   Option “DRI” “true”
   Option “AccelMethod” “exa”
   Option “MigrationHeuristic” “greedy”
   Option “ExaNoComposite” “false”
   EndSection

   Driver i810 (xserver-xorg-video-i810 package) was about 10-20 FPS slower than Driver intel (xserver-xorg-video-intel package) for me. The most important lines here are Option AccelMethod exa (which enables EXA acceleration instead of the older XAA; if you run 2.6.28 or later kernel, you can try UXA instead of EXA) and Option MigrationHeuristic greedy. Option ExaNoComposite false doesn’t make a difference for me, and Option DRI true might be redundant (didn’t bother trying to remove it).

4. at the end of xorg.conf I have two more sections:
   

   Section “DRI”
   Mode 0666
   EndSection

   Section “Extensions”
   Option “Composite” “enable”
   Option “MIT-SHM” “Yes”
   EndSection

5. sudo nano /etc/environment, and add one line: INTEL_BATCH=1 (if you have graphics stability issues or screen corruption – try removing this change first)
6. you can test if your xorg.conf is syntactically correct by running X -config /etc/X11/xorg.conf (provided that you edited this exact file)

This is it. Before these modifications, my FPS in glxgears was around 430; after these modifications it is ~915.

As time permits, I might try the XAA configuration, suggested at foolcontrol.

Update: with compiz enabled, XAA configuration recommended at foolcontrol is definitely faster in 2D.

Improvements, suggestions, corrections are welcome.

(…)
it is a very useful tool for performing installations if you’ve got something like a LiveCD which supports your hardware.
(…)

Installing new Debian systems with debootstrap, and also as a debootstrap PDF (with comments).

Two most common reasons (and fixes) for the MySQL server has gone away (error 2006) are:

1. Server timed out and closed the connection. How to fix: check that wait_timeout variable in your mysqld’s my.cnf configuration file is large enough. On Debian: sudo nano /etc/mysql/my.cnf, set wait_timeout = 600 seconds (you can tweak/decrease this value when error 2006 is gone), then sudo /etc/init.d/mysql restart. I didn’t check, but the maximal value for wait_timeout might be around 28800 seconds (8 hours).
2. Server dropped an incorrect or too large packet. If mysqld gets a packet that is too large or incorrect, it assumes that something has gone wrong with the client and closes the connection. You can increase the maximal packet size limit by increasing the value of max_allowed_packet in my.cnf file. On Debian: sudo nano /etc/mysql/my.cnf, set max_allowed_packet = 64M (you can tweak/decrease this value when error 2006 is gone), then sudo /etc/init.d/mysql restart.

If you get MySQL server has gone away (error 2006) when using MySQL ODBC driver – give this hint a try.

Though I prefer ‘testing’ (which is currently codenamed Lenny) over ‘stable’ (aka Etch).

For the case of ATI (and ATI Mobility) Radeon series, the easiest procedure would be (doing all as root, or prepending sudo to all commands):

1. aptitude update, to ensure you’ve got the list of latest packages
2. aptitude install fglrx-driver fglrx-control fglrx-kernel-src, for the actual driver; I also installed fglrx-atieventsd and fglrx-glx (these are driver-recommended packages)
3. aptitude install module-assistant, required for building the kernel module
4. module-assistant prepare, to verify that you have everything needed for the module build procedure
5. module-assistant update
6. module-assistant auto-install fglrx, to build and install the fglrx kernel module
7. depmod -a
8. modprobe fglrx, to load the fglrx kernel module
9. aticonfig ––initial, to configure ATI’s device section in /etc/X11/xorg.conf (for more options, see aticonfig ––help)
10. reboot

That should make your ATI hardware work as expected, and make fgl_glxgears show gears again.

If everything went fine, then your movies will play smoother ;), and the output of fglrxinfo will look similar to

display: :0.0 screen: 0
OpenGL vendor string: ATI Technologies Inc.
OpenGL renderer string: ATI MOBILITY RADEON X600
OpenGL version string: 2.1.7769 Release

This procedure should be also possible for the latest ATI-provided installer-generated Debian packages (as described elsewhere), but I decided to stick to the repository version – this requires no extra tweaking.

For much more information, see the ATI Debian Installation Guide.

This post is based on the iwanabeguru’s response at Linux Forums.