1. touch -d “Jun 01 00:00 2011″ /tmp/.date1
2. enter into your BIG dir
3. press C-x ! (External panelize)
4. add new command like a “find . -type f \( -newer /tmp/.date1 \) -print”

I’ve used a slightly different approach, specifying desired date right in the command line of External Panelize:

1. enter your directory with many files
2. press C-x ! (External Panelize)
3. add a command like find . -type f -newermt "2017-02-01 23:55:00" -print (man find for more details)

In both cases, the created panel will only have files matching your search condition.

This will duplicate both metadata and btrfs system data.
Verify with: sudo btrfs fi df /toplevel:

Data, single: total=10.00GiB, used=3.88GiB
System, DUP: total=64.00MiB, used=4.00KiB
Metadata, DUP: total=512.00MiB, used=286.18MiB
GlobalReserve, single: total=96.00MiB, used=0.00B

Explanation: on SSDs, mkfs.btrfs creates metadata in single mode (because of widely spread SSD deduplication algorithms negating duplicate entries). However, second copy of metadata increases recovery chances, especially so if your SSD does not deduplicate writes. Hence the desire to add metadata/systemdata duplication after the filesystem is created.

My current email server was configured eons ago, it works well,
but I have no desire to painfully transfer all the configuration…
Better install something new, shiny and exciting, right?

I had 3 #self-hosted, #mail-server bookmarks:

• Mail-in-a-box
• iRedMail
• Modoboa
• Sovereign

(Sovereign, the 4th one, was addded after reading more about Mail-in-a-box.)

Here are my notes on what seemed important about these 4.

• iRedMail
• has free and paid web-UIs
• no DNSSEC, DMARC, HSTS
• amavisd with clamav
• has useful manual parts
• containerized
• not attractive
• Modoboa
• less sophisticated than Sovereign or Mail-in-a-box
• web-UI, also for amavisd filters
• overall: focuses on better UI
• has useful manual parts
• recent (experimental?) LetsEncrypt support
• has (some) unit tests
• containerized
• not that attractive
• Sovereign
• has more than I need, but components can be deactivated
• has EncFS support (useful, but questionable because of reboots…)
• no dedicated web-interface, configs are text
• has proper testing against a vagrant virtual machine
• can be dockerized using github.com/kisamoto/dancible
• attractive as “the next solution”, or to borrow EncFS support
• Mail-in-a-box
• the most sophisticated email server (except for EncFS which is not used here)
• simple but useful web-UI
• no amavisd, clamav, UI for filters
• has good relaying manual
• more or less requires a separate machine (overwrites configs?)
• has no well-established testing, not even for development; this is being worked on as of New Year 2017
• problems with owncloud (which I don’t really need)
• hub.docker.com/r/mtrnord/mailinabox/ , github.com/mail-in-a-box/mailinabox/blob/docker/containers/docker/run
• postscreen is not yet configured, it is not obvious if it were beneficial
• the most attractive; might be reasonable to fork and modify (e.g. drop owncloud?)

MIAB appeared really attractive,
but then – do I really want to dedicate one of the VPS to the mail server only?
Not in my case – too low emails volume/traffic.

So running it in an LXC (or some other) container would make sense.
And this is actually possible, some of the users over at MIAB’s discussion forum
have been running MIAB inside docker container for over a year now with no issues.
(An extra upside is that web-UI can be left unexposed, preventing external access to it.)
A possible long-term downside is, of course, lack of tests – Sovereign looks much better in this regard.

Sovereign looks very good overall. In fact, MIAB feels like
“Sovereign’s email component + webui for it” (MIAB was inspired by Sovereign).

One extra MIAB-specific feature is DNSSEC support.
MIAB takes on the role of your nameserver, and thus is able to setup (and refresh, when necessary)
all the DKIM/DNSSEC/etc-relevant DNS records for you.

As soon as I’ve started adding “containerization” to the mix, dozens of other projects entered my field of view:

• github.com/indiehosters/email, inspired by MIAB, looks ok; lacks webmail, fail2ban, SPF, DANE, DNSSEC, but uses vimbadmin instead of a custom-coded MIAB UI
• github.com/tomav/docker-mailserver looks great! No UI, no SQL backend, only 2 text files (accounts and aliases) for all configuration – yay!
• github.com/lava/dockermail, much less active/polished, not really interesting
• github.com/frankh/docker-compose-mailbox adds roundcube and vimbadmin containers; uses SQL; not sure why it has only 10 stars on github…
• github.com/adaline/dockermail – looks ok, less active and seems simpler than docker-mailserver
• poste.io : has free (downloadable) and 2 paid versions; packed with many features and containerized; there is no Dockerfile, but of course you can examine what’s inside the public image anyway; actually, looks good – not sure how posteio-specific the data directory structure is, though… still something to try
• mailgun.com – SMTP service with a more than sufficient free quota for a few low-traffic websites; can be coupled with some forwarding service to avoid any need in an email server; but not this time, I want a mail-server
• yunohost.org : I’m not entirely sure why this is here, maybe it does have email support built-in? ok, yes it does – this is a debian-based “home-server” software, which also includes LDAP and SSO and XMPP and DNS and nginx. Hmm, not bad. I wonder how well it works out of the box.
• kolab.org : groupware; looks interesting as well, but I have no group (yet) to have a use for a full groupware solution
• not reviewed: mailcow.email, mailcow.email/dockerized, github.com/andryyy/mailcow

Finally, one can build an own LXC container, either by following this ArsTechnica series,
or after examining the install scripts of MIAB or Sovereign.
Then automate all of this, keep it well-maintained – and there you have it, one more mail-server solution!

To re-cap:

• MIAB looks very good – feature-rich, easy to install, and just works – you should try it!
• docker-mailserver looks great – I should try it!
• poste.io, yunohost.org and kolab.org are also some interesting solutions to try, along with Sovereign

Not much of a summary, but this is definitely an accurate reflection of reality.

So I have decided to add a swap partition.
It worked amazingly (and very easily), there was even no need to reboot – at all.
I still did restart, just to make sure the system is bootable – and all was perfectly fine.

My initial setup is very simple: a single /dev/sda1 partition on the /dev/sda disk, fully used by btrfs.
Different important paths/mountpoints are btrfs subvolumes, using flat hierarchy.
For this example, let us assume that /dev/sda (and /dev/sda1) is 25GB large, and that I want to add a 2GB swap /dev/sda2 after /dev/sda1.

Brief explanation before we start:

1. shrink btrfs filesystem by more than 2GB;
2. shrink btrfs partition by 2GB;
3. create new 2GB partition for the swap;
4. resize btrfs filesystem to full size of its new-size partition;
5. initialize swap and turn it on.

Here are the very easy steps! Just make sure you do not make mistakes anywhere

1. If your btrfs volume with ID 5 (top level) is a separate mountpoint: mount it now, e.g. sudo mount /toplevel.
2. Take note of your current partition label and UUID: sudo blkid.
3. Resize btrfs filesystem down (shrink) with a good margin; for example, if I want to add a 2 GB swap, then I can sudo btrfs fi resize -3g /toplevel – here, I’m shrinking btrfs filesystem by about a gigabyte more than necessary. The process is very quick if you have free space, so you can even use a larger margin – say, sudo btrfs fi resize -5g /toplevel.
4. sudo parted, then print to make sure what is the number of your btrfs partition, then resizepart 1 (where 1 is the partition number), and answer a few questions: yes, new_size_here (in our example: 23.0GB), yes. You can also create a swap partition from parted, then quit parted with q and Enter.
5. sudo partprobe to let the OS know that partitions have changed.
6. I have used cfdisk to create a 2GB swap partition: it has a very simple ncurses UI, and is very intuitive. After creating swap partition, do run sudo partprobe again.
7. Resize btrfs filesystem back up to take all of the partition: sudo btrfs fi resize max /toplevel.
8. Simply to be sure, run a scrub: sudo btrfs scrub start -B -r /toplevel.
9. Initialize swap; you can specify uuid and/or label which you may already have in your fstab: mkswap --label=swap --uuid=your1234-your-uuid-1234-youruuid1234 /dev/sda2.
10. sudo blkid to make sure your /dev/sda1 UUID stayed the same (or to get swap uuid/label if you haven’t specified any).
11. Optionally, add the swap line to your /etc/fstab. Then turn on swap with swapon -a.

That’s it! Amazing, isn’t it? On-the-fly filesystem and partition resizing!

I wanted to use it with compressed (gzipped/bzipped2) FASTQ input files.
As I do not need to store decompressed input files, the most efficient solution is to use pipes.
This works well for a single file, but not for 2 files (paired-end reads).

For 2 files, named pipes (also known as FIFOs) can be used.
You can create a named pipe in Linux with the help of mkfifo command, for example mkfifo R1_decompressed.fastq.
To use it, start decompressing something into it (either in a different terminal, or in background), for example zcat R1.fastq.gz > R1_decompressed.fastq &;
we can call this a writing/generating process, because it writes into a pipe.
(If you are writing software to use named pipes, any processes writing into them should be started in a new thread, as they will block until all the data is consumed.)
Now if you give the R1_decompressed.fastq as a file argument to some other program, it will see decompressed content (e.g. wc -l R1_decompressed.fastq will tell you the number of lines in the decompressed file); we can call program reading from the named pipe a reading/consuming process.
As soon as a consuming process had consumed (read) all of the data, the writing/generating process will finally exit.

This, however, does not work with prinseq-lite.pl (version 0.20.4 or earlier), with a broken pipe error.

Named pipes are very similar to usual files, with two major differences:

• named pipes are not seekable: you cannot move file pointer (at least not backwards, not sure about skipping forward);
• you cannot arbitrarily close/re-open a named pipe from the consuming end: closing a pipe on the consuming end also closes it for the writing/generating process.

The reason why prinseq-lite.pl does not work with named pipes is that it performs file format checking first – by opening the file, reading the first 3 lines, and closing it.
Closing a named pipe causes broken pipe for the writing process, and when prinseq-lite.pl attempts to open the pipe again – it succeeds, but there is no data there anymore, so it just sits and waits for data

I’m ok with a quick and dirty solution, so here it is: prinseq-lite.pl patch to enable mkfifo named pipes as input files (also local prinseq-lite.pl.patch).
WARNING: this patch simply disables file format checking!

I must admit the 16.04 demo does look very pleasant to work with.

However, bringing in ZFS into Linux reminded me of a fairly recent comparison of ZFS and btrfs that I had to do when building my home NAS.
At that time, few months ago, I’ve arrived (among others) at the following conclusions:

• ZFS on FreeBSD is reliable, though a memory hog;
• on Debian, OpenVault seems to be a good NAS web-management interface;
• on FreeBSD, FreeNAS is good (there is also Nas4Free fork of an older version, but I haven’t looked into it deep enough);
• running ZFS on linux (even as a kernel module) is the least efficient solution, at least partially because kernel’s file caching and ZFS’s ARC cache are two separate entities;
• although btrfs offers features very similar to ZFS, as of few months ago OpenVault did not offer btrfs volumes support from the web-interface.

In the end, I’ve decided to go with FreeNAS, and it seems to work well so far.

But had anything changed in the btrfs vs ZFS on Linux field?

Luckily, in the comments section of the original post Brian Mullan linked to a recent Bachelor degree project titled A performance comparison of ZFS and btrfs on Linux (local copy). Here’s an excerpt from the abstract:

The main conclusions that can be drawn from the analysis of the gathered data is that
Btrfs has improved greatly in recent years and is today showing great throughput whereas
ZFS on Linux is performing considerably worse than Btrfs.

The document is definitely worth looking at – actual results start on page 22, and there are figures for easy comprehension
XFS and ext4 are also added to the comparison, so if you are thinking about changing your FS – the document may help you, too.
The only area where ZFS excelled was database-like load; btrfs and XFS were the leaders in the majority of other tests.
(In fact, somewhat lower btrfs performance for database-like loads seems to be a well-known thing, and can be cured with the nodatacow mount option – see e.g. this nice rant about PostgreSQL on btrfs.)

I do believe that ZFS is still more stable than btrfs, but I’m also still not convinced that ZFS and Linux are a good combination.

I’ve purchased a fairly cheap VPS with an easy, one-click upgrade option for after I’m done configuring it.
It comes with a wide selection of OSes to pre-install; I’ve chosen Debian Jessie, version 8.3 as of this writing.

I wanted to use btrfs from the beginning, so could have installed Debian myself, but… VPS provider does some initial configuration (like their Debian mirror and some other things), so I’ve felt that converting to btrfs after the fact would be easier. Now that I’ve done this – I guess it was fairly easy, although preparation did take some time.

Below, I’m providing step-by-step instructions on how to convert your root filesystem from (most likely) ext4 to btrfs.

I’ll be using my provider’s rescue boot mode: this is a live Debian system which is network-booted on my own VPS, and thus has access to the SSD/HDD of my VPS. Hopefully, your provider has a similar feature.

Preparing for this conversion, I had mostly used two sources:

• an older howtoforge conversion tutorial, and
• Sysadmin Guide

Before we begin with actual conversion, here is a

List of hints and useful practices/commands for a future btrfs user

• Changing the default subvolume with btrfs subvolume set-default will make the top level of the filesystem inaccessible, except by use of the subvol=/ or subvolid=5 mount options.
• If top level is no longer your default subvolume, it is useful to have an fstab entry for the top level (note the noauto option!):
  LABEL=toplevel /root/btrfs-top-lvl btrfs subvol=/,defaults,noauto,noatime 0 0
• scurbbing and repairing btrfs; most of the hints below originate from Marc’s 2014 btrfs presentation
• scrub script: http://marc.merlins.org/linux/scripts/btrfs-scrub
• mount -o compress=lzo is fast and best for SSDs; mount -o compress=zlib compresses better but is slower
• you can turn off COW (copy-on-write) for specific files and directories with chattr ­-C /path (new files will inherit this)
• for a highly fragmented VM, btrfs filesystem defragment vbox.vdi could take hours; cp -reflink=never vbox.vdi vbox.vdi.new; rm vbox.vdi is much faster
• cp -reflink=always copies within and between subvolumes without duplicating data
• run scrub nightly or weekly on all btrfs filesystems
• noatime is the best fstab mount option for btrfs IF USING SNAPSHOTS, because relatime still causes at least 1 write every 24h
• if you have errors=remount-ro in your fstab for extX-filesystem, you should remove it after converting to btrfs as it does not understand the option.

We are almost ready to start! Just one last thing: partitioning!

On my previous server, I did have a certain partition scheme, with separate /boot, /home, /var, /usr, and everything else under / partition.
It worked very well for me, never ran into problems (although this is more like a Debian’s feature, not that of my partition scheme ).
However… With the current VPS, it started initially with just everything under /, in a single partition.
I could have re-partitioned everything from the rescue system while converting to btrfs, but what for? What would be the benefits?

Let’s have a look and evaluate

Advantages/disadvantages of keeping specific partitions separate.

• /boot comes the first. Normally I format it with ext2, and set aside just 100-200 megabytes for this small partition.
  This improves boot-reliability of the system. With btrfs, it was also the case that grub could not use /boot on a btrfs volume.
  However,

  Newer GRUBs can handle a /boot partition which is btrfs, so you need not have a separate /boot partition formatted as ext3/4.

  So this time I’m going without a separate /boot.
  I will not even create a separate subvolume for it: for snapshotting purposes, /boot fits perfectly under the / snapshots.

• Next comes /home.
  The reason to separate it is to safeguard user data when upgrading/reinstalling the OS.
  This, however, can be achieved with btrfs subvolumes – no need for partitions!
  I’m going with a “subvolume for home” here, then.
• Next comes /var, or more precisely /var/log.
  One of the reasons to keep /var/log separate is to prevent filesystem overfill if logs suddenly start growing too fast.
  This, however, can be achieved with subvolume quotas. No need for a partition!
• There is also /var/lib/ (and also /var/lib/lxc/…), which hold fairly big data files, often with random write access patterns.
  There are two reasons to make them subvolumes:
  1. for data-only snapshots
  2. to enable nocow/nodatacow at mount time, to avoid high fragmentation with random writes

  However, this can be done later, when these databases/LXC get installed – in just a few commands, without rebooting.

• Finally, / itself should be a separate subvolume – mostly for snapshot and reinstall/upgrade reasons.

Ok, now we are really going to

Convert the filesystem

1. Enable rescue system, write down access credentials. You may need to restart your VPS to boot into rescue system, but check with your VPS provider first.
2. Connect to and log in to the rescue system.
3. Install btrfs-tools in the rescue system, if not there yet. We’ll need btrfs-convert, which should be included in btrfs-tools.
4. I’m assuming that your SSD/HDD is device /dev/sda, and that the only (root) partition there is /dev/sda1; adjust the next steps if your setup is different. Only proceed with commands if you understand what they are doing!
5. It never hurts to fsck /dev/sda1 before doing filesystem conversion You should see something like /dev/sda1: clean, 27282/1564672 files, 351853/6249472 blocks.
6. btrfs-convert -l toplevel /dev/sda1; note that I’m adding a “toplevel” LABEL to the top level of btrfs, for easier mounting it later. The output should look like this:
   

   creating btrfs metadata.
   creating ext2fs image file.
   cleaning up system chunk.
   conversion complete.

   At this point the system is already btrfs!

7. If you forgot to set btrfs label, or decide to do it later: list the filesystems with btrfs filesystem show, or just look for the label with btrfs filesystem label /dev/sda1 (which will produce an empty line if there is no label), then simply btrfs filesystem label /dev/sda1 toplevel; it will produce no output on success.
8. Let’s change into the new system, and start configuring it:
   mount /dev/sda1 /mnt
for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done
chroot /mnt
9. ls now should show your root filesystem files and directories, plus the ext2_saved “directory” (which is a subvolume, actually):
   

   bin dev ext2_saved initrd.img installimage.debug lib64 media opt root sbin sys usr vmlinuz
   boot etc home installimage.conf lib lost+found mnt proc run srv tmp var

10. Run blkid /dev/sda1 to find out the UUID of the new btrfs top-level volume. You can also see it with btrfs filesystem show:
    

    /dev/sda1: LABEL=”toplevel” UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ UUID_SUB=”ffad5668-2ac7-4ea4-83ad-e6ccba7ccf96″ TYPE=”btrfs” PARTUUID=”750b31a7-01″

11. Edit /etc/fstab (with e.g. nano /etc/fstab):
    • change UUID of / to the new one
    • change the filesystem type from ext3/ext4 to btrfs
    • change the options to subvol=root,defaults,noatime,compress=lzo,ssd,discard (yes, subvol=root does not exist yet… also, ssd,discard are for SSDs only, and instead of adding discard here it might be better to setup an fstrim cronjob. Finally, noatime can be left out if you are not going to make snapshots.)
    • change the last number in the line from 1 to 0 (the fsck passes – btrfs does not want boot-time filesystem checking)
12. Some guides mention editing /etc/grub.d/00_header in a specific way, running grub-mkconfig | grep ” ro “ to see if it adds the proper rootflags=subvol=root, then even run update-grub and grub-install /dev/sda here… But all of this seems redundant or plain unnecessary. First of all, the root subvolume simply does not exist yet, so grub-mkconfig will not produce correct rootflags. Secondly, we are not yet done with setting up subvolumes, and we are not going to reboot right now, so running update-grub/grub-install is a bit premature at this stage
13. Now we are going to create some subvolumes. As I want to name the root subvolume simply root, I should first move the root user’s home files somewhere else (otherwise btrfs snapshot complains “ERROR: incorrect snapshot name ‘/’”):
    mv /root /root_orig
btrfs subvolume snapshot / /root
14. The same problem with /home, need to move it as well: mv /home /home_orig
15. I can NOT snapshot /home like I did with /, because it is just a directory, not a volume like /, so:
    btrfs subvolume create /home
    Note: my /home (/home_orig) is still empty, as this is a brand-new installation with only the root user, so double-check steps below, as I haven’t really run them.
    rsync --progress -aHAX /home_orig/* /home
ls -la /home/
ls -la /home_orig/
rm -fr /home_orig/*
ls -la /root/home/
rm -fr /root/home/* # be sure to leave the empty /root/home/ in place: it will be the mount point.
16. nano /etc/fstab to add the /home mount line, using the same UUID as earlier for the / partition, but a different subvol option:
    UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /home btrfs subvol=home,defaults 0 0
    Feel free to add more mount options, similar to those for root subvolume. While we are editing /etc/fstab, it also makes sense to add a non-automount /toplevel entry (UUID=3124c827-c3bd-4a62-843f-1d5a552f1858 /toplevel btrfs subvol=/,defaults,noauto 0 0) and create mountpoint for it: mkdir /root/toplevel
17. Follow the same procedure to create the /var/log subvolume. I’m using flat subvolumes layout here:
    btrfs subvolume create /log
rsync --progress -aHAX /var/log/* /log
ls -la /log/
ls -la /var/log/
rm -fr /var/log/*
ls -la /root/var/log/
rm -fr /root/var/log/*

18. Add /var/log entry to fstab:
    UUID=”3124c827-c3bd-4a62-843f-1d5a552f1858″ /var/log btrfs subvol=log,defaults,ssd,discard,compress=lzo,noatime 0 0
19. /var/lib/ – databases, LXC, nodatacow/nocow – can be created later, not doing it here and now.
20. cp /etc/fstab /root/etc/fstab
21. exit # chroot
22. for fs in proc sys dev/pts dev; do umount /mnt/$fs; done
umount /mnt
23. Let’s enter the new root subvolume!
    mount -o subvol=root /dev/sda1 /mnt
for fs in proc sys dev dev/pts; do mount --bind /$fs /mnt/$fs; done
chroot /mnt
24. Oh, we forgot to move back root_orig into /root! That’s easy: mv root_orig root. If you want your root’s aliases/profile to load, then just exit and chroot /mnt again (I had to do that for the nice pre-configured prompt colours )
25. btrfs subvolume list / to see our new and shiny subvolumes:
    

    ID 256 gen 8 top level 5 path ext2_saved
    ID 260 gen 49 top level 5 path root
    ID 261 gen 35 top level 5 path home
    ID 262 gen 42 top level 5 path log

26. Let’s make root the default subvolume! If suddenly rootflags get lost during grub configuration, then the system should still be able to boot. Subvol ID comes from the output of the command above (260 in my case):
    btrfs subvolume set-default 260 / # no output
    btrfs subvolume get-default / # to verify

    ID 260 gen 49 top level 5 path root

27. Now let’s check grub-mkconfig output: grub-mkconfig | grep " ro ". Now that the subvolume exists, it does show all the desired rootflags=subvol=root.
28. Let’s make sure our system will be bootable:
    update-grub
grub-install /dev/sda
29. Leave the chroot:
    exit
for fs in proc sys dev/pts dev; do umount /mnt/$fs; done
umount /mnt
30. Now let’s mount top-level again, for cleanup. If you remember, we had all the / files there – which are now in the root subvolume:
    mount -o subvol=/ /dev/sda1 /mnt
31. Navigate to /mnt (cd /mnt, yes ) and delete from here everything that is NOT a subvolume.
    The command which supposedly does this is find /mnt -xdev -delete (-xdev is supposed to not descend into mounts of other filesystems), but I haven’t used it – was a bit tired and did not want to screw up the so-far-successful conversion, so I just did rm -rf bin boot dev etc initrd.img lib lib64 lost+found media mnt opt proc run sbin srv sys tmp usr vmlinuz var root_orig – basically, deleted everything except for ext2_saved, home, root, and log – which are all subvolumes.
32. Confirm that only subvolumes are left now:
    ls -la:

    drwxr-xr-x 1 root root 10 Feb 13 21:30 ext2_saved
    drwxr-xr-x 1 root root 0 Feb 13 22:04 home
    drwxr-xr-x 1 root root 510 Feb 13 22:13 log
    drwxr-xr-x 1 root root 318 Feb 13 22:18 root

    btrfs subvolume list /mnt:

    ID 256 gen 8 top level 5 path ext2_saved
    ID 260 gen 52 top level 5 path root
    ID 261 gen 35 top level 5 path home
    ID 262 gen 42 top level 5 path log

    btrfs subvolume get-default /mnt:

    ID 260 gen 52 top level 5 path root

33. We can leave now: cd / && umount /mnt
34. Reboot into the new system. If this fails: can boot rescue again, chroot and check what went wrong…
35. In my case, conversion seemed to have succeeded, BUT: a) for some reason /toplevel was mounted at boot and full of root-like files which should not have been there anymore; b) I could not delete ext2_saved, at all, neither from / nor from /toplevel. The problem was with the toplevel mount line in /etc/fstab (initially, I forgot to specify the filesystem type). After fixing this and rebooting once more, everything was perfect: /toplevel not mounted, and when mounted it only had subvolumes in it. And of course the system booted without any problems. Now I could delete the previous filesystem snapshot: mount /toplevel; btrfs subvolume delete /toplevel/ext2_saved; umount /toplevel. Note that there was also another, different ext2_saved in the root subvolume: it is NOT a subvolume, which was not carried over when we created the / snapshot – only the empty directory was left in place. (Yes, snapshots do not descend into subvolumes.)

The last 2 commands to issue (after mount /toplevel) are btrfs filesystem defragment -r /toplevel and btrfs balance start /toplevel.
I’m not including them in the instructions above: they are not strictly required, and if your filesystem had some heavy use before conversion, you may want to run these 2 commands more intelligently, in phases (metadata first, then big files, then smaller, etc). Another reason is that it may produce significant system load. On my fresh system both finished very quickly.

…And we are done!

The simplest solution to forward requests from the web-frontend to your API is by using mod_proxy.
If you want to forward any requests to /api/* to your custom back-end server on port 8000, you just add the following lines to your VirtualHost configuration:

ProxyPass “/api” “http://localhost:8000″
ProxyPassReverse “/api” “http://localhost:8000″

I’d suggest not wrapping this fragment with the classical IfModule: as your application will not really work without its API back-end, you actually want Apache to fail as soon as possible if mod_proxy is missing.

That was easy, right? What, it doesn’t work? Can’t be! It’s dead simple! No way you could make a mistake in 2 lines of configuration!!! :mad_rage:

Oh wait… I remember I had this problem before…

Let’s check:

1. Step 1. Did you disable (using a2dissite default or a2dissite 000-default, depending on your Debian-based GNU/Linux) the default website? If your application and the default website are configured in a similar way, then it might be the default site which is serving your app’s pages. The most sure way is to just disable it.
2. Step 2. Did you enable also the proxy_http sub-module? (Using a2enmod proxy_http, followed by service apache2 restart) mod_proxy is only the core module, actual per-protocol work is done by these sub-modules.

Your requests to /api should now be passed on to your API server. If not – please write in the comments what was the problem in your case and how you solved it. HTH!

You can create hard links and symbolic links using C-x l and C-x s keyboard shortcuts. However, these two shortcuts invoke two completely different dialogs.

While for C-x s you get 2 pre-populated fields (path to the existing file, and path to the link – which is pre-populated with your opposite file panel path plus the name of the file under cursor; simply try it to see what I mean), for C-x l you only get 1 empty field: path of the hard link to create for a file under cursor. Symlink’s behaviour would be much more convenient…

Fortunately, a good man called Wiseman1024 created a feature request in the MC’s bug tracker 6 years ago. Not only had he done so, but he had also uploaded a sample mc user menu script (local copy), which works wonderfully! You can select multiple files, then F2 l (lower-case L), and hard-links to your selected files (or a file under cursor) will be created in the opposite file panel. Great, thank you Wiseman1024!

Word of warning: you must know what hard links are and what their limitations are before using this menu script. You also must check and understand the user menu code before adding it to your mc (by F9 C m u, and then pasting the script from the file).

Word of hope: 4 years ago Wiseman’s feature request was assigned to Future Releases version, so a more convenient C-x l will (sooner or later) become the part of mc. Hopefully.

More details about these tools and runlevels (in German).

Yesterday I got somewhat distracted, and thus found lbzip2 -

an independent, multi-threaded implementation of bzip2. It is commonly the fastest SMP (and uniprocessor) bzip2 compressor and decompressor

- as it says in the Debian package description. Is it really “commonly the fastest” one? How does it compare to pbzip2? Should I use lbzip2 instead of pbzip2?

This minor distraction had grown into a full-scale web-search and comparison, adding to the mix plzip (a parallel version of lzip), xz, and lrzip. After reading thousands of characters, all of these were put to a simple test: compressing an about 2 gigabyte FASTQ file with default options.

All the external links and benchmarks, as well as my own mini-benchmark results, are provided below.

The conclusion is that out of all the tested compressors lbzip2 is indeed the best one (for my practical use). It is only slightly better than the trusty pbzip2, which takes the second place. All the other compressors performed so poorly, that they do not get any place in my practical rating…

So, let us first ask internet wisdom/foolishness, if lbzip2 or pbzip2 is faster/better?

• this askubuntu question shows that lbzip2 is compressing faster (1:43) than pbzip2 (2:34)
• this nice benchmark also confirms that lbzip2 is indeed faster at compressing; lbzip2 also appears to use less RAM and a little bit less CPU during compression; during decompression, lbzip2 (reportedly) uses much more RAM. lbzip2 achieved at least as good (and even marginally better) compression ratios as pbzip2.
• lbzip2 github page and also this unrelated page both say that lbzip2 is fully cross-compatible with bzip2
• probably most importantly, lbzip2 github readme says that even bzip2-compressed archives get a decompression speedup (which is definitely not the case with pbzip2)
• lbzip2 also uses 100-900k blocks (900k by default)
• it is not clear if lbzip2 is somewhat less widely tested than pbzip2
• lbzip2‘s author has performed some testing (back in 2009, mind you!), and these were the most important results:
• lbzip2 is better when decompressing from a pipe, no matter the producer, and also when the compressed input coming from a regular file is single stream
• pbzip2 beats lbzip2 when the compressed input is coming from a regular file and is multi-stream (yes, pbzip2 can decompress even lbzip2′s compressed output faster than lbzip2 itself, when it’s coming from a regular file) note: if you check the vbsupport benchmark above, you’ll see that lbzip2 had probably fixed slight lagging behind pbzip2 for regular multi-stream files; this improvement is also confirmed by my testing

So, at least in theory lbzip2 is indeed better than pbzip2, even if only at faster decompression of bzip2-compressed files.

While looking for benchmarks, I’ve found this one (old but good), which highly praises lzop compressor. Apparently, lzop is noticeably faster than even gzip, and compresses only a little bit worse. However, I am not really interested in a faster gzip: I need something with much better compression, but still fast enough for multi-gigabyte files.

Next, I have stumbled upon lzip and plzip (.lz). What are these compressors?

• plzip is a parallel version of lzip, and fully lzip-compatible
• lzip is an LZMA compressor
• reading the documentation leaves an impression that [p]lzip achieves better compression, is slower, and needs much more RAM than competing compressors
• there is a special utility called lziprecover, which helps recover data from damaged lzip archives, by leveraging, on the one hand, CRC checksums of compressed blocks, and, on the other, multiple damaged copies of the archive (if available)
• from the official website:
  

  Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip is about as fast as gzip, compresses most files more than bzip2, and is better than both from a data recovery perspective.

• default “member” (compressed block/chunk) size is 4 petabytes, but can be set to a lower value (minimal 100kb), mimicking bzip2′s chunk size
• supports multiple, independent volumes (loosing one volume will still allow recovering data from all other volumes)
• with multiple cores, plzip creates multi-member files by default (but it is not clear, what is the size of these members? Default is said to be twice the dictionary size, but default for dictionary size is not specified in the manual – so lzip/plzip seem to require compression level -1…-9 specification)
• here lzip compresses a little bit better than xz without the --extreme option
• (l|p)bzip2 should still be faster than either lzip or xz
• I started mentioning xz, because lzip and xz (at least historically) are competing LZMA-based compressors
• a 1 year old opinion makes the following statements about lzip:
• lzip is a marginal archiver with no real benefits since the appearance of xz (note: xz is a successor of lzma-utils)
• xz is more popular, more widely accepted
• xz has a community, while lzip has 1 author
• performance of xz and lzip is comparable
• xz has more features
• but lzip does indeed have a recovery utility that xz doesn’t

That doesn’t really tell us much on how plzip/lzip compare to, say, pbzip2. But before performance, let us pay some more attention to long-term storage features of lzip:

The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability:

• The lzip format provides very safe integrity checking and some data recovery means. The lziprecover program can repair bit-flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked merging of damaged copies of a file.
• The lzip format is as simple as possible (but not simpler). The lzip manual provides the code of a simple decompressor along with a detailed explanation of how it works, so that with the only help of the lzip manual it would be possible for a digital archaeologist to extract the data from a lzip file long after quantum computers eventually render LZMA obsolete.
• Additionally, the lzip reference implementation is copylefted, which guarantees that it will remain free forever.

(I really liked the part about the digital archaeologist! And the copyleft, to a lesser extent.)

Looks really attractive! Because what I am using compressors for is, essentially, longer-term archiving, with unpredictable needs to sometimes decompress some of the files. And, of course, storage media will fail fully or partially, so recovering is important, too. But what is this xz compressor?.. I’ve seen it before, in the contexts with words “overtake the world” or similar…

xz

• much more complex file format than lzip, but maybe it has some benefits for client programs and/or recovery?
• supports integrity checks and multiple compressed blocks
• according to this post from 2012, xz (single-threaded) both compressed and decompressed much faster than lzip… and lrzip (depends on settings, of course)
• lzip is older than xz, and was better than xz predecessor – lzma-utils
• xz is adopted by some linux distributions and software projects for package compression
• xz does not seem to have an equivalent of lziprecover
• tar supports both --lzip and --xz, also with --auto-compress

This hasn’t really added any clarity, has it? Moreover, we now have one more unknown – the lrzip compressor. lrzip is a redundancy compressor with LZO, gzip, bzip2, ZPAQ and LZMA back-ends. It is highly efficient for highly redundant data, even if redundancies are separated with long stretches of other data. (FASTQ files are fairly redundant, though bzip2 seems to utilize that fairly well already; can lrzip do better?)

However, what if a part of the archive is damaged? How much information is lost then? Is it at all possible to recover some of the data from damaged .lrz archives?
Author’s benchmarks showcase how good lrzip is at redundant data compression (although lrzip is multithreaded, so comparison in the benchmark to non-multithreaded algorithm implementations is not quite correct…). Damaged archive recovery concerns would have prevented me from using lrzip anyway, but I was really interested if a “long-range redundancy” compressor can do better than usual, “short-range redundancy” compressors.

My testing setup

• Debian testing 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1 (2015-03-01) x86_64 GNU/Linux
• Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 physical cores with HT enabled: 8 hardware threads)
• 16GiB RAM
• test file name: test.fastq
• test file size: 2 223 860 346 bytes (a little over 2 gigabytes)
• test file was copied once to RAM-mounted /tmp, to exclude any I/O bottleneck effects on compression speeds
• bzip2: 1.0.6
• lbzip2: 2.5
• pbzip2: 1.1.9
• xz: 5.1.0alpha
• plzip: 1.2
• lrzip: 0.616
• command execution time and maximal process RSS memory were measured with /usr/bin/time -f '%C: %e s, %M Kb' compressor arguments (note: this is not bash’s built-in time); please note that memory measurement can be incorrect for multithreaded compressors

Below come testing results. I have not put them into a single table, but I do comment the results in a few places. Entire testing followed this pattern:

• compress test.fastq, deleting the original
• test compressed archive (note: this was done only for some compressors, not all)
• decompress archive back to test.fastq, delete archive
• if 3 previous steps are fast enough: repeat 1-2 more times (but only show the best result below); otherwise continue
• repeat with the next compressor

bzip2: 309 159 275 bytes
bzip2 was used as a baseline, to highlight speed benefits of both lbzip2 and pbzip2.

test.fastq: 7.193:1, 1.112 bits/byte, 86.10% saved, 2223860346 in, 309159275 out.
bzip2 -v test.fastq: 190.63 s, 7608 Kb
bzip2 -v -d test.fastq.bz2: 51.58 s, 4620 Kb

Bzip2 is neither particularly slow, nor particularly fast. It also seems to have modest memory requirements.

pbzip2: 310 462 610 bytes
pbzip2 is the currently used reference. For any other compressor to become a successor of pbzip2, that other compressor must be either a little faster (while compressing as good as pbzip2), or a little better compressor (while being as fast as pbzip2), or both. Note that compressed file size is only a tiny bit larger than with bzip2.

“test.fastq.bz2″: compression ratio is 1:7.163, space savings is 86.04%
pbzip2 -v test.fastq: 46.22 s, 67436 Kb
pbzip2 -dv test.fastq.bz2: 19.80 s, 46672 Kb

Interestingly, pbzip2 --test uses 1 thread only (but also consumes only 6MB RAM), resulting in decompression times similar to those of bzip2. lbzip2 uses all 8 threads also during testing.

lbzip2: 311 040 543 bytes

lbzip2: compressing “test.fastq” to “test.fastq.bz2″
lbzip2: “test.fastq”: compression ratio is 1:7.150, space savings is 86.01%
lbzip2 -v test.fastq: 22.67 s, 49812 Kb

lbzip2: decompressing “test.fastq.bz2″ to “test.fastq”
lbzip2: “test.fastq.bz2″: compression ratio is 1:7.150, space savings is 86.01%
lbzip2 -vd test.fastq.bz2: 18.86 s, 46652 Kb

I repeated pbzip2 and lbzip2 tests several times, and it was always that lbzip2 compressed this same file about twice as fast… Wow! Decompression speed is about the same, compressed file size is marginally larger than with pbzip2. Overall, lbzip2 does look like a new drop-in replacement of bzip2/pbzip2 for me.

xz -0 ––threads=8: 517 967 372 bytes
I would call this one major test disappointment. Default setting, -6, was way too slow (estimated 28 minutes to compress!!!). Even the fastest -0 setting was still too slow! And here’s one of the reasons, straight from the xz man page:

Multithreaded compression and decompression are not implemented yet, so this option has no effect for now. As of writing (2010-09-27), it hasn’t been decided if threads will be used by default on multicore systems once support for threading has been implemented.

Also, I forgot to use the --block-size=900k option, but that seems to be of no concern with such results:

100 % 492.5 MiB / 2,120.8 MiB = 0.232 18 MiB/s 1:59
xz -0 -v test.fastq: 119.25 s, 4780 Kb
xz ––test ––verbose ––threads=8 test.fastq.xz: 36.00 s, 2568 Kb
100 % 492.5 MiB / 2,120.8 MiB = 0.232 58 MiB/s 0:36
xz -d -v test.fastq.xz: 36.54 s, 2500 Kb

xz -0 was both slower and had significantly worse compression when compared to lbzip2 and pbzip2. xz -0 was faster than good old bzip2, but had significantly worse compression… Really, major test disappointment.

plzip: between 407 696 562 and 498 708 539 bytes
One more major test disappointment. (Or am I somehow using these compressors in a wrong way?…) I haven’t found a way to set block/member size (for lzip, that would be the -b option). Default speed setting -6 was also way too slow, but settings -1 to -3 were comparable to pbzip2, so I did all three.

plzip -1: 498 708 539 bytes

test.fastq: 4.459:1, 1.794 bits/byte, 77.57% saved, 2223860346 in, 498708539 out.
plzip -1 ––verbose ––threads=8 test.fastq: 30.27 s, 126360 Kb (this seems to be per-thread memory…)
plzip ––test ––verbose ––threads=8 test.fastq.lz: 6.86 s, 11640 Kb
plzip -d ––verbose ––threads=8 test.fastq.lz: 7.24 s, 11644 Kb

Compression speed and ratio: both worse than lbzip2. But the fastest testing and decompression so far.

plzip -2: 456 301 558 bytes

test.fastq: 4.874:1, 1.641 bits/byte, 79.48% saved, 2223860346 in, 456301558 out.
plzip -2 ––verbose ––threads=8 test.fastq: 38.81 s, 193416 Kb
plzip ––test ––verbose ––threads=8 test.fastq.lz: 6.26 s, 14828 Kb
plzip -d ––verbose ––threads=8 test.fastq.lz: 6.38 s, 14736 Kb

Compression time worse than lbzip2, a little better than pbzip2, but compression ratio worse than any one of these. But even faster testing and decompression.

plzip -3: 407 696 562 bytes

test.fastq: 5.455:1, 1.467 bits/byte, 81.67% saved, 2223860346 in, 407696562 out.
plzip -3 ––verbose ––threads=8 test.fastq: 63.74 s, 245756 Kb
plzip ––test ––verbose ––threads=8 test.fastq.lz: 5.82 s, 18936 Kb
plzip -d ––verbose ––threads=8 test.fastq.lz: 6.10 s, 18944 Kb

Even faster testing and decompression! But compression ratio and speed are still worse than lbzip2 and pbzip2.

And the final contestant, lrzip! All 5 back-ends were tested: LZO, gzip, bzip2, LZMA, ZPAQ.

lrzip has several peculiarities, which hinder its use as a drop-in replacement for, say, bzip2. Most importantly, when a file is compressed, it is not deleted, unless a -D options is specified. Unlike pbzip2 and lbzip2, which use all available CPUs/cores by default, lrzip only uses 2 by default (-p 8 in the results below requests use of 8 cores). Another unusual feature is that during testing a file is uncompressed to a storage medium, and then deleted; almost all the other compressors only verify the decompressed data stream, which is then immediately discarded and never written to storage medium. Related feature is a -c option, which performs file verification after decompression by reading the decompressed file from storage medium and comparing it to the decompressed stream. lrzip also stores MD5 hashes of data, and allows verifying these. lrzip comes with several helper scripts – for example, one which allows tarballing and lrzipping a chosen directory in a single command. Actually, lrzip is more of an archive utility, and not just a compressor.

lrzip -D -p 8: 334 504 383 bytes
In this default (LZMA) mode, lrzip starts with 1 thread, but eventually uses more and more cores (though never all 8, or I haven’t noticed this). Decompressing seems to use more threads, but that also depends on the back-end used (the slower it is – the more threads will be used, e.g. ZPAQ versus LZO).

test.fastq – Compression Ratio: 6.648. Average Compression Speed: 3.113MB/s.
Total time: 00:11:21.85
lrzip -D -p 8 test.fastq: 681.84 s, 3331080 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 124.706MB/s
[OK] – 2223860346 bytes
Total time: 00:00:17.13
lrzip -t -p 8 test.fastq.lrz: 17.21 s, 2567608 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 117.778MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:17.59
lrzip -d -p 8 -D test.fastq.lrz: 17.67 s, 2567664 Kb

In the default LZMA mode, lrzip is significantly slower than even bzip2, and has somewhat worse compression ratio. Yes, this is the 3rd major test disappointment.

gzip back-end: lrzip -g -L 9 -D -p 8: 430 013 769 bytes
Despite specifying -p 8, lrzip mostly operates in 1 thread, and only sometimes in 2 (probably invokes gzip library). Testing is also done with 1 thread only, but is very fast (but slower than plzip). The -L 9 option is supposed to be translated into -9 for gzip; as this normally has nearly no effect, it wasn’t used in the following lrzip tests.

test.fastq – Compression Ratio: 5.172. Average Compression Speed: 0.704MB/s.
Total time: 00:50:11.34
lrzip -p 8 -g -L 9 -D test.fastq: 3011.34 s, 2745520 Kb

100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 163.077MB/s
[OK] – 2223860346 bytes
Total time: 00:00:12.71
lrzip -t -p 8 test.fastq.lrz: 12.79 s, 2577632 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 163.077MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:12.88
lrzip -d -p 8 -D test.fastq.lrz: 12.95 s, 2577728 Kb

And again, compression speed and ratio are worse than for bzip2…

LZO back-end: lrzip -l -D -p 8: 766 520 776 bytes

test.fastq – Compression Ratio: 2.901. Average Compression Speed: 4.690MB/s.
Total time: 00:07:32.89
lrzip -l -D -p 8 test.fastq: 452.88 s, 2714452 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 212.000MB/s
[OK] – 2223860346 bytes
Total time: 00:00:10.58
lrzip -t -p 8 test.fastq.lrz: 10.66 s, 2582516 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 192.727MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:11.32
lrzip -d -p 8 -D test.fastq.lrz: 11.39 s, 2582504 Kb

No comments.

bzip2 back-end: lrzip -b -D -p 8: 353 473 476 bytes

test.fastq – Compression Ratio: 6.291. Average Compression Speed: 4.473MB/s.
Total time: 00:07:53.95
lrzip -b -D -p 8 test.fastq: 473.94 s, 2781104 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 68.387MB/s
[OK] – 2223860346 bytes
Total time: 00:00:30.69
lrzip -t -p 8 test.fastq.lrz: 30.77 s, 2583156 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 66.250MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:31.92
lrzip -d -p 8 -D test.fastq.lrz: 32.00 s, 2583108 Kb

Hadn’t I done all of these simple tests myself, by now I’d think that this test was rigged to show how good pbzip2 and lbzip2 are at compressing FASTQ files

ZPAQ back-end: lrzip -z -D -p 8: 292 380 439 bytes

test.fastq – Compression Ratio: 7.606. Average Compression Speed: 2.804MB/s.% 7:100%
Total time: 00:12:36.51
lrzip -z -D -p 8 test.fastq: 756.51 s, 3585740 Kb

Decompressing…
100% 2120.84 / 2120.84 MB 1:100% 2:100% 3:100% 4:100% 5:100% 6:100% 7:100%
Average DeCompression Speed: 3.970MB/s
[OK] – 2223860346 bytes
Total time: 00:08:54.57
lrzip -t -p 8 test.fastq.lrz: 534.65 s, 2583424 Kb

Decompressing…
100% 2120.84 / 2120.84 MB 1:100% 2:100% 3:100% 4:100% 5:100% 6:100% 7:100%
Average DeCompression Speed: 3.759MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:09:24.27
lrzip -d -p 8 -D test.fastq.lrz: 564.36 s, 2583460 Kb

Finally!!! We have compression better than bzip2! But it is also much slower than bzip2 (and some 12 times slower than pbzip2), so not really an option. Alas. And decompression time is the worst in the test – almost 10 minutes for what plzip does in under 7 seconds! (I do realize that compression ratio is also different – but not that much.) I wonder if slow lrzip speeds have anything to do with test.fastq being effectively in RAM? I do not know if there are any performance penalties to mmaping a file which is already on a RAM-mounted partition.

The test.fastq file that I’ve used was somehow really hard for the tested compressors to tackle as fast and as good as lbzip2 and pbzip2 could…

Questions? Comments? Improvements, including plots of these figures? Comment below.

DEFAULT -d sat -H -f -p -t -W 0,40,45 -n standby -S on -m example@example.com
# Attributes 1, 230, and 231 are very important (-r 1! -r 230! -R 230! -r 231! -R 231!), but likely covered by -t.
/dev/sda -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00) -C 0 -I 189 -I 194
# -a implies -f and -p (through -t)
DEFAULT -d sat -a -I 194 -W 0,40,45 -n standby -o on -S on -m example@example.com
/dev/sdb -s (S/../../6/02|L/../(01|02|03|04|05|06|07)/7/02)
# This drive does not decrement Offline_Uncorrectable (198) after re-allocation,
# so only monitoring for increase, not for non-zero value.
/dev/sdc -s (S/../../6/03|L/../(01|02|03|04|05|06|07)/7/04) -U 198+
# This drive has 40 “normally”.
/dev/sdd -s (S/../../6/04|L/../(01|02|03|04|05|06|07)/7/06) -W 0,42,45

Note: explanations below are intentionally simplified; please consult man smartd.conf for more precise, complete, and up-to-date information.

Ok, so what do these settings mean, and how is this different from default settings?

By default, smartd assumes a DEVICESCAN directive, which auto-detects all HDDs, and enables reasonable default monitoring of SMART attributes.
However, there are several benefits to individually specifying your disks:

• less verbose smartd startup log messages (no messages about auto-detection and missing attributes)
• ability to run scheduled offline, short, and long SMART self-tests
• ability to monitor or exclude attributes individually for each drive (including temperature)

Reasonable default mentioned above is the -a option, equivalent to the following individual options:

-H -f -t -l error -l selftest -C 197 -U 198

This is important to know, because my only SSD has no attribute 197, no self-test log, no error log, and no automatic offline testing.
But I still want to start with quasi-default settings, and that is why the first configuration line includes all the options from -a, except those that my SSD does not support:

DEFAULT -d sat -H -f -p -t -W 0,40,45 -n standby -S on -m example@example.com

Here and in the -a options above,

• -H: monitor overall health status (passed/failed)
• -d sat: HDD type is SATA
• -f: check if any of the Usage attributes (those not marked as Pre-fail) are below the manufacturer-set thresholds
• -p: report changes in Pre-fail attributes (implied by -t below, so can be omitted)
• -t: same as -p (above) with -u (report changes in Usage attributes)
• -W 0,40,45: log a message if drive’s temperature goes above 40 degrees Celsius; log a critical messages if above 45
• -n standby: do not wake-up (spin-up) the HDD if it is in sleep or standby mode (in which platters do not spin)
• -S on: enable attributes auto-saving
• -m example@example.com: address (or several comma-separated addresses) to receive warnings from smartd

The DEFAULT directive is for convenience: options set by this directive apply to all the individual disk configuration lines below, until a different DEFAULT line is encountered.
Here, I had used it to separate all /dev/sda options into 2 logical groups: supported defaults, and drive-specific configuration.
SSD’s configuration is

/dev/sda -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00) -C 0 -I 189 -I 194

Here,

• -C 0: explicitly disable attribute 197 monitoring (which is not present in this SSD)
• -I 189 and -I 194: ignore attributes 189 and 194 (they both show temperature in this SSD)
• -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00): schedule for short and long tests

As for the tests…
I want short self-tests every Saturday night, between 1 AM and 5 AM (shifted by 1H for every disk).
I want long self-tests on the 1st Sunday of every month, between midnight and 8 AM (shifted by 2H for every disk).
This is exactly what is encoded in my configuration for all drives. The easiest way to figure out the format is to read the relevant section of man smartd.conf.

The remaining 3 drives are all HDDs, so I had defined a different common DEFAULT for them.
The only new option that we see is -U 198+, which instructs smartd to only report increases of the Offline_Uncorrectable (198) attribute.
This is necessary because my /dev/sdc does not decrement this attribute after sector re-allocation.

I hope you found this post helpful.

Running a short SMART self-test (sudo smartctl -t short /dev/sdc) produced a read error at sector 1289:

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
1 Short offline Completed: read failure 80% 22339 1289

Looking at the partition table of /dev/sdc, we see that this sector is outside of the only RAID partition on the disk, which starts at sector 2048:

Device Boot Start End Blocks Id System
/dev/sdc1 2048 117209087 58605088 fd Lnx RAID auto

To make sure that sector 1289 is re-allocated, some data needs to be written to it, e.g. with sudo dd if=/dev/zero of=/dev/sdc count=1 seek=1289.
You may also try to read the sector first, then – if successful – write it back to the disk:
i=1289 ; sudo dd if=/dev/sdc of=/tmp/sector count=1 skip=$i && sleep 1 && sudo dd if=/tmp/sector of=/dev/sdc count=1 seek=$i

Another solution (untested!) would be to read/write a bunch of sectors around the problematic one (this is similar to what badblocks -n does):

export i=1280
while [ $i -lt 1300 ]
do echo $i
# read once (count=1) 512 bytes (default ibs/obs values of dd) to a temporary file, skipping first $i ibs-sized blocks (skip=$i);
# if successful, then (wait a bit and) write the same data back to disk, skipping $i obs-sized blocks (seek=$i)
dd if=/dev/sdc of=/tmp/sector count=1 skip=$i && sleep 1 && dd if=/tmp/sector of=/dev/sdc count=1 seek=$i
let i+=1
done


After sector re-allocation both Reallocated_Sector_Ct (5) and Reallocated_Event_Count (196) SMART attributes increased from 0 to 1, while Current_Pending_Sector (197) decreased from 1 to 0. In addition, running badblocks /dev/sdc and diskscan --output diskscan-sdc-out-25-02-2015.json /dev/sdc (both in read-only mode, of course) has not shown any read errors, and another short SMART self-test also finished successfully. So, is the problem solved?

Unfortunately, Offline_Uncorrectable (198) stayed at 1, and I kept getting warning emails. Apparently, my HDD simply does not decrease the Offline_Uncorrectable (198) attribute after sector re-allocation.

In this case the proper solution is to edit /etc/smartd.conf so that it only sends emails if Offline_Uncorrectable (198) attribute increases, and not if it is non-zero. Just add this option to your HDD scan line in smartd.conf: -U 198+.

I am updating a multisite Drupal 6 installation. To the best of my knowledge, the only difference for Drupal 7 is that instead of the site_offline D6 variable the maintenance_mode variable is used in D7.

On Debian stable and later, you can sudo aptitude install drush and then just use it immediately after that.

Note: I recommend su webuser (or sudo -s followed by sudo -s -u webuser) before you run any non-testing drush commands, where webuser is the user which owns your web-exposed files (e.g. Debian’s default is, I think, www-data). I’ve seen a lot of recommendations to run drush as a super-user, but that does not make sense, and may actually cause problems with file ownership.

One last thing before we start: if your drush seems to work fine but hangs when untarring modules – check this solution.

1. Run some innocent command in drush to see if it produces any PHP warnings/errors you may want to fix before running actual update: drush @sites core-status. In my case, all the sites had the CacheRouter module for in-RAM caching with a server daemon back-end, which was not initialized properly when drush bootstrapped Drupal from the command line. In my case, the only working solution was to edit settings.php files of every site to comment out the CacheRouter configuration for the period of update. If you get no warnings/errors, proceed to the next step. Note: I was running drush from the Drupal’s root (directory which has top-level index.php and .htaccess files), but this should also work if you run from sites/ or even sites/sitename.
2. Here would be several more steps – copying your production website(s) to a dev-server (if you do not have one already), performing an update on the dev-server first to see if anything breaks and needs fixes, then migrating updated website(s) from the dev-server to production server. Drush actually has tools to simplify all of these procedures. However, the websites I was updating were not critical, and short downtime was not a problem, so I was updating live websites. Modify these steps as you see fit to make the process more reliable.
3. Backup databases of all your sites. With drush: drush @sites sql-dump --result-file --gzip. This puts backups somewhere into the home directory of your webuser. Backups are named with a human-readable timestamp. Of course, you can also create a manual Backup and Migrate backup, or use phpMyAdmin, or just mysqldump.
4. Backup your site’s files. This step might be unnecessary, as drush seems to backup modules it is upgrading. I would still recommend making a backup, e.g. with tar -acf multidrupal.tar.bz2 html, where html is the directory containing your multisite Drupal’s root index.php.
5. Put the websites into maintenance mode and clear all caches; see the D7-specific note above: drush @sites variable-set site_offline 1 ; drush @sites cache-clear all.
6. The actual update! The easiest way would probably be to drush @sites pm-update, but I haven’t tested that and used a process which I understand better, and which seems more reliable to me (if anything goes wrong). If in your drupal root you have sites/site1 and sites/site2, then run:
   
drush site1 pm-updatecode
drush @sites updatedb
drush site2 pm-updatecode
drush @sites updatedb

   The pm-updatecode command only updates files, and does not run database update. So with these commands I am first updating modules from site1, then running database update on all sites, then update modules of site2, and run database update on all sites again. Running drush @sites updatedb multiple times, even when there are no updates, should be safe. Take note of any warnings/errors reported, you will want to fix them later, for example:

   WARNING: Updating core will discard any modifications made to Drupal core files, most noteworthy among these are .htaccess and robots.txt. If you have made any modifications to these files, please back them up before updating so that you can re-create your modifications in the updated version of the file.

7. Disable maintenance mode. Cleaning the cache seems unnecessary, as updatedb command does that. drush @sites variable-set site_offline 0.
8. Finalize: re-enable anything disabled before the updates, fix warnings/errors you noted during the update.

This worked well for me, and I hope it works well for you.

Symptoms

• running drush @sites pm-update results in normal execution up to after answering ‘y[es]‘; then drush seems to hang indefinitely (haven’t waited beyond about 10 minutes, maybe it does produce an error after a long while);
• running the same command with --debug shows that drush hangs when trying to untar the downloaded module.tar.gz archive; there are no errors/warnings, it just hangs with no CPU usage;
• trying to untar any of the modules downloaded from drupal.org manually is also unsuccessful: tar -xzvf module.tar.gz seems to do nothing, it also hangs with zero CPU usage/time and no warnings/errors;
• interestingly, if I create some test.tar.gz locally, tar does happily extract that;
• finally, running strace tar -xzvf module.tar.gz shows a number of unexpected lines, such as references to NSS and libnss files (I am only showing some of the lines of strace output, including the last line):
  

  open(“/etc/nsswitch.conf”, O_RDONLY) = 4
  read(4, “# /etc/nsswitch.conf\n#\n# Example”…, 4096) = 683
  open(“/lib/x86_64-linux-gnu/libnss_nis.so.2″, O_RDONLY) = 4
  open(“/lib/x86_64-linux-gnu/libnss_files.so.2″, O_RDONLY) = 4
  open(“/etc/passwd”, O_RDONLY|O_CLOEXEC) = 4
  open(“/usr/lib/x86_64-linux-gnu/libnss_mysql.so.2″, O_RDONLY) = 4
  open(“/etc/group”, O_RDONLY|O_CLOEXEC) = 4
  open(“/etc/libnss-mysql.cfg”, O_RDONLY) = -1 EACCES (Permission denied)
  open(“/etc/libnss-mysql-root.cfg”, O_RDONLY) = -1 EACCES (Permission denied)
  futex(0x7fd0816e8c48, FUTEX_WAIT_PRIVATE, 2, NULL

Analysis
strace output provided enough information to understand the issue and generate a workaround. Briefly, we see tar querying users and groups information. On the system where this problem was identified, MySQL is used as a name-service back-end. This is why we see references to mysql libraries in the trace. Apparently, tar is trying to resolve some user/groups information, but for some reason does not get what it is asking in a timely manner, or possibly never gets it and will only fail/proceed when the request times out.

Workaround
Not a solution, but works: tar -xzv --numeric-owner -f module.tar.gz. The --numeric-owner switch asks tar to use numeric file/directory owner information as-is, without trying to resolve the name of the owner. This works. I have not checked strace for the workaround, but I expect to see no MySQL/NSS references in it with the switch.

To actually be able to use drush with this workaround, I had to edit drush.inc somewhere under /usr/share/drush/; look for ‘tar ‘ string, and add --numeric-owner where necessary. Do not forget that -f has to be just in front of the archive filename, otherwise your edits will not work.

Remove packages that take up too much space and are not currently needed (you can always reinstall them after the upgrade). If you have popularity-contest installed, you can use sudo popcon-largest-unused to list the packages you do not use that occupy the most space. You can find the packages that just take up the most disk space with dpigs (available in the debian-goodies package) or with wajig (running wajig size). They can also be found with aptitude. Start aptitude in “visual mode”, select Views → New Flat Package List, press l and enter ~i, then press S and enter ~installsize. This will give you a handy list to work with.

• First comes make itself – the oldest and the most widely used software build tool. Stable and powerful. Still, even people who got used to using make, have some gripes about it. The most detailed list of gripes is probably here.
• SCons is a build tool written in Python. I guess I like that “configuration files are Python scripts” – maybe knowing Python is enough to use SCons, which makes SCons a better choice than make for me. SCons seems to have gained some support (scroll down for comments/discussion). There were some doubts about SCons performance (1, 2, and 3); not sure where SCons is at right now in that regard.
• waf, a Python-based framework for configuring, compiling and installing applications.
• pyDoIt is a Python automation tool. It seems to use Python syntax. It aims at bringing the power of build-tools to execute any kind of task, where a task describes some computation to be done (actions), and contains some extra meta-data. Based on the description alone, I’m quite intrigued! I wonder if anyone had already worked with pyDoIt and can share experiences?…
• Rake – Ruby make – is a simple build program with capabilities similar to those of make. Had seen a lot of positive feedback about this one – mostly regarding simplicity of use. Still [py]DoIt so far looks more attractive to me personally.
• Ruffus is a lightweight python module for running computational pipelines. Sounds like some good competition to [py]DoIt!
• Anduril is an open source component-based workflow framework for scientific data analysis. Sounds promising, though the latest downloadable version is over 400 MBs… It probably already contains a bunch of binaries and maybe even data and complete workflows for data analysis. Probably worth a look, but may turn out a little overweight for simple pipelining.
• snakemake is a scalable bioinformatics workflow engine. I get the feeling that Python is truly dominating the pipelines/workflows world: snakemake, as even the name suggests, is in Python, too. The front-page example is so simple and clear, that snakemake immediately pushes DoIt down from the 1st place! Awesome.
• Paver is a yet-another Python-based software project scripting tool along the lines of Make or Rake, designed to help out with repetitive tasks with the convenience of Python’s syntax. Sounds similar to DoIt. Have no idea how they actually compare to each other.

That is it for now.

What were your experiences with automating repetitive tasks and building simple pipelines?

As VihangD points out in his serverfault answer, the same can be achieved with aptitude, while also excluding dependent, automatically installed packages (which are included by the classical method). To create the list of packages, run aptitude search -F '%p' '~i!~M' > package_list. Here, -F '%p' asks aptitude to only print package names (instead of the default output, which also contains package state and description); search term ‘~i!~M’ asks for all non-automatically installed packages.

To install packages using the created list, run xargs aptitude --schedule-only install < package_list; aptitude install. The first of these two commands instructs aptitude to mark all the packages from the list as scheduled for installation. The second command actually performs the installation.

Hamish Downer suggests an alternative way of getting the initial package_list: using the deborphan utility, deborphan -a --no-show-section > package_list. This command asks deborphan to show a list of packages, which have no dependencies on them. Sounds very similar to what we did with aptitude above, but using deborphan will most likely result in a much shorter list of packages (on my system, deborphan printed 291 package names, aptitude printed 847, and dpkg printed 3650 package names). One more potentially important difference between aptitude- and deborphan-produced package lists is that aptitude only specifies package architecture when it is different from native (e.g. 'googleearth:i386' on a 64-bit system), while deborphan specifies architectures for all the packages (resulting in e.g. 'google-talkplugin:amd64' and 'googleearth-package:all' on a 64-bit system).

My personal choice was RStudio: it is good-looking, intuitive, easy-to-use, while powerful.

Next step would be using some R-equivalent of the excellent ipython’s Mathematica-like Notebook webinterface…