Your software needs tons of RAM, and runs a bit too slow on your super-duper HPC cluster? Read this: Slow memory allocation due to Transparent Huge Pages (THP)

If you ever need to select lots (hundreds, thousands) of files by their modification date, and your directory contains many more files (thousands, tens of thousands), then angel_il has the answer for you:

1. touch -d “Jun 01 00:00 2011″ /tmp/.date1
2. enter into your BIG dir
3. press C-x ! (External panelize)
4. add new command like a “find . -type f \( -newer /tmp/.date1 \) -print”

I’ve used a slightly different approach, specifying desired date right in the command line of External Panelize:

1. enter your directory with many files
2. press C-x ! (External Panelize)
3. add a command like find . -type f -newermt "2017-02-01 23:55:00" -print (man find for more details)

In both cases, the created panel will only have files matching your search condition.

Just one command: sudo btrfs balance start -v -mconvert=dup /toplevel/
where /toplevel/ is your mountpoint of the btrfs root, -v is there for verbosity (not too verbose, don’t worry), and -mconvert=dup literally says act on metadata only, convert data profile to DUP.

This will duplicate both metadata and btrfs system data.
Verify with: sudo btrfs fi df /toplevel:

Data, single: total=10.00GiB, used=3.88GiB
System, DUP: total=64.00MiB, used=4.00KiB
Metadata, DUP: total=512.00MiB, used=286.18MiB
GlobalReserve, single: total=96.00MiB, used=0.00B

Explanation: on SSDs, mkfs.btrfs creates metadata in single mode (because of widely spread SSD deduplication algorithms negating duplicate entries). However, second copy of metadata increases recovery chances, especially so if your SSD does not deduplicate writes. Hence the desire to add metadata/systemdata duplication after the filesystem is created.

Preparing to dismantle my physical server (and move different hosted things to one or more VPS),
I’ve realized that an email server is necessary: to send website-generated emails, and also
receive a few rare contact requests arriving at the websites.

My current email server was configured eons ago, it works well,
but I have no desire to painfully transfer all the configuration…
Better install something new, shiny and exciting, right?

I had 3 #self-hosted, #mail-server bookmarks:

• Mail-in-a-box
• iRedMail
• Modoboa
• Sovereign

(Sovereign, the 4th one, was addded after reading more about Mail-in-a-box.)

Here are my notes on what seemed important about these 4.
Read the rest of this entry »

Recently I had a need to add a swap file to my Debian installation.
However, I am now using btrfs, and – as with any other COW filesystem – it is not possible to simply create a swap file and use it.
There are workarounds (creating a file with a COW attribute removed, and then loop-mounting it), but I just did not like them.

So I have decided to add a swap partition.
It worked amazingly (and very easily), there was even no need to reboot – at all.
I still did restart, just to make sure the system is bootable – and all was perfectly fine.

My initial setup is very simple: a single /dev/sda1 partition on the /dev/sda disk, fully used by btrfs.
Different important paths/mountpoints are btrfs subvolumes, using flat hierarchy.
For this example, let us assume that /dev/sda (and /dev/sda1) is 25GB large, and that I want to add a 2GB swap /dev/sda2 after /dev/sda1.

Brief explanation before we start:

1. shrink btrfs filesystem by more than 2GB;
2. shrink btrfs partition by 2GB;
3. create new 2GB partition for the swap;
4. resize btrfs filesystem to full size of its new-size partition;
5. initialize swap and turn it on.

Here are the very easy steps! Just make sure you do not make mistakes anywhere
Read the rest of this entry »

prinseq-lite.pl is a utility written in Perl for preprocessing NGS reads, also in FASTQ format.
It can read sequences both from files and from stdin (if you only have 1 sequence).

I wanted to use it with compressed (gzipped/bzipped2) FASTQ input files.
As I do not need to store decompressed input files, the most efficient solution is to use pipes.
This works well for a single file, but not for 2 files (paired-end reads).

For 2 files, named pipes (also known as FIFOs) can be used.
You can create a named pipe in Linux with the help of mkfifo command, for example mkfifo R1_decompressed.fastq.
To use it, start decompressing something into it (either in a different terminal, or in background), for example zcat R1.fastq.gz > R1_decompressed.fastq &;
we can call this a writing/generating process, because it writes into a pipe.
(If you are writing software to use named pipes, any processes writing into them should be started in a new thread, as they will block until all the data is consumed.)
Now if you give the R1_decompressed.fastq as a file argument to some other program, it will see decompressed content (e.g. wc -l R1_decompressed.fastq will tell you the number of lines in the decompressed file); we can call program reading from the named pipe a reading/consuming process.
As soon as a consuming process had consumed (read) all of the data, the writing/generating process will finally exit.

This, however, does not work with prinseq-lite.pl (version 0.20.4 or earlier), with a broken pipe error. Read the rest of this entry »

Recently in hacker news the following was posted: ZFS is the FS for Containers in Ubuntu 16.04.

I must admit the 16.04 demo does look very pleasant to work with.

However, bringing in ZFS into Linux reminded me of a fairly recent comparison of ZFS and btrfs that I had to do when building my home NAS.
At that time, few months ago, I’ve arrived (among others) at the following conclusions:

• ZFS on FreeBSD is reliable, though a memory hog;
• on Debian, OpenVault seems to be a good NAS web-management interface;
• on FreeBSD, FreeNAS is good (there is also Nas4Free fork of an older version, but I haven’t looked into it deep enough);
• running ZFS on linux (even as a kernel module) is the least efficient solution, at least partially because kernel’s file caching and ZFS’s ARC cache are two separate entities;
• although btrfs offers features very similar to ZFS, as of few months ago OpenVault did not offer btrfs volumes support from the web-interface.

In the end, I’ve decided to go with FreeNAS, and it seems to work well so far.

But had anything changed in the btrfs vs ZFS on Linux field?
Read the rest of this entry »