Autarchy of the Private Cave

Tiny bits of bioinformatics, [web-]programming etc

    • Archives

    • Recent comments

    Archive for the '*nix' Category

    Slow memory allocation due to Transparent Huge Pages (THP)

    6th August 2018

    Your software needs tons of RAM, and runs a bit too slow on your super-duper HPC cluster? Read this: Slow memory allocation due to Transparent Huge Pages (THP)

    Share

    Posted in *nix, Links | No Comments »

    Midnight Commander: panelize or select all files newer than specified date

    3rd February 2017

    If you ever need to select lots (hundreds, thousands) of files by their modification date, and your directory contains many more files (thousands, tens of thousands), then angel_il has the answer for you:

    1. touch -d “Jun 01 00:00 2011″ /tmp/.date1
    2. enter into your BIG dir
    3. press C-x ! (External panelize)
    4. add new command like a “find . -type f \( -newer /tmp/.date1 \) -print”

    I’ve used a slightly different approach, specifying desired date right in the command line of External Panelize:

    1. enter your directory with many files
    2. press C-x ! (External Panelize)
    3. add a command like find . -type f -newermt "2017-02-01 23:55:00" -print (man find for more details)

    In both cases, the created panel will only have files matching your search condition.

    Share

    Posted in *nix, Notepad | No Comments »

    How to: enable metadata duplication on an existing btrfs filesystem

    30th December 2016

    Just one command: sudo btrfs balance start -v -mconvert=dup /toplevel/
    where /toplevel/ is your mountpoint of the btrfs root, -v is there for verbosity (not too verbose, don’t worry), and -mconvert=dup literally says act on metadata only, convert data profile to DUP.

    This will duplicate both metadata and btrfs system data.
    Verify with: sudo btrfs fi df /toplevel:

    Data, single: total=10.00GiB, used=3.88GiB
    System, DUP: total=64.00MiB, used=4.00KiB
    Metadata, DUP: total=512.00MiB, used=286.18MiB
    GlobalReserve, single: total=96.00MiB, used=0.00B

    Explanation: on SSDs, mkfs.btrfs creates metadata in single mode (because of widely spread SSD deduplication algorithms negating duplicate entries). However, second copy of metadata increases recovery chances, especially so if your SSD does not deduplicate writes. Hence the desire to add metadata/systemdata duplication after the filesystem is created.

    Share

    Posted in *nix, how-to, Notepad, Software | No Comments »

    Mail-in-a-box, Sovereign, Modoboa, iRedMail, etc

    28th December 2016

    Preparing to dismantle my physical server (and move different hosted things to one or more VPS),
    I’ve realized that an email server is necessary: to send website-generated emails, and also
    receive a few rare contact requests arriving at the websites.

    My current email server was configured eons ago, it works well,
    but I have no desire to painfully transfer all the configuration…
    Better install something new, shiny and exciting, right? :)

    I had 3 #self-hosted, #mail-server bookmarks:

    (Sovereign, the 4th one, was addded after reading more about Mail-in-a-box.)

    Here are my notes on what seemed important about these 4.
    Read the rest of this entry »

    Share

    Posted in *nix, Comparison, Links, Notepad, Software, Web | 5 Comments »

    How to: easily add swap partition to a live system on btrfs

    14th April 2016

    Recently I had a need to add a swap file to my Debian installation.
    However, I am now using btrfs, and – as with any other COW filesystem – it is not possible to simply create a swap file and use it.
    There are workarounds (creating a file with a COW attribute removed, and then loop-mounting it), but I just did not like them.

    So I have decided to add a swap partition.
    It worked amazingly (and very easily), there was even no need to reboot – at all.
    I still did restart, just to make sure the system is bootable – and all was perfectly fine.

    My initial setup is very simple: a single /dev/sda1 partition on the /dev/sda disk, fully used by btrfs.
    Different important paths/mountpoints are btrfs subvolumes, using flat hierarchy.
    For this example, let us assume that /dev/sda (and /dev/sda1) is 25GB large, and that I want to add a 2GB swap /dev/sda2 after /dev/sda1.

    Brief explanation before we start:

    1. shrink btrfs filesystem by more than 2GB;
    2. shrink btrfs partition by 2GB;
    3. create new 2GB partition for the swap;
    4. resize btrfs filesystem to full size of its new-size partition;
    5. initialize swap and turn it on.

    Here are the very easy steps! Just make sure you do not make mistakes anywhere ;)
    Read the rest of this entry »

    Share

    Posted in *nix, how-to | No Comments »

    How to use mkfifo named pipes with prinseq-lite.pl

    24th February 2016

    prinseq_logo_1prinseq-lite.pl is a utility written in Perl for preprocessing NGS reads, also in FASTQ format.
    It can read sequences both from files and from stdin (if you only have 1 sequence).

    I wanted to use it with compressed (gzipped/bzipped2) FASTQ input files.
    As I do not need to store decompressed input files, the most efficient solution is to use pipes.
    This works well for a single file, but not for 2 files (paired-end reads).

    For 2 files, named pipes (also known as FIFOs) can be used.
    You can create a named pipe in Linux with the help of mkfifo command, for example mkfifo R1_decompressed.fastq.
    To use it, start decompressing something into it (either in a different terminal, or in background), for example zcat R1.fastq.gz > R1_decompressed.fastq &;
    we can call this a writing/generating process, because it writes into a pipe.
    (If you are writing software to use named pipes, any processes writing into them should be started in a new thread, as they will block until all the data is consumed.)
    Now if you give the R1_decompressed.fastq as a file argument to some other program, it will see decompressed content (e.g. wc -l R1_decompressed.fastq will tell you the number of lines in the decompressed file); we can call program reading from the named pipe a reading/consuming process.
    As soon as a consuming process had consumed (read) all of the data, the writing/generating process will finally exit.

    This, however, does not work with prinseq-lite.pl (version 0.20.4 or earlier), with a broken pipe error. Read the rest of this entry »

    Share

    Posted in *nix, Bioinformatics, Software | No Comments »

    ZFS is the FS for Containers in Ubuntu 16.04 and how it compares to btrfs

    20th February 2016

    Recently in hacker news the following was posted: ZFS is the FS for Containers in Ubuntu 16.04.

    I must admit the 16.04 demo does look very pleasant to work with.

    However, bringing in ZFS into Linux reminded me of a fairly recent comparison of ZFS and btrfs that I had to do when building my home NAS.
    At that time, few months ago, I’ve arrived (among others) at the following conclusions:

    • ZFS on FreeBSD is reliable, though a memory hog;
    • on Debian, OpenVault seems to be a good NAS web-management interface;
    • on FreeBSD, FreeNAS is good (there is also Nas4Free fork of an older version, but I haven’t looked into it deep enough);
    • running ZFS on linux (even as a kernel module) is the least efficient solution, at least partially because kernel’s file caching and ZFS’s ARC cache are two separate entities;
    • although btrfs offers features very similar to ZFS, as of few months ago OpenVault did not offer btrfs volumes support from the web-interface.

    In the end, I’ve decided to go with FreeNAS, and it seems to work well so far.

    But had anything changed in the btrfs vs ZFS on Linux field?
    Read the rest of this entry »

    Share

    Posted in *nix, Software | No Comments »