Autarchy of the Private Cave

Tiny bits of bioinformatics, [web-]programming etc

    • Archives

    • Recent comments

    How to use mkfifo named pipes with prinseq-lite.pl

    24th February 2016

    prinseq_logo_1prinseq-lite.pl is a utility written in Perl for preprocessing NGS reads, also in FASTQ format.
    It can read sequences both from files and from stdin (if you only have 1 sequence).

    I wanted to use it with compressed (gzipped/bzipped2) FASTQ input files.
    As I do not need to store decompressed input files, the most efficient solution is to use pipes.
    This works well for a single file, but not for 2 files (paired-end reads).

    For 2 files, named pipes (also known as FIFOs) can be used.
    You can create a named pipe in Linux with the help of mkfifo command, for example mkfifo R1_decompressed.fastq.
    To use it, start decompressing something into it (either in a different terminal, or in background), for example zcat R1.fastq.gz > R1_decompressed.fastq &;
    we can call this a writing/generating process, because it writes into a pipe.
    (If you are writing software to use named pipes, any processes writing into them should be started in a new thread, as they will block until all the data is consumed.)
    Now if you give the R1_decompressed.fastq as a file argument to some other program, it will see decompressed content (e.g. wc -l R1_decompressed.fastq will tell you the number of lines in the decompressed file); we can call program reading from the named pipe a reading/consuming process.
    As soon as a consuming process had consumed (read) all of the data, the writing/generating process will finally exit.

    This, however, does not work with prinseq-lite.pl (version 0.20.4 or earlier), with a broken pipe error. Read the rest of this entry »

    Posted in *nix, Bioinformatics, Software | No Comments »

    ZFS is the FS for Containers in Ubuntu 16.04 and how it compares to btrfs

    20th February 2016

    Recently in hacker news the following was posted: ZFS is the FS for Containers in Ubuntu 16.04.

    I must admit the 16.04 demo does look very pleasant to work with.

    However, bringing in ZFS into Linux reminded me of a fairly recent comparison of ZFS and btrfs that I had to do when building my home NAS.
    At that time, few months ago, I’ve arrived (among others) at the following conclusions:

    • ZFS on FreeBSD is reliable, though a memory hog;
    • on Debian, OpenVault seems to be a good NAS web-management interface;
    • on FreeBSD, FreeNAS is good (there is also Nas4Free fork of an older version, but I haven’t looked into it deep enough);
    • running ZFS on linux (even as a kernel module) is the least efficient solution, at least partially because kernel’s file caching and ZFS’s ARC cache are two separate entities;
    • although btrfs offers features very similar to ZFS, as of few months ago OpenVault did not offer btrfs volumes support from the web-interface.

    In the end, I’ve decided to go with FreeNAS, and it seems to work well so far.

    But had anything changed in the btrfs vs ZFS on Linux field?
    Read the rest of this entry »

    Posted in *nix | No Comments »

    How to: convert your VPS root filesystem to btrfs (using rescue boot)

    15th February 2016

    I’m moving from (a kind of…) a dedicated server to a VPS, to decrease my frightful anticipation of hardware failures.
    Honestly though, that server had been freezing up and restarting spontaneously for several months now, causing sometimes really long down-times…
    That server is now about 6-7 years old, built with off-the-shelf components, some of which (the HDD :) ) had weird noises from the very start.
    Definitely time to move!

    I’ve purchased a fairly cheap VPS with an easy, one-click upgrade option for after I’m done configuring it.
    It comes with a wide selection of OSes to pre-install; I’ve chosen Debian Jessie, version 8.3 as of this writing.

    I wanted to use btrfs from the beginning, so could have installed Debian myself, but… VPS provider does some initial configuration (like their Debian mirror and some other things), so I’ve felt that converting to btrfs after the fact would be easier. Now that I’ve done this – I guess it was fairly easy, although preparation did take some time.

    Below, I’m providing step-by-step instructions on how to convert your root filesystem from (most likely) ext4 to btrfs.
    Read the rest of this entry »

    Posted in *nix, how-to | No Comments »

    How to fix: mod_proxy’s ProxyPass directive does not work

    10th February 2016

    So… You had finally built a nice LXC container for your web-facing application, and even configured Apache (Debian package version 2.14.18-1 in my case) to serve some static/web-only components.
    From your client-side JavaScript UI you talk (in JSON) to the API, which is implemented as a separate node.js/Python/etc server – say, on port 8000 in the same LXC container.

    The simplest solution to forward requests from the web-frontend to your API is by using mod_proxy.
    If you want to forward any requests to /api/* to your custom back-end server on port 8000, you just add the following lines to your VirtualHost configuration:

    ProxyPass “/api” “http://localhost:8000″
    ProxyPassReverse “/api” “http://localhost:8000″

    I’d suggest not wrapping this fragment with the classical IfModule: as your application will not really work without its API back-end, you actually want Apache to fail as soon as possible if mod_proxy is missing.

    That was easy, right? What, it doesn’t work? Can’t be! It’s dead simple! No way you could make a mistake in 2 lines of configuration!!! :mad_rage: :)

    Oh wait… I remember I had this problem before… Read the rest of this entry »

    Posted in *nix, how-to, Web | No Comments »

    Recommended: “Parasyte: The Maxim”

    30th January 2016

    Parasyte: The Maxim (2014) on IMDb

    This anime is the best that I have watched since (all seasons/full features of) Ghost in the Shell. In fact, I like this one more than Steins;Gate – which I can also recommend, but it takes patience to watch the first 3-4 episodes. With Parasyte, no patience is necessary: it is a very smooth flow from the very start until the very end. Luckily, season 1 is all there is. Just like with the legendary single season of Firefly :)

    Posted in Links, Movies | No Comments »

    Smiling Amsterdam: under surveillance

    27th January 2016

    Coat of Arms of Amsterdam

    Coat of Arms of Amsterdam by Ssolbergj

    I had recently undertaken a spontaneous trip to Amsterdam, the Netherlands. There were a few strong first-time impressions, which I want to document before they fade.

    The strongest first impression is from the check in and check out system in trams (and, likely, other public transportation nets). Tickets are all NFC-based, and you must swipe/touch special verifying devices upon entering and exiting a tram. If your ticket is valid, green light and a sound tone will notify you (and everyone around you) about this. If your ticket is no longer valid, you’ll get red light and a different (louder also? not sure) sound.

    These chipcards come in a variety of forms: non-reloadable cards for 1/24/48/etc hours (from the moment of the 1st check in), anonymous reloadable cards, and other card types.

    Upon entering a tram with an invalid (train :) ) ticket, we were greeted by… a woman in a cage! :) She was selling tickets to those who didn’t have them. It was already late evening, but she was smiling a lot and also tried to figure out if we really need just a 1h ticket, or if a multi-day ticket would be better for us.

    I am not entirely sure why checking out is necessary, other than for collecting loads of data on passenger flows (oh, sweet sweet data to analyze!). I also haven’t checked what happens if your ticket runs out while you are in a tram: will it fail to check out, forcing you to buy another one? Most likely not: should still let you out (there are actual tourniquets at tram exit doors).
    Read the rest of this entry »

    Posted in Life | No Comments »

    What is in 2016 for you and me and everyone?

    26th December 2015

    There will be 366 days in 2016 (yes, a leap year). 366 steps towards your goal. That’s a lot! With the right goal and persistence, a lot can be done and achieved.

    There are lots of good, real, helpful self-development/self-improvement techniques. These can often be practiced non-stop in one’s everyday life. These can often also be practiced in parallel, for a likely synergy.

    There are lots of success books and biographies of successful self-made people. Some stories are quite well-known, while some remain hidden gems. Likely enough reading for several months, non-stop!

    There are many good books on business, data analysis, engineering, software development, and other fields. Likely many years of reading non-stop!

    Some people say that in 5 years one can master a common subject of choice good enough to become an expert. Others point out that many activities (and their co-activities) can be monetized or turned into profitable versions of said activities.

    All in all, the only thing required is action.

    Which books do you have on your 2016 reading list?
    What is the field you will be an expert in by 2021, assuming you start in 2016?
    Information consumption aside, what will you produce in 2016? What will be your actions?

    Posted in Books, Life | No Comments »