Autarchy of the Private Cave

Tiny bits of bioinformatics, [web-]programming etc

    • Archives

    • Recent comments

    Archive for the '*nix' Category

    How to fix: mod_proxy’s ProxyPass directive does not work

    10th February 2016

    So… You had finally built a nice LXC container for your web-facing application, and even configured Apache (Debian package version 2.14.18-1 in my case) to serve some static/web-only components.
    From your client-side JavaScript UI you talk (in JSON) to the API, which is implemented as a separate node.js/Python/etc server – say, on port 8000 in the same LXC container.

    The simplest solution to forward requests from the web-frontend to your API is by using mod_proxy.
    If you want to forward any requests to /api/* to your custom back-end server on port 8000, you just add the following lines to your VirtualHost configuration:

    ProxyPass “/api” “http://localhost:8000″
    ProxyPassReverse “/api” “http://localhost:8000″

    I’d suggest not wrapping this fragment with the classical IfModule: as your application will not really work without its API back-end, you actually want Apache to fail as soon as possible if mod_proxy is missing.

    That was easy, right? What, it doesn’t work? Can’t be! It’s dead simple! No way you could make a mistake in 2 lines of configuration!!! :mad_rage: :)

    Oh wait… I remember I had this problem before… Read the rest of this entry »


    Posted in *nix, how-to, Web | No Comments »

    Midnight Commander (mc): convenient hard links creation from user menu

    3rd December 2015

    Midnight Commander is a convenient two-panel file manager with tons of features.

    You can create hard links and symbolic links using C-x l and C-x s keyboard shortcuts. However, these two shortcuts invoke two completely different dialogs.

    While for C-x s you get 2 pre-populated fields (path to the existing file, and path to the link – which is pre-populated with your opposite file panel path plus the name of the file under cursor; simply try it to see what I mean), for C-x l you only get 1 empty field: path of the hard link to create for a file under cursor. Symlink’s behaviour would be much more convenient…

    Fortunately, a good man called Wiseman1024 created a feature request in the MC’s bug tracker 6 years ago. Not only had he done so, but he had also uploaded a sample mc user menu script (local copy), which works wonderfully! You can select multiple files, then F2 l (lower-case L), and hard-links to your selected files (or a file under cursor) will be created in the opposite file panel. Great, thank you Wiseman1024!

    Word of warning: you must know what hard links are and what their limitations are before using this menu script. You also must check and understand the user menu code before adding it to your mc (by F9 C m u, and then pasting the script from the file).

    Word of hope: 4 years ago Wiseman’s feature request was assigned to Future Releases version, so a more convenient C-x l will (sooner or later) become the part of mc. Hopefully.


    Posted in *nix, how-to, Notepad | No Comments »

    Tools to manage Debian services and start-up scripts

    8th April 2015

    • sysv-rc-conf: text-UI, allows editing multiple runlevels, but seems to be failing recently
    • rcconf: another good-looking text-UI alternative; can only modify current runlevel
    • bum: GUI tool (BootUp Manager)

    More details about these tools and runlevels (in German).


    Posted in *nix, Notepad | No Comments »

    Compressors galore: pbzip2, lbzip2, plzip, xz, and lrzip tested on a FASTQ file

    28th March 2015

    About 2 years ago I had already reviewed some parallel (and not) compressing utilities, settling at that time on pbzip2 – it scales quasi-linearly with the number of CPUs/cores, stores compressed data in relatively small 900k blocks, is fast, and has good compression ratio. pbzip2 was (and still is) a very good choice.

    Yesterday I got somewhat distracted, and thus found lbzip2 -

    an independent, multi-threaded implementation of bzip2. It is commonly the fastest SMP (and uniprocessor) bzip2 compressor and decompressor

    - as it says in the Debian package description. Is it really “commonly the fastest” one? How does it compare to pbzip2? Should I use lbzip2 instead of pbzip2?

    This minor distraction had grown into a full-scale web-search and comparison, adding to the mix plzip (a parallel version of lzip), xz, and lrzip. After reading thousands of characters, all of these were put to a simple test: compressing an about 2 gigabyte FASTQ file with default options.

    All the external links and benchmarks, as well as my own mini-benchmark results, are provided below.

    The conclusion is that out of all the tested compressors lbzip2 is indeed the best one (for my practical use). It is only slightly better than the trusty pbzip2, which takes the second place. All the other compressors performed so poorly, that they do not get any place in my practical rating…

    So, let us first ask internet wisdom/foolishness, if lbzip2 or pbzip2 is faster/better?
    Read the rest of this entry »


    Posted in *nix, Comparison, Links, Misc, Software | 6 Comments »

    My smartd.conf, explained

    28th February 2015

    After fixing offline uncorrectable sector warning email, I have taken a closer look at my /etc/smartd.conf, and now it looks like this:

    DEFAULT -d sat -H -f -p -t -W 0,40,45 -n standby -S on -m
    # Attributes 1, 230, and 231 are very important (-r 1! -r 230! -R 230! -r 231! -R 231!), but likely covered by -t.
    /dev/sda -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00) -C 0 -I 189 -I 194
    # -a implies -f and -p (through -t)
    DEFAULT -d sat -a -I 194 -W 0,40,45 -n standby -o on -S on -m
    /dev/sdb -s (S/../../6/02|L/../(01|02|03|04|05|06|07)/7/02)
    # This drive does not decrement Offline_Uncorrectable (198) after re-allocation,
    # so only monitoring for increase, not for non-zero value.
    /dev/sdc -s (S/../../6/03|L/../(01|02|03|04|05|06|07)/7/04) -U 198+
    # This drive has 40 “normally”.
    /dev/sdd -s (S/../../6/04|L/../(01|02|03|04|05|06|07)/7/06) -W 0,42,45

    Note: explanations below are intentionally simplified; please consult man smartd.conf for more precise, complete, and up-to-date information.

    Ok, so what do these settings mean, and how is this different from default settings?
    Read the rest of this entry »


    Posted in *nix, how-to, Notepad, Software | No Comments »

    How to fix Offline Uncorrectable sector outside of a partition

    26th February 2015

    A few days ago my smartd daemon (from the smartmontools package) notified me about a +1 increase in Current_Pending_Sector (197) and Offline_Uncorrectable (198) SMART attributes. The 2.5″ Fujitsu laptop hard-drive these appeared on is very old, and it also has been working 24/365 since a little over a year.

    Running a short SMART self-test (sudo smartctl -t short /dev/sdc) produced a read error at sector 1289:

    Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
    1 Short offline Completed: read failure 80% 22339 1289

    Looking at the partition table of /dev/sdc, we see that this sector is outside of the only RAID partition on the disk, which starts at sector 2048:

    Device Boot Start End Blocks Id System
    /dev/sdc1 2048 117209087 58605088 fd Lnx RAID auto

    Read the rest of this entry »


    Posted in *nix, how-to, Software | 1 Comment »

    How to update a multisite Drupal 6/7 installation using Drush

    25th August 2014

    There are quite a lot of posts on how to do this, but my differs a tiny little bit, so I’m saving it for my own future reference, and also for the benefits of the wider audience.

    I am updating a multisite Drupal 6 installation. To the best of my knowledge, the only difference for Drupal 7 is that instead of the site_offline D6 variable the maintenance_mode variable is used in D7.

    On Debian stable and later, you can sudo aptitude install drush and then just use it immediately after that.

    Note: I recommend su webuser (or sudo -s followed by sudo -s -u webuser) before you run any non-testing drush commands, where webuser is the user which owns your web-exposed files (e.g. Debian’s default is, I think, www-data). I’ve seen a lot of recommendations to run drush as a super-user, but that does not make sense, and may actually cause problems with file ownership.

    One last thing before we start: if your drush seems to work fine but hangs when untarring modules – check this solution.

    Read the rest of this entry »


    Posted in *nix, Drupal, how-to, Notepad, PHP, Programming, Software, Web | 1 Comment »