Autarchy of the Private Cave

Tiny bits of bioinformatics, [web-]programming etc

    • Archives

    • Recent comments

    Archive for the 'how-to' Category

    Debian: how to whitelist IP addresses in tumgrey-SPF

    7th August 2013

    SPF is nice for protecting your mail server from spam, but sometimes there is a need to bypass SPF checking. For example, if you rely on 3rd party servers to do spam protection for you :)

    Current setup:

    • MX records point to the spam protection mail servers, which then
    • connect to my server and deliver (hopefully spam-free) mail.

    Problem: some senders (like last.fm) do have proper, strict SPF records. Tumgreyspf on my server then rejects emails relayed through the spam-protection service.

    If these spam protection relay servers are the only which send mail to your server, then it makes sense to fully disable/uninstall tumgreyspf. Putting tumgreyspf into the permanent “learning mode” (set defaultSeedOnly = 1 in /etc/tumgreyspf/tumgreyspf.conf) may not fix the SPF problem described above, as SeedOnly seems to only affect greylisting, and not rejecting unauthorized senders.

    Solution: whitelist relay server IPs.
    Read the rest of this entry »

    Share

    Posted in *nix, how-to, Software | No Comments »

    How to remotely convert live 1xHDD/LVM Linux server to 2xHDD RAID1/LVM (GRUB2, GPT)

    17th May 2011

    Assumptions:

    • current HDD is /dev/sda, it has a GPT (with bios_grub being /dev/sda1), separate /boot partition (/dev/sda2), and a physical LVM volume (/dev/sda3), where LVM holds all the remaining partitions (root, /home, /srv, …); LVM is properly configured, and system reboots with no problems
    • your new drive is /dev/sdb, it is identical to /dev/sda, and it comes empty from the manufacturer (this is important! wipe the drive if it is not empty, especially if it used to be a part of another RAID)
    • your system is Debian or Debian-based; in this exact example I’ve been using Ubuntu Server 10.04
    • your LVM volume group is named vg0
    • make sure you understand what each command does before executing it
    • you do have an external backup of all your important data, and you do understand that the following operations are potentially dangerous to your data integrity

    Inspired by: Debian Etch RAID guide, serverfault question.
    Read the rest of this entry »

    Share

    Posted in *nix, how-to, Software | 6 Comments »

    How to truncate git history (sample script included)

    28th March 2011

    Under a few assumptions (most importantly – you do not have any non-merged branches,), it is very easy to throw away git repository commits older than an arbitrarily-chosen commit.

    Here’s a sample script (call it e.g. git-truncate and put into your ~/bin or whichever location you have in PATH).


    #!/bin/bash
    git checkout --orphan temp $1
    git commit -m "Truncated history"
    git rebase --onto temp $1 master
    git branch -D temp
    # The following 2 commands are optional - they keep your git repo in good shape.
    git prune --progress # delete all the objects w/o references
    git gc --aggressive # aggressively collect garbage; may take a lot of time on large repos

    Invocation: cd to your repository, then git-truncate refspec, where refspec is either a commit’s SHA1 hash-id, or a tag.

    Expected result: a git repository starting with “Truncated history” initial commit, and continuing to the tip of the branch you were on when calling the script.

    If you truncate repositories often, then consider adding an optional 2nd argument (truncate-commit message) and also some safeguards against improper use – currently, even if refspec is wrong, the script will not abort after a failed checkout.

    Thanks for posting any improvements you may have.

    Source: Tekkub’s post on github discussions.
    See also: how to remove a single file from all of git’s commits.

    Share

    Posted in how-to, Notepad | 12 Comments »

    How to fix: Nokia Ovi Suite could not connect to the Nokia account server

    10th March 2011

    I’ve been getting this message for a long while, when trying to log into Ovi from within my Ovi Suite:

    Nokia Ovi Suite could not connect to the Nokia account server. Make sure the internet connection is working properly and try again.

    However, both my internet connection, and logging into ovi.com using a browser work fine. Even looking for updates from within Ovi Suite works fine!

    Here’s the solution (tested on Nokia Ovi Suite 3.0.0.290):
    Read the rest of this entry »

    Share

    Posted in how-to, Misc | 21 Comments »

    How to easily install any PyPi/easy_install python module on Debian

    16th February 2011

    Imagine you need to install pycassa (which uses easy_install). Here are the 2 (at maximum) very simple steps to have it properly debianized and installed on your Debian/Ubuntu:

    • if you don’t have the python-stdeb package: sudo aptitude install python-stdeb
    • pypi-install pycassa

    That’s it.

    Refer to stdeb readme for more information. You will need that if there are dependencies – which might not be resolved automatically by stdeb.

    Before stdeb, it wasn’t exactly trivial to make a .deb from python module.

    Share

    Posted in *nix, how-to, Notepad, Python, Software | 1 Comment »

    How to relay outgoing postfix emails via another mail server (e.g. your ISP)

    4th December 2010

    Here’s a simple and clear guide for gmail, which also definitely works with other relay hosts. I’ve used it to configure my ISP’s mail relay (they block outgoing port 25) on a Debian Squeeze laptop.

    Share

    Posted in *nix, how-to, Links, Notepad, Software | No Comments »

    How to replace newlines with commas, tabs etc (merge lines)

    16th November 2010

    Imagine you need to get a few lines from a group of files with missing identifier mappings. I have a bunch of files with content similar to this one:

    ENSRNOG00000018677 1368832_at 25233
    ENSRNOG00000002079 1369102_at 25272
    ENSRNOG00000043451 25353
    ENSRNOG00000001527 1388013_at 25408
    ENSRNOG00000007390 1389538_at 25493

    In the example above I need ’25353′, which does not have corresponding affy_probeset_id in the 2nd column.

    It is clear how to do that:

    1. sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}'

    This outputs a column of required IDs (EntrezGene in this example):

    116720
    679845
    309295
    364867
    298220
    298221
    25353

    However, I need these IDs as a comma-separated list, not as newline-separated list.

    There are several ways to achieve the desired result (only the last pipe commands differ):

    1. sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | gawk '$1=$1' ORS=', '
    1. sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | tr '\n' ','
    1. sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | sed ':a;N;$!ba;s/\n/, /g'
    1. sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | sed ':q;N;s/\n/, /g;t q'
    1. sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | paste -s -d ","

    These solutions differ in efficiency and (slightly) in output. sed will read all the input into its buffer to replace newlines with other separators, so it might not be best for large files. tr might be the most efficient, but I haven’t tested that. paste will re-use delimiters, so you cannot really get comma-space “, ” separation with it.

    Sources: linuxquestions 1 (explains used sed commands), linuxquestions 2, nixcraft.

    Share

    Posted in *nix, Bioinformatics, how-to, Notepad, Software | 2 Comments »