Autarchy of the Private CaveNotepad »

MyISAM vs InnoDB vs Postgres benchmark

Bogdan — Thu, 06 Oct 2011 11:08:13 +0000

For some reason, I believed that MyISAM storage engine should be very fast - faster than InnoDB and Postgres. After all, MyISAM does not support transactions, has no logging, and is overall simpler than "true" storage engines/databases.

I was surprised to find out that this isn't true, at least for the specific (simple!) query I'm interested in:

PLAIN TEXT

SQL:

SELECT primary_id FROM tablename WHERE indexed_varchar = %s AND intcol1 <= %d AND intcol2> %d

While at it, I also tested Postgres on that same database, using the same method. With only 1 thread, Postgres was somewhere between MyISAM and InnoDB in queries per second, while with more threads Postgres was getting way ahead. This was expected, because I've known that Postgres generally has much better multi-threading scalability than MySQL.

I should have trusted the results posted back in 2007 on mysqlperformanceblog.

Versions tested: MySQL 5.1.41, PostgreSQL 8.4.8. Postgres config was not modified, MySQL had key_buffer and query_buffer increased to values allowing to contain the whole table which was tested.

Note: be very careful when testing with different engines. Initially, I had completely different numbers for Postgres, because when manually porting MySQL schema to Postgres, indices for columns intcol1 and intcol2 were ported differently from MySQL version. That made Postgres results look terrible - 4k q/s at 8 threads.
Also note, that for MySQL the native mysqli PHP driver was used, while for Postgres I've used PDO.
Code for this benchmark is based on phptestsuite.stable.tar.gz.
Hardware used: core i7 (4 cores + 4 HT "cores"), frequency steps within 1.6-4.6 GHz; 16 GB RAM.

Python performance: set vs list

Bogdan — Mon, 15 Aug 2011 09:29:04 +0000

Sometimes there is a need to be sure that no identifier is processed twice - for example, when parsing a file into a database, with file potentially containing duplicate records. An obvious solution is to properly wrap the DB insertion code into try...except block, and process duplicate primary ID exceptions. Another, sometimes more desired solution is to maintain a set/list of processed IDs internally, and check against that list prior to attempting the insertion of anything. So is it a set or a list?

There are already quite a few internet resources discussing "python set vs list", but probably the simplest while elegant way to test that is below.

First, test the speed of adding/appending to a set or a list (here, I'm mimicking the real-life application, thus the test case has an optional loop):

PLAIN TEXT

CODE:

$python -mtimeit -s 'myset = set()' 'for x in xrange(1000): myset.add(x)'
10000 loops, best of 3: 133 usec per loop
$python -mtimeit -s 'tmp = list()' 'for x in xrange(1000): tmp.append(x)'
10000 loops, best of 3: 116 usec per loop

As we can see, set and list are comparable in the speed of adding new items, with list being slightly (~12%) faster than set.

Now, the speed of membership testing: 'x in tmp'. For this test, I've deliberately chosen an imbalance of True (1%) and False (99%) results for the test - again, mimicking the real problem I have at hand:

PLAIN TEXT

CODE:

$python -mtimeit -s 'tmp = set()' -s 'for x in xrange(1000): tmp.add(x)' 'for x in xrange(100000): x in tmp'
100 loops, best of 3: 7.27 msec per loop
$python -mtimeit -s 'tmp = list()' -s 'for x in xrange(1000): tmp.append(x)' 'for x in xrange(100000): x in tmp'
10 loops, best of 3: 2.12 sec per loop

List is much slower for membership testing, while sets were designed to be fast for doing just that.

Phusion Passenger Apache users guide

Bogdan — Sun, 14 Aug 2011 21:07:40 +0000

Phusion Passenger Apache users guide

Also as a PDF.

HandBrake profile for Nokia E71 default player

Bogdan — Sat, 13 Aug 2011 14:41:24 +0000

Inspired by video encoding with handbrake.

HandBrake is a very high-quality piece of software - next time you need recoding something into H.264/MPEG-4 (using MKV or MP4 containers) - try HandBrake. It easily saturated all my CPU cores - which I failed to achieve with ffmpeg, which even with threads=8 was only saturating 2 cores.

Attached to this post are 2 profiles for recoding movies for Nokia E71. The "_best" profile has exhaustive motion detection, otherwise is identical to the base profile.
E71.plist
E71_best.plist

Generate .mood moodbar files for your whole music collection

Bogdan — Sun, 10 Apr 2011 20:21:46 +0000

Amarok moodbar wiki page has 2 nice scripts to generate .mood files for your whole music collection (to be displayed by amarok when playing).

Both scripts should be started from the directory where you keep your music. The .mood files will be generated next to the source music files (in the same directories). You can modify the scripts to have the moods stored to $HOME/.kde/share/apps/amarok/moods/, so as not to clutter your music directories.

Script for multicore CPUs (will not overwrite existing mood files):

PLAIN TEXT

CODE:

#!/bin/bash
NUMCPU="$(grep ^processor /proc/cpuinfo | wc -l)"
find . -type f -regextype posix-awk -iregex '.*\.(mp3|ogg|flac|wma)' | while read i ; do
while [ `jobs -p | wc -l` -ge $NUMCPU ] ; do
sleep 0.1
done
TEMP="${i%.*}.mood"
OUTF=`echo "$TEMP" | sed 's#$.*$/$[^,]*$#\1/.\2#'`
if [ ! -e "$OUTF" ] ; then
moodbar -o "$OUTF" "$i" &
fi
done

Script for single-threaded moods generation (will only overwrite the last-generated mood file from a previous run):

PLAIN TEXT

CODE:

#!/bin/bash
DIR=${1:-.}
LAST=.moodbar-lastreadsong
C_RET=0
control_c() # run if user hits control-c
{
echo "$1"> "$LAST"
echo "Exiting..."
exit
}
if [ -e "$LAST" ]; then
read filetodelete <"$LAST"
rm "$filetodelete" "$LAST"
fi
exec 9<<(find "$DIR" -type f -regextype posix-awk -iregex '.*\.(mp3|ogg|flac|wma)') # you may need to add m4a and mp4
while read i
do
TEMP="${i%.*}.mood"
OUTF=`echo "$TEMP" | sed 's#$.*$/$[^,]*$#\1/.\2#'`
trap 'control_c "$OUTF"' INT
if [ ! -e "$OUTF" ] || [ "$i" -nt "$OUTF" ]; then
moodbar -o "$OUTF" "$i" || { C_RET=1; echo "An error occurred!">&2; }
fi
done <&9
exec 9<&-
exit $C_RET

That same page has a link to an older, somewhat more polished ruby script for the same task. It requires a running instance of Amarok, and uses it's media collection to find music files and create mood files for them. This script can potentially (after a little editing) store .mood files either with the music or to the $HOME/.kde/... path mentioned above. It is for an older version of Amarok, and thus is not likely to work with e.g. Amarok 2.4

PLAIN TEXT

RUBY:

#!/usr/bin/ruby
# Simple moodbar file management utility by Joe Rabinoff
# This is also my first ever ruby script so bear with me
$moodbar_folder = "#{ENV['HOME']}/.kde/share/apps/amarok/moods/"
def usage()
print "This is the moodbar file management utility\n"
print "\n"
print "Usage is:\n"
print " moodbar_util.rb -rename Rename mood files" \
+ " from any old naming scheme to the current one\n" \
+ " (not applicable when mood" \
+ " files are stored with music)\n"
print " moodbar_util.rb -calcall Calculate all un-calculated" \
+ " mood files\n"
exit(1)
end
def check_amarok_running()
result = `dcop 'amarok*'`.split("\n")
return (result.length> 0)
end
def get_devices()
result = `dcop amarok collection query 'SELECT id, lastmountpoint FROM devices;'`.split("\n")
ret = Hash.new()
result.each_index do |i|
ret[result[i].to_i] = result[i+1] if i % 2 == 0
end
return ret
end
def mood_name(url, deviceid)
ret = url.gsub(/\//, ',')
return deviceid.to_s + "," + ret
end
def rename()
print "Renaming mood files from outdated naming schemes...\n"
# As far as I can tell, the best way to do this is just select all
# tracks, check the old possible moodbar names, and rename them
tracks = `dcop amarok collection query 'SELECT url, deviceid FROM tags;'`.split("\n")
devices = get_devices
0.step(tracks.length-1, 2) do |i|
url = tracks[i].sub(/\.[^\.]*$/, '.mood')
deviceid = tracks[i+1].to_i
path = url.sub(/\./, '')
if devices.has_key?(deviceid)
path = devices[deviceid] + path
end
moodfile = $moodbar_folder + mood_name(url, deviceid)
moodfile1 = $moodbar_folder + path.gsub(/\//, ',')
next if FileTest.exists?(moodfile)
if FileTest.exists?(moodfile1)
system("mv", "-f", moodfile1, moodfile)
print "Moved ", moodfile1, " to ", moodfile, "\n"
end
end
end
def calcall()
print "Calculating all nonexisting moodbars...\n\n"
tracks = `dcop amarok collection query 'SELECT url, deviceid FROM tags;'`.split("\n")
devices = get_devices
withMusic = (`dcop amarok script readConfig MoodsWithMusic`.chomp == "true")
0.step(tracks.length-1, 2) do |i|
url = tracks[i]
deviceid = tracks[i+1].to_i
moodfile = ""
songpath = url.sub(/\./, '')
if devices.has_key?(deviceid)
songpath = devices[deviceid] + songpath
end
if withMusic
moodfile = songpath.sub(/\.[^\.]*$/, '.mood')
moodfile = File.dirname(moodfile) + "/." + File.basename(moodfile)
else
moodfile = $moodbar_folder \
+ mood_name(url.sub(/\.[^\.]*$/, '.mood'), deviceid)
end
next if FileTest.exists?(moodfile) && FileTest.size(moodfile)> 0
next unless FileTest.exists?(songpath)
print "Saving moodbar for ", songpath, "\n"
print " to file ", moodfile, "\n"
system("moodbar", "-o", moodfile, songpath)
print "\n"
end
end
$mode = ARGV[0]
if !check_amarok_running
print "Amarok is not running! Please start Amarok and re-run " \
"this script.\n"
end
if $mode == "-rename"
rename
elsif $mode == "-calcall"
calcall
else
usage
end

How to truncate git history (sample script included)

Bogdan — Mon, 28 Mar 2011 18:17:21 +0000

Under a few assumptions (most importantly - you do not have any non-merged branches,), it is very easy to throw away git repository commits older than an arbitrarily-chosen commit.

Here's a sample script (call it e.g. git-truncate and put into your ~/bin or whichever location you have in PATH).

PLAIN TEXT

CODE:

#!/bin/bash
git checkout --orphan temp $1
git commit -m "Truncated history"
git rebase --onto temp $1 master
git branch -D temp

Invocation: cd to your repository, then git-truncate refspec, where refspec is either a commit's SHA1 hash-id, or a tag.

Expected result: a git repository starting with "Truncated history" initial commit, and continuing to the tip of the branch you were on when calling the script.

If you truncate repositories often, then consider adding an optional 2nd argument (truncate-commit message) and also some safeguards against improper use - currently, even if refspec is wrong, the script will not abort after a failed checkout.

Thanks for posting any improvements you may have.

Source: Tekkub's post on github discussions.
See also: how to remove a single file from all of git's commits.

How to easily install any PyPi/easy_install python module on Debian

Bogdan — Wed, 16 Feb 2011 17:22:20 +0000

Imagine you need to install pycassa (which uses easy_install). Here are the 2 (at maximum) very simple steps to have it properly debianized and installed on your Debian/Ubuntu:

if you don't have the python-stdeb package: sudo aptitude install python-stdeb
pypi-install pycassa

That's it.

Refer to stdeb readme for more information. You will need that if there are dependencies - which might not be resolved automatically by stdeb.

Before stdeb, it wasn't exactly trivial to make a .deb from python module.

My favourite command line for mirroring with wget

Bogdan — Sun, 09 Jan 2011 12:30:27 +0000

wget ––continue ––mirror ––page-requisites ––adjust-extension ––convert-links ––backup-converted ––limit-rate=500k ––wait=2 URL

(short form: wget -cmpEkK ––limit-rate=500k -w 2 URL)

––continue does no harm on initial run, but when run again this instructs wget to pick up where it was interrupted the previous time
––mirror turns on several options useful for mirroring (read wget manual for details)
––page-requisites instructs wget to also fetch linked resources, which would otherwise be skipped
––adjust-extension (or ––html-extension in older wget versions) will append the .html extension to files without it, but with document type text/html or equivalent
––convert-links will convert the links for convenient local viewing
––backup-converted helps to avoid re-downloading some files, especially when ––convert-links is specified
––limit-rate=500k and ––wait=2 prevent overloading the target website (being nice to someone offering information you need is a must)

How to relay outgoing postfix emails via another mail server (e.g. your ISP)

Bogdan — Sat, 04 Dec 2010 18:01:37 +0000

Here's a simple and clear guide for gmail, which also definitely works with other relay hosts. I've used it to configure my ISP's mail relay (they block outgoing port 25) on a Debian Squeeze laptop.

How to replace newlines with commas, tabs etc (merge lines)

Bogdan — Tue, 16 Nov 2010 08:20:45 +0000

Imagine you need to get a few lines from a group of files with missing identifier mappings. I have a bunch of files with content similar to this one:

ENSRNOG00000018677 1368832_at 25233
ENSRNOG00000002079 1369102_at 25272
ENSRNOG00000043451 25353
ENSRNOG00000001527 1388013_at 25408
ENSRNOG00000007390 1389538_at 25493

In the example above I need '25353', which does not have corresponding affy_probeset_id in the 2nd column.

It is clear how to do that:

PLAIN TEXT

CODE:

sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}'

This outputs a column of required IDs (EntrezGene in this example):

116720
679845
309295
364867
298220
298221
25353

However, I need these IDs as a comma-separated list, not as newline-separated list.

There are several ways to achieve the desired result (only the last pipe commands differ):

PLAIN TEXT

CODE:

sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | gawk '$1=$1' ORS=', '

PLAIN TEXT

CODE:

sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | tr '\n' ','

PLAIN TEXT

CODE:

sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | sed ':a;N;$!ba;s/\n/, /g'

PLAIN TEXT

CODE:

sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | sed ':q;N;s/\n/, /g;t q'

PLAIN TEXT

CODE:

sort -u *_affy_ensembl.txt | grep -v '_at' | awk '{print $2}' | paste -s -d ","

These solutions differ in efficiency and (slightly) in output. sed will read all the input into its buffer to replace newlines with other separators, so it might not be best for large files. tr might be the most efficient, but I haven't tested that. paste will re-use delimiters, so you cannot really get comma-space ", " separation with it.

Sources: linuxquestions 1 (explains used sed commands), linuxquestions 2, nixcraft.

Batch-retrieve EntrezGene homologs using NCBI’s HomoloGene and R’s annotationTools

Bogdan — Wed, 27 Oct 2010 10:49:01 +0000

Install the annotationTools R package:
source("http://bioconductor.org/biocLite.R")
biocLite("annotationTools")
Download full HomoloGene data file from ftp://ftp.ncbi.nlm.nih.gov/pub/HomoloGene/current
library(annotationTools)
homologene = read.delim("homologene.data", header=FALSE)
mygenes = read.table("file with one entrez ID of the source organism per line.txt")
getHOMOLOG(unlist(mygenes), taxonomy_ID_of_target_organism, homologene) [alternatively, wrap the call to getHOMOLOG into unlist to get a vector]

It might be easier to achieve the same results with a Perl script calling NCBI's e-utils.

The world’s shortest marketing plan (4P cheat sheet)

Bogdan — Sun, 24 Oct 2010 14:57:35 +0000

Source: Kelly's think tank.

Workplace or gamer’s HQ? ;)

Bogdan — Sat, 16 Oct 2010 20:23:00 +0000

Find out more about the depicted office. That would be a nice setup for flight and/or space simulators, I guess.

TinEye Firefox extension helped finding more nice workplaces.

And Stefan in his office description provided some more links to multi-display workplaces - Mitch Haile's and Kevin Connollie's among others.

Simple and efficient Drupal upgrades: patch!

Bogdan — Sun, 03 Jan 2010 13:50:37 +0000

Just a quick note: upgrading Drupal using a patch file is a really efficient and fast method, especially because diff/patch files are available for different Drupal version combinations.

Search and replace in a MySQL table

Bogdan — Tue, 27 Oct 2009 07:37:34 +0000

This query performs a table-wide search-and-repalce:

UPDATE `table_name` SET `table_field` = REPLACE(`table_field`,'string to search for and replace','replacement string');

If you need a database-wide search-and-replace, you could try this script (I haven't tested/used it myself).

Beware of the following gotchas:

wrong query syntax may ruin the field you are performing replace on, so always backup first!
be sure to provide "search-for" string as specific as possible, or you will get unexpected replacements (e.g. replacing mini with little will also convert all minivans into littlevans); also, do use WHERE clause when necessary to limit the number of rows modified
the function in the example is case-sensitive, so replacing all minivans with vehicles won't replace Minivans. However, I believe there exists a case-insensitive version of REPLACE function

Configuring web-server: for production and for development

Bogdan — Sun, 25 Oct 2009 15:16:10 +0000

Production: see http://www.howtoforge.com/how-to-set-up-apache2-with-mod_fcgid-and-php5-on-debian-etch - it is for Debian Etch (which is old-stable), but many of the steps apply equally well to Debian Lenny (current-stable). Also, this is a very basic guide, as if you are going to host multiple sites from multiple clients, you most definitely will need some hosting control panel.

Development: see http://www.ruzee.com/blog/2009/01/apache-virtual-hosts-a-clean-setup-for-php-developers. This setup works very well, unless you need to create several virtual hosts every day - in which case necessary actions could be partially scripted.

R under Debian testing/i386: permanently set pdfviewer option

Bogdan — Wed, 21 Oct 2009 14:06:37 +0000

If you get this message when opening vignettes:

Error in openPDF(vif) :
getOption('pdfviewer') is ''; please use 'options(pdfviewer=...)'

and you are tired of running this command every time:

> options(pdfviewer="okular")

then you should check if your system-wide Renviron file has proper PDF viewer set:

grep -i pdf /etc/R/Renviron

R_PDFLATEXCMD=${R_PDFLATEXCMD-${PDFLATEX-'/usr/bin/pdflatex'}}
R_RD4PDF=${R_RD4PDF-'times,hyper'}
## Default PDF viewer
R_PDFVIEWER=${R_PDFVIEWER-''}

It wasn't in my case.

To set one, either edit the system-wide Renviron (e.g. by editing the R_PDFVIEWER=${R_PDFVIEWER-''} line to look like R_PDFVIEWER=${R_PDFVIEWER-'/usr/bin/xdg-open'} - this will use MIME types to open your preferred PDF viewer), or one of the per-user/per-directory Renviron files to fix this minor annoyance (format is the same, e.g. R_PDFVIEWER=okular ).

Convenient design and debugging of regular expressions under Linux

Bogdan — Sat, 10 Oct 2009 15:51:18 +0000

Regular expressions (regexps) are powerful indeed. But debugging non-trivial regexps is a burden even if you understand how regexps work, and remember most (if not all) regexp syntax.

Miscellaneous tools exist to ease this task. This post was inspired by redet's comparison of regexp helper tools - it could be sufficient to read only that, if you're going to try the mentioned tools yourself. Otherwise, read on.

Kodos is an easy-to-use (but powerful enough) Python tool to verify regular expressions against sample user-provided strings. Available for both Linux and Windows. Generates ready-to-use Python regexp code.

redet assumes some knowledge of REs. Unlike all the other helper tools, it has the widest code generation coverage - just go to "Tools->Save command information" to generate code (Python, Perl, C, grep - just to name a few of the 57 (!) languages/tools/editors supported). At the same time, redet does not depend on specific versions of all supported RE engines - instead, redet automatically tests for features in available versions.

visual-regexp is similar to kodos in features.

txt2regex is a regex wizard written in bash, capable of generating code for 24 languages/editors/programs (by default, these are enabled: perl, php, postgres, python, sed and vim). It is really simple and convenient, though some understanding of "groups" and "character ranges" is highly recommended.

KRegExpEditor was a part of kde3-utils, and seems to be missing from current KDE 4.3. It is similar to kodos and visual-regexp in its philosophy and features.

I would recommend using one of kodos/visual-regexp/kregexpeditor to quickly test or design your regular expression, then use redet to generate appropriate source code. As an alternative (and when working in a console), just use txt2regexp to generate the code you need.

Tools I haven't tried include regex coach and regexpviewer.

Please also see this extensive list of regular expression and pattern-matching related software.

Best method to recursively chmod/process files or directories

Bogdan — Mon, 08 Jun 2009 14:33:51 +0000

Found here.

Recursively set directories only to drwx-rx-rx (755):

find . -type d -exec chmod 755 {} \;

Recursively set files only to rwx-r-r (644):

find . -type f -exec chmod 644 {} \;

Recursively remove carriage returns (^M) from the end of all *.php files:

find . -type f -name "*.php" -exec /home/user/dos2unix.sh {} \;

In all these cases, {} is replaced with the filename/directory find has found matching your parameters; \; at the end just stops exec processing.

Drupal Views: how to display random nodes/content

Bogdan — Mon, 04 May 2009 11:19:19 +0000

Today I had a task of displaying random node in a Views-generated sidebar block.

This is how to do that in Drupal 7 (Views 3):

edit the view which makes the block available (follow http://your.site/admin/build/views/viewname/edit)
in the Sort Criteria section (under Filter), look for and add Global:Random.

This is how to do that in Drupal 6 (Views 2):

edit the view which makes the block available (follow http://your.site/admin/build/views/viewname/edit)
in the Sort Criteria section, add the Random criteria.

It can't be simpler than that.