I do not really recollect anything that large. Well, not counting dropbox’s 4-hour window of “any password fits all accounts”, but that was different.

Here are some of the news items about this seriously large hacking incident:

• NYTimes (Dec. 25, 2011)
• WSJ (Dec. 25, 2011)
• CNN (Dec. 25, 2011)
• relatively above-average write-up from Wired (Dec. 26, 2011)
• ABCNews (Dec. 26, 2011)
• The Register (Jan. 3, 2012)

Here come more technical reports:

• short pastebin document, supposedly by the hackers
• cryptome keeps track of the data being removed from the internet
• a 1MB report by the hackers
• TheTechGerald has some analysis of the leaked stratfor passwords (Jan. 2, 2012)

TheTechGerald’s analysis linked to above got my attention. Unfortunately, a while ago I’ve subscribed to stratfor’s “free intelligence mailing list”, and was wondering if my account information is now publicly available. I was the most worried about the password I’ve used to subscribe, because of the risk of using the same password somewhere else.

Unlike TheTechGerald, I haven’t used any dictionaries – just the default configuration of a well-known tool for finding weak passwords. Within a single hour, ~100k passwords were decrypted (~12% of all). Till the end of the day, ~50k more passwords were decrypted (totalling 17.4% of 860k). At this point my password was still safe, and I’ve found a way to verify that it is not used anywhere else, so I’ve aborted further decryption.

There are a few simple conclusions:

• anybody who had a stratfor account must verify that he/she isn’t using that password anywhere else, because if 1 PC can get 17% of all the passwords in less than a day, it is only a matter of short time until all the leaked passwords will be decrypted and made publicly available in various “md5 decryption databases”
• system owners should run periodic screenings for weak passwords (and implement policies to prevent creating obviously weak passwords from the very beginning)
• md5 is very fast to decrypt/bruteforce – a much slower hashing function wouldn’t hurt; also, using a more complex hashing approach, maybe even with a closed-source shared library, could help
• single-factor authentication (password-based) is likely to get replaced with 2-factor authentication in the nearest future
• one may enjoy increased personal data safety by using throw-away passwords in conjunction with antispam mailboxes like spam.la and mailinator.com (at least 1600 users – 0.186% – did use these services).

Fortunately, the top 10 passwords (by their counts) were exclusively “throw-away”, and added up to ~10% of the decrypted passwords. (I’m not showing any, as that would unnecessarily simplify further decryption – maybe thetechgerald should have also been more vague about actual passwords.)

Sooner or later this significant-size corpus of real-life passwords will find its way (after decryption by those who would actually use leaked passwords to gain unauthorized access) into various wordlists and wordlist mutation rules, making it even easier to decrypt any future leaks. This is where 2-factor authentication will, hopefully, come in handy to protect against similar leaks.

I wonder if I should put up a page “Check if my password was among those 860k”, to help people easily identify if they should change theirs – not even necessarily being a Stratfor subscriber. Unless similar pages/services had already been put up by others.

It is also unclear what will the future of Stratfor be, taking into account that their website is still dysfunctional.

It is sad to see Drupal (stratfor.com’s CMS) involved here. However, I have no idea if their installation was up to date, and if their website was the point of entry. The hacklog suggests that attackers somehow obtained the password of one of the system administrators, and then used it for SSH access, which would save Drupal’s face (Drupal’s security record to date was pretty reassuring).

The original Ukrainian text tells the tale of a swallow flying into a household to proclaim the plentiful and bountiful year that the family will have. The title shchedryk is derived from the Ukrainian word for “bountiful”. This follows a tradition of praising the hosts of festivities in the songs during those festivities, or when coming to get sweets, small money bills or presents in exchange for nice singing by a group of children.

English text was written separately, and is copyrighted.

All the derived music uses the original’s four-note pattern by Mykola Leontovych. Folk song/chant was the basis for Leontovych’s work on this piece. I believe the original song had a similar musical (vocal) pattern, and that “ostinato” figure of music was already present in the song, so Leontovych’s work was probably to smooth out any uneven moments, and formalize the music in notes. Citing wikipedia article, “ostinato motif, a repeated four-note pattern within the range of a minor third is thought to be of prehistoric origins”.

If you (like me) love ncurses – you will love tudu Just give it a try.

Screenshot copyright: Ruben Pollan (tudu author).

I was surprised to find out that this isn't true, at least for the specific (simple!) query I'm interested in:

While at it, I also tested Postgres on that same database, using the same method. With only 1 thread, Postgres was somewhere between MyISAM and InnoDB in queries per second, while with more threads Postgres was getting way ahead. This was expected, because I've known that Postgres generally has much better multi-threading scalability than MySQL.

I should have trusted the results posted back in 2007 on mysqlperformanceblog.

Versions tested: MySQL 5.1.41, PostgreSQL 8.4.8. Postgres config was not modified, MySQL had key_buffer and query_buffer increased to values allowing to contain the whole table which was tested.

Note: be very careful when testing with different engines. Initially, I had completely different numbers for Postgres, because when manually porting MySQL schema to Postgres, indices for columns intcol1 and intcol2 were ported differently from MySQL version. That made Postgres results look terrible - 4k q/s at 8 threads.
Also note, that for MySQL the native mysqli PHP driver was used, while for Postgres I've used PDO.
Code for this benchmark is based on phptestsuite.stable.tar.gz.
Hardware used: core i7 (4 cores + 4 HT "cores"), frequency steps within 1.6-4.6 GHz; 16 GB RAM.

There are already quite a few internet resources discussing "python set vs list", but probably the simplest while elegant way to test that is below.

First, test the speed of adding/appending to a set or a list (here, I'm mimicking the real-life application, thus the test case has an optional loop):

As we can see, set and list are comparable in the speed of adding new items, with list being slightly (~12%) faster than set.

Now, the speed of membership testing: 'x in tmp'. For this test, I've deliberately chosen an imbalance of True (1%) and False (99%) results for the test - again, mimicking the real problem I have at hand:

List is much slower for membership testing, while sets were designed to be fast for doing just that.

Also as a PDF.

HandBrake is a very high-quality piece of software - next time you need recoding something into H.264/MPEG-4 (using MKV or MP4 containers) - try HandBrake. It easily saturated all my CPU cores - which I failed to achieve with ffmpeg, which even with threads=8 was only saturating 2 cores.

Attached to this post are 2 profiles for recoding movies for Nokia E71. The "_best" profile has exhaustive motion detection, otherwise is identical to the base profile.
E71.plist
E71_best.plist

Related:

• x264 ffmpeg mapping and options guide
• ffmpeg audio/video encoding cheat sheet

• current HDD is /dev/sda, it has a GPT (with bios_grub being /dev/sda1), separate /boot partition (/dev/sda2), and a physical LVM volume (/dev/sda3), where LVM holds all the remaining partitions (root, /home, /srv, ...); LVM is properly configured, and system reboots with no problems
• your new drive is /dev/sdb, it is identical to /dev/sda, and it comes empty from the manufacturer (this is important! wipe the drive if it is not empty, especially if it used to be a part of another RAID)
• your system is Debian or Debian-based; in this exact example I've been using Ubuntu Server 10.04
• your LVM volume group is named vg0
• make sure you understand what each command does before executing it
• you do have an external backup of all your important data, and you do understand that the following operations are potentially dangerous to your data integrity

Inspired by: Debian Etch RAID guide, serverfault question.

1. Create the GPT on the new drive:
   parted /dev/sdb mklabel gpt
2. Get the list of partitions on /dev/sda:
   parted -m /dev/sda print
3. Create /dev/sdb partitions similarly to what you have on /dev/sda (my example numbers follow, use your numbers here):
   parted /dev/sdb mkpart bios_grub 1049kB 2097kB
   parted /dev/sdb mkpart boot 2097kB 258MB
   parted /dev/sdb mkpart lvm 258MB 2000GB
4. Set proper flags on partitions:
   parted /dev/sdb set 1 bios_grub on (GPT doesn't have MBR, so you create a 1-MB partition instead to hold grub2's boot code)
   (possibly optional) parted /dev/sdb set 2 raid on
   (possibly optional) parted /dev/sdb set 3 raid on
5. (possibly optional) To make sure /dev/sdb1 (the bios_grub) indeed contains grub's boot code, I did dd if=/dev/sda1 of=/dev/sdb1
6. apt-get install mdadm
7. Note: at this point, older tutorials suggest adding a bunch of raid* kernel modules to /etc/modules and to grub's list of modules to load. I'm not sure this is really necessary, but do see the tutorials mentioned at the top for more information. If you do modify the lists of modules - don't forget to run update-initramfs -u.
8. Create two initially-degraded RAID1 devices (one for /boot, another for LVM):
   mdadm ––create /dev/md0 ––level=1 ––raid-devices=2 /dev/sdb2 missing
   mdadm ––create /dev/md1 ––level=1 ––raid-devices=2 /dev/sdb3 missing
9. Store the configuration of your RAID1 to the mdadm.conf file (important! this is not done automatically!)
   mdadm -Es >> /etc/mdadm/mdadm.conf
10. Verify the contents of your mdadm.conf:
    cat /etc/mdadm/mdadm.conf
    dpkg-reconfigure mdadm, and enable booting in degraded mode
11. Copy your current /boot (/dev/sda2) to the new /dev/md0 /boot partition:
    (one can use dd here as well, but for some reason my attempt at dd failed writing 1 last byte of data)
    mkdir /mnt/md0
    mount /dev/md0 /mnt/md0
    cp -a /boot/* /mnt/md0/
    umount /dev/md0
    rmdir /mnt/md0
12. Now extend your existing volume group to include the newly-created /dev/md1:
    pvcreate /dev/md1
    vgextend vg0 /dev/md1
13. Verify the list of logical volumes you curently have: enter lvm shell, and type lvs. Here's what I had:
    LV VG Attr LSize Origin Snap% Move Log Copy% Convert
    home vg0 -wi-ao 1.70t
    logs vg0 -wi-ao 4.66g
    root vg0 -wi-ao 10.24g
    srv vg0 -wc-ao 100.00g
    swap vg0 -wi-ao 1.86g
    tmp vg0 -wi-ao 4.66g
14. Now, you can move all the logical volumes to new physical volume in one command: pvmove /dev/sda3 /dev/md1. Personally, remembering the problem I had with dd from /dev/sda2 to /dev/md0, I decided to move all logical volumes one-by-one; as this takes time, you may consider joining these operations with ; or &&, and putting the /tmp last (as the easiest one to re-create if it fails to move):
    pvmove ––name home /dev/sda3 /dev/md1
    pvmove ––name srv /dev/sda3 /dev/md1
    pvmove ––name logs /dev/sda3 /dev/md1
    pvmove ––name swap /dev/sda3 /dev/md1
    pvmove ––name root /dev/sda3 /dev/md1
    pvmove ––name tmp /dev/sda3 /dev/md1
15. To be safer, I ran FS check on a few volumes I could umount:
    umount /dev/mapper/vg0-srv
    fsck -f /dev/mapper/vg0-srv
    mount /dev/mapper/vg0-srv
    umount /dev/mapper/vg0-tmp
    fsck -f /dev/mapper/vg0-tmp
    mount /dev/mapper/vg0-tmp
16. Remove /dev/sda3 from the physical space available to your volume group:
    vgreduce vg0 /dev/sda3
17. Install grub2 to both drives, so as to make them both bootable in case of failure:
    grub-install '(hd0)'
    grub-install '(hd1)'
18. Edit /etc/fstab, pointing /boot to /dev/md0. You may use UUIDs here, but please do not use UUIDs from mdadm.conf - those are different from FS-UUIDs, instead do ls -l /dev/disk/by-uuid to find the UUID of /dev/md0. Personally, I had no problems just using /dev/md0.
19. Now is the time to add your original /dev/sda to the RAID1; be absolutely sure you have moved all the data off that drive, because these commands will destroy it:
    mdadm ––manage ––add /dev/md0 /dev/sda2
    mdadm ––manage ––add /dev/md1 /dev/sda3
    Re-syncing array will take some time.
20. To be on the safe side, you may want to run again update-initramfs -u and update-grub; I have also edited /etc/grub.d/40_custom, adding there 2 more boot options: from /dev/sda2 and /dev/sdb2 (/boot on both drives) - have no idea if that will work, but having more boot options didn't hurt
21. Reboot into your new system. Actually, at this point reboot is only necessary to verify that your system is bootable - you may delay this reboot as long as you want to.
22. Many tutorials also suggest testing your RAID1 by manually "degrading" it, trying to boot, and then rebuilding it back. I haven't done that, but you may want to.

Improvement suggestions, criticism and thank-you are welcome in the comments.

• Jython - Python for the Java platform, compile your python scripts into Java bytecode
• Groovy - not Python, but still a scripting language which compiles to jars

Check this list of 10 programming fonts - you may find your love there.

Both scripts should be started from the directory where you keep your music. The .mood files will be generated next to the source music files (in the same directories). You can modify the scripts to have the moods stored to $HOME/.kde/share/apps/amarok/moods/, so as not to clutter your music directories.

Script for multicore CPUs (will not overwrite existing mood files):

Script for single-threaded moods generation (will only overwrite the last-generated mood file from a previous run):

That same page has a link to an older, somewhat more polished ruby script for the same task. It requires a running instance of Amarok, and uses it's media collection to find music files and create mood files for them. This script can potentially (after a little editing) store .mood files either with the music or to the $HOME/.kde/... path mentioned above. It is for an older version of Amarok, and thus is not likely to work with e.g. Amarok 2.4

There is an excellent, in-depth, well-argumented article by Geoff Chappell on the issue. Highly recommended in its entirety to those who want a complete understanding (additional side-reading and facts verification might be necessary).

A single citation to get you started:

There is already on the Internet and elsewhere an awful lot of rubbish to read about this question. Hardly any of it would be worth citing even if I didn’t want to spare the authors the embarrassment. A surprising number of people who claim some sort of attention as expert commentators would have you believe that using more than 4GB of memory is mathematically impossible for any 32-bit operating system because 2 to the power of 32 is 4G and a 32-bit register can’t form an address above 4GB. If nothing else, these experts don’t know enough history: 2 to the 16 is only 64K and yet the wealth of Microsoft is founded on a 16-bit operating system that from its very first version was designed to use 640KB of RAM plus other memory in a physical address space of 1MB. Some remember this history and add seemingly plausible qualifications that exceeding 4GB is possible only at the price of nasty hacks that require everyone—well, all programmers—to jump through hoops. Fortunately, Intel’s processors are a lot more advanced than the 8086 from all those years ago.

P.S. Unfortunately, patching the kernel won't help make Windows XP see more than 4GB RAM: even though the kernel itself does support more RAM (with PAE), starting with SP2 the HAL was modified in a way prohibiting access to any RAM beyond 4GB. Patching may only be suggested to devoted geeks with Vista's and 7's.

There are a few highly-popular patches, which haven't yet made it into the Squeeze's rtorrent version 0.8.6.

Note: ArchLinux already has an rtorrent-extended package, so you may want to look at that one (instead of my Debian package below). I might use Arch's patchset next time to generate a truly extended rtorrent package.

I've incorporated 3 of such patches (#239 ipfilter, #2064 GeoIP support, and #2171 colours/eyecandy), and built a squeeze amd64 rtorrent-enhanced package.

A few excerpts to explain how these extra features are to be configured and used (all come from the above-mentioned trac tickets, some were edited/extended).

ipfilter

ipfilter allows to selectively blacklist/whitelist peers - based on IP address range files (so-called ipfilter files). The ipfilter files may, for example, come from bluetack.co.uk. Each line of the file contains a record in this format: range_description:start_IP-end_IP, where start_IP should be less than or equal to end_IP. Multiple files can be used. Overlapping ranges are merged automatically. Both incoming and outgoing connections are checked against the filter. Exclusions are not supported, so connection to/from IPs in all the loaded ranges will be disallowed and dropped.

Include "ip_filter=" directive in .rtorrent.rc. For example (paths are specified relative to user's home directory):
ip_filter=ipfilter/level1,ipfilter/level2

It is probably a good idea to reload ipfilter files once in a while, so you can also include "reload_ip_filter" directive on schedule to refresh the filter. The same files named in "ip_filter" will be reloaded.

schedule = filter,18:30:00,24:00:00,reload_ip_filter=

To support the feature you may want to setup cron job to reload and unzip files from bluetack. Examples here and here.

GeoIP

In the Peers view of each torrent a new column - CC, country code - is added, which shows peer's country code. When examining each peer, you will see more data from the GeoIP database (if you have the relevant files installed): AS num and city. At the very least, you need to have geoip-database installed. You can get free "lite" versions of city/AS-databases, rename them (removing "lite") and put into an appropriate location to make rtorrent+geoip use them as well. Geop-isp data support isn't integrated into this rtorrent package.

Colours

This patch somewhat changes the way rtorrent shows torrents. Seeding torrents have bold titles, there are no half-displayed torrents at the top/bottom when scrolling, and you can configure colours for active/done torrents (ticket libtorrent.rakshasa.no/ticket/1382, which seems to be gone now). Source code and testing hint at these configurable colours: done_fg_color, done_bg_color, active_fg_color, active_bg_color, and at these possible values: 1 (red), 2 (green), 3 (yellow).

Geek's cellar
A related (though not used in any way for the preparation of the package) resource is rtorrent mods page.

Relatively schematically, applying patches and building the package was performed in these steps (starting within some newly-created directory):

1. sudo aptitude install cdbs devscripts [and whatever else you find you're missing]
2. apt-get source rtorrent
3. dpkg-source -x rtorrent_0.8.6-1.dsc
4. cd rtorrent-0.8.6
5. cdbs-edit-patch 01-ipfilter.patch
6. patch -p1 < /path/to/patch/239
7. exit 0
8. cdbs-edit-patch 02-geoip.patch
9. patch -p1 < /path/to/patch/2064-after-ipfilter
10. exit 0
11. cdbs-edit-patch 03-eyecandy.patch
12. patch -p1 < /path/to/patch/2171-mod-with-canvas
13. exit 0 [you could do all 3 patches together, but I prefer cleaner and manageable approaches]
14. [some weird operation to be explained below]
15. dpkg-buildpackage

Weird operation: I didn't know how to make dpkg-buildpackage run autorun.sh (which is required for one of the patches I've used) before configure (could someone enlighten me how to do that?), so I started a new patch with cdbs-edit-patch, then ran one by one `autoreconf -if`, `./autorun.sh`, `./configure`, exited with zero status and was done with that problem. The resulting "patch" was 2MB large

Here's a sample script (call it e.g. git-truncate and put into your ~/bin or whichever location you have in PATH).

Invocation: cd to your repository, then git-truncate refspec, where refspec is either a commit's SHA1 hash-id, or a tag.

Expected result: a git repository starting with "Truncated history" initial commit, and continuing to the tip of the branch you were on when calling the script.

If you truncate repositories often, then consider adding an optional 2nd argument (truncate-commit message) and also some safeguards against improper use - currently, even if refspec is wrong, the script will not abort after a failed checkout.

Thanks for posting any improvements you may have.

Source: Tekkub's post on github discussions.
See also: how to remove a single file from all of git's commits.

Nokia Ovi Suite could not connect to the Nokia account server. Make sure the internet connection is working properly and try again.

However, both my internet connection, and logging into ovi.com using a browser work fine. Even looking for updates from within Ovi Suite works fine!

Here's the solution (tested on Nokia Ovi Suite 3.0.0.290):

Important: before trying the solution below, try downloading and running root certificates update program from Microsoft, then restarting Ovi Suite to see if the problem is gone. (Thanks Finn for sharing this one.)

• Navigate to Start - Control panel - Internet options (or: start Internet Explorer - Tools - Internet options). It doesn't matter that your default browser is not IE.
• Select Content tab.
  
• Click Certificates, select Trusted root certification authorities tab, and sort ascending by expiration date.
  
• Now delete all GTE CyberTrust certificates whose expiration date has passed.
• Click Close, then OK
• Restart Ovi suite.

If that didn't help: try removing all expired certificates:

A word of warning

Deleting all expired trusted root certs is not a good idea. You could end up with vital parts of the system not working, or unable to access some documents (especially if you have encryption turned on). Thing is, expired certs can still be valid for anything signed or encrypted before they expired.

Sources used:

• http://discussions.europe.nokia.com/t5/Nokia-Ovi-Suite/NOKIA-OVI-Suite-could-not-connect-to-the-nokia-account-server/m-p/878427
• http://betalabs.nokia.com/apps/nokia-ovi-suite-3-0-beta/bugreport/19143/ovi-sign-in-failed
• http://betalabs.nokia.com/node/5251

Paradoxically, the tremendous downpour of microarray results prevents a simple use of expression data. Therefore, we propose a thematic entry to public transcriptomes: you may for instance query a gene on a "Stem Cells page", where you will see the expression of your favorite gene across selected microarray experiments related to stem cell biology. This selection of samples can be customized at will among the 6462 samples currently present in the database.

Every transcriptome study results in the identification of lists of genes relevant to a given biological condition. In order to include this valuable information in any new query in the Amazonia! database, we indicate for each gene in which lists it is included. This is a straightforward and efficient way to synthesize hundreds of microarray publications.

A special feature of Amazonia! is the field of human stem cells, notably embryonic stem cells.