There’s still a ton of maintenance work needed, but at least it’s accessible again .

The blog went offline in early April 2021 – because the trusty physical server at home, built sometime before 2008 from off-the-shelf components, finally malfunctioned badly enough to not be fixable remotely over ssh.
(Or maybe it was still fixable, but at 13+ years old I thought it’s better not to fix anymore.)

It had previously survived (and recovered from) several hardware failures:

• (there might have been earlier failures that I no longer remember)
• PSU: after showing higher-than-normal deviations from standard voltages (+.3V, 5V, and 12V), the PSU died with a puff of smoke. It was replaced with a comparably cheap ATX PSU, that served fine for many more years.
• CPU fan failure: as the CPU heatsink was rather small, even with powersave CPU mode it was still getting too hot – so I had to shut it down and wait until I was able to replace the fan.
• OS disk: the server started with an old 320GB Seagate. When SMART data started deteriorating (unreadable/remapped sectors), I have swapped it out for a small and cheap 60GB Kingston SSD.
• Second CPU core: that server used a rather old dual-core AMD Athlon X2. I think it was old back when it was installed . At some point second core (#1) was showing 100% usage, and the server would restart within some minutes after booting. I am still surprised and impressed this was fixable remotely! Maybe the issue wasn’t too bad if the server could still boot and last for a few minutes. The fix was to disable the problematic core permanently from within Linux.

That chapter is over now.
Will the new chapter bring more regular posting?
Other, non-text content?…

We’ll see

There is a special font, called Sans Forgetica, designed to better retain the text that you read… Wow.

I can see how this may become abused – for example, for advertising

Anyway, you can download the font, and even a Chrome extension to show any text chunk in this Unforgettable font from the font’s website: http://sansforgetica.rmit/.

Here’s my blog URL for you to remember, hehe

But I always keep looking for new/improved tools, as right now I feel the best one does not exist…
(If the best one can exist at all – requirements and conditions change all the time, so there is no fixed perfect immovable target.)

I have been contemplating trying out the TSW methodology, but neither Keep nor Trello are quite there yet.
I ended up using Evernote; after recent management changes and actually trying to become profitable it may as well last long enough.

Everything was fine and calm until I have found workflowy yesterday.
In essence, it is very similar to the text-file-based system that I have been using for at least half a year.

Briefly, it is a web-based text editor on steroids, with possibly infinite nesting lists and seemingly full keyboard shortcuts control – no mouse needed.
I recommend that you try the demo – it seems to be fully functional, and there is no need to sign up.

This discovery made me read through pages and pages of this class of software tools.
Here is a very brief summary of my findings:

• org-mode for emacs: taking notes, organizing to-do lists and projects, writing structured text; I can definitely see the benefits – especially for tasks/projects management; however, for rich content – with attached/embedded files/images – this probably won’t work that well; see also orgzly;
• TagSpaces (also on github) is a tags-based files manager; cross-platform, offline, stores tags in filenames (in square brackets) – this is probably the least interfering/locking-in solution, the only things changing are file names; TagSpaces users are expected to synchronize the tag-controlled filesystem (which can be some specific directory tree) using 3rd-party tools such as [BT]Sync, SyncThing, Dropbox, Box, etc;
• many kinds of Evernote-like tools and wikis, that people also use to take notes; the two most mature and usable and feature-rich are probably Laverna (looks great, uses markdown, can be self-hosted, unclear how to search by multiple tags) and Paperwork (same thing with multiple tags); WizNote also deserves a mention; among other tools: git-backed magpie, yipgo, Marks, KeepNote, zim, etc;
• hamster-GTD is a (yet another?) variation of TSW/GTD – not a tool, but a system;
• Artificer (formerly Overlord S-RAMP), a system for any kind of interconnected, hierarchical data; see also Artificer GTD example;
• magnolial, workflowy clone (haven’t tried it yet); another clone is HackFlowy – trying its offline demo shows that only a basic list functionality is present.

Now that I think of it, TagSpaces is a neat idea…
Especially for photos – assigning tags actually updates filenames, which is great for photos.
And you can easily search by those tags later in TagSpaces.
In fact, TagSpaces looks very interesting for organizing lots of directory/file-based data.

Laverna looks quite exciting! And seems to be actively developed.
But there seems to be nothing quite comparable to WorkFlowy… Need to test it some more.

bogdan.org.ua:80 130.193.51.57 – - [09/Apr/2016:15:53:22 +0300] “GET /categories/programming?_SERVER[DOCUMENT_ROOT]=http://www.daedongfur.co.kr/shop/log/.logs/id1.txt HTTP/1.1″ 200 13158 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”

Client’s IP 130.193.51.57 does belong to Yandex network range.

So…

• Had Yandex started looking for vulnerabilities in the web-sites it scans?
• Does it only look for vulnerabilities in the .UA web-sites/domains?
• Does Yandex really use a Korean web-site to host malicious code?

In fact, there are more entries like that one, also from one of Yandex IPs:

bogdan.org.ua:80 130.193.51.25 – - [04/Apr/2016:00:14:22 +0300] “GET /categories/programming/page/5?_SERVER%5BDOCUMENT_ROOT%5D=http%3A%2F%2Fwww.daedongfur.co.kr%2Fshop%2Flog%2F.logs%2Fid1.txt HTTP/1.1″ 200 12607 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”
bogdan.org.ua:80 130.193.51.25 – - [04/Apr/2016:00:19:31 +0300] “GET /categories/programming/page/4?_SERVER%5BDOCUMENT_ROOT%5D=http%3A%2F%2Fwww.daedongfur.co.kr%2Fshop%2Flog%2F.logs%2Fid1.txt HTTP/1.1″ 200 12174 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”

I can see 3 explanations, and all of them are bad for Yandex:

• Yandex now belongs to KGB, and it does scan [.UA] web-sites for vulnerabilities;
• some/many of Yandex crawler servers are compromised, and are used by malicious 3rd parties;
• there was a public malicious link somewhere (???) to my blog, and Yandex blindly followed it.

This post is not intended to be an in-depth comparison, but should be sufficient to choose one of the three file formats for your purposes. For more format details and history simply follow the links provided. File formats are reviewed roughly in “historical” order.

PNG (Portable Network Graphics) was designed as GIF replacement.

• It is lossless.
• It is suitable for photos.
• PNG is more space-efficient in the case of images with many pixels of the same color, such as diagrams/plots (as compared to PGF and JPEG2000). However, PNG photos are almost always larger than lossless PGF/JPEG2000 photos (real photo example: 9.9 MB in PNG, 7.0 MB in JPEG 2000).
• PNG is fairly fast at (en|de)coding.
• PNG is widely supported by web-browsers, image editors, and other software.
• PNG uses CRCs internally for each data block, so if damage occurs only the damaged block(s) should be lost – theoretically. However, in practice, according to the Just One Bit paper (local copy), PNG is actually much less damage-resilient than JPEG 2000.

JPEG 2000 (JPEG 2000) was designed as JPEG replacement.

• It has both lossless and lossy modes. Lossy mode is “better” (perceptually at the same file size) than JPEG.
• Lossless mode is the smallest of all 3 file formats tested.
• JPEG 2000 is slower at (en|de)coding than PGF and PNG.
• JPEG 2000 has several associated ISO and other standards. Software support for JPEG 2000 is not as good as for PNG, but better than for PGF.
• JPEG 2000 has good bit errors resilience (local copy).

PGF (Progressive Graphics File) was also designed to replace/enhance JPEG.

• PGF uses the same wavelet-based method as JPEG 2000, so it shares a lot of features with JPEG 2000, including support for lossless and lossy modes, with lossy being “better” than JPEG.
• Lossless mode file is marginally larger than the same image in JPEG 2000 (real photo example: 7.0 MB in JPEG 2000, 7.4 MB in PGF).
• It is much faster at (en|de)coding than JPEG 2000 (this is major difference #1 of 2).
• Software support for PGF is not as good as for JPEG 2000 (this is major difference #2 of 2).
• Being under-appreciated, PGF doesn’t seem to have received any error resilience testing, yet. However, one should expect error resilience similar to that of JPEG 2000, because essentially the same compression method is used (albeit with different “parameters”, resulting in a speed/size trade-off/gain). It is not clear if PGF has any resilience features like JPEG 2000.

Fun fact #1 from the links above: best error resilience was observed for bitmap files.
Fun fact #2: after about 1% of data damage none of the compressed formats is able to reliably reproduce the original image.

I would be happy to use PGF for its speed, compression ratio, and features, but lacking software support is detrimental (in a self-reinforcing manner) to widespread adoption of PGF; being quite similar to the more popular (or better promoted) JPEG 2000, I do not know if PGF will gain sufficient traction to get e.g. browser support.

PNG has very good support, but produces noticeably larger files, and exhibits significantly lower error resilience than JPEG 2000.

So as of today my choice is JPEG 2000 (lossless, with resilience features).

Casio G-Shock

Looking for a replacement, I found that:

• Casio seems to no longer make affordable models with the same functionality and a metal band – only polymer;
• a similar used model from Casio (MTG-something) costs upwards of 70 EUR.

I was quite sad about that. Any survivalist-minded person can easily see why:

• Solar-powered, with integrated rechargeable battery lasting 8 years or more. On full charge, it can operate up to 6 (or 9?) months without further exposure to light. Note: there exist non-solar watches claiming up to 8-10 years off a single battery. And yes, mechanical watch is above any electronic competition in this regard.
• Digital display (for precise time readout) and functionality: stopwatch, timer, alarms, multiple timezones. Mechanical watches cannot match this.
• Precision: +/- 15 seconds a month.
• Radio time synchronization with atomic clocks (feature currently referred to as Waveceptor, but previously also known as Atomic). Modern Casio Waveceptor watches can receive signals from China, Japan, USA, and UK/Germany/Europe. I write UK/Germany/Europe, because these radio signals (depending on weather conditions) can travel thousands of kilometers. If the watch synchronizes successfully every night, then you are guaranteed to have +/- 1 second precision for the entire following day – neat!
• And, of course, shock/vibration protection, and 200m water resistance.
• Adding to the above, metal band should last way longer than any plastic or leather or fabric band.

This is why Casio G-Shocks are awesome
And no, I am in no way affiliated with Casio, and I do not get any personal benefits from writing this post.

Fortunately, my watch was found and returned to me a few weeks after it was lost – so yes, people are good, on average. It will easily serve me many more years. I may, though, consider buying a newer Casio model (with similar functionality) if the accumulator in this one loses capacity, because battery replacement may worsen waterproof properties… But that is still some years away from now.

Yesterday I got somewhat distracted, and thus found lbzip2 -

an independent, multi-threaded implementation of bzip2. It is commonly the fastest SMP (and uniprocessor) bzip2 compressor and decompressor

- as it says in the Debian package description. Is it really “commonly the fastest” one? How does it compare to pbzip2? Should I use lbzip2 instead of pbzip2?

This minor distraction had grown into a full-scale web-search and comparison, adding to the mix plzip (a parallel version of lzip), xz, and lrzip. After reading thousands of characters, all of these were put to a simple test: compressing an about 2 gigabyte FASTQ file with default options.

All the external links and benchmarks, as well as my own mini-benchmark results, are provided below.

The conclusion is that out of all the tested compressors lbzip2 is indeed the best one (for my practical use). It is only slightly better than the trusty pbzip2, which takes the second place. All the other compressors performed so poorly, that they do not get any place in my practical rating…

So, let us first ask internet wisdom/foolishness, if lbzip2 or pbzip2 is faster/better?

• this askubuntu question shows that lbzip2 is compressing faster (1:43) than pbzip2 (2:34)
• this nice benchmark also confirms that lbzip2 is indeed faster at compressing; lbzip2 also appears to use less RAM and a little bit less CPU during compression; during decompression, lbzip2 (reportedly) uses much more RAM. lbzip2 achieved at least as good (and even marginally better) compression ratios as pbzip2.
• lbzip2 github page and also this unrelated page both say that lbzip2 is fully cross-compatible with bzip2
• probably most importantly, lbzip2 github readme says that even bzip2-compressed archives get a decompression speedup (which is definitely not the case with pbzip2)
• lbzip2 also uses 100-900k blocks (900k by default)
• it is not clear if lbzip2 is somewhat less widely tested than pbzip2
• lbzip2‘s author has performed some testing (back in 2009, mind you!), and these were the most important results:
• lbzip2 is better when decompressing from a pipe, no matter the producer, and also when the compressed input coming from a regular file is single stream
• pbzip2 beats lbzip2 when the compressed input is coming from a regular file and is multi-stream (yes, pbzip2 can decompress even lbzip2′s compressed output faster than lbzip2 itself, when it’s coming from a regular file) note: if you check the vbsupport benchmark above, you’ll see that lbzip2 had probably fixed slight lagging behind pbzip2 for regular multi-stream files; this improvement is also confirmed by my testing

So, at least in theory lbzip2 is indeed better than pbzip2, even if only at faster decompression of bzip2-compressed files.

While looking for benchmarks, I’ve found this one (old but good), which highly praises lzop compressor. Apparently, lzop is noticeably faster than even gzip, and compresses only a little bit worse. However, I am not really interested in a faster gzip: I need something with much better compression, but still fast enough for multi-gigabyte files.

Next, I have stumbled upon lzip and plzip (.lz). What are these compressors?

• plzip is a parallel version of lzip, and fully lzip-compatible
• lzip is an LZMA compressor
• reading the documentation leaves an impression that [p]lzip achieves better compression, is slower, and needs much more RAM than competing compressors
• there is a special utility called lziprecover, which helps recover data from damaged lzip archives, by leveraging, on the one hand, CRC checksums of compressed blocks, and, on the other, multiple damaged copies of the archive (if available)
• from the official website:
  

  Lzip is a lossless data compressor with a user interface similar to the one of gzip or bzip2. Lzip is about as fast as gzip, compresses most files more than bzip2, and is better than both from a data recovery perspective.

• default “member” (compressed block/chunk) size is 4 petabytes, but can be set to a lower value (minimal 100kb), mimicking bzip2′s chunk size
• supports multiple, independent volumes (loosing one volume will still allow recovering data from all other volumes)
• with multiple cores, plzip creates multi-member files by default (but it is not clear, what is the size of these members? Default is said to be twice the dictionary size, but default for dictionary size is not specified in the manual – so lzip/plzip seem to require compression level -1…-9 specification)
• here lzip compresses a little bit better than xz without the --extreme option
• (l|p)bzip2 should still be faster than either lzip or xz
• I started mentioning xz, because lzip and xz (at least historically) are competing LZMA-based compressors
• a 1 year old opinion makes the following statements about lzip:
• lzip is a marginal archiver with no real benefits since the appearance of xz (note: xz is a successor of lzma-utils)
• xz is more popular, more widely accepted
• xz has a community, while lzip has 1 author
• performance of xz and lzip is comparable
• xz has more features
• but lzip does indeed have a recovery utility that xz doesn’t

That doesn’t really tell us much on how plzip/lzip compare to, say, pbzip2. But before performance, let us pay some more attention to long-term storage features of lzip:

The lzip file format is designed for data sharing and long-term archiving, taking into account both data integrity and decoder availability:

• The lzip format provides very safe integrity checking and some data recovery means. The lziprecover program can repair bit-flip errors (one of the most common forms of data corruption) in lzip files, and provides data recovery capabilities, including error-checked merging of damaged copies of a file.
• The lzip format is as simple as possible (but not simpler). The lzip manual provides the code of a simple decompressor along with a detailed explanation of how it works, so that with the only help of the lzip manual it would be possible for a digital archaeologist to extract the data from a lzip file long after quantum computers eventually render LZMA obsolete.
• Additionally, the lzip reference implementation is copylefted, which guarantees that it will remain free forever.

(I really liked the part about the digital archaeologist! And the copyleft, to a lesser extent.)

Looks really attractive! Because what I am using compressors for is, essentially, longer-term archiving, with unpredictable needs to sometimes decompress some of the files. And, of course, storage media will fail fully or partially, so recovering is important, too. But what is this xz compressor?.. I’ve seen it before, in the contexts with words “overtake the world” or similar…

xz

• much more complex file format than lzip, but maybe it has some benefits for client programs and/or recovery?
• supports integrity checks and multiple compressed blocks
• according to this post from 2012, xz (single-threaded) both compressed and decompressed much faster than lzip… and lrzip (depends on settings, of course)
• lzip is older than xz, and was better than xz predecessor – lzma-utils
• xz is adopted by some linux distributions and software projects for package compression
• xz does not seem to have an equivalent of lziprecover
• tar supports both --lzip and --xz, also with --auto-compress

This hasn’t really added any clarity, has it? Moreover, we now have one more unknown – the lrzip compressor. lrzip is a redundancy compressor with LZO, gzip, bzip2, ZPAQ and LZMA back-ends. It is highly efficient for highly redundant data, even if redundancies are separated with long stretches of other data. (FASTQ files are fairly redundant, though bzip2 seems to utilize that fairly well already; can lrzip do better?)

However, what if a part of the archive is damaged? How much information is lost then? Is it at all possible to recover some of the data from damaged .lrz archives?
Author’s benchmarks showcase how good lrzip is at redundant data compression (although lrzip is multithreaded, so comparison in the benchmark to non-multithreaded algorithm implementations is not quite correct…). Damaged archive recovery concerns would have prevented me from using lrzip anyway, but I was really interested if a “long-range redundancy” compressor can do better than usual, “short-range redundancy” compressors.

My testing setup

• Debian testing 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1 (2015-03-01) x86_64 GNU/Linux
• Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 physical cores with HT enabled: 8 hardware threads)
• 16GiB RAM
• test file name: test.fastq
• test file size: 2 223 860 346 bytes (a little over 2 gigabytes)
• test file was copied once to RAM-mounted /tmp, to exclude any I/O bottleneck effects on compression speeds
• bzip2: 1.0.6
• lbzip2: 2.5
• pbzip2: 1.1.9
• xz: 5.1.0alpha
• plzip: 1.2
• lrzip: 0.616
• command execution time and maximal process RSS memory were measured with /usr/bin/time -f '%C: %e s, %M Kb' compressor arguments (note: this is not bash’s built-in time); please note that memory measurement can be incorrect for multithreaded compressors

Below come testing results. I have not put them into a single table, but I do comment the results in a few places. Entire testing followed this pattern:

• compress test.fastq, deleting the original
• test compressed archive (note: this was done only for some compressors, not all)
• decompress archive back to test.fastq, delete archive
• if 3 previous steps are fast enough: repeat 1-2 more times (but only show the best result below); otherwise continue
• repeat with the next compressor

bzip2: 309 159 275 bytes
bzip2 was used as a baseline, to highlight speed benefits of both lbzip2 and pbzip2.

test.fastq: 7.193:1, 1.112 bits/byte, 86.10% saved, 2223860346 in, 309159275 out.
bzip2 -v test.fastq: 190.63 s, 7608 Kb
bzip2 -v -d test.fastq.bz2: 51.58 s, 4620 Kb

Bzip2 is neither particularly slow, nor particularly fast. It also seems to have modest memory requirements.

pbzip2: 310 462 610 bytes
pbzip2 is the currently used reference. For any other compressor to become a successor of pbzip2, that other compressor must be either a little faster (while compressing as good as pbzip2), or a little better compressor (while being as fast as pbzip2), or both. Note that compressed file size is only a tiny bit larger than with bzip2.

“test.fastq.bz2″: compression ratio is 1:7.163, space savings is 86.04%
pbzip2 -v test.fastq: 46.22 s, 67436 Kb
pbzip2 -dv test.fastq.bz2: 19.80 s, 46672 Kb

Interestingly, pbzip2 --test uses 1 thread only (but also consumes only 6MB RAM), resulting in decompression times similar to those of bzip2. lbzip2 uses all 8 threads also during testing.

lbzip2: 311 040 543 bytes

lbzip2: compressing “test.fastq” to “test.fastq.bz2″
lbzip2: “test.fastq”: compression ratio is 1:7.150, space savings is 86.01%
lbzip2 -v test.fastq: 22.67 s, 49812 Kb

lbzip2: decompressing “test.fastq.bz2″ to “test.fastq”
lbzip2: “test.fastq.bz2″: compression ratio is 1:7.150, space savings is 86.01%
lbzip2 -vd test.fastq.bz2: 18.86 s, 46652 Kb

I repeated pbzip2 and lbzip2 tests several times, and it was always that lbzip2 compressed this same file about twice as fast… Wow! Decompression speed is about the same, compressed file size is marginally larger than with pbzip2. Overall, lbzip2 does look like a new drop-in replacement of bzip2/pbzip2 for me.

xz -0 ––threads=8: 517 967 372 bytes
I would call this one major test disappointment. Default setting, -6, was way too slow (estimated 28 minutes to compress!!!). Even the fastest -0 setting was still too slow! And here’s one of the reasons, straight from the xz man page:

Multithreaded compression and decompression are not implemented yet, so this option has no effect for now. As of writing (2010-09-27), it hasn’t been decided if threads will be used by default on multicore systems once support for threading has been implemented.

Also, I forgot to use the --block-size=900k option, but that seems to be of no concern with such results:

100 % 492.5 MiB / 2,120.8 MiB = 0.232 18 MiB/s 1:59
xz -0 -v test.fastq: 119.25 s, 4780 Kb
xz ––test ––verbose ––threads=8 test.fastq.xz: 36.00 s, 2568 Kb
100 % 492.5 MiB / 2,120.8 MiB = 0.232 58 MiB/s 0:36
xz -d -v test.fastq.xz: 36.54 s, 2500 Kb

xz -0 was both slower and had significantly worse compression when compared to lbzip2 and pbzip2. xz -0 was faster than good old bzip2, but had significantly worse compression… Really, major test disappointment.

plzip: between 407 696 562 and 498 708 539 bytes
One more major test disappointment. (Or am I somehow using these compressors in a wrong way?…) I haven’t found a way to set block/member size (for lzip, that would be the -b option). Default speed setting -6 was also way too slow, but settings -1 to -3 were comparable to pbzip2, so I did all three.

plzip -1: 498 708 539 bytes

test.fastq: 4.459:1, 1.794 bits/byte, 77.57% saved, 2223860346 in, 498708539 out.
plzip -1 ––verbose ––threads=8 test.fastq: 30.27 s, 126360 Kb (this seems to be per-thread memory…)
plzip ––test ––verbose ––threads=8 test.fastq.lz: 6.86 s, 11640 Kb
plzip -d ––verbose ––threads=8 test.fastq.lz: 7.24 s, 11644 Kb

Compression speed and ratio: both worse than lbzip2. But the fastest testing and decompression so far.

plzip -2: 456 301 558 bytes

test.fastq: 4.874:1, 1.641 bits/byte, 79.48% saved, 2223860346 in, 456301558 out.
plzip -2 ––verbose ––threads=8 test.fastq: 38.81 s, 193416 Kb
plzip ––test ––verbose ––threads=8 test.fastq.lz: 6.26 s, 14828 Kb
plzip -d ––verbose ––threads=8 test.fastq.lz: 6.38 s, 14736 Kb

Compression time worse than lbzip2, a little better than pbzip2, but compression ratio worse than any one of these. But even faster testing and decompression.

plzip -3: 407 696 562 bytes

test.fastq: 5.455:1, 1.467 bits/byte, 81.67% saved, 2223860346 in, 407696562 out.
plzip -3 ––verbose ––threads=8 test.fastq: 63.74 s, 245756 Kb
plzip ––test ––verbose ––threads=8 test.fastq.lz: 5.82 s, 18936 Kb
plzip -d ––verbose ––threads=8 test.fastq.lz: 6.10 s, 18944 Kb

Even faster testing and decompression! But compression ratio and speed are still worse than lbzip2 and pbzip2.

And the final contestant, lrzip! All 5 back-ends were tested: LZO, gzip, bzip2, LZMA, ZPAQ.

lrzip has several peculiarities, which hinder its use as a drop-in replacement for, say, bzip2. Most importantly, when a file is compressed, it is not deleted, unless a -D options is specified. Unlike pbzip2 and lbzip2, which use all available CPUs/cores by default, lrzip only uses 2 by default (-p 8 in the results below requests use of 8 cores). Another unusual feature is that during testing a file is uncompressed to a storage medium, and then deleted; almost all the other compressors only verify the decompressed data stream, which is then immediately discarded and never written to storage medium. Related feature is a -c option, which performs file verification after decompression by reading the decompressed file from storage medium and comparing it to the decompressed stream. lrzip also stores MD5 hashes of data, and allows verifying these. lrzip comes with several helper scripts – for example, one which allows tarballing and lrzipping a chosen directory in a single command. Actually, lrzip is more of an archive utility, and not just a compressor.

lrzip -D -p 8: 334 504 383 bytes
In this default (LZMA) mode, lrzip starts with 1 thread, but eventually uses more and more cores (though never all 8, or I haven’t noticed this). Decompressing seems to use more threads, but that also depends on the back-end used (the slower it is – the more threads will be used, e.g. ZPAQ versus LZO).

test.fastq – Compression Ratio: 6.648. Average Compression Speed: 3.113MB/s.
Total time: 00:11:21.85
lrzip -D -p 8 test.fastq: 681.84 s, 3331080 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 124.706MB/s
[OK] – 2223860346 bytes
Total time: 00:00:17.13
lrzip -t -p 8 test.fastq.lrz: 17.21 s, 2567608 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 117.778MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:17.59
lrzip -d -p 8 -D test.fastq.lrz: 17.67 s, 2567664 Kb

In the default LZMA mode, lrzip is significantly slower than even bzip2, and has somewhat worse compression ratio. Yes, this is the 3rd major test disappointment.

gzip back-end: lrzip -g -L 9 -D -p 8: 430 013 769 bytes
Despite specifying -p 8, lrzip mostly operates in 1 thread, and only sometimes in 2 (probably invokes gzip library). Testing is also done with 1 thread only, but is very fast (but slower than plzip). The -L 9 option is supposed to be translated into -9 for gzip; as this normally has nearly no effect, it wasn’t used in the following lrzip tests.

test.fastq – Compression Ratio: 5.172. Average Compression Speed: 0.704MB/s.
Total time: 00:50:11.34
lrzip -p 8 -g -L 9 -D test.fastq: 3011.34 s, 2745520 Kb

100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 163.077MB/s
[OK] – 2223860346 bytes
Total time: 00:00:12.71
lrzip -t -p 8 test.fastq.lrz: 12.79 s, 2577632 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 163.077MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:12.88
lrzip -d -p 8 -D test.fastq.lrz: 12.95 s, 2577728 Kb

And again, compression speed and ratio are worse than for bzip2…

LZO back-end: lrzip -l -D -p 8: 766 520 776 bytes

test.fastq – Compression Ratio: 2.901. Average Compression Speed: 4.690MB/s.
Total time: 00:07:32.89
lrzip -l -D -p 8 test.fastq: 452.88 s, 2714452 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 212.000MB/s
[OK] – 2223860346 bytes
Total time: 00:00:10.58
lrzip -t -p 8 test.fastq.lrz: 10.66 s, 2582516 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 192.727MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:11.32
lrzip -d -p 8 -D test.fastq.lrz: 11.39 s, 2582504 Kb

No comments.

bzip2 back-end: lrzip -b -D -p 8: 353 473 476 bytes

test.fastq – Compression Ratio: 6.291. Average Compression Speed: 4.473MB/s.
Total time: 00:07:53.95
lrzip -b -D -p 8 test.fastq: 473.94 s, 2781104 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 68.387MB/s
[OK] – 2223860346 bytes
Total time: 00:00:30.69
lrzip -t -p 8 test.fastq.lrz: 30.77 s, 2583156 Kb

Decompressing…
100% 2120.84 / 2120.84 MB
Average DeCompression Speed: 66.250MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:00:31.92
lrzip -d -p 8 -D test.fastq.lrz: 32.00 s, 2583108 Kb

Hadn’t I done all of these simple tests myself, by now I’d think that this test was rigged to show how good pbzip2 and lbzip2 are at compressing FASTQ files

ZPAQ back-end: lrzip -z -D -p 8: 292 380 439 bytes

test.fastq – Compression Ratio: 7.606. Average Compression Speed: 2.804MB/s.% 7:100%
Total time: 00:12:36.51
lrzip -z -D -p 8 test.fastq: 756.51 s, 3585740 Kb

Decompressing…
100% 2120.84 / 2120.84 MB 1:100% 2:100% 3:100% 4:100% 5:100% 6:100% 7:100%
Average DeCompression Speed: 3.970MB/s
[OK] – 2223860346 bytes
Total time: 00:08:54.57
lrzip -t -p 8 test.fastq.lrz: 534.65 s, 2583424 Kb

Decompressing…
100% 2120.84 / 2120.84 MB 1:100% 2:100% 3:100% 4:100% 5:100% 6:100% 7:100%
Average DeCompression Speed: 3.759MB/s
Output filename is: test.fastq: [OK] – 2223860346 bytes
Total time: 00:09:24.27
lrzip -d -p 8 -D test.fastq.lrz: 564.36 s, 2583460 Kb

Finally!!! We have compression better than bzip2! But it is also much slower than bzip2 (and some 12 times slower than pbzip2), so not really an option. Alas. And decompression time is the worst in the test – almost 10 minutes for what plzip does in under 7 seconds! (I do realize that compression ratio is also different – but not that much.) I wonder if slow lrzip speeds have anything to do with test.fastq being effectively in RAM? I do not know if there are any performance penalties to mmaping a file which is already on a RAM-mounted partition.

The test.fastq file that I’ve used was somehow really hard for the tested compressors to tackle as fast and as good as lbzip2 and pbzip2 could…

Questions? Comments? Improvements, including plots of these figures? Comment below.

• Google Keep, an awesome note-taking and to-do lists application with a really good web-interface, and free;
• Trello, convenient lists/projects/tasks management platform (especially for group work), and free;
• Google Calendar, the de facto calendar standard for Android phones, and free;
• my A5 format weekly paper planner, and… the only not free component.

It is easy to see that I am using too many tools.

In an effort to use less tools, and also to try some of the features of TSW, I’ve performed a brief search for GTD/TSW-compatible Android apps.

TSW website is built around the Evernote app. However, I am not sure if this would be a good solution for me, as I have been already using Evernote since several years for longer-term note-keeping, and thus already have a bunch of notepads, notes, and tags there. Moreover, Evernote’s website mentions something about “offline notes” in the Premium (non-free) tier for mobile apps; this hints at the requirement to have internet connectivity to be able to work with TSW+Evernote efficiently through the day.

Oh, before I forget: all the 4 tools that I am using have their purpose, with overlap between Keep and Trello.
My A5 format paper planner (weekview compact 2015) is not a simple weekly planner; it has a structure that stimulates goal-oriented planning.
More specifically, it provides means to plan:

• the entire life, by specifying (succinct) goals in several categories (personal, work, family, social, and some others);
• the next several years (there is enough space for just a few keywords for each year);
• the entire current year (as an overview or a list of goals, without too many details);
• each quarter of the current year (with more details: goals/tasks can have specific days or date ranges assigned, and have 3 priorities);
• each week has 3 priorities for what you would like to accomplish;
• there are also other important, useful, and well-designed elements, all with high attention to details.

I mostly use the paper planner for quarter-level goals and tasks.

Trello is my primary project and task management tool, both for work and personal matters (using different boards).
It also really simplifies my weekly reports: I only have to check the Done list of the primary/project work board,
and show it to my supervisor – which (showing/sharing) is also easy with Trello.

I’m using Calendar for all the events which have specific dates/times, like meetings, deadlines, celebrations, etc.

Finally, I’ve started using Keep not that long ago as a to-do list and note-taking application. It is extremely easy and quick to use, which explains this new adoption. I use it mostly as a short-term buffer for quick (shorter than 2 hours) tasks. I have 3 separate lists: home, work, and shopping The only component which is missing if I want to use TSW is tagging of individual checklist items, together with tags search. Other than that, Google Keep is plain perfect.

The first app I had a look at was DGT GTD (alpha):

• has “@-contexts”;
• has tags;
• has search for arbitrary tag combinations (both AND and OR logic);
• no web interface, uses toodledo/dropbox/ftp for sync;
• web-interface might be available through toodledo (which has its own limitations, see below);
• overall: alpha, no easy-to-use web-interface, unclear future… though otherwise seems good.

Next, I had a look at Toodledo:

• has a free, but (seemingly quite strongly) limited version; in addition, it felt
• somehow not easy to register, thus I have not tried it.

Next was MyLifeOrganized:

• way too commercial all over – you seem to need many pieces of (paid) software, (paid) cloud sync, (paid) plans…
• no web-interface and no Linux support, only Win/Mac/iOS/Android, thus have not tried this one as well.

I was leaving better contestants (like RTM, Remember The Milk) for later:

• nice, convenient, light, keyboard-friendly web-interface;
• free version syncs with web, but only once every 24 hours;
• has locations (GPS-based) and tags;
• has inbox, personal, work, study, sent pre-defined lists of tasks; you can define your own (and delete 3 of the pre-defined, if you wish);
• can search for multiple tags using brackets, logical operators, and multiple per-task attribute filters (like timeEstimate, dueDate, etc – many of these!);
• can save searches as smart lists (at least in the Android app);
• tasks cannot be ordered manually, they can only be sorted by priority, due date, or name.

I can see myself using RTM, which feels like a quality tasks-management environment. Syncing once every 24 hours is the only free version limitation that I am sensitive to, because I tend to use web-versions (Trello, Keep) while at the computer. If you are using for planning your phone only, then RTM might be a very good fit for you.

Another detail which I find inconvenient is the inability to manually sort tasks. As I know from using Keep, manually sorting smaller tasks into their logical order by dragging is quick and easy. This RTM drawback could be worked around by sorting task list by name, and devoting the first 2 characters of the task text to its number (e.g. ’06 start scaffolding’). I am still unsure about RTM.

Given the failure of the new contestants to fit my needs, I also had a quick formal look at the tools I am already using.

Evernote:

• recommended by TSW website;
• has tags and saved tag searches;
• not sure if it keeps all notes available offline – it likely needs connection to function; it may keep the most recent notes offline, though – still have to test this;
• not exactly a to-do list, thus (much more?) cumbersome to use than Google Keep (again, this wasn’t tested yet – consider this a prejudice);
• free version has a 60 MB/month data upload limit, which should be more than enough for tasks management.

I am going to try Evernote with TSW, and see if that works good enough. I’ll update the post with the results.

Google Keep:

• very easy and convenient to use checklists;
• keeps all tasks local and always available; works offline, syncs when you have connection;
• has an efficient, quick-to-use web-interface;
• does not have tagging (only colors for notes);
• lightweight in terms of size and resources needed.

I’ll keep using keep, even if some other app becomes my primary for tasks management. It is simply too good not to use.

Trello:

• has a fairly convenient web-interface (though more complicated than Keep because of more features);
• allows easy collaboration;
• supports multiple boards, containing task lists, containing tasks, containing checklists and other elements, which all together enable fairly complex project management;
• phone app caches tasks/boards that you access while online, and can later show those while offline, but
• phone app does not allow changes while offline – you must have connection for the changes to have effect;
• has tags (labels), but these are board-specific, so it is impossible to get a flat list of all tasks from all boards filtered by some labels/criteria.

Right now I have tons of tasks in Trello, so I am not going to abandon it any time soon (unless I find a perfect alternative solution). I have already seen recipes online to adapt Trello to TSW/GTD use. This will not fix the necessity for internet connection for the app to work, though. It is also quite possible that tags + global flat list of all tasks from all boards might get introduced as new features into Trello, as it is developing dynamically and new features do get added quite often… Maybe I should leave a feature request for the developers, together with a thank-you for their excellent product.

That’s it for now, I’ll update after some more app testing.

Update 1: Trello can be quite convenient as a general GTD-like (but not quite TSW-like) app. I’ve set up a separate GTD board, with lists inbox, now, next, later, 8 project/watching/reading lists, 2 goal lists (one for the current year, one with general goals), some day, contemplate (no clear decision if an item has to be done at all), waiting, done, and discarded (something from any of the other lists which is [no longer] worth doing).

Update 2: Google Keep now has labels (aka tags)! (It now also has recurring reminders, which is cool as well.) Tags seemed to be the only thing keeping (no pun intended) Keep from being a perfectly simple and lightweight GTD/TSW app! Or so I thought. Unfortunately, there does not appear to be any way to search by several labels right now. You can search by note colors, and can select a single label to list all notes that have it, but no multiple labels… One last step missing to perfection?

Leave comments if some of my statements seem wrong, or if you know a solution which would be capable of satisfying my needs

I buy a lot on amazon.de – this is easy and convenient. Most of the time it also works good, with items arriving as promised or (mostly) earlier. Sometimes there are minor problems, but nothing worth remembering or mentioning. However, my latest order, “delivered” by Hermes Paketversand, was such a spectacular failure, that it deserves a blog post.

My item was expected to arrive on Tuesday, the 5th of August. In the evening of that day we found an important message from Hermes in our mailbox, saying that We were unable to deliver your package, we come again tomorrow, on the 6th of August. Hmmm. That is strange. My wife was at home in the 1st half of the day, when Hermes, supposedly, was delivering my package (I learned about their delivery time later from delivery tracking page). This is what I would call a minor problem, not really an issue. Maybe, for some reason, the door bell wasn’t heard, and our delivery did not reach us. But it comes next day, right? Right?…

There was absolutely nothing – not even a note – delivered in the following 4 days. On Saturday I started worrying. I checked Amazon package delivery status (detailed), which said that we have missed 3 (!) delivery attempts between 9:00 and 13:00 on August 5-7. I could have believed one miss, but three in a row, with my wife at home… that is extremely strange, to say the least. Anyway, I was ready to forgive this, as long as I get the package.

Previous packages, when their delivery failed, were usually left at the nearest Paketshop (pickup place), which is about 7 minutes walking from where I live. There were absolutely no notes left about the whereabouts of my package, so I assumed it is in that Paketshop, and walked there. But the package was not there.

I then explored the Hermes Paketversand website, and did find a form to ask about the location of the package. I had thoroughly, as I usually do, filled in the form, including my email and phone number. I have explained the absence of any notification about the package whereabouts, and asked to deliver the package to that Paketshop not far from me. I have never received any response to that form.

To be extra safe, I also enabled Amazon‘s SMS notifications about delivery status changes. Never needed them before, but this was already a special case.

After a day or two, I received an SMS from Amazon, saying that delivery requires my interaction to proceed. On the amazon.de I was told that delivery failed and I need to contact Hermes to resolve the issue. Amazon does give the phone number. Interestingly, on the Hermes Paketversand website there are no phone numbers, only the (apparently ignored or dysfunctional) feedback form.

The phone number costs up to 0.6 EUR to call from cell phone. I had already paid for shipping when ordering, so it is really surprising that I have to pay again to ask where the shiny paradise my package is.

Anyway, it was a small gift for a friend, so I called and arranged the delivery to that Paketshop I like.

I was already deeply unsatisfied with Hermes, but had this failure ended here, I would have never written this piece.

On the 14th of August I get 2 SMSs from Amamzon saying that my package is ready to be picked up. “Hurray!”, I think. Next day was a holiday, so I planned the pickup for early Saturday. After getting the package I intended to have a haircut, and then depart for my vacations.

The package was not there. Not in the Paketshop I was told it will arrive to, making a toll phone call.

I had no choice other than to call Hermes again. (Hermes seems to enjoy leaving customers no choice.) This time, I was told that the package is in a different Paketshop. I kept, in vain, asking “But why is it in a different Paketshop?…”, only to be told again that I can take it there. In response to my complaint that I go on vacations today and, thus, may have no time to get the package, I was informed that the package will then be returned to the sender, and was suggested to print some proxy/trust form from Hermes website for someone else to get the package. Well, thanks! 2 hours before I leave, and I need to think about some stupid form, and where to print it, and who could get the package – all because of Hermes delivering the package to a wrong Paketshop.

I ended up walking to that other Paketshop (~3km further from where I had already been) and getting the package.

Obviously, I no longer had enough time for a haircut. I was also quite angry and destabilized – just what I needed before a day-long trip. Thanks, Hermes.

I will be contacting Amazon and the specific shop to report this failure of Hermes, and to ask to never use Hermes for my deliveries.

A friend of mine had similar experiences with Hermes, so I guess this is not a one in a thousand failure.

I have three simple questions to Hermes, if it ever bothers to answer them:

1. Why do I have to seek contact with Hermes to find my package? Why Hermes delivery left absolutely no notifications about the package whereabouts when it failed to deliver for the 3rd time?
2. Why was there absolutely no response to my feedback form?
3. Why was the package delivered to a different Paketshop? Why wasn’t I notified about the change?

I have come to a conclusion never to use Hermes, even if the alternative is more expensive.

I wish you timely and successful deliveries.

This information is not available in one piece, but there are some of the pieces at the following URLs:

A mistery hidden in the Genesis Block
Who is Satoshi Nakamoto?
Four years and $100 million later, Bitcoin’s mysterious creator remains anonymous
Ted Nelson Says That Bitcoin’s Satoshi Nakamoto Is Shinichi Mochizuki
Where in the World is Satoshi Nakamoto?
The Rise and Fall of Bitcoin

Below is a huge graphical brief history of Bitcoin, including the continued growth of its exchange rates.

you can easily fix this problem by going to Control Panel, clicking on Mail, then Show Profiles button.
Remove everything that’s there. Now start outlook again.

Note: removing all the mail profiles will disable your Novell Groupwise client.
If you still want to use non-Outlook email profiles, then the better solution is to manually create a new mail profile for Outlook.

Source: CNET forums.

With automated spam, automated spam-fighting systems might be the only choice. Sending rightfully angry emails to ISPs to notify about their customers violating service agreements is probably a waste of effort (something tells me most of these complaints end up in the trash folder, or even in the… spam folder). However, I get a feeling that some spam is not automated – it appears to have been actually prepared and sent by a human. (Alternatively, spammers behind those spams simply have better software.) Anyway, some spams seem to contain valid contact data of the advertized entity – like an email.

The resulting idea is very simple and was probably already implemented somewhere by someone: simply publish online contact emails of the entities which, apparently, had chosen spam as the primary means of advertising. These emails will be sooner or later harvested by spammers, added to spam databases, and will start getting progressively more spam.

There are a few drawbacks to this approach:

• knowing spam-collection points enables “black PR”-like mass-mailings in the name of one’s competitor, double-hurting the innocents; I do not see a clear method of preventing this, other than by concealing spam collection methods;
• human intelligence is required to identify if the contained email truly belongs to the advertised entity; this is fairly time-consuming, especially when scaled up; a possible solution (with its own problems) would be to build an online gateway for submitting curated spam samples, thus distributing the workload to all the participating volunteers;
• the next logical step is actually harvesting and then publishing all the emails from the advertised website;
• the biggest drawback, however, is low efficiency of this approach; increasing spam percentage will only be a mild nuisance, which isn’t likely to propagate high enough to affect spam-deciders; also, indirectly spamming someone’s mailbox will result in the loss of time, which could have been otherwise used for facebook and other important activities

What do you think? Should such a method be used?

Below I provide a few sample records from real spam comments, which had true-looking emails. I’m including some extra meta-data. Ideally, this should be stored in some kind of a database.

Submitted on 2013/11/13 at 15:23 GMT
Author : Виктор (IP: 95.134.110.37 , 37-110-134-95.pool.ukrtel.net)
E-mail : aionind@yandex.ru
E-mail : sale@aion-industry.ru
E-mail : info@aion-industry.ru
Submitted on 2013/11/26 at 8:53 GMT
Author : Виктор (IP: 95.134.146.235 , 235-146-134-95.pool.ukrtel.net)
E-mail : kvazargr@yandex.ru
E-mail : info@kvazar-gr.ru
Submitted on 2013/11/28 at 7:24 GMT
Author : Виктор (IP: 95.134.117.155 , 155-117-134-95.pool.ukrtel.net)
E-mail : relevater@yandex.ru
E-mail : info@relevate.ru
E-mail : support@relevate.ru
E-mail : billing@relevate.ru

There’s definitely a need for a public database, API keys, and quorum algorithms…

Author : casinoworka (IP: 91.207.4.201 , 201.4.207.91.unknown.SteepHost.Net)
E-mail : pharmacywork7777777@gmail.com
E-mail : info@prowessmedical.com

This is just a “Go to” dialog of the really good Notepad++ editor.

I do not really recollect anything that large. Well, not counting dropbox’s 4-hour window of “any password fits all accounts”, but that was different.

Here are some of the news items about this seriously large hacking incident:

• NYTimes (Dec. 25, 2011)
• WSJ (Dec. 25, 2011)
• CNN (Dec. 25, 2011)
• relatively above-average write-up from Wired (Dec. 26, 2011)
• ABCNews (Dec. 26, 2011)
• The Register (Jan. 3, 2012)

Here come more technical reports:

• short pastebin document, supposedly by the hackers
• cryptome keeps track of the data being removed from the internet
• a 1MB report by the hackers
• TheTechGerald has some analysis of the leaked stratfor passwords (Jan. 2, 2012)

TheTechGerald’s analysis linked to above got my attention. Unfortunately, a while ago I’ve subscribed to stratfor’s “free intelligence mailing list”, and was wondering if my account information is now publicly available. I was the most worried about the password I’ve used to subscribe, because of the risk of using the same password somewhere else.

Unlike TheTechGerald, I haven’t used any dictionaries – just the default configuration of a well-known tool for finding weak passwords. Within a single hour, ~100k passwords were decrypted (~12% of all). Till the end of the day, ~50k more passwords were decrypted (totalling 17.4% of 860k). At this point my password was still safe, and I’ve found a way to verify that it is not used anywhere else, so I’ve aborted further decryption.

There are a few simple conclusions:

• anybody who had a stratfor account must verify that he/she isn’t using that password anywhere else, because if 1 PC can get 17% of all the passwords in less than a day, it is only a matter of short time until all the leaked passwords will be decrypted and made publicly available in various “md5 decryption databases”
• system owners should run periodic screenings for weak passwords (and implement policies to prevent creating obviously weak passwords from the very beginning)
• md5 is very fast to decrypt/bruteforce – a much slower hashing function wouldn’t hurt; also, using a more complex hashing approach, maybe even with a closed-source shared library, could help
• single-factor authentication (password-based) is likely to get replaced with 2-factor authentication in the nearest future
• one may enjoy increased personal data safety by using throw-away passwords in conjunction with antispam mailboxes like spam.la and mailinator.com (at least 1600 users – 0.186% – did use these services).

Fortunately, the top 10 passwords (by their counts) were exclusively “throw-away”, and added up to ~10% of the decrypted passwords. (I’m not showing any, as that would unnecessarily simplify further decryption – maybe thetechgerald should have also been more vague about actual passwords.)

Sooner or later this significant-size corpus of real-life passwords will find its way (after decryption by those who would actually use leaked passwords to gain unauthorized access) into various wordlists and wordlist mutation rules, making it even easier to decrypt any future leaks. This is where 2-factor authentication will, hopefully, come in handy to protect against similar leaks.

I wonder if I should put up a page “Check if my password was among those 860k”, to help people easily identify if they should change theirs – not even necessarily being a Stratfor subscriber. Unless similar pages/services had already been put up by others.

It is also unclear what will the future of Stratfor be, taking into account that their website is still dysfunctional.

It is sad to see Drupal (stratfor.com’s CMS) involved here. However, I have no idea if their installation was up to date, and if their website was the point of entry. The hacklog suggests that attackers somehow obtained the password of one of the system administrators, and then used it for SSH access, which would save Drupal’s face (Drupal’s security record to date was pretty reassuring).

The original Ukrainian text tells the tale of a swallow flying into a household to proclaim the plentiful and bountiful year that the family will have. The title shchedryk is derived from the Ukrainian word for “bountiful”. This follows a tradition of praising the hosts of festivities in the songs during those festivities, or when coming to get sweets, small money bills or presents in exchange for nice singing by a group of children.

English text was written separately, and is copyrighted.

All the derived music uses the original’s four-note pattern by Mykola Leontovych. Folk song/chant was the basis for Leontovych’s work on this piece. I believe the original song had a similar musical (vocal) pattern, and that “ostinato” figure of music was already present in the song, so Leontovych’s work was probably to smooth out any uneven moments, and formalize the music in notes. Citing wikipedia article, “ostinato motif, a repeated four-note pattern within the range of a minor third is thought to be of prehistoric origins”.

HandBrake is a very high-quality piece of software – next time you need recoding something into H.264/MPEG-4 (using MKV or MP4 containers) – try HandBrake. It easily saturated all my CPU cores – which I failed to achieve with ffmpeg, which even with threads=8 was only saturating 2 cores.

Attached to this post are 2 profiles for recoding movies for Nokia E71. The “_best” profile has exhaustive motion detection, otherwise is identical to the base profile.
E71.plist
E71_best.plist

Related:

• x264 ffmpeg mapping and options guide
• ffmpeg audio/video encoding cheat sheet

There is an excellent, in-depth, well-argumented article by Geoff Chappell on the issue. Highly recommended in its entirety to those who want a complete understanding (additional side-reading and facts verification might be necessary).

A single citation to get you started:

There is already on the Internet and elsewhere an awful lot of rubbish to read about this question. Hardly any of it would be worth citing even if I didn’t want to spare the authors the embarrassment. A surprising number of people who claim some sort of attention as expert commentators would have you believe that using more than 4GB of memory is mathematically impossible for any 32-bit operating system because 2 to the power of 32 is 4G and a 32-bit register can’t form an address above 4GB. If nothing else, these experts don’t know enough history: 2 to the 16 is only 64K and yet the wealth of Microsoft is founded on a 16-bit operating system that from its very first version was designed to use 640KB of RAM plus other memory in a physical address space of 1MB. Some remember this history and add seemingly plausible qualifications that exceeding 4GB is possible only at the price of nasty hacks that require everyone—well, all programmers—to jump through hoops. Fortunately, Intel’s processors are a lot more advanced than the 8086 from all those years ago.

P.S. Unfortunately, patching the kernel won’t help make Windows XP see more than 4GB RAM: even though the kernel itself does support more RAM (with PAE), starting with SP2 the HAL was modified in a way prohibiting access to any RAM beyond 4GB. Patching may only be suggested to devoted geeks with Vista’s and 7′s.