Andy Dustman gave a presentation on Python and MySQL at the MySQL Users Conference 2005, Santa Clara, CA. The presentation is an excellent collection of examples for those who use the MySQLdb Python module.
Read the rest of this entry »

Here’s an excellent description of various methods to handle date and time in Python scripts.

I had just installed the latest ATI video drivers for Linux onto my Slackware 11 (with kernel 2.4.33.3); I have Radeon Mobility X600 with 64MiB of non-shared memory.

Running glxgears, I got ~2050 FPS (glxgears window focused) and ~4900 FPS (console window focused, glxgears window out of focus).

But running fgl_glxgears yielded values as low as 1 FPS! As fgl_glxgears is much more similar to the real-world game visuals, it’s apparent that no game will be playable at e.g. 0.2 FPS.

Also, there were numerous repetitions of a single error in the console window:

FGLTexMgr: open of shared memory object failed (Function not implemented) __FGLTexMgrCreateObject: __FGLTexMgrSHMmalloc failed! fglX11AllocateManagedSurface: __FGLTexMgrCreateObject failed!

I found the solution which worked perfectly for me:
Read the rest of this entry »

If you happen to need to check your linux filesystem, which is mounted read-write, and for some reason you do not want to reboot, then the simple sequence of commands listed below should help you. Note, that the commands provided put you into single-user mode, which kills web-server and mysql daemon.

So, running e2fsck on a live (mounted) filesystem isn’t recommended - and f2sck asks if you really want to check the rw-mounted FS (be sure to answer ‘n’o). If the FS you want to check is root (/), you cannot also re-mount it read-only - unless first you go to single-user mode:

init 1

Now you can re-mount your FS read-only:

mount -o ro,remount /dev/cobd0

(/dev/cobd0 is my device, replace it for your device when repeating step-by-step).
Filesystem is now read-only, and it’s safe to run e2fsck:

e2fsck -D -C 0 -f -t -v /dev/cobd0

e2fsck options are optional:
-D: optimize directory structure
-C 0: show progress
-f: force check (use if you get “volume is clean” with no check)
-t: e2fsck timing statistics
-v: verbose mode

Finally, return to your previous user mode:

init 3

… on eBay. I wonder when will finally PayPal allow accepting payments for people registered as the residents of Ukraine. It’s fairly discriminating to be unable to use PayPal, thus loosing a good percentage of potential buyers.

Update: didn’t sell. Now I’m asking 120$ for the bundle of all three domains:
banners-network.com
banners-network.net
banners-network.org

or will sell each one for 50$.

On purchase agreement (when buying all three domains), I may setup the domains to point to your IP - so no need to wait until the payment is cleared. When payment is cleared, the domains will be transferred to your account. Payment options are: direct to-Visa payment, check, wire transfer (SWIFT). Other payment options are negotiable.

Refreshing my scarce knowledge of Apache’s mod-rewrite, I read through the mod_rewrite guide, and found an extremely interesting section, titled

On-the-fly Content-Regeneration

Here’s the theoretical problem:

1. we are building a high-traffic site with lots of once-per-(hour|day) updated items
2. we have a CMS with just all the features we need, but it’s really CPU/DB-consuming and slow (does it sound familiar? )
3. there’s a need to serve static files

And here’s the ‘radical alternative’ solution:

1. install the CMS of choice
2. tweak the CMS’s output layer to both produce/write to disk (or update) static HTML files, and to dump those same pages directly to browser
3. use the “On-the-fly Content-Regeneration” mod_rewrite rules set

This is it, in short. The “On-the-fly Content-Regeneration” will read the static files if they exist, or will query the CMS, which will create/update the static files and output the necessary page. You can also setup a cron-job to remove all static files older than XX minutes, to force content refresh.

Below is the copy of “On-the-fly Content-Regeneration” from the mod_rewrite guide.
Read the rest of this entry »

Came across toonel.net project. It’s free for private, non-commercial use.
Citing the site:

toonel.net is an experiment in a technique for conserving network bandwidth. If you are paying a fee based on the amount of data you transfer (e.g. 3G/GPRS/DSL/CDMA) and have monthly download limit, or if you have narrowband connection (e.g. dialup modem) then you are most likely to appreciate the service toonel.net offers. There are number of toonel.net client application packages. All solutions are 100% Java based and therefore this service can be used on any platform with an installed Java Runtime Environment, including mobile phones.

toonel.net: what it is good for?

• expensive transports (e.g. GPRS, CDMA…)
• narrowband or slow connections (e.g. dial-up)
• stretching download limits

Why toonel.net is different?

• cross platform
• built from open source components
• allows to compress virtually any traffic

As soon as I switch to some java-enabled personal communication device, (and if toonel.net will still exist by that time ), I would definitely try it. Or if I switch to some speedy but traffic-limited internet-access provider.

Nonanticipatory (system or predictor) is a (system or predictor) where the output y(t) at some specific instant t₀ only depends on the input x(t) for values of t less than or equal to t₀. Therefore these kinds of (systems or predictors) have outputs and internal states that depend only on the current and previous input values.

In simpler words, nonanticipatory systems can “take into account” only past and present, and cannot base their behaviour/decisions on future expectations.

Nonanticipatory systems are also known as causal systems.

Reference and further reading: article at answers.com.

There is an excellent mod_rewrite cheat sheet.
I have been using it for over a year now, as from time to time there’s a need to try to remember how to use mod_rewrite (e.g. in the case of no-www Class B web-site).

To help preserve this useful bit of information, and for the cases of the original article inaccessible, below is a link to the cheat sheet stored on my server.
Read the rest of this entry »

When using PHP, the simplest way to find the absolute path of your files/folders on the server is by creating a simple path.php file with the following contents (click on the "Plain text" box header for copy-pasting):

Put the new file anywhere in the web-accessible folder on your server,
then just access that file from your favourite web-browser - and you'll have the absolute path shown to you.

Alternatively, you may use the following code:

This also should display the absolute path on your server.

PrivatBank is one of the largest Ukrainian banks, and is very active in the field of e-commerce.

Today I came across their new blog, which is a WordPress-powered one.
From the first glance I'd say that it's here where PrivatBank now publishes the news - before the blog launch, news were published on the privatbank's website. The main and only difference appears to be in comments - now visitors can comment on those news, and even get response.

I wonder how many "comment-responders" PrivatBank had to hire to launch this blog :). Or is that tech-support which was given one more duty?

Yes, it should.
Read the rest of this entry »

Being heavily loaded with work, and with a number of tasks overdue, I'll temporally stop regular updates of this blog until approximately 23rd of April. However, if something emergent occurs, I'll post it here. If not - await a bunch of posts in late April!

The blog isn't currently updated, as I'm quite busy with several abrupt opportunities, main of which is the Google Summer of Code 2007 (announcement here). I applied for the WordPress with "Improve the performance of WordPress" project. Currently I'm working on the detailed weekly 3-month-long plan of implementation for the project. I do feel that I should have learned about GSoC-2007 earlier than Friday, March 23rd . Well, it's good at least that the deadline for applications was extended until the 27th of March, and I could at least register and submit the only application which I consider to be a good fit for my current activities and skills.

This year the competition doesn't seem overly tough, with around 3000 students, over 6100 applications, and 800 stipends for successful applicants. However, among the 131 OpenSource projects, some will definitely enjoy slightly more attention and applications than others. I wonder how many applications there are for WordPress

Update: WordPress 2.5+ does not require adding rewrite rules to .htaccess, as it now redirects the browser to the correct (configured) URL itself. However, if you would like the redirection to be made by apache's mod_rewrite rather than by PHP's header() instruction (which I suspect to be slower than mod_rewrite), then you can still use the instructions below. (Another consideration to stick to mod_rewrite is the presence of other software installed into the root of the same domain as the WP blog; mod_rewrite solution works for all, while WP's own redirect works only for WP.)

See important update at the end of this post!

In the early days of my acquaintance with internet, I considered it obligatory to add the "www." part in front of every site (domain) name. As a matter of fact, without those three mysterious letters most of the websites "didn't work".

Now, nearly a decade later, it appears clear to me that the www part is redundant. But it was only today, that I finally switched my blog to the use of Class B no-www policy. Earlier it was Class A, the most common.
Read the rest of this entry »

This is a copy of the Top 1000 SQL Performance Tips by Jay Pipes Sheeri Kritzer Bill Karwin Ronald ("Jeremy Basher") Bradford Farhan "Frank Mash" Mashraqi Taso Du Val Ron Hu Klinton Lee Rick James Alan Kasindorf Eric Bergen Kaj Arno Joel Seligstein Amy Lee. Some tips dropped as not useful for me personally. This post is kept as a personal note/reference.

Specific Query Performance Tips:
1. Use EXPLAIN to profile the query execution plan
2. Use Slow Query Log (always have it on!)
3. Don't use DISTINCT when you have or could use GROUP BY
4. Insert performance
1. Batch INSERT and REPLACE
2. Use LOAD DATA instead of INSERT
5. LIMIT m,n may not be as fast as it sounds
6. Don't use ORDER BY RAND() if you have > ~2K records
7. Use SQL_NO_CACHE when you are SELECTing frequently updated data or large sets of data
8. avoid wildcards at the start of LIKE queries
9. avoid correlated subqueries and in select and where clause (try to avoid in)
10. no calculated comparisons -- isolate indexed columns
11. ORDER BY and LIMIT work best with equalities and covered indexes
12. separate text/blobs from metadata, don't put text/blobs in results if you don't need them
13. derived tables (subqueries in the FROM clause) can be useful for retrieving BLOBs w/out sorting them. (self-join can speed up a query if 1st part finds the IDs and use it to fetch the rest)
14. ALTER TABLE...ORDER BY can take data sorted chronologically and re-order it by a different field -- this can make queries on that field run faster
15. Know when to split a complex query and join smaller ones
16. Delete small amounts at a time if you can
17. make similar queries consistent so cache is used
18. Have good SQL query standards
19. Don't use deprecated features
20. Turning OR on multiple index fields (<5.0) into UNION may speed things up (with LIMIT), after 5.0 the index_merge should pick stuff up.
21. Don't use COUNT * on Innodb tables for every search, do it a few times and/or summary tables, or if you need it for the total # of rows, use SQL_CALC_FOUND_ROWS and SELECT FOUND_ROWS()
22. Use INSERT ... ON DUPLICATE KEY update (INSERT IGNORE) to avoid having to SELECT
23. use groupwise maximum instead of subqueries

Scaling Performance Tips:
1. Use benchmarking
2. isolate workloads don't let administrative work interfere with customer performance. (ie backups)
3. as your data grows, indexing may change (cardinality and selectivity change). Structuring may want to change. Make your schema as modular as your code. Make your code able to scale. Plan and embrace change, and get developers to do the same.

Network Performance Tips:
1. Minimize traffic by fetching only what you need.
1. Paging/chunked data retrieval to limit
2. Don't use SELECT *
3. Be wary of lots of small quick queries if a longer query can be more efficient
2. use multi_query if appropriate to reduce round-trips

OS Performance Tips:
1. Use proper data partitions
1. For Cluster. Start thinking about Cluster *before* you need them
2. Keep the database host as clean as possible. Do you really need a windowing system on that server?
3. Utilize the strengths of the OS
4. pare down cron scripts
5. create a test environment
6. source control schema and config files
7. for LVM innodb backups, restore to a different instance of MySQL so Innodb can roll forward
8. partition appropriately
9. partition your database when you have real data -- do not assume you know your dataset until you have real data

MySQL Server Overall Tips:
1. innodb_flush_commit=0 can help slave lag
2. Optimize for data types, use consistent data types. Use PROCEDURE ANALYSE() to help determine the smallest data type for your needs.
3. use optimistic locking, not pessimistic locking. try to use shared lock, not exclusive lock. share mode vs. FOR UPDATE
4. if you can, compress text/blobs
5. compress static data
6. don't back up static data as often
7. enable and increase the query and buffer caches if appropriate
8. config params -- http://docs.cellblue.nl/easy_mysql_performance_tweaks/ is a good reference
9. Config variables & tips:
1. use one of the supplied config files
2. key_buffer, unix cache (leave some RAM free), per-connection variables, innodb memory variables
3. be aware of global vs. per-connection variables
4. check SHOW STATUS and SHOW VARIABLES (GLOBAL|SESSION in 5.0 and up)
5. be aware of swapping esp. with Linux, "swappiness" (bypass OS filecache for innodb data files, innodb_flush_method=O_DIRECT if possible (this is also OS specific))
6. defragment tables, rebuild indexes, do table maintenance
7. If you use innodb_flush_txn_commit=1, use a battery-backed hardware cache write controller
8. more RAM is good so faster disk speed
9. use 64-bit architectures
10. --skip-name-resolve
11. increase myisam_sort_buffer_size to optimize large inserts (this is a per-connection variable)
12. look up memory tuning parameter for on-insert caching
13. increase temp table size in a data warehousing environment (default is 32Mb) so it doesn't write to disk (also constrained by max_heap_table_size, default 16Mb)
14. Run in SQL_MODE=STRICT to help identify warnings
15. /tmp dir on battery-backed write cache
16. consider battery-backed RAM for innodb logfiles
17. use --safe-updates for client

Storage Engine Performance Tips:
1. InnoDB ALWAYS keeps the primary key as part of each index, so do not make the primary key very large
2. Utilize different storage engines on master/slave ie, if you need fulltext indexing on a table.
3. BLACKHOLE engine and replication is much faster than FEDERATED tables for things like logs.
4. Know your storage engines and what performs best for your needs, know that different ones exist.
1. ie, use MERGE tables ARCHIVE tables for logs
2. Archive old data -- don't be a pack-rat! 2 common engines for this are ARCHIVE tables and MERGE tables
5. use row-level instead of table-level locking for OLTP workloads
6. try out a few schemas and storage engines in your test environment before picking one.

Database Design Performance Tips:
1. Design sane query schemas. don't be afraid of table joins, often they are faster than denormalization
2. Don't use boolean flags
3. Use Indexes
4. Don't Index Everything
5. Do not duplicate indexes
6. Do not use large columns in indexes if the ratio of SELECTs:INSERTs is low.
7. be careful of redundant columns in an index or across indexes
8. Use a clever key and ORDER BY instead of MAX
9. Normalize first, and denormalize where appropriate.
10. use INET_ATON and INET_NTOA for IP addresses, not char or varchar
11. make it a habit to REVERSE() email addresses, so you can easily search domains (this will help avoid wildcards at the start of LIKE queries if you want to find everyone whose e-mail is in a certain domain)
12. In 5.1 BOOL/BIT NOT NULL type is 1 bit, in previous versions it's 1 byte.
13. A NULL data type can take more room to store than NOT NULL
14. Choose appropriate character sets & collations -- UTF16 will store each character in 2 bytes, whether it needs it or not, latin1 is faster than UTF8.
15. Use Triggers wisely
16. use min_rows and max_rows to specify approximate data size so space can be pre-allocated and reference points can be calculated.
17. Use HASH indexing for indexing across columns with similar data prefixes
18. Use myisam_pack_keys for int data
19. be able to change your schema without ruining functionality of your code
20. segregate tables/databases that benefit from different configuration variables

Other:
1. Read and post to MySQL Planet at http://www.mysqlplanet.org/

Came across the 419 eater scambaiting site. I did get tons of scam emails, but never thought of them as of an opportunity to unveil the identity (or at least get a photo) of the scammer. This must be fun - unveiling scammers , especially when the scammer is pushed to waste his money for some fictional "hotel booking" or "identity proof donations". It's like reversing the roles of cat and mouse . Nice work, 419eater!

Brian Valentine is said to be the author of the motto: "Decisions in 10 minutes or less, or the next one is free".

I came across this motto accidentally while surfing the web, and think it's worth writing down.

For more details you can see the story behind the Brian Valentine managing Windows 2000 release.

There is also the "quotes" page at WarpedVisions, which has the one about decisions in 10 minutes, as well as many other quotes.

I had been to Poland for a month (until mid-November), and was too busy doing bioinformatics there to make any updates to my blog. Now, though at home, I also do not have enough time - my wife had one of her back bones broken in a car accident, so now she cannot walk, and needs care and attention. Thus I cannot afford to have some spare time to update this blog. Sorry, if you are a regular visitor. I expect to add two new rather interesting articles within the first week of December - one will be the sequel of the HTTP caching article, and the other might be about headers handling in PHP-CGI installations - but did not decide on this one yet, so this can be something else as well.

Since the 16th of October I am on bioinformatics training in Gliwice, Poland, at the Comprehensive Cancer Centre Maria Sklodowska-Curie Memorial Institute, Gliwice branch. I am training in microarrays data analysis, with focus on R + BioConductor usage for analysis.

I am already more-or-less comfortable with Polish - though understanding is still a much easier task than expressing own thoughts, thus most commonly I address colleagues in English, and perceive responses in Polish

Here there is an excellent lunch location - Bar U Piotra ("At the Peter's"). For 6.5 PLN (roughly 2.2 USD or 10.4 UAH) you get soup, meat with some garnish, and - not always - some salad. However, 6.5 PLN is a special price, available only if you buy at least 7 (or 10?) meals ahead. Food is simple but tastes like home-cooked, and helpings size is well sufficient for me - which means "more than sufficient" for an average person.

I still cannot get used to other people coming to work at 8 a.m., but the idea of leaving at 4 p.m. adds a kernel of good sense to the early-rising idea.

Generally, Gliwice is not a bad place to stay, at least for up to 2-3 months. Autumn here is beautiful - see my gallery for some samples of what is looks like here at the end of October.