Development: see http://www.ruzee.com/blog/2009/01/apache-virtual-hosts-a-clean-setup-for-php-developers. This setup works very well, unless you need to create several virtual hosts every day – in which case necessary actions could be partially scripted.

As a result, I’m now using Eclipse PDT with Xdebug and Xdebug Helper (Firefox extension). Now I don’t understand how I used to debug my PHP programs before!

After proper configuration (I’m using local Apache, but it is also possible to debug remotely), my work flow is rather simple:

• use my web-app as usual, e.g. tweaking and testing here and there
• if something server-side goes wrong: click the XDebug helper icon in Firefox, and perform some server-request action (e.g. load a page)
• debugging is started in Eclipse PDT, where I can step through the code, set breakpoints, and examine all variables
• as soon as the problem is fixed – click the XDebug helper icon again to continue using the site normally (w/o invoking the debugger)

It takes some time to get used to, but then it’s a breeze.

Some advice:

• don’t use apt-get/aptitude to install Eclipse; it will be much easier both in the short and long run to use some all-in-one package from the Eclipse PDT site; all you need to do – download, extract, run!
• before actually starting to do anything, tweak the eclipse.ini file by increasing heap size from 40 MiB (default) to some larger value (I used 128MiB). If you don’t do this, then at some point your debugging will become painfully sloooow, and then you’ll start getting tons of “out of heap memory” errors, each one suggesting that you quit Eclipse immediately
• install XDebug with apt-get/aptitude – worked perfectly, and there’s /etc/php5/conf.d/xdebug.ini not to mess with php.ini
• do read XDebug guide for PDT 2.x (I’m assuming you got the 2.x version); it should be the only document you will really need to configure everything

I only wish Eclipse was faster – that is, written not in Java but e.g. C or C++.

I assume that the problem is caused by Google Analytics, namely the "track outgoing clicks" feature (as recalled, might be inaccurate feature name). "Track outgoing links" adds some JavaScript code to all outgoing links, and that script has tick characters like this one ' which, incidentally, are also used for delimiting the values of comment anchor tags.

To fix:

1. locate file wp-includes/comment-template.php
2. in that file, find the get_comment_author_link function (lines 140-150 in WP 2.7.1)
3. replace the line
   PLAIN TEXT
   PHP:

   1. $return = "<a href='$url' rel='external nofollow' class='url'>$author</a>";

   with

   PLAIN TEXT
   PHP:

   1. $return = '<a href="'.$url.'" rel="external nofollow" class="url">'.$author.'</a>';

This helped me and might help you.

It appears that it is very easy to use ExpressionEngine's contact form (which uses Email module) to send emails to arbitrary addresses - simply put, send spam using someone's EE.

And here's why:

• recipients hidden field is passed to the client; it is encrypted, but with access to the mod.email.php code, it is a matter of several minutes to write your own email-encoding function which will produce a completely valid recipients field
• there's also XID field, which seems to be unique for each page load

Spamming algorithm is clear, so I won't elaborate. (I could have missed some session variables, though - didn't check them.)

This information is valid as of ExpressionEngine 1.6.6, but nothing in the change-logs indicates that this mechanism was modified in the newer versions of EE.

Update: I've tested, and this vulnerability does exist. The simplest prevention measure is to enable Captcha for Contact Form.

I've notified the developers.

Step-by-step:

• Download the compiled ffmpeg with libmp3lame support (direct download links: older version ffmpeg.with.lame and newer version ffmpeg.2007-10-28.with-libmp3lame-support).
• Rename the downloaded executable file to "ffmpeg" (no extension), upload it to the directory on your server (in this example /home/myusername/).
• Download libmp3lame.so.0.
• Upload libmp3lame.so.0 to the same directory where you placed ffmpeg in.
• Create a php file with the following code (remember to change the paths to your own, where you actually uploaded binaries in previous steps):
  PLAIN TEXT
  PHP:

  1. <?php
  2. exec("export LD_LIBRARY_PATH=/home/myusername/");
  3. echo passthru("/home/myusername/ffmpeg -formats");
  4. ?>

• If you access that PHP file with your browser, you should be able to see a list of formats which are supported by ffmpeg, and if you find "EA libmp3lame" somewhere in the output, then it means you now can Encode Audio in libmp3lame!
• If that doesn't work for you: LD_LIBRARY_PATH can be a protected variable in PHP (you can check for this by searching for the "safe_mode_protected_env_vars" value in phpinfo() output). The workaround here can be to write your commands to a file from php, then chmod that file with 0755 permissions (rwx-rx-rx), and run the file through exec(). In this way, PHP is not changing the LD_LIBRARY_PATH, but telling the server to run a file, which tells the server to change it. This method worked for the original author of these instructions.

For passing arguments to the PHP wrapper script - check out PHP's $argv variable (more on this here). If you decided to use shell-script as a wrapper for "export LD_LIBRARY_PATH", then using $1, $2 etc will allow you to pass parameters to the shell script, e.g. in the example below

you could pass "input file" as first argument, "parameters" as second and "output file" as third to make such a wrapper script work.

Update: the original script had an error, it would generate queries likes this one (note the bold part):

ALTER TABLE `links` CHANGE `link_rel` `link_rel` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT NULL;

This is clearly wrong syntax (and logic). I fixed this by making comparison to NULL strict (with three equal signs instead of two):

// Does the field default to null, a string, or nothing?
if ($row['Default'] === NULL)

Update 2: based on comment by banesto, I modified the script; now it does not require specifying the from_collation, it's sufficient to specify to_collation (which will be used for all the fields and tables). The modified code is:

if ($row['Collation'] == '' || $row['Collation'] == $convert_to)
continue;

Update 3: the long-lasting, re-appearing NOT NULL DEFAULT NULL problem is finally fixed.

Update 4: incorporated Russ's fix to skip numeric fields (in order to leave autoincrement values intact).

Here's the script itself: (to copy-paste: first click the "Plain text" header)

Do pay attention to the comments, especially these two:

I had issues with this script (and others) returning 0 for the bytesTotal in flash. Basically, the Content-Length header was absent from the response. By simply adding
header("Content-length: ".strlen($response)) before the echo, it resolved the issue. I don't know if there is a more appropriate fix to account for character encoding, etc, but it seems to work.

@Schimmi: Well, if you can add some checks there (like who is referring your script) and allow the access to whitelisted clients (served from your domain)... I think, you can totally make it used applications from same-domain....So it would not be open to world. Yeah above script doesn't have those things.

In case the original disappears, here's a "backup":

So here's that tiny missing bit of information.

Windows:
Launch regedit, navigate to HKEY_CURRENT_USER\Environment, and create a string value called PHP_PEAR_HTTP_PROXY. Modify that new value to hold the string like: http://proxy_username:proxy_password@proxy_server_address:proxy_port.

Linux:
In the Terminal/Konsole, execute (for a system-wide pear configuration)

sudo pear config-set http_proxy http://proxy_username:proxy_password@proxy_server_address:proxy_port

If your proxy password has symbols, special for the shell (e.g. question or exclamation mark) - enclose full proxy specifications with single-quotes, e.g.

sudo pear config-set http_proxy 'http://proxy_username:proxy_password@proxy_server_address:proxy_port'

If your HTTP proxy server does not require authentication, then use http://proxy_server_address:proxy_port instead.

I think the strings are completely self-explanatory; however, here's an example of proxy (with authentication) specification: http://john.smith:CrAzYP433WoRd@192.168.0.1:3128.

The script is simple, but still it's easier to use the ready solution than to write your own .
It is heavily commented, and should be easy to understand.

To copy-paste, first click the "PLAIN TEXT" box header.

How to use: insert this code into your PHP file where you need to output the list of images. Do not forget to apply some pretty CSS styling to IMG tags!

The major drawback of this solution is inefficiency: reading the complete directory of image files on every page display is not good at all. However, I inserted this code snippet into the CMS's PHP-enabled template, which allows to turn on caching. This actually solves efficiency problem, as directory will be read (and images randomized...) only as often as long your configured cache is.

Another option is to add in some caching options. It might be reasonable as well to use MySQL HEAP-table (in-memory table) to store the directory-reading result and refresh it only once in an hour; this way images would be randomized on each page display, but the directory would be re-read less than once in an hour.

But all of the above-described enhancements are optional, and can be left as an exercise to the reader

Here's sample code without explanations:

To run something like hmmsearch from the HMMER package, you’d do this:

Put the new file anywhere in the web-accessible folder on your server,
then just access that file from your favourite web-browser - and you'll have the absolute path shown to you.

Alternatively, you may use the following code:

This also should display the absolute path on your server.

i18n module
i18n: Getting the whole thing to work : http://drupal.org/node/81094
Patch: Translations of menu titles and descriptions: http://drupal.org/node/70919
Translated links: http://drupal.org/node/67814
i18n: menu not expanding with URL-Alias: http://drupal.org/node/80820

There was an alternative module to i18n, but I cannot find it at the moment.

Update 2: fsmullin suggested a method to fix a similar error in index_table.php, the files_count_max cannot be found error when you click the FILES tab/menu item. His suggestion is now incorporated into the post.

Update 3: this post is still relevant for dotProject 2.1.3.

In one of my recent posts about project management software I stated the desire to extensively test dotProject 2.0.4. However, many shared hosting providers appear incompatible with dotProject: the right to CREATE TEMPORARY TABLES in MySQL is not granted, but is needed by dotProject.

Here's sample error (2.0.4):

query failed(CREATE TEMPORARY TABLE tasks_sum SELECT task_project, COUNT(distinct task_id) AS total_tasks,
SUM(task_duration * task_percent_complete * IF(task_duration_type = 24, 8.0, task_duration_type))/
SUM(task_duration * IF(task_duration_type = 24, 8.0, task_duration_type)) AS project_percent_complete FROM `tasks` GROUP BY task_project)

For 2.1.2, sample error would be:

Table 'tasks_total' doesn't exist

I tried looking for solutions, and here's what I found...

First of all, I tried issuing the "grant" command to try enabling temporary tables:

GRANT CREATE TEMPORARY TABLES ON dot_projects.* to dot_projects@x.myserver.com;

Expectedly, this didn't work: "access denied for user ....".

Then I tried looking for a ready-to-apply solution. Here are the links relevant to the topic, if you happen to prefer "the source", and not the retelling: link 1, link 2, link 3, and link 4.

However, none of the solutions described behind the links worked for me.

Here's my solution (error fix).

To fix, instead of creating temporary tables each time they are needed (what is impossible on many shared hostings), we need to create the necessary tables once. Then, instead of dropping temporary tables, we just truncate our "static" tables.

Here's how we proceed.

1. Issue once the following MySQL statements (using, e.g., phpMyAdmin):
   PLAIN TEXT
   SQL:

   1. CREATE TABLE IF NOT EXISTS `tasks_problems` (
   2. `task_project` INT( 11 ) DEFAULT NULL ,
   3. `task_log_problem` TINYINT(1) DEFAULT NULL
   4. ) TYPE=MyISAM;
   5.  
   6. CREATE TABLE IF NOT EXISTS `tasks_critical` (
   7. `task_project` INT( 11 ) DEFAULT NULL ,
   8. `critical_task` INT( 11 ) DEFAULT NULL ,
   9. `project_actual_end_date` DATETIME DEFAULT NULL
   10. ) TYPE=MyISAM;
   11.  
   12. CREATE TABLE IF NOT EXISTS `tasks_sum` (
   13. `task_project` INT( 11 ) DEFAULT NULL ,
   14. `total_tasks` INT( 6 ) DEFAULT NULL ,
   15. `project_percent_complete` VARCHAR( 11 ) DEFAULT NULL,
   16. `project_duration` VARCHAR( 11 ) DEFAULT NULL
   17. ) TYPE=MyISAM;
   18.  
   19. CREATE TABLE IF NOT EXISTS `tasks_summy` (
   20. `task_project` INT(11) DEFAULT NULL,
   21. `my_tasks` varchar(10) DEFAULT NULL
   22. ) TYPE=MyISAM;
   23.  
   24. CREATE TABLE IF NOT EXISTS `tasks_users` (
   25. `task_project` INT(11) DEFAULT NULL,
   26. `user_id` INT(11) DEFAULT NULL
   27. ) TYPE=MyISAM;
   28.  
   29. CREATE TABLE IF NOT EXISTS `tasks_total` (
   30. `task_project` INT( 11 ) DEFAULT NULL ,
   31. `total_tasks` INT( 6 ) DEFAULT NULL
   32. ) TYPE=MyISAM;
   33.  
   34. CREATE TABLE IF NOT EXISTS `files_count_max` (
   35. `file_versions` INT(11) DEFAULT 0,
   36. `file_lastversion` float DEFAULT '0',
   37. `file_version_id` INT(11) DEFAULT 0,
   38. `file_project` INT(11) DEFAULT 0
   39. ) TYPE=MyISAM;

   This will create all the necessary tables.
   Note: tasks_total and files_count_max tables are 2.1.2-specific.
   Another note: for 2.1.2, the tasks_sum table is one field shorter:

   PLAIN TEXT
   SQL:

   1. CREATE TABLE IF NOT EXISTS `tasks_sum` (
   2.   `task_project` int(11) DEFAULT NULL,
   3.   `project_percent_complete` varchar(11) DEFAULT NULL,
   4.   `project_duration` varchar(11) DEFAULT NULL
   5. ) TYPE=MyISAM;

2. Now, in classes/query.class.php at line 393 (for dotProject 2.0.4; just search for the matching line in 2.1.2) replace this line:
   PLAIN TEXT
   PHP:

   1. $q = 'CREATE TEMPORARY TABLE ' . $this->_table_prefix . $this->create_table;

   with these:

   PLAIN TEXT
   PHP:

   1. //$q = 'CREATE TEMPORARY TABLE ' . $this->_table_prefix . $this->create_table;
   2. $q = 'INSERT INTO ' . $this->_table_prefix . $this->create_table;

   Now we won't get any errors about the tables which fail to be created.

3. Next, replace the following text in modules/projects/projects.class.php (lines 388-392 for 2.0.4):
   PLAIN TEXT
   PHP:

   1. // Let's delete temporary tables
   2. $q  = new DBQuery;
   3. $q->dropTemp('tasks_sum, tasks_summy, tasks_critical, tasks_problems, tasks_users');
   4. $q->exec();
   5. $q->clear();

   with this code:

   PLAIN TEXT
   PHP:

   1. // Let's delete temporary tables
   2. $sql = "TRUNCATE TABLE tasks_sum"; db_exec($sql);
   3. $sql = "TRUNCATE TABLE tasks_summy"; db_exec($sql);
   4. $sql = "TRUNCATE TABLE tasks_critical"; db_exec($sql);
   5. $sql = "TRUNCATE TABLE tasks_problems"; db_exec($sql);
   6. $sql = "TRUNCATE TABLE tasks_users"; db_exec($sql);
   7. $sql = "TRUNCATE TABLE tasks_total"; db_exec($sql);
   8. $q  = new DBQuery;

   Note: $sql = "TRUNCATE TABLE tasks_sum"; db_exec($sql); line is 2.1.2-specific.
   At this point, you will no longer receive error messages while on the "projects" tab.

4. (this point applies to 2.1.2 only) Replace lines 222 and 223 in modules/files/index_table.php
   PLAIN TEXT
   PHP:

   1. $r->dropTemp('files_count_max');
   2. $r->exec();

   with:

   PLAIN TEXT
   PHP:

   1. //$r->dropTemp('files_count_max');
   2. //$r->exec();
   3. $sql = "TRUNCATE TABLE files_count_max"; db_exec($sql);

5. Optionally, if you want to improve the performance of dotProject specifically for these tables, change the 'engine' for the above-mentioned tables to 'memory'. To do that, execute the following SQL statements:
   PLAIN TEXT
   SQL:

   1. ALTER TABLE tasks_problems engine=memory;
   2. ALTER TABLE tasks_critical engine=memory;
   3. ALTER TABLE tasks_sum engine=memory;
   4. ALTER TABLE tasks_summy engine=memory;
   5. ALTER TABLE tasks_users engine=memory;
   6. ALTER TABLE tasks_total engine=memory;
   7. ALTER TABLE files_count_max engine=memory;

   This change will make MySQL keep these tables in RAM. As they are used as temporary tables, this decreases disk I/O a lot while working with dotProject. Of course, YMMV - to apply this change or not depends on where is your bottleneck.

Comments and further improvements are welcome.



]]> http://bogdan.org.ua/2007/01/22/dotproject-204-on-godaddy-shared-hosting.html/feed 10 http://bogdan.org.ua/2006/12/09/http-caching-universal-approach-and-sample-code.html http://bogdan.org.ua/2006/12/09/http-caching-universal-approach-and-sample-code.html#comments Sat, 09 Dec 2006 09:16:53 +0000 Bogdan http://www.bogdan.org.ua/2006/12/09/http-caching-universal-approach-and-sample-code.html As described in my previous post, there are some rather simple mechanisms to enable visitor's browser to cache content, and avoid unnecessary load on your servers. In this post I'll take a look at some parts of the practical implementation of the caching mechanism on the server, using PHP.

I think of caching as a set of relatively independent procedures:
1. Process browser's request - find out, if the browser is asking for content for the first time, or is looking for an updated version. If the browser's content is fresh, just reply that it's fresh. Otherwise, proceed to step 2. Also proceed to step 2, if this is a first-time request.
2. If this is a first-time request, check if we have a fresh locally cached content, and send it in response, to bypass de novo contet generation. If local cache is stale, proceed to step 3.
3. Generate new (fresh) content cache item, and send it to browser, together with cache-enabling headers.

I decided to put all the header-related processing into one function, and all the cache-creation and sending - into another. If you prefer, you can make an object with two or more methods, but I see no need in doing so.

Here's the headers part:

PLAIN TEXT
PHP:
1. function HTTP_caching($timestamp)
2. {
3.     /*
4.      * HTTP_caching(v0.3)
5.      * checks headers for eTag or Modified request,
6.      * and replies with HTTP 304 Not Modified
7.      * if $timestamp is equal to the requested time/tag
8.      * returns TRUE if 304, FALSE otherwise
9.      * produces Last-Modified and ETag headers from $timestamp
10.     */
11.  
12.     $latest = gmdate('D, d M Y H:i:s', $timestamp) . ' GMT';
13.     $etag = md5($latest);
14.  
15.     $not = false;
16.  
17.     if ( isset( $_SERVER['HTTP_IF_MODIFIED_SINCE'] ) )
18.     {
19.         if ( $_SERVER['HTTP_IF_MODIFIED_SINCE'] == $latest )
20.         {
21.             header("HTTP/1.1 304 Not Modified");
22.             header("Status: 304 Not Modified");
23.             $not = true;
24.         }
25.     }
26.     elseif ( isset( $_SERVER['HTTP_IF_NONE_MATCH'] ) )
27.     {
28.         if ( $_SERVER['HTTP_IF_NONE_MATCH'] == $etag )
29.         {
30.             header("HTTP/1.1 304 Not Modified");
31.             header("Status: 304 Not Modified");
32.             $not = true;
33.         }
34.     }
35.     if (!$not)
36.     {
37.         header("Last-Modified: " . $latest );
38.         header("ETag: " . $etag);
39.     }
40.     return $not;
41. }

This function checks two caching-related request headers: If-None-Match and If-Modified-Since. Use of two headers is redundant, but this was done with an idea to provide different caching methods for different content types in the future. Timestamp-based approach should be sufficient in most cases, but having ETag as well does not hurt.

The parameter this function takes - $timestamp - is the date of the locally cached content item. We should check if the cached item exists before calling HTTP_caching; and if cached item does not exist, we should create it and then call HTTP_caching with the timestamp of the newly generated cached content item.

Function is utterly simple, but needs some explaining comments:

• There is a convenient function "apache_request_headers", but as its name implies, it works only under apache. Thus I had to use global PHP variables for fetching specific request headers. However, this approach also doesn't work on all hosting providers - you just do not get those headers, though they are sent by the browser. But you still benefit from local caching, and the only drawback of being unable to process request headers is that you waste traffic sending data which could have been unsent.
• I'm also sending two equal headers - standard HTTP/1.1, and outdated Status header. This is redundant (sending Status might be removed), but depending on how your PHP is installed (as apache-httpd module or CGI) you may want to leave both in place, or experiment and leave the one which actually works.

Now, let's move on to actual caching. The simplest and quite reliable method of identifying any object within your cache is md5(url) - that is, the hash of the request URL. Note, that you might want to hash not the complete URL (starting with http://), but only the part after the TLD's slash, e.g. for complete URL http://bogdan.org.ua/2006/10/27/xnameorg-down-largest-ddos-they-ever-had.html you would hash only the "xnameorg-down-largest-ddos-they-ever-had.html" part (or "2006/10/27/xnameorg-down-largest-ddos-they-ever-had.html", if the filename part of the path might be non-unique). Evidently, this will save you from generating cache both for "http://www.bogdan.org.ua/2006/10/27/xnameorg-down-largest-ddos-they-ever-had.html" and for "http://bogdan.org.ua/2006/10/27/xnameorg-down-largest-ddos-they-ever-had.html" (differing only in "www." part).

Here's what we need to do (some evident micro-tunings dropped for clarity):

1. Hash the request URL.
2. Check if that cache item exists. If yes - go to step 4. If not - step 3.
3. Create cache item (cache file), set the file name to the hash of the request URL, and store the file in your cache directory (e.g. '/cache').
4. Read cache file modification timestamp.
5. Call HTTP_caching with that timestamp. If it returns true - do nothing. Else - continue to step 6.
6. Read cache file and send it to browser.

To make this part of caching system wrapped in a single universally applicable function, you will need to define some "content-generating" handler function, which will be called when local cache file must be regenerated. "Content-generation" might be as simple as just reading some file from the disk; or you could wrap your index.php file in cache-checking block, so that page call and generation occur only if there is no readily available cached page.

PLAIN TEXT
PHP:
1. function caching($handler_function)
2. {
3.  // first, check if we have existing cache
4.  // of currently requested resource in the 'cache' directory
5.  $cachedfile = 'cache/' . md5($_SERVER['REQUEST_URI']);
6.  if (file_exists($cachedfile) && is_file($cachedfile))
7.  { // cache exists. now check if it is up-to-date
8.     $filetime = filemtime($cachedfile);
9.     $modif = time() - $filetime;
10.     // CACHE_PERIOD is a constant defined somewhere else,
11.     // e.g. define('CACHE_PERIOD',300)
12.     if ($modif <= (int)CACHE_PERIOD)
13.     {
14.      if ( HTTP_caching($filetime) === false )
15.      { // we need to send cached content to browser - no
16.        // if-modified-since or if-none-match request headers
17.       $expires = gmdate('D, d M Y H:i:s', ($filetime + CACHE_PERIOD) ) . ' GMT';
18.       send_cached_file($cachedfile, CACHE_PERIOD, $expires)
19.      }
20.      // if HTTP_caching returned 'true' - we are done with '304' header.
21.      // finally, ensure nothing else happens after sending contents...
22.      exit();
23.     }
24.     // if cache is old - just call $handler_function,
25.     // assuming that it returns all the content
26.     $new_cache = $handler_function();
27.  
28.     // if instead of using $handler_function you wrap some code:
29.     // ob_start();
30.     // include "page_generation.php";
31.     // $new_cache = ob_get_contents();
32.     // ob_end_clean();
33.    
34.     $fp = fopen ($cachedfile, "w");
35.     fwrite ($fp, $new_cache);
36.     fclose ($fp);
37.     $filetime = filemtime($cachedfile);
38.  
39.     $expires = gmdate('D, d M Y H:i:s', ($filetime + CACHE_PERIOD) ) . ' GMT';
40.     send_cached_file($cachedfile, CACHE_PERIOD, $expires);
41.  }
42. }

Below is a function which sends cached files to the browser. Note, that it lacks content-type parameter (or content-type detection code).

PLAIN TEXT
PHP:
1. function send_cached_file($cachedfile, $cache, $expires)
2. {
3.     // I omitted Content-Length and Content-Type for simplicity,
4.     // though it's better to send them
5.     header("HTTP/1.1 200 OK");
6.     header("Status: 200 OK");
7.     header("Pragma: cache");
8.     header("Cache-Control: max-age=".$cache.", min-fresh=".$cache.", no-transform");
9.     // Content-Type needs adjustment on a per-content-item basis
10.     header("Content-Type: text/html");
11.     header("Content-Length: " . filesize($cachedfile));
12.     header("Expires: $expires");
13.     readfile($cachedfile);
14. }

(Note: function 'caching' was not tested; if you find any errors, or if it just doesn't work for you - let me know via the contact form, I'll try to fix my error.)

For the explanation of what headers mean, refer to my previous post.

This is it for now. Coming next might be some useful information on differences in HTTP headers handling between PHP installed as a module and PHP as CGI.

As always, comments/suggestions are welcome.



]]> http://bogdan.org.ua/2006/12/09/http-caching-universal-approach-and-sample-code.html/feed 1 http://bogdan.org.ua/2006/10/18/http-caching-request-and-response-headers.html http://bogdan.org.ua/2006/10/18/http-caching-request-and-response-headers.html#comments Wed, 18 Oct 2006 15:22:25 +0000 Bogdan http://www.bogdan.org.ua/2006/10/18/http-caching-request-and-response-headers.html In this post: some caching-related HTTP request and response headers discussed.

Modern websites are "dynamic" by nature - content you get depends on a number of variables and conditions. The simplest example - being a "guest" or a "registered" user of some forum; in both cases you get content generated by the same script/program, but it differs because of your "registered" state.

Another example - web-photogallery. By design, it is wise to always keep the original photo (the largest and presumably of the highest quality). When gallery visitor's browser requests any smaller version of the photo - we can dynamically resize (often downsize) the original and feed the resulting photo to the browser. This scheme works OK until your gallery gets more and more visitors - CPU load climbs up and increases wait time when accessing your gallery. Evidently, caching is needed.

Caching is a process of storing some integral (independent, atomic) piece of data in order to quickly retrieve it later, when asked for, in cases, when creating/accessing that piece of data without storing would require more time. Caching provides means to bypass calculation-intensive steps of content creation, and thus decreases server load, and allows your server to generate content faster, and deliver it to more visitors.

Probably the most important parameter of caching is "cache period" - time, during which cached piece of data is considered "fresh" and is read of the cache; if the cached piece of data exists more than "cache period" - it is considered "stale", and should be replaced with newly recreated piece of data.

When web-server sends some content to the browser, there are three main "RFC-compliant" headers it may use to instruct the browser how to cache that content. These headers are Last-Modified, ETag, and Cache-Control. Two of these - Last-Modified and ETag - force the compliant browser to generate specific request tags: If-Modified-Since and If-None-Match. The Cache-Control header instructs the browser's cache on how to handle data. For complete technical description, please refer to RFC2616.

Last-Modified informs the browser about the date when the document (or image) was last updated (modified). Correct Last-Modified header must be issued using GMT time, and might look like this example:

PLAIN TEXT
CODE:
1. Last-Modified: Wed, 10 May 2006 08:24:39 GMT

Sending this response header makes the browser to add corresponding If-Modified-Since request header next time it needs to load the resource. Following our example, browser's request would include:

PLAIN TEXT
CODE:
1. If-Modified-Since: Wed, 10 May 2006 08:24:39 GMT

Note: browsers are recommended to send back time exactly as it was sent by the server, without modifications. At the moment, I do not know whether and which browsers follow this recommendation. In any case, if you do send time as GMT, you are more likely to get the desired caching effect, than if you send any unique string in the Last-Modified header.

If a web-server receives If-Modified-Since header, it might reply either with a HTTP/1.1 200 OK (followed by resource data) or HTTP/1.1 304 Not Modified (header itself, no data following). Server response depends on the "fresh" or "stale" status of the resource, as described above.

Time is not always a good measure of resource "freshness", especially in the case of dynamic sites, which send and process headers themselves, "bypassing" server behaviours. In order to identify some resource, an ETag response header can be used:

PLAIN TEXT
CODE:
1. ETag: "some-double-quoted-string"

ETag is used as a unique resource identifier, which is expected to change if the resource changes. (Here, cyclic redundancy checks come to memory - and that is correct, hashing algorithms like CRC32 and md5 can be used on resource content to generate it's ETag.) When a browser receives ETag header, then on next request it should send "If-None-Match" header:

PLAIN TEXT
CODE:
1. If-None-Match: "some-double-quoted-string"

Obviously, string sent by the browser must be equal to one received from the server. Server behaviour in this case is determined by the equality or non-equality of local (server) and received (from browser in If-None-Match) tags: equality hints sending the 304 Not Modified header only, non-equality requires sending 200 OK with content following.

ETag header value also has weak/strong modifiers, and If-None-Match request header is not the only one for handling ETag, but I will not cover those topics here. Please refer to RFC2616 for more.

Finally, Cache-Control response header is used to tell the browser how to cache (and if to cache at all) the resource being sent. Cache-Control has a number of possible values (comma-separated if there are more than one), but I will review only some of the values.
no-cache: forces resource re-validation on each request.
no-store: (applies to both request and response) forces the resource not to be stored at all; this is usually important for sensitive data.
max-age: (in seconds) informs about the time period when resource stays "fresh".
must-revalidate: as the name implies, effect is similar to no-cache.
no-transform: forbids any resource transformations by proxies/servers in the chain to the end user (don't think it's still relevant, though).

That is it. Next post will continue the caching topic with the discussion of the universal approach to caching for web-sites.



]]> http://bogdan.org.ua/2006/10/18/http-caching-request-and-response-headers.html/feed 2 http://bogdan.org.ua/2006/09/08/php-nuke-6-06-5-to-drupal-4-7-x5-x-migration-conversion.html http://bogdan.org.ua/2006/09/08/php-nuke-6-06-5-to-drupal-4-7-x5-x-migration-conversion.html#comments Fri, 08 Sep 2006 15:29:56 +0000 Bogdan http://www.bogdan.org.ua/2006/09/08/php-nuke-60-to-drupal-473-migration-conversion.html Post last updated: April 18, 2010.

Now there is a Drupal 6.x module available. It is in no way related to the migrate script(s) below.

The newest script version migrates from PHP-Nuke 6.5 to Drupal 5.x.
Download the latest version of the migration script.

In 2002 I set up a PHPNuke-6.0 - based portal. Eventually it died due to the lack of time investments and support from collaborators. Now, when time came to revive the project, I made a search and decided to use Drupal as a base CMS for the portal.
In order to migrate userbase from an old portal to the new Drupal-powered one, and following the topic at drupal.org, I found a script and its modification.
I used it to migrate only users, and made some cosmetic changes:

• added options for custom phpnuke table prefixes
• default user name is now = uname (login), not 'temp_name', as before
• I replaced hard-coded links to 'migrate.php' with links to $_SERVER['PHP_SELF'], so that if you rename the script you don't have any problems with that
• now forum topics should not be promoted to the main page (changed 1 to 0 as hinted by Alexis)

Finally, I would like to thank both Karthik Kumar for the original script and Alexis Bellido for the 6.0_to_4.7 modification.


Update: Waldo updated the migration script.
Here are the changes:

• imports from (a security-patched) Nuke 6.5 -> Drupal 5.1
• imports story categories into Drupal taxonomy
• incorporates phpnuke "Contributor writes..." into story
• incorporates phpnuke admin story "note"
• imports polls (does not import record of who already voted. You can easily "close" old phpnuke polls via Admin page. I left them open.)
• imports poll comments
• imports *anonymous* comments for stories, fora, & polls
• imports anonymous forum posts
• filters BBCode to make semi-compatible with Drupal BBCode module
• uses Drupal 5.0 API, and must be run from drupal root directory (/drupal-5/, for example)

Thanks to Waldo for his work!

Latest update: Waldo sent the newest, improved version of the PHP-Nuke-2-Drupal conversion script. Here is Waldo's description:

New Fixes:

• retain original user registration date & last visit 4/22/07
• fixed user migration bug which could result in screwed max_uid in session table 4/22/07
• store anonymous forum posts names in the log for future compatibility w/Drupal 6 7/16/07
• uses the "tidy" extension (if installed) in php 5.x to clean up the HTML of postings & stories 7/30/07
• bug fixes (possibly) related to drupal 5.2 where the fora had the wrong "changed" date resulting in misordered node lists. 7/30/07
• polls properly remember date of last comment 7/30/07

If you have the "tidy" PHP extension, it should close all open HTML tags and stuff in stories.

update: I added to the script some more debug messages in case something goes wrong (as recommended by me) - that pushed the version up to 0.03.

Also, now MySQL connection collation is set to utf8 before doing anything - this should fix the problems with non-English characters in the imported texts (especially accented/umlauted/etc characters). Thus I upped the version to 0.04. For reference and in case 0.04 doesn't work for you, I keep the older versions as well.


Download the latest phpnuke2drupal-5.x-0.04-W.zip or the latest tar-gzipped version of the phpnuke2drupal migration script.

update: to import Reviews and WebLinks from PHPNuke to Drupal, have a look at this and this comments; you will need CCK, and might need CCK Link Field to import reviews/weblinks.

Previous (older) versions:
phpnuke2drupal-5.x-0.03-W.zip
phpnuke2drupal-5.x-0.03-W.tar.gz
phpnuke2drupal-5.x-0.02-w.zip
phpnuke2drupal-5.x-0.02-w.tar.gz
phpnuke2drupal-5.x-0.01 (VValdo's huge improvements first appear here)
first version with my minor modifications

IMPORTANT: for migration to work you will need MySQL version to be >= 4.1.1. Earlier versions do not support the STR_TO_DATE() function, used in the migration script.

IMPORTANT: if you want to migrate forum entries from PHPNuke to Drupal, be sure to setup forum in your Drupal installation first - otherwise migration of forum entries will fail. See comments 13-17 below for user experiences.

If you do not get any help here, there is also a drupal.org migration thread. I will keep this page updated for as long as I get new information about the migration script.

If you encounter any problems during migration - just search/scan the comments, your problem might have been already solved by someone else.



]]> http://bogdan.org.ua/2006/09/08/php-nuke-6-06-5-to-drupal-4-7-x5-x-migration-conversion.html/feed 74 http://bogdan.org.ua/2006/09/06/allow-posting-duplicate-form-name-entries-with-different-values.html http://bogdan.org.ua/2006/09/06/allow-posting-duplicate-form-name-entries-with-different-values.html#comments Wed, 06 Sep 2006 09:04:41 +0000 Bogdan http://www.bogdan.org.ua/2006/09/06/allow-posting-duplicate-form-name-entries-with-different-values.html Sometimes, writing automatic HTML forms processors, you need to post several values with the same name of the form field, e.g.:
collection_gene = str_chrom_name
collection_gene = gene_stable_id

This is against the RFC on form fields design and submitting, but this approach is used - for example, by Ensembl. I spent some time to figure out how to make HTTP_Client and HTTP_Request submit multiple 'name-value' pairs instead of one (the latest defined, which overrides the previous). The solution is extremely simple:


PLAIN TEXT
PHP:
1. /*
2. allow posting duplicate form-name
3. entries with different values
4. */
5. $params['useBrackets'] = false;
6. $req = &new HTTP_Client($params, $headers);

After setting 'useBrackets' parameter to 'false', HTTP_Client/HTTP_Request do post multiple lines as you would want it.

Hope this saves you some time.



]]> http://bogdan.org.ua/2006/09/06/allow-posting-duplicate-form-name-entries-with-different-values.html/feed 0 http://bogdan.org.ua/2006/09/06/avoiding-out-of-memory-fatal-error-when-using-http_client-or-http_request.html http://bogdan.org.ua/2006/09/06/avoiding-out-of-memory-fatal-error-when-using-http_client-or-http_request.html#comments Wed, 06 Sep 2006 08:58:29 +0000 Bogdan http://www.bogdan.org.ua/2006/09/06/avoiding-out-of-memory-fatal-error-when-using-http_client-or-http_request.html If you fetch large amounts of data (e.g. over 2MB per request) using HTTP_Client (or HTTP_request), you may get "out of memory" fatal errors, especially if:
1. memory_limit is set to default 8M, and
2. you process multiple pages using single non-reset instance of HTTP_Client object.

This problem can manifest itself by producing fatal error after a couple of cycles of successful page retrieval - but always, if run with the same parameters, after some constant or only slightly variable number of cycles.

In my case the problem was that HTTP_Request (a dependancy of HTTP_Client) was holding in memory all the previously fetched pages of the current session (the 'history' feature). To force HTTP_Request to hold only the most recent page, you need to 'disable' history after creating the HTTP_Client or HTTP_Request object instance:

PLAIN TEXT
PHP:
1. $req = &new HTTP_Client($params, $headers);
2. // disable history to save memory
3. $req->enableHistory(false);

Hope this helps you.



]]> http://bogdan.org.ua/2006/09/06/avoiding-out-of-memory-fatal-error-when-using-http_client-or-http_request.html/feed 0 http://bogdan.org.ua/2006/09/06/using-pear-http_client-or-http_request-with-http-proxy-example.html http://bogdan.org.ua/2006/09/06/using-pear-http_client-or-http_request-with-http-proxy-example.html#comments Wed, 06 Sep 2006 08:00:46 +0000 Bogdan http://www.bogdan.org.ua/2006/09/06/using-pear-http_client-or-http_request-with-http-proxy.html If you happen to write PHP script, which uses either HTTP_Client or its dependancy HTTP_Request from PEAR, and the script is supposed to work through the HTTP proxy - here are the sample settings you need:
PLAIN TEXT
PHP:
1. $params['proxy_user'] = 'proxy_username';
2. $params['proxy_pass'] = 'proxy_password';
3. $params['proxy_host'] = 'proxy_hostname_or_ip';
4. $params['proxy_port'] = 8080; // 3128, ...
5.  
6. // I assume $headers were set somewhere else
7. $req = &new HTTP_Client($params, $headers);

If your proxy does not need authorization - just drop the proxy_user and proxy_pass parameters.



]]> http://bogdan.org.ua/2006/09/06/using-pear-http_client-or-http_request-with-http-proxy-example.html/feed 1