Autarchy of the Private Cave

Tiny bits of bioinformatics, [web-]programming etc

    Avoiding out of memory fatal error when using HTTP_Client or HTTP_Request

    6th September 2006

    If you fetch large amounts of data (e.g. over 2MB per request) using HTTP_Client (or HTTP_request), you may get “out of memory” fatal errors, especially if:

    1. memory_limit is set to default 8M, and
    2. you process multiple pages using single non-reset instance of HTTP_Client object.

    This problem can manifest itself by producing fatal error after a couple of cycles of successful page retrieval – but always, if run with the same parameters, after some constant or only slightly variable number of cycles.

    In my case the problem was that HTTP_Request (a dependancy of HTTP_Client) was holding in memory all the previously fetched pages of the current session (the ‘history’ feature). To force HTTP_Request to hold only the most recent page, you need to ‘disable’ history after creating the HTTP_Client or HTTP_Request object instance:

    1. $req = &new HTTP_Client($params, $headers);
    2. // disable history to save memory
    3. $req->enableHistory(false);

    Hope this helps you.

    Share

    Leave a Reply

    XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>