<?xml version="1.0" encoding="UTF-8"?> <rss
version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
> <channel><title>Autarchy of the Private CaveProgramming &#187;</title> <atom:link href="http://bogdan.org.ua/categories/programming/feed" rel="self" type="application/rss+xml" /><link>http://bogdan.org.ua</link> <description>Tiny bits of bioinformatics, [web-]programming etc</description> <lastBuildDate>Tue, 15 May 2012 21:56:55 +0000</lastBuildDate> <language>en</language> <sy:updatePeriod>hourly</sy:updatePeriod> <sy:updateFrequency>1</sy:updateFrequency> <generator>http://wordpress.org/?v=3.3.2</generator> <item><title>Beanstalkd and related tools for easy parallelizing and backgrounding</title><link>http://bogdan.org.ua/2012/02/18/beanstalkd-related-tools-for-easy-parallelizing-backgrounding.html</link> <comments>http://bogdan.org.ua/2012/02/18/beanstalkd-related-tools-for-easy-parallelizing-backgrounding.html#comments</comments> <pubDate>Sat, 18 Feb 2012 11:07:56 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Python]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[beanstalk]]></category> <category><![CDATA[beanstalkd]]></category> <category><![CDATA[queue]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1818</guid> <description><![CDATA[beanstalkd: a simple, fast work queue. Jack and the Beanstalkd: a web-app for basic work queue administration. beanstalkc: a simple beanstalkd client library for Python. queueit: a CLI interface tool which helps to integrate beanstalkd into shell scripts.]]></description> <content:encoded><![CDATA[<p><a
href="http://kr.github.com/beanstalkd/" rel="nofollow" >beanstalkd</a>: a simple, fast work queue.<br
/> <a
href="https://github.com/andreisavu/django-jack" rel="nofollow" >Jack and the Beanstalkd</a>: a web-app for basic work queue administration.<br
/> <a
href="https://github.com/earl/beanstalkc" rel="nofollow" >beanstalkc</a>: a simple beanstalkd client library for Python.<br
/> <a
href="https://github.com/chexov/queueit" rel="nofollow" >queueit</a>: a CLI interface tool which helps to integrate beanstalkd into shell scripts.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2012/02/18/beanstalkd-related-tools-for-easy-parallelizing-backgrounding.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2012/02/18/beanstalkd-related-tools-for-easy-parallelizing-backgrounding.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2012%2F02%2F18%2Fbeanstalkd-related-tools-for-easy-parallelizing-backgrounding.html&amp;title=Beanstalkd%20and%20related%20tools%20for%20easy%20parallelizing%20and%20backgrounding" id="wpa2a_4"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2012/02/18/beanstalkd-related-tools-for-easy-parallelizing-backgrounding.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Python performance: set vs list</title><link>http://bogdan.org.ua/2011/08/15/python-performance-set-vs-list.html</link> <comments>http://bogdan.org.ua/2011/08/15/python-performance-set-vs-list.html#comments</comments> <pubDate>Mon, 15 Aug 2011 09:29:04 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Notepad]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Python]]></category> <category><![CDATA[list]]></category> <category><![CDATA[membership]]></category> <category><![CDATA[performance]]></category> <category><![CDATA[set]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1673</guid> <description><![CDATA[Sometimes there is a need to be sure that no identifier is processed twice - for example, when parsing a file into a database, with file potentially containing duplicate records. An obvious solution is to properly wrap the DB insertion code into try...except block, and process duplicate primary ID exceptions. Another, sometimes more desired solution [...]]]></description> <content:encoded><![CDATA[<p>Sometimes there is a need to be sure that no identifier is processed twice - for example, when parsing a file into a database, with file potentially containing duplicate records. An obvious solution is to properly wrap the DB insertion code into try...except block, and process <em>duplicate primary ID</em> exceptions. Another, sometimes more desired solution is to maintain a set/list of processed IDs internally, and check against that list prior to attempting the insertion of anything. So is it a set or a list?</p><p>There are already quite a few internet resources discussing "python set vs list", but probably the simplest while elegant way to test that is below.<br
/> <span
id="more-1673"></span><br
/> First, test the speed of adding/appending to a set or a list (here, I'm mimicking the real-life application, thus the test case has an optional loop):</p><div
class="igBar"><span
id="lcode-3"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('code-3'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">CODE:</span><div
id="code-3"><div
class="code"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">$python -mtimeit -s <span
style="color:#CC0000;">'myset = set()'</span> <span
style="color:#CC0000;">'for x in xrange(1000): myset.add(x)'</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#800000;color:#800000;">10000</span> loops, best of <span
style="color:#800000;color:#800000;">3</span>: <span
style="color:#800000;color:#800000;">133</span> usec per loop</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">$python -mtimeit -s <span
style="color:#CC0000;">'tmp = list()'</span> <span
style="color:#CC0000;">'for x in xrange(1000): tmp.append(x)'</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#800000;color:#800000;">10000</span> loops, best of <span
style="color:#800000;color:#800000;">3</span>: <span
style="color:#800000;color:#800000;">116</span> usec per loop</div></li></ol></div></div></div><p></p><p>As we can see, set and list are comparable in the speed of adding new items, with list being slightly (~12%) faster than set.</p><p>Now, the speed of membership testing: 'x in tmp'. For this test, I've deliberately chosen an imbalance of True (1%) and False (99%) results for the test - again, mimicking the real problem I have at hand:</p><div
class="igBar"><span
id="lcode-4"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('code-4'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">CODE:</span><div
id="code-4"><div
class="code"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">$python -mtimeit -s <span
style="color:#CC0000;">'tmp = set()'</span> -s <span
style="color:#CC0000;">'for x in xrange(1000): tmp.add(x)'</span> <span
style="color:#CC0000;">'for x in xrange(100000): x in tmp'</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#800000;color:#800000;">100</span> loops, best of <span
style="color:#800000;color:#800000;">3</span>: <span
style="color:#800000;color:#800000;">7</span>.<span
style="color:#800000;color:#800000;">27</span> msec per loop</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">$python -mtimeit -s <span
style="color:#CC0000;">'tmp = list()'</span> -s <span
style="color:#CC0000;">'for x in xrange(1000): tmp.append(x)'</span> <span
style="color:#CC0000;">'for x in xrange(100000): x in tmp'</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#800000;color:#800000;">10</span> loops, best of <span
style="color:#800000;color:#800000;">3</span>: <span
style="color:#800000;color:#800000;">2</span>.<span
style="color:#800000;color:#800000;">12</span> sec per loop</div></li></ol></div></div></div><p></p><p>List is much slower for membership testing, while <a
href="http://en.wikipedia.org/wiki/Collection_(computing)#Sets" rel="nofollow" >sets were designed to be fast for doing just that</a>.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2011/08/15/python-performance-set-vs-list.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2011/08/15/python-performance-set-vs-list.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2011%2F08%2F15%2Fpython-performance-set-vs-list.html&amp;title=Python%20performance%3A%20set%20vs%20list" id="wpa2a_8"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2011/08/15/python-performance-set-vs-list.html/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> <item><title>Using Python in a Java project</title><link>http://bogdan.org.ua/2011/05/17/using-python-in-a-java-project.html</link> <comments>http://bogdan.org.ua/2011/05/17/using-python-in-a-java-project.html#comments</comments> <pubDate>Tue, 17 May 2011 11:33:52 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[Movies]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Python]]></category> <category><![CDATA[Groovy]]></category> <category><![CDATA[java]]></category> <category><![CDATA[Jython]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1589</guid> <description><![CDATA[If you are a Python zealot, and Java doesn't feel right, but the project you are working on is a Java project - try Jython - Python for the Java platform, compile your python scripts into Java bytecode Groovy - not Python, but still a scripting language which compiles to jars]]></description> <content:encoded><![CDATA[<p>If you are a Python zealot, and Java <em>doesn't feel right</em>, but the project you are working on is a Java project - try</p><ul><li><a
href="http://www.jython.org/" rel="nofollow" >Jython</a> - Python for the Java platform, compile your python scripts into Java bytecode</li><li><a
href="http://groovy.codehaus.org/" rel="nofollow" >Groovy</a> - not Python, but still a scripting language which compiles to jars</li></ul><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2011/05/17/using-python-in-a-java-project.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2011/05/17/using-python-in-a-java-project.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2011%2F05%2F17%2Fusing-python-in-a-java-project.html&amp;title=Using%20Python%20in%20a%20Java%20project" id="wpa2a_12"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2011/05/17/using-python-in-a-java-project.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Introduction to Python for bioinformatics</title><link>http://bogdan.org.ua/2011/02/25/introduction-to-python-for-bioinformatics.html</link> <comments>http://bogdan.org.ua/2011/02/25/introduction-to-python-for-bioinformatics.html#comments</comments> <pubDate>Fri, 25 Feb 2011 12:03:55 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Bioinformatics]]></category> <category><![CDATA[Links]]></category> <category><![CDATA[Python]]></category> <category><![CDATA[Software]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1451</guid> <description><![CDATA[This overview presentation is two years old, but still a highly valuable resource: modules and tools mentioned are alive and useful. I think this is the second presentation by Giovanni I'm embedding (first one being about GNU/make for bioinformatics). Introduction to python for bioinformatics]]></description> <content:encoded><![CDATA[<p>This overview presentation is two years old, but still a highly valuable resource: modules and tools mentioned are alive and useful.<br
/> I think this is the second presentation by Giovanni I'm embedding (first one being about GNU/make for bioinformatics).</p><div
style="width:425px" id="__ss_1320208"><strong
style="display:block;margin:12px 0 4px"><a
href="http://www.slideshare.net/giovanni/introduction-to-python-for-bioinformatics" rel="nofollow"  title="Introduction to python for bioinformatics">Introduction to python for bioinformatics</a></strong><object
id="__sse1320208" width="425" height="355"><param
name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=introduction-090421044444-phpapp02&#038;stripped_title=introduction-to-python-for-bioinformatics&#038;userName=giovanni" /><param
name="allowFullScreen" value="true"/><param
name="allowScriptAccess" value="always"/><embed
name="__sse1320208" src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=introduction-090421044444-phpapp02&#038;stripped_title=introduction-to-python-for-bioinformatics&#038;userName=giovanni" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object></div><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2011/02/25/introduction-to-python-for-bioinformatics.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2011/02/25/introduction-to-python-for-bioinformatics.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2011%2F02%2F25%2Fintroduction-to-python-for-bioinformatics.html&amp;title=Introduction%20to%20Python%20for%20bioinformatics" id="wpa2a_16"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2011/02/25/introduction-to-python-for-bioinformatics.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>How to easily install any PyPi/easy_install python module on Debian</title><link>http://bogdan.org.ua/2011/02/16/how-to-easily-install-any-pypi-easy_install-python-module-on-debian.html</link> <comments>http://bogdan.org.ua/2011/02/16/how-to-easily-install-any-pypi-easy_install-python-module-on-debian.html#comments</comments> <pubDate>Wed, 16 Feb 2011 17:22:20 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[*nix]]></category> <category><![CDATA[how-to]]></category> <category><![CDATA[Notepad]]></category> <category><![CDATA[Python]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[Debian]]></category> <category><![CDATA[debianize]]></category> <category><![CDATA[easy_install]]></category> <category><![CDATA[pycassa]]></category> <category><![CDATA[PyPi]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1430</guid> <description><![CDATA[Imagine you need to install pycassa (which uses easy_install). Here are the 2 (at maximum) very simple steps to have it properly debianized and installed on your Debian/Ubuntu: if you don't have the python-stdeb package: sudo aptitude install python-stdeb pypi-install pycassa That's it. Refer to stdeb readme for more information. You will need that if [...]]]></description> <content:encoded><![CDATA[<p>Imagine you need to install <a
href="https://github.com/pycassa/pycassa" rel="nofollow" >pycassa</a> (which uses easy_install). Here are the 2 (at maximum) very simple steps to have it properly debianized and installed on your Debian/Ubuntu:</p><ul><li>if you don't have the python-stdeb package: <strong>sudo aptitude install python-stdeb</strong></li><li><strong>pypi-install pycassa</strong></li></ul><p>That's it.</p><p>Refer to <a
href="https://github.com/astraw/stdeb#readme" rel="nofollow" >stdeb readme</a> for more information. You will need that if there are dependencies - which might not be resolved automatically by stdeb.</p><p>Before stdeb, it wasn't exactly trivial to <a
href="http://showmedo.com/videotutorials/video?name=linuxJensMakingDeb" rel="nofollow" >make a .deb from python module</a>.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2011/02/16/how-to-easily-install-any-pypi-easy_install-python-module-on-debian.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2011/02/16/how-to-easily-install-any-pypi-easy_install-python-module-on-debian.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2011%2F02%2F16%2Fhow-to-easily-install-any-pypi-easy_install-python-module-on-debian.html&amp;title=How%20to%20easily%20install%20any%20PyPi%2Feasy_install%20python%20module%20on%20Debian" id="wpa2a_20"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2011/02/16/how-to-easily-install-any-pypi-easy_install-python-module-on-debian.html/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> <item><title>DreamPie: the ultimate Python shell</title><link>http://bogdan.org.ua/2010/08/11/dreampie-the-ultimate-python-shell.html</link> <comments>http://bogdan.org.ua/2010/08/11/dreampie-the-ultimate-python-shell.html#comments</comments> <pubDate>Wed, 11 Aug 2010 14:54:40 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Python]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1107</guid> <description><![CDATA[DreamPie: the Python shell you've always dreamt about! • Type your code in the lower pane of the window. To execute, press Ctrl+Enter. One-liners can be executed by simply pressing Enter; If you don't want them executed, press Space and then Enter. • Use Ctrl+Up and Ctrl+Down to navigate between code segments you've already executed. [...]]]></description> <content:encoded><![CDATA[<p><a
href="http://dreampie.sourceforge.net/" rel="nofollow" >DreamPie: the Python shell you've always dreamt about!</a></p><blockquote><p>• Type your code in the lower pane of the window. To execute, press Ctrl+Enter. One-liners can be executed by simply pressing Enter; If you don't want them executed, press Space and then Enter.</p><p>• Use Ctrl+Up and Ctrl+Down to navigate between code segments you've already executed. You can write a few letters before pressing Ctrl+Up, and DreamPie will only search through code segments starting with those letters.</p><p>• Press Tab or Ctrl+Space to show a list of completions to the current expression. It will also complete file names!</p><p>• Your results are stored in variables named _0, _1, and so on.</p><p>• Type a function name and press the space key and DreamPie will automatically add parentheses for you!</p></blockquote><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2010/08/11/dreampie-the-ultimate-python-shell.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2010/08/11/dreampie-the-ultimate-python-shell.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2010%2F08%2F11%2Fdreampie-the-ultimate-python-shell.html&amp;title=DreamPie%3A%20the%20ultimate%20Python%20shell" id="wpa2a_24"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2010/08/11/dreampie-the-ultimate-python-shell.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Good online Python book with code examples</title><link>http://bogdan.org.ua/2010/03/01/good-online-python-book-with-code-examples.html</link> <comments>http://bogdan.org.ua/2010/03/01/good-online-python-book-with-code-examples.html#comments</comments> <pubDate>Mon, 01 Mar 2010 18:03:11 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Python]]></category> <category><![CDATA[book]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=1008</guid> <description><![CDATA[Building Skills in Python: A Programmer's Introduction to Python by Steven F. Lott (© 2002, 2005, 2007, 2008 Steven F. Lott).]]></description> <content:encoded><![CDATA[<p><a
href="http://www.linuxtopia.org/online_books/programming_books/python_programming/index.html" rel="nofollow" >Building Skills in Python: A Programmer's Introduction to Python</a> by Steven F. Lott (© 2002, 2005, 2007, 2008 Steven F. Lott).</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2010/03/01/good-online-python-book-with-code-examples.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2010/03/01/good-online-python-book-with-code-examples.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2010%2F03%2F01%2Fgood-online-python-book-with-code-examples.html&amp;title=Good%20online%20Python%20book%20with%20code%20examples" id="wpa2a_28"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2010/03/01/good-online-python-book-with-code-examples.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>R script to filter probesets with log-expression values below the lowest spike-in</title><link>http://bogdan.org.ua/2010/01/27/r-script-to-filter-probesets-with-log-expression-values-below-the-lowest-spike-in.html</link> <comments>http://bogdan.org.ua/2010/01/27/r-script-to-filter-probesets-with-log-expression-values-below-the-lowest-spike-in.html#comments</comments> <pubDate>Wed, 27 Jan 2010 12:44:02 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Bioinformatics]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Science]]></category> <category><![CDATA[Affymetrix]]></category> <category><![CDATA[filter]]></category> <category><![CDATA[log-expression]]></category> <category><![CDATA[microarray]]></category> <category><![CDATA[probeset]]></category> <category><![CDATA[R]]></category> <category><![CDATA[spike-in]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=985</guid> <description><![CDATA[Sometimes there is a need to remove all the probesets, which have expression values below the minimal spike-in intensity on the Affymetrix microarray. The reasoning behind this procedure is simple: minimal-expression spike-ins represent the bottom margin of microarray sensitivity, and anything below that margin cannot be reliably quantified - which also means that both fold-change [...]]]></description> <content:encoded><![CDATA[<p>Sometimes there is a need to remove all the probesets, which have expression values below the minimal spike-in intensity on the <a
href="http://www.affymetrix.com/" rel="nofollow" >Affymetrix</a> microarray. The reasoning behind this procedure is simple: minimal-expression spike-ins represent the bottom margin of microarray sensitivity, and anything below that margin cannot be reliably quantified - which also means that both fold-change and p-value of expression variance will be unreliable for these probesets.</p><p>Here's a simple <a
href="http://www.r-project.org/" rel="nofollow" >R</a> script to do just that. It is abundantly commented, and also contains an optional (commented out) fragment which allows the removal of more low-variance, low-intensity probesets.</p><p><span
id="more-985"></span><br
/> <em>Hint: click the "plain text" box header to be able to right-click the code, "Select All", and then "Copy".</em></p><div
class="igBar"><span
id="lcode-6"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('code-6'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">CODE:</span><div
id="code-6"><div
class="code"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">filter_below_spikes = function<span
style="color:#006600; font-weight:bold;">&#40;</span>eset<span
style="color:#006600; font-weight:bold;">&#41;</span> <span
style="color:#006600; font-weight:bold;">&#123;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Finds max<span
style="color:#006600; font-weight:bold;">&#40;</span>lowest<span
style="color:#006600; font-weight:bold;">&#41;</span> AFFX/spike-in intensity, and removes rows consisting entirely of values below max<span
style="color:#006600; font-weight:bold;">&#40;</span>lowest<span
style="color:#006600; font-weight:bold;">&#41;</span>.</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # @param eset</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # ExpressionSet</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # @returns</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # exprs<span
style="color:#006600; font-weight:bold;">&#40;</span>ExpressionSet<span
style="color:#006600; font-weight:bold;">&#41;</span>, filtered</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Without Biobase exprs<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#006600; font-weight:bold;">&#41;</span> will not work.</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; <span
style="">require</span><span
style="color:#006600; font-weight:bold;">&#40;</span>Biobase<span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; expr = exprs<span
style="color:#006600; font-weight:bold;">&#40;</span>eset<span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # <span
style="color:#CC0000;">'expr'</span> sample:</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span
style="color:#800000;color:#800000;">1</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # <span
style="color:#800000;color:#800000;">1367452</span>_at <span
style="color:#800000;color:#800000;">10</span>.<span
style="color:#800000;color:#800000;">880208</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # <span
style="color:#800000;color:#800000;">1367453</span>_at <span
style="color:#800000;color:#800000;">10</span>.<span
style="color:#800000;color:#800000;">554647</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; cat<span
style="color:#006600; font-weight:bold;">&#40;</span>nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"rows before filtering.<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Make a vector of spike row names.</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; <span
style="">spikes</span> = grep<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#CC0000;">"AFFX"</span>, rownames<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, value = TRUE<span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; cat<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#CC0000;">"Expression matrix has"</span>, length<span
style="color:#006600; font-weight:bold;">&#40;</span>spikes<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"spike-in rows.<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; cat<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#CC0000;">"Summary of spike-in values distribution follows:<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; print<span
style="color:#006600; font-weight:bold;">&#40;</span>summary<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#91;</span>spikes, <span
style="color:#006600; font-weight:bold;">&#93;</span><span
style="color:#006600; font-weight:bold;">&#41;</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Find max<span
style="color:#006600; font-weight:bold;">&#40;</span>lowest<span
style="color:#006600; font-weight:bold;">&#41;</span> spike-in values.</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; <span
style="">minval_max</span> = max<span
style="color:#006600; font-weight:bold;">&#40;</span>as.<span
style="">double</span><span
style="color:#006600; font-weight:bold;">&#40;</span>substr<span
style="color:#006600; font-weight:bold;">&#40;</span>grep<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#CC0000;">"Min"</span>, summary<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#91;</span>spikes, <span
style="color:#006600; font-weight:bold;">&#93;</span><span
style="color:#006600; font-weight:bold;">&#41;</span>, value = TRUE<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#800000;color:#800000;">10</span>, <span
style="color:#800000;color:#800000;">14</span><span
style="color:#006600; font-weight:bold;">&#41;</span><span
style="color:#006600; font-weight:bold;">&#41;</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; cat<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#CC0000;">"max(minimal spike-in log-intensity values) ="</span>, minval_max, <span
style="color:#CC0000;">"<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Remove spike-ins from expr.</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; <span
style="">expr</span> = expr<span
style="color:#006600; font-weight:bold;">&#91;</span>grep<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#CC0000;">"AFFX"</span>, rownames<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, value = TRUE, invert = TRUE<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#006600; font-weight:bold;">&#93;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; nospike_rows = nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; cat<span
style="color:#006600; font-weight:bold;">&#40;</span>nospike_rows, <span
style="color:#CC0000;">"rows remaining after the removal of"</span>, length<span
style="color:#006600; font-weight:bold;">&#40;</span>spikes<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"spike-in probesets.<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Optional: calculate max<span
style="color:#006600; font-weight:bold;">&#40;</span>SD<span
style="color:#006600; font-weight:bold;">&#41;</span> of all removed rows.</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #bad_sds_max = max<span
style="color:#006600; font-weight:bold;">&#40;</span>apply<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#91;</span>!apply<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#006600; font-weight:bold;">&#40;</span>expr&gt; minval_max<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#800000;color:#800000;">1</span>, any<span
style="color:#006600; font-weight:bold;">&#41;</span>,<span
style="color:#006600; font-weight:bold;">&#93;</span>, <span
style="color:#800000;color:#800000;">1</span>, sd<span
style="color:#006600; font-weight:bold;">&#41;</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Now remove all rows, where each value is &lt;= minval_max.</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; <span
style="">expr</span> = expr<span
style="color:#006600; font-weight:bold;">&#91;</span>!apply<span
style="color:#006600; font-weight:bold;">&#40;</span><span
style="color:#006600; font-weight:bold;">&#40;</span>expr &lt;= minval_max<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#800000;color:#800000;">1</span>, all<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#006600; font-weight:bold;">&#93;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; cat<span
style="color:#006600; font-weight:bold;">&#40;</span>nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"rows remaining after filtering out"</span>, nospike_rows - nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"probesets with all values below"</span>, minval_max, <span
style="color:#CC0000;">"<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #cat<span
style="color:#006600; font-weight:bold;">&#40;</span>bad_sds_max, <span
style="color:#CC0000;">"is max(SD) of all"</span>, nospike_rows - nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"filtered probesets with all values below"</span>, minval_max, <span
style="color:#CC0000;">"<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; # Optional: Remove *some* of the rows, which have at least one value below minval_max, and row_SD &lt;= bad_sds_max.</div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #pre_final_rows = nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #expr = expr<span
style="color:#006600; font-weight:bold;">&#91;</span><span
style="color:#006600; font-weight:bold;">&#40;</span>apply<span
style="color:#006600; font-weight:bold;">&#40;</span>expr, <span
style="color:#800000;color:#800000;">1</span>, sd<span
style="color:#006600; font-weight:bold;">&#41;</span>&gt; bad_sds_max<span
style="color:#006600; font-weight:bold;">&#41;</span> | <span
style="color:#006600; font-weight:bold;">&#40;</span>apply<span
style="color:#006600; font-weight:bold;">&#40;</span>expr, <span
style="color:#800000;color:#800000;">1</span>, min<span
style="color:#006600; font-weight:bold;">&#41;</span>&gt; minval_max<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#006600; font-weight:bold;">&#93;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #cat<span
style="color:#006600; font-weight:bold;">&#40;</span>pre_final_rows-nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"rows with SD &lt;="</span>, bad_sds_max, <span
style="color:#CC0000;">"and min(row) &lt;="</span>, minval_max, <span
style="color:#CC0000;">"were removed.<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; #cat<span
style="color:#006600; font-weight:bold;">&#40;</span>nrow<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span>, <span
style="color:#CC0000;">"final rows returned.<span
style="color:#000099; font-weight:bold;">\n</span>"</span><span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp;</div></li><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;">&nbsp; &nbsp; return<span
style="color:#006600; font-weight:bold;">&#40;</span>expr<span
style="color:#006600; font-weight:bold;">&#41;</span></div></li><li
style="font-weight: bold;color:#26536A;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#006600; font-weight:bold;">&#125;</span></div></li></ol></div></div></div><p></p><p>Sample use:</p><blockquote><p> > source("script.R")<br
/> > expr.filtered = filter_below_spikes(eset)</p></blockquote><p>Comments and suggestions are welcome.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2010/01/27/r-script-to-filter-probesets-with-log-expression-values-below-the-lowest-spike-in.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2010/01/27/r-script-to-filter-probesets-with-log-expression-values-below-the-lowest-spike-in.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2010%2F01%2F27%2Fr-script-to-filter-probesets-with-log-expression-values-below-the-lowest-spike-in.html&amp;title=R%20script%20to%20filter%20probesets%20with%20log-expression%20values%20below%20the%20lowest%20spike-in" id="wpa2a_32"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2010/01/27/r-script-to-filter-probesets-with-log-expression-values-below-the-lowest-spike-in.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Cloud-based bruteforcing, slowloris, and Golang: links</title><link>http://bogdan.org.ua/2009/11/13/cloud-based-bruteforcing-slowloris-and-golang-links.html</link> <comments>http://bogdan.org.ua/2009/11/13/cloud-based-bruteforcing-slowloris-and-golang-links.html#comments</comments> <pubDate>Fri, 13 Nov 2009 16:34:51 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[Misc]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Security]]></category> <category><![CDATA[Web]]></category> <category><![CDATA[Apache]]></category> <category><![CDATA[bruteforce]]></category> <category><![CDATA[complexity]]></category> <category><![CDATA[cost]]></category> <category><![CDATA[DOS]]></category> <category><![CDATA[EC]]></category> <category><![CDATA[go]]></category> <category><![CDATA[golang]]></category> <category><![CDATA[mod]]></category> <category><![CDATA[password]]></category> <category><![CDATA[security]]></category> <category><![CDATA[slowloris]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=931</guid> <description><![CDATA[A nice report on the cost of bruteforcing variable-length and variable-complexity passwords using cloud computing services (e.g. Amazon's EC). There's a kind of a tutorial in their previous post. Slow DoS attack with just 1 computer against a number of web servers, including Apache: slowloris. There is a solution for Apache, packaged for RedHat and [...]]]></description> <content:encoded><![CDATA[<p>A nice report on the <a
href="http://news.electricalchemy.net/2009/10/password-cracking-in-cloud-part-5.html" rel="nofollow" >cost of bruteforcing variable-length and variable-complexity passwords</a> using cloud computing services (e.g. Amazon's <abbr
title="Elastic Cloud">EC</abbr>). There's a kind of a tutorial in their previous post.</p><p>Slow <abbr
title="Denial of Service">DoS</abbr> attack with just 1 computer against a number of web servers, including Apache: <a
href="http://ha.ckers.org/slowloris/" rel="nofollow" >slowloris</a>. There is a solution for Apache, packaged for <a
href="ftp://ftp.monshouwer.eu/pub/linux/mod_antiloris/" rel="nofollow" >RedHat</a> and also available for <a
href="http://www.liranuna.com/securing-your-debian-server-against-slowloris/" rel="nofollow" >Debian</a>.</p><p>Finally, there's <a
href="http://golang.org/" rel="nofollow" >Go programming language</a>. The most inspiring promise to me personally is the ease of execution parallelization with language's built-in syntactic constructs. That is something highly desired. Also, I like that it is a compiled language. However, it might be 10%-20% slower than pure C. Let's see how it grows.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/11/13/cloud-based-bruteforcing-slowloris-and-golang-links.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/11/13/cloud-based-bruteforcing-slowloris-and-golang-links.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F11%2F13%2Fcloud-based-bruteforcing-slowloris-and-golang-links.html&amp;title=Cloud-based%20bruteforcing%2C%20slowloris%2C%20and%20Golang%3A%20links" id="wpa2a_36"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/11/13/cloud-based-bruteforcing-slowloris-and-golang-links.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Configuring web-server: for production and for development</title><link>http://bogdan.org.ua/2009/10/25/configuring-web-server-for-production-and-for-development.html</link> <comments>http://bogdan.org.ua/2009/10/25/configuring-web-server-for-production-and-for-development.html#comments</comments> <pubDate>Sun, 25 Oct 2009 15:16:10 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[Notepad]]></category> <category><![CDATA[PHP]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[development]]></category> <category><![CDATA[how-to]]></category> <category><![CDATA[production]]></category> <category><![CDATA[web-server]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=680</guid> <description><![CDATA[Production: see http://www.howtoforge.com/how-to-set-up-apache2-with-mod_fcgid-and-php5-on-debian-etch - it is for Debian Etch (which is old-stable), but many of the steps apply equally well to Debian Lenny (current-stable). Also, this is a very basic guide, as if you are going to host multiple sites from multiple clients, you most definitely will need some hosting control panel. Development: see http://www.ruzee.com/blog/2009/01/apache-virtual-hosts-a-clean-setup-for-php-developers. [...]]]></description> <content:encoded><![CDATA[<p><strong>Production</strong>: see <a
href="http://www.howtoforge.com/how-to-set-up-apache2-with-mod_fcgid-and-php5-on-debian-etch" rel="nofollow" >http://www.howtoforge.com/how-to-set-up-apache2-with-mod_fcgid-and-php5-on-debian-etch</a> - it is for Debian Etch (which is old-stable), but many of the steps apply equally well to Debian Lenny (current-stable). Also, this is a very basic guide, as if you are going to host multiple sites from multiple clients, you most definitely will need some <a
href="http://bogdan.org.ua/2008/09/27/which-hosting-control-panel-to-use-for-a-debian-server.html">hosting control panel</a>.</p><p><strong>Development</strong>: see <a
href="http://www.ruzee.com/blog/2009/01/apache-virtual-hosts-a-clean-setup-for-php-developers" rel="nofollow" >http://www.ruzee.com/blog/2009/01/apache-virtual-hosts-a-clean-setup-for-php-developers</a>. This setup works very well, unless you need to create several virtual hosts every day - in which case necessary actions could be partially scripted.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/10/25/configuring-web-server-for-production-and-for-development.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/10/25/configuring-web-server-for-production-and-for-development.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F10%2F25%2Fconfiguring-web-server-for-production-and-for-development.html&amp;title=Configuring%20web-server%3A%20for%20production%20and%20for%20development" id="wpa2a_40"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/10/25/configuring-web-server-for-production-and-for-development.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Convenient design and debugging of regular expressions under Linux</title><link>http://bogdan.org.ua/2009/10/10/convenient-design-and-debugging-of-regular-expressions-under-linux-tools-comparison.html</link> <comments>http://bogdan.org.ua/2009/10/10/convenient-design-and-debugging-of-regular-expressions-under-linux-tools-comparison.html#comments</comments> <pubDate>Sat, 10 Oct 2009 15:51:18 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[*nix]]></category> <category><![CDATA[Notepad]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[debugging]]></category> <category><![CDATA[kodos]]></category> <category><![CDATA[linux]]></category> <category><![CDATA[RE]]></category> <category><![CDATA[redet]]></category> <category><![CDATA[regexp]]></category> <category><![CDATA[regular expression]]></category> <category><![CDATA[txt2regexp]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=890</guid> <description><![CDATA[Regular expressions (regexps) are powerful indeed. But debugging non-trivial regexps is a burden even if you understand how regexps work, and remember most (if not all) regexp syntax. Miscellaneous tools exist to ease this task. This post was inspired by redet's comparison of regexp helper tools - it could be sufficient to read only that, [...]]]></description> <content:encoded><![CDATA[<p><a
href="http://bogdan.org.ua/wp-content/uploads/2009/10/redet-supported-languages.png"><img
src="http://bogdan.org.ua/wp-content/uploads/2009/10/redet-supported-languages-199x171.png" alt="redet-supported languages" title="redet-supported languages" width="199" height="171" class="alignleft size-thumbnail wp-image-896" /></a>Regular expressions (regexps) are powerful indeed. But debugging non-trivial regexps is a burden even if you understand how regexps work, and remember most (if not all) regexp syntax.</p><p>Miscellaneous tools exist to ease this task. This post was inspired by <a
href="http://billposer.org/Software/redet.html#comparison" rel="nofollow" >redet's comparison of regexp helper tools</a> - it could be sufficient to read only that, if you're going to try the mentioned tools yourself. Otherwise, read on.<br
/> <span
id="more-890"></span><br
/> <a
href="http://kodos.sourceforge.net/" rel="nofollow" >Kodos</a> is an easy-to-use (but powerful enough) Python tool to verify regular expressions against sample user-provided strings. Available for both Linux and Windows. Generates ready-to-use Python regexp code.</p><p><a
href="http://billposer.org/Software/redet.html" rel="nofollow" >redet</a> <a
href="http://billposer.org/Software/redet.html#comparison" rel="nofollow" >assumes some knowledge</a> of REs. Unlike all the other helper tools, it has the widest code generation coverage - just go to "Tools->Save command information" to generate code (Python, Perl, C, grep - just to name a few of the <strong>57 (!)</strong> languages/tools/editors supported). At the same time, redet does not depend on specific versions of all supported RE engines - instead, redet automatically tests for features in available versions.</p><p><a
href="http://laurent.riesterer.free.fr/regexp/" rel="nofollow" >visual-regexp</a> is similar to kodos in features.</p><p><a
href="http://txt2regex.sourceforge.net/" rel="nofollow" >txt2regex</a> is a regex wizard written in bash, capable of generating code for <strong>24</strong> languages/editors/programs (by default, these are enabled: perl, php, postgres, python, sed and vim). It is really simple and convenient, though some understanding of "groups" and "character ranges" is highly recommended.</p><p><a
href="http://websvn.kde.org/trunk/playground/utils/kregexpeditor/" rel="nofollow" class="broken_link">KRegExpEditor</a> was a part of kde3-utils, and seems to be missing from current KDE 4.3. It is similar to kodos and visual-regexp in its philosophy and features.</p><p>I would recommend using one of kodos/visual-regexp/kregexpeditor to quickly test or design your regular expression, then use redet to generate appropriate source code. As an alternative (and when working in a console), just use txt2regexp to generate the code you need.</p><p>Tools I haven't tried include <a
href="http://www.weitz.de/regex-coach/" rel="nofollow" >regex coach</a> and <a
href="http://freshmeat.net/projects/regexpviewer/" rel="nofollow" >regexpviewer</a>.</p><p>Please also see this extensive <a
href="http://billposer.org/Linguistics/Computation/Resources.html#patterns" rel="nofollow" >list of regular expression and pattern-matching related</a> software.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/10/10/convenient-design-and-debugging-of-regular-expressions-under-linux-tools-comparison.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/10/10/convenient-design-and-debugging-of-regular-expressions-under-linux-tools-comparison.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F10%2F10%2Fconvenient-design-and-debugging-of-regular-expressions-under-linux-tools-comparison.html&amp;title=Convenient%20design%20and%20debugging%20of%20regular%20expressions%20under%20Linux" id="wpa2a_44"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/10/10/convenient-design-and-debugging-of-regular-expressions-under-linux-tools-comparison.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Screem HTML editor dies with &#8216;GSlice: assertion failed: sinfo-&gt;n_allocated &gt; 0&#8242;</title><link>http://bogdan.org.ua/2009/08/20/screem-html-editor-dies-with-gslice-assertion-failed-sinfo-n_allocated.html</link> <comments>http://bogdan.org.ua/2009/08/20/screem-html-editor-dies-with-gslice-assertion-failed-sinfo-n_allocated.html#comments</comments> <pubDate>Thu, 20 Aug 2009 11:51:20 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[*nix]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[XHTML/CSS]]></category> <category><![CDATA[CSS]]></category> <category><![CDATA[HTML]]></category> <category><![CDATA[screem]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=847</guid> <description><![CDATA[Screem HTML/XML editor has tag-specific auto-complete, and is a nice editor for web-developers (at least as long as Quanta is not available for Debian testing). However, version 0.16.1 is very unstable, and dies with ***MEMORY-ERROR***: screem[5527]: GSlice: assertion failed: sinfo->n_allocated > 0 As a workaround (initially suggested for the highly similar Firestarter crashes), try running [...]]]></description> <content:encoded><![CDATA[<p><a
href="http://www.screem.org/" rel="nofollow" >Screem HTML/XML editor</a> has tag-specific auto-complete, and is a nice editor for web-developers (at least as long as Quanta is not available for Debian testing).</p><p>However, version 0.16.1 is very unstable, and dies with</p><blockquote><p>***MEMORY-ERROR***: screem[5527]: GSlice: assertion failed: sinfo->n_allocated > 0</p></blockquote><p>As a workaround (initially suggested for the highly similar Firestarter crashes), try running screem with this command:</p><blockquote><p>G_SLICE=always-malloc screem</p></blockquote><p>Too bad last development version of Screem is dated March 2006.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/08/20/screem-html-editor-dies-with-gslice-assertion-failed-sinfo-n_allocated.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/08/20/screem-html-editor-dies-with-gslice-assertion-failed-sinfo-n_allocated.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F08%2F20%2Fscreem-html-editor-dies-with-gslice-assertion-failed-sinfo-n_allocated.html&amp;title=Screem%20HTML%20editor%20dies%20with%20%E2%80%98GSlice%3A%20assertion%20failed%3A%20sinfo-%3En_allocated%20%3E%200%E2%80%B2" id="wpa2a_48"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/08/20/screem-html-editor-dies-with-gslice-assertion-failed-sinfo-n_allocated.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Bad memory handling and server stability</title><link>http://bogdan.org.ua/2009/07/13/bad-memory-handling-and-server-stability.html</link> <comments>http://bogdan.org.ua/2009/07/13/bad-memory-handling-and-server-stability.html#comments</comments> <pubDate>Mon, 13 Jul 2009 09:29:27 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[*nix]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[memory]]></category> <category><![CDATA[mtrace]]></category> <category><![CDATA[RAM]]></category> <category><![CDATA[valgrind]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=813</guid> <description><![CDATA[The two graphs below (clickable) are for CPU and RAM use during a period of a program going wild between 23:17 and 23:41 (24+ minutes of server's downtime). The program was run non-root, it just consumed all the memory it could. It was killed by kernel, so the server started responding without any interventions - [...]]]></description> <content:encoded><![CDATA[<p>The two graphs below (clickable) are for CPU and RAM use during a period of a program going wild between 23:17 and 23:41 (24+ minutes of server's downtime). The program was run non-root, it just consumed all the memory it could. It was killed by kernel, so the server started responding without any interventions - which were hard to perform, because none of the services (including ssh) were responding during downtime.<br
/> <a
href="http://bogdan.org.ua/wp-content/uploads/2009/07/cpugraph.png"><img
src="http://bogdan.org.ua/wp-content/uploads/2009/07/cpugraph-500x130.png" alt="cpugraph" title="cpugraph" width="500" height="130" class="aligncenter size-medium wp-image-814" /></a><br
/> <a
href="http://bogdan.org.ua/wp-content/uploads/2009/07/memgraph.png"><img
src="http://bogdan.org.ua/wp-content/uploads/2009/07/memgraph-500x138.png" alt="memgraph" title="memgraph" width="500" height="138" class="aligncenter size-medium wp-image-815" /></a></p><p>If you happen to be developing a C/C++ program - do use mtrace and valgrind, those are huge helpers against the problems akin to that shown on the graphs.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/07/13/bad-memory-handling-and-server-stability.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/07/13/bad-memory-handling-and-server-stability.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F07%2F13%2Fbad-memory-handling-and-server-stability.html&amp;title=Bad%20memory%20handling%20and%20server%20stability" id="wpa2a_52"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/07/13/bad-memory-handling-and-server-stability.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>C: how to specify comparison operators floating precision</title><link>http://bogdan.org.ua/2009/06/11/c-how-to-specify-comparison-operators-floating-precision.html</link> <comments>http://bogdan.org.ua/2009/06/11/c-how-to-specify-comparison-operators-floating-precision.html#comments</comments> <pubDate>Thu, 11 Jun 2009 17:47:37 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[how-to]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[C]]></category> <category><![CDATA[comparison]]></category> <category><![CDATA[double]]></category> <category><![CDATA[example]]></category> <category><![CDATA[float]]></category> <category><![CDATA[floating]]></category> <category><![CDATA[operator]]></category> <category><![CDATA[precision]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=774</guid> <description><![CDATA[There is no way I'm aware of to do what the title says. However... I'm sure that you are aware of the fact that floats representation in any programming language is limited by the precision of the internal binary representations. In other words, you can never have an exact float representation - there will always [...]]]></description> <content:encoded><![CDATA[<p>There is no way I'm aware of to do what the title says. However...</p><p>I'm sure that you are aware of the fact that floats representation in any programming language is limited by the precision of the internal binary representations. In other words, you can never have an <strong>exact</strong> float representation - there will always be some precision associated with the float you are working with. The simplest example is the difference in precision between the <em>float</em> and <em>double</em> types in <strong>C</strong>.</p><p>Suppose I have the following code fragment:</p><div
class="igBar"><span
id="lc-10"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('c-10'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">C:</span><div
id="c-10"><div
class="c"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color: #b1b100;">if</span> <span
style="color: #66cc66;">&#40;</span> result.<span
style="color: #202020;">score</span> &gt;= input-&gt;raw_cut_off <span
style="color: #66cc66;">&#41;</span></div></li></ol></div></div></div><p></p><p>Both <em>result.score</em> and <em>input->raw_cut_off</em> are of type <strong>float</strong>, and can have positive and negative values. When compared with the greater than or equal ( >= ) operator, it is not always that condition is true - for the precision reasons shortly mentioned above.</p><p>As I already said, there is no precision specification for equality operators in <strong>C</strong>. But it is quite simple to "invent" precision specification; e.g. if I wanted to test for equality only, I could write</p><div
class="igBar"><span
id="lc-11"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('c-11'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">C:</span><div
id="c-11"><div
class="c"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color: #b1b100;">if</span> <span
style="color: #66cc66;">&#40;</span> fabsf<span
style="color: #66cc66;">&#40;</span> result.<span
style="color: #202020;">score</span> - input-&gt;raw_cut_off <span
style="color: #66cc66;">&#41;</span> &lt; <span
style="color: #cc66cc;color:#800000;">0</span>.<span
style="color: #cc66cc;color:#800000;">000001</span> <span
style="color: #66cc66;">&#41;</span></div></li></ol></div></div></div><p></p><p>In this example, I'm effectively asking for 6-digit precision for the equality comparison of floating-point values. Note, that if you replace that 0.000001 with the actual precision limit of the floating type you are using, you will be "exactly" comparing floating-point numbers - up to that type's precision, of course <img
src='http://bogdan.org.ua/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> .</p><p>The first-most example with the >= operator can be rewritten as</p><div
class="igBar"><span
id="lc-12"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('c-12'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">C:</span><div
id="c-12"><div
class="c"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color: #b1b100;">if</span> <span
style="color: #66cc66;">&#40;</span> result.<span
style="color: #202020;">score</span> &gt; <span
style="color: #66cc66;">&#40;</span> input-&gt;raw_cut_off - precision<span
style="color: #66cc66;">&#41;</span> <span
style="color: #66cc66;">&#41;</span></div></li></ol></div></div></div><p> where <em>precision</em> is exactly what it is named, e.g. <em>precision</em> = 0.000001.</p><p>Sources used:</p><ul><li>comment by <a
href="http://bytes.com/topic/c-sharp/answers/233215-float-double-arithmetic-precision-error#post953273" rel="nofollow" >Randy A. Ynchausti</a></li><li><a
href="http://www.shokhirev.com/nikolai/abc/sciprog/sciprognum.html" rel="nofollow" >scientific programming</a></li></ul><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/06/11/c-how-to-specify-comparison-operators-floating-precision.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/06/11/c-how-to-specify-comparison-operators-floating-precision.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F06%2F11%2Fc-how-to-specify-comparison-operators-floating-precision.html&amp;title=C%3A%20how%20to%20specify%20comparison%20operators%20floating%20precision" id="wpa2a_56"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/06/11/c-how-to-specify-comparison-operators-floating-precision.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>Debugging PHP: Eclipse PDT + XDebug + XDebug helper</title><link>http://bogdan.org.ua/2009/06/08/debugging-php-like-a-pro-eclipse-pdt-xdebug-helper.html</link> <comments>http://bogdan.org.ua/2009/06/08/debugging-php-like-a-pro-eclipse-pdt-xdebug-helper.html#comments</comments> <pubDate>Sun, 07 Jun 2009 22:59:43 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Links]]></category> <category><![CDATA[PHP]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[bug]]></category> <category><![CDATA[debugging]]></category> <category><![CDATA[development]]></category> <category><![CDATA[Eclipse]]></category> <category><![CDATA[FireFox]]></category> <category><![CDATA[PDT]]></category> <category><![CDATA[xdebug]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=735</guid> <description><![CDATA[Stimulated by a bug in a complex and unfamiliar web PHP application with heaps of custom tweaks by other programmers, I decided to try a more professional approach to PHP programming and debugging than the standard var_dump() and family. As a result, I'm now using Eclipse PDT with Xdebug and Xdebug Helper (Firefox extension). Now [...]]]></description> <content:encoded><![CDATA[<p>Stimulated by a bug in a complex and unfamiliar web PHP application with heaps of custom tweaks by other programmers, I decided to try a more professional approach to PHP programming and debugging than the standard var_dump() and family.</p><p>As a result, I'm now using Eclipse <abbr
title="PHP Development Tools">PDT</abbr> with <a
href="http://xdebug.org/" rel="nofollow" >Xdebug</a> and <a
href="https://addons.mozilla.org/uk/firefox/addon/3960" rel="nofollow" >Xdebug Helper</a> (Firefox extension). Now I don't understand how I used to debug my PHP programs before!</p><p>After proper configuration (I'm using local Apache, but it is also possible to debug remotely), my work flow is rather simple:</p><ul><li>use my web-app as usual, e.g. tweaking and testing here and there</li><li>if something server-side goes wrong: click the <a
href="https://addons.mozilla.org/uk/firefox/addon/3960" rel="nofollow" >XDebug helper</a> icon in Firefox, and perform some server-request action (e.g. load a page)</li><li>debugging is started in Eclipse PDT, where I can step through the code, set breakpoints, and examine all variables</li><li>as soon as the problem is fixed - click the XDebug helper icon again to continue using the site normally (w/o invoking the debugger)</li></ul><p>It takes some time to get used to, but then it's a breeze.</p><p>Some advice:</p><ul><li>don't use apt-get/aptitude to install Eclipse; it will be much easier both in the short and long run to use some all-in-one package from the <a
href="http://www.eclipse.org/pdt/downloads/" rel="nofollow" >Eclipse PDT site</a>; all you need to do - download, extract, run!</li><li>before actually starting to do anything, tweak the eclipse.ini file by increasing heap size from 40 MiB (default) to some larger value (I used 128MiB). If you don't do this, then at some point your debugging will become painfully sloooow, and then you'll start getting tons of "out of heap memory" errors, each one suggesting that you quit Eclipse immediately</li><li>install <a
href="http://xdebug.org/" rel="nofollow" >XDebug</a> with apt-get/aptitude - worked perfectly, and there's /etc/php5/conf.d/xdebug.ini not to mess with php.ini</li><li>do <strong>read</strong> <a
href="http://www.eclipse.org/pdt/documents/XDebugGuideForPDT2.0.pdf" rel="nofollow" >XDebug guide for PDT 2.x</a> (I'm assuming you got the 2.x version); it should be the only document you will really need to configure everything</li></ul><p>I only wish Eclipse was faster - that is, written not in Java but e.g. C or C++.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/06/08/debugging-php-like-a-pro-eclipse-pdt-xdebug-helper.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/06/08/debugging-php-like-a-pro-eclipse-pdt-xdebug-helper.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F06%2F08%2Fdebugging-php-like-a-pro-eclipse-pdt-xdebug-helper.html&amp;title=Debugging%20PHP%3A%20Eclipse%20PDT%20%2B%20XDebug%20%2B%20XDebug%20helper" id="wpa2a_60"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/06/08/debugging-php-like-a-pro-eclipse-pdt-xdebug-helper.html/feed</wfw:commentRss> <slash:comments>3</slash:comments> </item> <item><title>GNU make for bioinformatics presentation</title><link>http://bogdan.org.ua/2009/06/05/gnu-make-for-bioinformatics-presentation.html</link> <comments>http://bogdan.org.ua/2009/06/05/gnu-make-for-bioinformatics-presentation.html#comments</comments> <pubDate>Fri, 05 Jun 2009 09:37:21 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Bioinformatics]]></category> <category><![CDATA[Links]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[make]]></category> <category><![CDATA[presentation]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=725</guid> <description><![CDATA[Giovanni Dall’olio has recently posted a presentation on using make. Although it has "bioinformatics" on the title page, this is a good and very easy to understand make intro. Original post is here.]]></description> <content:encoded><![CDATA[<p>Giovanni Dall’olio has recently posted a presentation on using <strong>make</strong>.</p><p>Although it has "bioinformatics" on the title page, this is a good and very easy to understand <strong>make</strong> intro.<br
/> <img
style="visibility:hidden;width:0px;height:0px;" border=0 width=0 height=0 src="http://counters.gigya.com/wildfire/IMP/CXNID=2000002.0NXC/bT*xJmx*PTEyNDQxOTQxMzg2NzImcHQ9MTI*NDE5NDE5ODg4NCZwPTEwMTkxJmQ9c3NfZW1iZWQmZz*yJnQ9Jm89ZTA5MGYzZTEyYzUzNDRiZmEyZTJjNWJhMmE2MTE*NGMmb2Y9MA==.gif" /><div
style="width:425px;text-align:left" id="__ss_1111258"><object
style="margin:0px" width="425" height="355"><param
name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=makefilesbioinfo-090306113207-phpapp02&#038;stripped_title=makefiles-bioinfo" /><param
name="allowFullScreen" value="true"/><param
name="allowScriptAccess" value="always"/><embed
src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=makefilesbioinfo-090306113207-phpapp02&#038;stripped_title=makefiles-bioinfo" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object></div><p>Original post is <a
href="http://bioinfoblog.it/2009/10/a-seminar-on-makefile-and-pipelines-of-shell-scripts/" rel="nofollow" >here</a>.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/06/05/gnu-make-for-bioinformatics-presentation.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/06/05/gnu-make-for-bioinformatics-presentation.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F06%2F05%2Fgnu-make-for-bioinformatics-presentation.html&amp;title=GNU%20make%20for%20bioinformatics%20presentation" id="wpa2a_64"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/06/05/gnu-make-for-bioinformatics-presentation.html/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> <item><title>Ada GNAT GPL 2009 released</title><link>http://bogdan.org.ua/2009/05/30/ada-gnat-gpl-2009-released.html</link> <comments>http://bogdan.org.ua/2009/05/30/ada-gnat-gpl-2009-released.html#comments</comments> <pubDate>Sat, 30 May 2009 21:55:07 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[Ada]]></category> <category><![CDATA[Links]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[download]]></category> <category><![CDATA[GNAT]]></category> <category><![CDATA[release]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=708</guid> <description><![CDATA[We are pleased to announce the release of GNAT GPL 2009, the Ada Toolset for Academic users and FLOSS developers. It introduces many new features including: - Ability to generate byte code for the JVM - Improved support for the .NET Framework - Addition of the Ada-Java Interfacing Suite (AJIS) that enables native Ada code [...]]]></description> <content:encoded><![CDATA[<blockquote><p> We are pleased to announce the release of GNAT GPL 2009, the Ada Toolset for Academic users and FLOSS developers. It introduces many new features including:</p><p>- Ability to generate byte code for the JVM</p><p>- Improved support for the .NET Framework</p><p>- Addition of the Ada-Java Interfacing Suite (AJIS) that enables native Ada code to be called from Java:<br
/> <a
href="http://www.adacore.com/2008/06/17/ada-java_interfacing_suite" rel="nofollow" >http://www.adacore.com/2008/06/17/ada-java_interfacing_suite</a></p><p>- Availability on the Mac OS X (64 bit) platform</p><p>- Automatic C/C++ binding generators</p><p>- Addition of the GNAT Component Collection (GNATcoll) providing new APIs that can be extended by the user community:<br
/> <a
href="http://www.adacore.com/2008/06/17/gnat_component_collection" rel="nofollow" >http://www.adacore.com/2008/06/17/gnat_component_collection</a></p><p>GNAT GPL 2009 comes with version 4.3.1 of the GNAT Programming Studio IDE and GNATbench 2.3, the GNAT plug-in for Eclipse.</p><p>It is available for the GNU Linux, Mac OS X (64 bit), .NET, JVM and Windows platforms.</p><p>GNAT GPL 2009 can be downloaded from the "Download" section on the new Libre website:<br
/> <a
href="http://libre.adacore.com/libre/" rel="nofollow" >https://libre.adacore.com</a>.</p></blockquote><p>I wonder if the new JVM bytecode generation feature was frequently requested by Ada developers, or is just a move towards popularizing Ada as a highly capable programming language. Either way, it's good.</p><p>Hopefully, I will find time and a matching project to finally learn Ada properly - since a couple of years I believe Ada is a very good programming language. And the <strong>D</strong> language is better than C and C++ <img
src='http://bogdan.org.ua/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> (holy war, anyone? <img
src='http://bogdan.org.ua/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> )</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/05/30/ada-gnat-gpl-2009-released.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/05/30/ada-gnat-gpl-2009-released.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F05%2F30%2Fada-gnat-gpl-2009-released.html&amp;title=Ada%20GNAT%20GPL%202009%20released" id="wpa2a_68"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/05/30/ada-gnat-gpl-2009-released.html/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item><title>WordPress and Google Analytics external nofollow problem in comment links</title><link>http://bogdan.org.ua/2009/02/13/wordpress-and-google-analytics-external-nofollow-problem-in-comment-links.html</link> <comments>http://bogdan.org.ua/2009/02/13/wordpress-and-google-analytics-external-nofollow-problem-in-comment-links.html#comments</comments> <pubDate>Fri, 13 Feb 2009 19:41:55 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[CMS]]></category> <category><![CDATA[how-to]]></category> <category><![CDATA[PHP]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[Web]]></category> <category><![CDATA[comment]]></category> <category><![CDATA[fix]]></category> <category><![CDATA[link]]></category> <category><![CDATA[problem]]></category> <category><![CDATA[wordpress]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=594</guid> <description><![CDATA[Since some WP release, the comment author's link in comments is broken - it has ' external nofollow' attached straight to the href attribute (which breaks the link). I assume that the problem is caused by Google Analytics, namely the "track outgoing clicks" feature (as recalled, might be inaccurate feature name). "Track outgoing links" adds [...]]]></description> <content:encoded><![CDATA[<p>Since some WP release, the comment author's link in comments is broken - it has ' external nofollow' attached straight to the href attribute (which breaks the link).</p><p>I assume that the problem is caused by Google Analytics, namely the "track outgoing clicks" feature (as recalled, might be inaccurate feature name). "Track outgoing links" adds some JavaScript code to all outgoing links, and that script has tick characters like this one ' which, incidentally, are also used for delimiting the values of comment anchor tags.</p><p>To fix:<br
/> <span
id="more-594"></span></p><ol><li>locate file <strong>wp-includes/comment-template.php</strong></li><li>in that file, find the <strong>get_comment_author_link</strong> function (lines 140-150 in WP 2.7.1)</li><li>replace the line<div
class="igBar"><span
id="lphp-15"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('php-15'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">PHP:</span><div
id="php-15"><div
class="php"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#0000FF;">$return</span> = <span
style="color:#FF0000;">"&lt;a href='$url' rel='external nofollow' class='url'&gt;$author&lt;/a&gt;"</span>;</div></li></ol></div></div></div><p> with</p><div
class="igBar"><span
id="lphp-16"><a
href="#" rel="nofollow"  onclick="javascript:showPlainTxt('php-16'); return false;">PLAIN TEXT</a></span></div><div
class="syntax_hilite"><span
class="langName">PHP:</span><div
id="php-16"><div
class="php"><ol><li
style="font-family: 'Courier New', Courier, monospace; color: black; font-weight: normal; font-style: normal;color:#3A6A8B;"><div
style="font-family: 'Courier New', Courier, monospace; font-weight: normal;"><span
style="color:#0000FF;">$return</span> = <span
style="color:#FF0000;">'&lt;a href=&quot;'</span>.<span
style="color:#0000FF;">$url</span>.<span
style="color:#FF0000;">'&quot; rel=&quot;external nofollow&quot; class=&quot;url&quot;&gt;'</span>.<span
style="color:#0000FF;">$author</span>.<span
style="color:#FF0000;">'&lt;/a&gt;'</span>;</div></li></ol></div></div></div><p></li></ol><p>This helped me and might help you.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/02/13/wordpress-and-google-analytics-external-nofollow-problem-in-comment-links.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/02/13/wordpress-and-google-analytics-external-nofollow-problem-in-comment-links.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F02%2F13%2Fwordpress-and-google-analytics-external-nofollow-problem-in-comment-links.html&amp;title=WordPress%20and%20Google%20Analytics%20external%20nofollow%20problem%20in%20comment%20links" id="wpa2a_72"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/02/13/wordpress-and-google-analytics-external-nofollow-problem-in-comment-links.html/feed</wfw:commentRss> <slash:comments>2</slash:comments> </item> <item><title>Git: how to remove file and commit from history</title><link>http://bogdan.org.ua/2009/02/13/git-how-to-remove-file-commit-from-history.html</link> <comments>http://bogdan.org.ua/2009/02/13/git-how-to-remove-file-commit-from-history.html#comments</comments> <pubDate>Fri, 13 Feb 2009 19:21:32 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[how-to]]></category> <category><![CDATA[Links]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[file]]></category> <category><![CDATA[git]]></category> <category><![CDATA[history]]></category> <category><![CDATA[remove]]></category> <category><![CDATA[repository]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=590</guid> <description><![CDATA[Once I accidentally added circa 300 MiB of archive files to one of my git repositories (which was as small as 5 MiB). I removed those files as soon as I noticed them, but the .git directory still preserved commits with those files, and still occupied over 300 MiB. I have found the solution at [...]]]></description> <content:encoded><![CDATA[<p>Once I accidentally added circa 300 MiB of archive files to one of my git repositories (which was as small as 5 MiB). I removed those files as soon as I noticed them, but the .git directory still preserved commits with those files, and still occupied over 300 MiB.</p><p>I have found the <a
href="http://stackoverflow.com/questions/307828/git-remove-file-accidentally-added-to-the-repository" rel="nofollow" >solution</a> at <a
href="http://stackoverflow.com/" rel="nofollow" >stackoverflow</a> (see also this <a
href="http://stackoverflow.com/questions/250238/collapsing-a-git-repositorys-history" rel="nofollow" >question</a>).</p><p>This method worked for me, but I couldn't push my rebased repository to the gitosis. I would need to re-init the gitosis repository from my rebased, but I'm not yet prepared to do that.</p><p>There is also a slightly different method (which relies on a temporary tag instead of a temporary branch), documented in Git online manual pages; I prefer the temporary branch method.</p><p>Below is a full copy-paste of the winning answer by Charles Bailey:</p><blockquote><p> # create and check out a temporary branch at the location of the bad merge<br
/> git checkout -b tmpfix <sha1-of-bad-merge-or-commit></p><p># remove the incorrectly added file<br
/> git rm somefile.orig</p><p># commit the amended merge<br
/> git commit --amend</p><p># go back to the master branch<br
/> git checkout master</p><p># replant the master branch onto the corrected merge<br
/> git rebase tmpfix</p><p># delete the temporary branch<br
/> git branch -d tmpfix</p></blockquote><p>Also, in my case <a
href="http://stackoverflow.com/questions/495345/git-removing-selected-commits-from-repository" rel="nofollow" >this thread</a> at stackoverflow was highly useful. I start enjoying the concise and compact style of Charles Bailey <img
src='http://bogdan.org.ua/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> :<br
/> <span
id="more-590"></span></p><blockquote><p> # detach head and move to D commit<br
/> git checkout <SHA1-for-D></p><p># move HEAD to A, but leave the index and working tree as for D<br
/> git reset --soft <SHA1-for-A></p><p># Redo the D commit re-using the commit message, but now on top of A<br
/> git commit -C <SHA1-for-D></p><p># Re-apply everything from the old D onwards onto this new place<br
/> git rebase --onto HEAD <SHA1-for-D> master</p></blockquote><p>Note: this example is for a branch of commits R--A--B--C--D--E--HEAD, where commits B &#038; C should be removed from commit history.</p><p>Another option is to use git rebase --interactive (for the same example):</p><blockquote><p> git rebase -i HEAD~5</p></blockquote><p>Unfortunately, rebasing on a public (pushed out) branch has no effect for other users of the current gitosis repository.</p><p>An alternative to re-initializing gitosis repository from rebased scratch could be switching to a different (rebased) branch, and deleting the outdated master branch; but I haven't investigated if this works.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/02/13/git-how-to-remove-file-commit-from-history.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/02/13/git-how-to-remove-file-commit-from-history.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F02%2F13%2Fgit-how-to-remove-file-commit-from-history.html&amp;title=Git%3A%20how%20to%20remove%20file%20and%20commit%20from%20history" id="wpa2a_76"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/02/13/git-how-to-remove-file-commit-from-history.html/feed</wfw:commentRss> <slash:comments>2</slash:comments> </item> <item><title>ExpressionEngine contact form (email module) spam vulnerability</title><link>http://bogdan.org.ua/2009/01/26/expressionengine-contact-form-email-module-spam-vulnerability.html</link> <comments>http://bogdan.org.ua/2009/01/26/expressionengine-contact-form-email-module-spam-vulnerability.html#comments</comments> <pubDate>Mon, 26 Jan 2009 09:50:05 +0000</pubDate> <dc:creator>Bogdan</dc:creator> <category><![CDATA[CMS]]></category> <category><![CDATA[PHP]]></category> <category><![CDATA[Programming]]></category> <category><![CDATA[Software]]></category> <category><![CDATA[Web]]></category> <category><![CDATA[EE]]></category> <category><![CDATA[ExpressionEngine]]></category> <category><![CDATA[spam]]></category> <category><![CDATA[vulnerability]]></category> <guid
isPermaLink="false">http://bogdan.org.ua/?p=575</guid> <description><![CDATA[Yesterday I had a look at mod.email.php - the Email module of ExpressionEngine CMS. It appears that it is very easy to use ExpressionEngine's contact form (which uses Email module) to send emails to arbitrary addresses - simply put, send spam using someone's EE. And here's why: recipients hidden field is passed to the client; [...]]]></description> <content:encoded><![CDATA[<p>Yesterday I had a look at mod.email.php - the Email module of ExpressionEngine CMS.</p><p>It appears that it is very easy to use ExpressionEngine's contact form (which uses Email module) to send emails to arbitrary addresses - simply put, send spam using someone's EE.</p><p>And here's why:</p><ul><li><em>recipients</em> hidden field is passed to the client; it is encrypted, but with access to the mod.email.php code, it is a matter of several minutes to write your own email-encoding function which will produce a completely valid <em>recipients</em> field</li><li>there's also <em>XID</em> field, which seems to be unique for each page load</li></ul><p>Spamming algorithm is clear, so I won't elaborate. (I could have missed some session variables, though - didn't check them.)</p><p>This information is valid as of ExpressionEngine 1.6.6, but nothing in the change-logs indicates that this mechanism was modified in the newer versions of EE.</p><p><ins
datetime="2009-01-26T13:12:42+00:00">Update:</ins> I've tested, and this vulnerability does exist. The simplest prevention measure is to enable Captcha for Contact Form.</p><p>I've <a
href="http://expressionengine.com/archived_forums/viewthread/103537/" rel="nofollow" >notified</a> the developers.</p><p><a
class="a2a_button_google_plusone addtoany_special_service" data-href="http://bogdan.org.ua/2009/01/26/expressionengine-contact-form-email-module-spam-vulnerability.html"></a><a
class="a2a_button_facebook_like addtoany_special_service" data-href="http://bogdan.org.ua/2009/01/26/expressionengine-contact-form-email-module-spam-vulnerability.html"></a><a
class="a2a_dd a2a_target addtoany_share_save" href="http://www.addtoany.com/share_save#url=http%3A%2F%2Fbogdan.org.ua%2F2009%2F01%2F26%2Fexpressionengine-contact-form-email-module-spam-vulnerability.html&amp;title=ExpressionEngine%20contact%20form%20%28email%20module%29%20spam%20vulnerability" id="wpa2a_80"><img
src="http://bogdan.org.ua/wp-content/plugins/add-to-any/share_save_120_16.png" width="120" height="16" alt="Share"/></a></p>]]></content:encoded> <wfw:commentRss>http://bogdan.org.ua/2009/01/26/expressionengine-contact-form-email-module-spam-vulnerability.html/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> </channel> </rss>
