Autarchy of the Private Cave » Python

Graphs in Python

Bogdan — Sat, 13 Jul 2013 18:06:26 +0000

Sooner or later, everyone has to deal with graphs. Some people have to do programming with graphs, and a subset of those – do that in Python.

NetworkX is a pure Python implementation, where anything can be nodes. Both nodes and edges have attributes. NetworkX supports directed graphs and multigraphs (where there are multiple edges between nodes). It might be slower than other implementations, but you may even not notice that – especially when working with smaller graphs, or not applying computationally-intensive algorithms to your graphs.

graph-tool uses the Boost graph library (C++), so it should be really fast. It could be the only multi-threaded graph library for Python. It supports pickling the graphs, allows interactive graph drawing, and has well-illustrated documentation. If performance and efficiency are of utmost importance – could be the best choice.

igraph is also really fast – just like graph-tool when using 1 CPU; graph-tool only wins conclusively when it is run on multiple CPUs/cores. igraph has an R package bindings to C.

Pure python is also an option for really smaller cases.

Finally, there’s a discussion around Python Graph API to simplify the inter-changeability and inter-operability of various existing Python graph modules. It also has a list of some less-known Python graph libraries, so check it out.

Beanstalkd and related tools for easy parallelizing and backgrounding

Bogdan — Sat, 18 Feb 2012 11:07:56 +0000

beanstalkd: a simple, fast work queue.
Jack and the Beanstalkd: a web-app for basic work queue administration.
beanstalkc: a simple beanstalkd client library for Python.
queueit: a CLI interface tool which helps to integrate beanstalkd into shell scripts.

Python performance: set vs list

Bogdan — Mon, 15 Aug 2011 09:29:04 +0000

Sometimes there is a need to be sure that no identifier is processed twice – for example, when parsing a file into a database, with file potentially containing duplicate records. An obvious solution is to properly wrap the DB insertion code into try…except block, and process duplicate primary ID exceptions. Another, sometimes more desired solution is to maintain a set/list of processed IDs internally, and check against that list prior to attempting the insertion of anything. So is it a set or a list?

There are already quite a few internet resources discussing “python set vs list”, but probably the simplest while elegant way to test that is below.

First, test the speed of adding/appending to a set or a list (here, I’m mimicking the real-life application, thus the test case has an optional loop):

$python -mtimeit -s 'myset = set()' 'for x in xrange(1000): myset.add(x)'
10000 loops, best of 3: 133 usec per loop
$python -mtimeit -s 'tmp = list()' 'for x in xrange(1000): tmp.append(x)'
10000 loops, best of 3: 116 usec per loop

As we can see, set and list are comparable in the speed of adding new items, with list being slightly (~12%) faster than set.

Now, the speed of membership testing: ‘x in tmp’. For this test, I’ve deliberately chosen an imbalance of True (1%) and False (99%) results for the test – again, mimicking the real problem I have at hand:

$python -mtimeit -s 'tmp = set()' -s 'for x in xrange(1000): tmp.add(x)' 'for x in xrange(100000): x in tmp'
100 loops, best of 3: 7.27 msec per loop
$python -mtimeit -s 'tmp = list()' -s 'for x in xrange(1000): tmp.append(x)' 'for x in xrange(100000): x in tmp'
10 loops, best of 3: 2.12 sec per loop

List is much slower for membership testing, while sets were designed to be fast for doing just that.

Using Python in a Java project

Bogdan — Tue, 17 May 2011 11:33:52 +0000

If you are a Python zealot, and Java doesn’t feel right, but the project you are working on is a Java project – try

Jython – Python for the Java platform, compile your python scripts into Java bytecode
Groovy – not Python, but still a scripting language which compiles to jars

Introduction to Python for bioinformatics

Bogdan — Fri, 25 Feb 2011 12:03:55 +0000

This overview presentation is two years old, but still a highly valuable resource: modules and tools mentioned are alive and useful.
I think this is the second presentation by Giovanni I’m embedding (first one being about GNU/make for bioinformatics).

Introduction to python for bioinformatics

How to easily install any PyPi/easy_install python module on Debian

Bogdan — Wed, 16 Feb 2011 17:22:20 +0000

Imagine you need to install pycassa (which uses easy_install). Here are the 2 (at maximum) very simple steps to have it properly debianized and installed on your Debian/Ubuntu:

if you don’t have the python-stdeb package: sudo aptitude install python-stdeb
pypi-install pycassa

That’s it.

Refer to stdeb readme for more information. You will need that if there are dependencies – which might not be resolved automatically by stdeb.

Before stdeb, it wasn’t exactly trivial to make a .deb from python module.

DreamPie: the ultimate Python shell

Bogdan — Wed, 11 Aug 2010 14:54:40 +0000

DreamPie: the Python shell you’ve always dreamt about!

â€¢ Type your code in the lower pane of the window. To execute, press Ctrl+Enter. One-liners can be executed by simply pressing Enter; If you don’t want them executed, press Space and then Enter.
â€¢ Use Ctrl+Up and Ctrl+Down to navigate between code segments you’ve already executed. You can write a few letters before pressing Ctrl+Up, and DreamPie will only search through code segments starting with those letters.
â€¢ Press Tab or Ctrl+Space to show a list of completions to the current expression. It will also complete file names!
â€¢ Your results are stored in variables named _0, _1, and so on.
â€¢ Type a function name and press the space key and DreamPie will automatically add parentheses for you!

Good online Python book with code examples

Bogdan — Mon, 01 Mar 2010 18:03:11 +0000

Python: passing by value vs passing by reference

Bogdan — Mon, 11 Feb 2008 16:06:22 +0000

Note: this a collection of scraps, describing how values (de)referencing works in Python, and describing when your variable is either a value or a reference. Primary source of knowledge for this post (also here).

Python passes references-to-objects by value (like Java), and everything in Python is an object. This sounds simple, but then you will notice that some data types seem to exhibit pass-by-value characteristics, while others seem to act like pass-by-reference… what’s the deal?

It is important to understand mutable and immutable objects. Some objects, like strings, tuples, and numbers, are immutable. Altering them inside a function/method will create a new instance and the original instance outside the function/method is not changed. Other objects, like lists and dictionaries are mutable, which means you can change the object in-place. Therefore, altering an object inside a function/method will also change the original object outside.

Immutable variables – such as integers [strings, numerics and tuples are immutables] – are passed by value. That is, if your function accepts some integer argument, you are safe assuming that your function won’t be able to modify your integer. Mutable variables – such as dictionaries and lists – are passed by reference, and so if your function accepts mutable argument, it may modify the contents of that mutable variable outside the scope of the function.

When doing :
s = “Hello ”
s += “World”
… you are not modifying the string object bound to s, but creating a new string object and binding it to s.

If using object’s methods within a called function, variable is considered “passed by reference” – it is modified out of the function’s scope. If using assignment on a mutable object, it is created a-new within the function, and global value isn’t modified.

When you call a function with an arg, a “local variable” is created, which references the object passed as the argument. (well… an entry with the formal parameter name as key and a reference to the object passed in is created in the ‘local’ dict).

So, rebinding this local symbol does not impact the binding in the caller’s namespace – because the symbol lives in another namespace.

*But* – and if the object referenced is mutable of course – modifying the object in the function… well, just modifies the object, because it’s the *same* object that is bound to (‘referenced by’, if you prefer) both symbols (the one in the caller’s namespace and the one in the function’s namespace). So yes, the object *is* modified when the function returns.

Useful Python looping techniques

Bogdan — Wed, 26 Sep 2007 09:21:00 +0000

These are all excerpts from the Python documentation.

To synchronously and simultaneously loop over two sequences:

questions = ['name', 'quest', 'favourite colour']
answers = ['Lancelot', 'the holy grail', 'blue']
for q, a in zip(questions, answers):
print 'What is your %s? It is %s.' % (q, a)

To loop over a sequence with both key and value:

for i, v in enumerate(['tic', 'tac', 'toe']):
print i, v

To loop over dictionary, again with key and value:

knights = {'gallahad': 'the pure', 'robin': 'the brave'}
for k, v in knights.iteritems():
print k, v

To loop over some sorted sequence, but without modifying the original sequence (beware the use of set() in this example):

basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
for f in sorted(set(basket)):
print f

For looping sorted dictionaries, see also How to sort Python dict (dictionary).

How to sort Python dict (dictionary)

Bogdan — Wed, 26 Sep 2007 06:20:57 +0000

Sample script (copypasted from Well House Consultants training course):
click the PLAIN TEXT header for copy-pasteable version

#!/usr/local/bin/python
author = {"php":"Rasmus Lerdorf",\
"perl":"Larry Wall",\
"tcl":"John Ousterhout",\
"awk":"Brian Kernighan",\
"java":"James Gosling",\
"parrot":"Simon Cozens",\
"python":"Guido van Rossum"}
langs = author.keys()
langs.sort()
for language in langs:
print language,"is the child of",author[language]

You can also define the Python ksort() function similar to that found in PHP:

def ksort(d, func = None):
keys = d.keys()
keys.sort(func)
return keys
# example use, assume 'd' is a dictionary
for k in ksort(d):
print k, d[k]

(example taken from TheScript forum)

MySQL – Python: good MySQLdb tutorial (examples)

Bogdan — Fri, 07 Sep 2007 12:32:25 +0000

Andy Dustman (used to blog at dustman.net) gave a presentation on Python and MySQL at the MySQL Users Conference 2005, Santa Clara, CA. The presentation is an excellent collection of examples for those who use the MySQLdb Python module.

For the purpose of maintaining a personal archive of that excellent presentation, I created a PDF version of it. All copyrights belong to Andy Dustman.

Date and time in Python scripts

Bogdan — Fri, 07 Sep 2007 12:07:45 +0000

Here’s an excellent description of various methods to handle date and time in Python scripts.

Elegantly converting Python list into another list with changed items

Bogdan — Thu, 30 Aug 2007 09:06:50 +0000

I just had a small problem with my Python script:

1. I have a list of tuples, called records_cache; each tuple looks like this:
(note: for copy-pasting, click the PLAIN TEXT box headers)

(ensembl_transcript_id, ensembl_gene_id, chrom_start, chrom_end, utr_start, utr_end, sequence, strand)

2. I need to INSERT only some of those values into MySQL table, using the MySQLdb executemany() method. The values I need are utr_start, utr_end, ensembl_gene_id (in exactly this order).

To do this, I need to create another list of tuples, but with shorter 3-item tuples. Let us call the new list genes_update.

Here is the elegant, though probably not the most efficient, solution:

genes_update = [(onetuple[4],onetuple[5],onetuple[1]) for onetuple in records_cache]]

Let’s see what is happening here:

First, note the brackets around the r-expression. These brackets force the result to be a list.
Next, note the construct used, it is from the Python manual: list_element.method() for list_element in list. This construct iterates all items of the list, in our case – of the records_cache list. This construct also applies the list_element.method() to each list method.
Instead of applying some list_element.method(), I just use tuple indices to select the values I need, and then (using parenthesis) join those values into a tuple again.
So for each tuple in the records_cache list, another (shorter) tuple is returned, and those shorter tuples are joined into the new list, which is assigned to the genes_update list.

After this conversion, I can happily write that list of tuples into MySQL table, using cursor.executemany():
(note: error_log() is a small custom function for appending to script’s error.log file)

try:
cursor.executemany("""UPDATE genes SET
5utr_start = %s,
5utr_end = %s
WHERE ensembl_gene_id = %s""", genes_update)
db.commit()
except MySQLdb.Error, (errno, strerror):
db.rollback()
error_log("Error %d: %s (%s)\n" % (errno, strerror, insertq))
return False # failed to update
return True # updated

Comments are welcome, especially on the efficiency improvement.

Non-Programmer’s Tutorial for Python

Bogdan — Sun, 12 Aug 2007 19:11:33 +0000

Non-Programmer’s Tutorial for Python

Python: iterate (and read) all files in a directory (folder)

Bogdan — Sun, 12 Aug 2007 11:33:57 +0000

To iterate through all the files within the specified directory (folder), with ability to use wildcards (*, ?, and [ ]-style ranges), use the following code snippet:

import os
import glob
path = 'sequences/'
for infile in glob.glob( os.path.join(path, '*.fasta') ):
print "current file is: " + infile

If you do not need wildcards, then there is a simpler way to list all items in a directory:

import os
path = 'sequences/'
listing = os.listdir(path)
for infile in listing:
print "current file is: " + infile

print was promoted from a statement to a function in Python 3 (use print(infile) instead of print infile).

One should use ‘os.path.join()’ part to make the script cross-platform-portable (different OS use different path separators, and hard-coding path separator would stop the script from executing under a different OS).

Python docs mention that there is also iglob(), which is an iterator and thus working on directories with way too many files it will save memory by returning only single result per iteration, and not the whole list of files – as glob() does.

Why Zope 3 is just great

Bogdan — Thu, 15 Mar 2007 17:30:58 +0000

Recently I learned about Zope, which is an “open source web application server”, primarily written in Python.

Then Django and Turbogears were seen as web-development frameworks akin to Zope. Search revealed an interesting anti-Zope rant at Zope vs Django. Reading until the end, and then following the comments, I came across the comment by Holger Froebe, which is a huge one (probably the longest comment I had ever seen), and represents a detailed explanation with examples of the reasons to use Zope 3. I found that comment to be a really good-written one, so if you are deciding on whether you should use Zope 3 or not, then read the comment here (scroll down or just search for “Holger Froebe”).

For convenience and in order to preserve this worhty piece of work from vanishing, below is the 99%-exact copy-paste of that comment.

Holger Froebe February 8, 2006 5:32 pm

Hello Jason,

when I read your post, I smiled a little.
Because I was in a similar situation in January/February
2005. Let me introduce: My name is Holger
Froebe, I work for the IT of a university hospital
here in Germany (does this count for enterprise

requirements? I hope so.) – sorry for my crude
english.

â€¦â€¦â€¦and now for something completely different â€¦â€¦â€¦.

We had some web-applications written in plone and
wanted to extend this stuff throughout the company
which failed via some of the reasons
(Integration of Oracle/User management) which you adressed
in your rant. The main goal of this extension project
was to ensure integrity of data stemming from

different sources (mainly Legacy systems, like SAP,
Oracle, stuff from File/FTP server, MySQL, SQLServer -
the whole spectrum

Why did we fail ?
First let me get one point clear: I think Plone has
its strenghts and merits and if you stay close to
the main street of its framework layout and its
original intention (see below), you can get very
satisfactory results – see Oxfam, Ebay-Developer Plattform,

Motorola and a lot of other impressive projects.

But if you look at its history, you see a pattern:
Plone started as a replacement for the User Interface of
the Content Management Framework (CMF) of Zope2. But
over the years more and more architectural
stuff slipped into the original Skin Package – which
suddenly became a framework of its own.
It started as one package – now
you have 13 (or so) Zope packages which constitutes Plone.

And this software stack got bigger with evâ€™ry release: There
is python, then comes zope, then plone, then put archetypes and
on top of it ArchetypesContentTypes. To add it, all this
stuff has strong inner dependencies.
So with all those dependencies
Plone slowly drifted away from its original goal (getting a more usable
+ visually appealing UI for Zope-CMF) and now does a lot of stuff
which should be done better deep down
in the software stack – may it be a pure python library

or a standard Zope Component/Product.
And that – to my totally personal mind – was the reason we failed
with a complex enterprise scenario.
If youâ€™re interested in this point of view,
Iâ€™ll recommend you Chris Withers insightful talk
â€œPlone rocks my worldâ€ – http://www.simplistix.co.uk/presentations

â€¦â€¦â€¦and now for something completely different â€¦â€¦â€¦.

But – What was the solution ?

We tried different approaches, but stayed
closely in well-known Python territory and rounded up
the usual suspects: TurboGears, Django, Zope3, to name
the most prominent.

And the winner was â€¦ Zope3 (now its time
to put on my fireproof suit, right

We started to work with it in Spring 2005 -
and we never looked back.

Man, this is such an amazing piece of software !

For me itâ€™s like a piece of art
Everyday I work with it Iâ€™m getting more
enthusiastic about it.

Zope3 is build on some values which in my mind
really counts if you want to build an enterprise system:

Quality
Dynamics + Extensibility
Flexibility
Reuse, reuse and reuse again
Clear and concise separation of concerns
Focus on core competencies
= Avoid the Not-invented-here-syndrom
- integrate proven solutions from outside your world

Zop3 is not the monolithic big framework like many
other appservers today, but a collection of loosely

coupled pieces where every piece has a clear and defined
responsibility and quality. There are a some
thousand tests to ensure the last one. You can use nearly every
of this little pieces (Zopies call them â€œcomponentsâ€)
outside Zope. This separation of concerns to the extreme
leads in a fast way to locate, isolate and remove errors -
that really saved some of my days in the last few months.

At the same time the Zope3-Guys build via ruthless refactoring a framework
which brings Zope-World closer to Python standard world

(stuff like WSGI, relational Database connectivity etc. – see
below).

Heck, you can even use Zope-Code for a client
app which doesnâ€™t know anything Persistency or ZODB.
Yes, itâ€™s true: You can write pure desktop applications
using Zope-Code without caring about ZODB – Example:
The CCPublisher2 rewrite of the CreativeCommons-Project,
see http://svn.berlios.de/svnroot/repos/cctools/publisher/trunk

or http://wiki.python.org/moin/PyCon2006/Talks#48
Or you want to use a collaborative tool for groups with
instant + slick visual feeling – try Bebop which strongly
relies on Zope3-Technologies on server AND client side.

â€¦â€¦.and now for something completely different â€¦..

The Zope3-Team discussed very thoroughly: What are the
core competencies of Zope and what not ?
For example: Zope2 had its own webserver – ZServer.

But why write and maintain such a beast when there
is something better in Python-World ?
So the Zope3-Guys integrated Twisted and
got best of both worlds.

â€¦ and now for something completely different â€¦

Relational Databases and Zope3

You described very well in your rant the problems with Oracle.
This was nearly 100% identical to our experience,
so my smile from the beginning of this post

comes a little bit from the pain I left behind me.

With Zope3 there are two approaches:

a) Use cxOracleDA-adapter (http://svn.zope.org/cxoracleda/)
Looking at this source code I was amazed how easy and
straightforward it is to integrate a well-known and proven
python-Standard DB-Adapter into Zope3-world.
The same pattern again: Take a proven quality piece

of software from standard python world, put on a small wrapper
and – whoa – use it right away in Zope3.

Well, and then you could put sql-expressions into your templates
via sql-expressions (uhm, not really recommended or use
your zsql-scripts from Zope2.

But then we thought there should be a better solution,
which breathes the spirit of Zope3 â€¦ which led us to

b) an OR-Mapper-Solution

We tried SQLObject first, since it had a â€œnativeâ€ Zope3-Integration
via sqlos. But Oracle connectivity is not one of the main targets
of SQLObject, despite there is a Oracle-SQL-Object-0.6.1-branch
in the repository.

PyDo2 from the Skunkweb-Project looked really promising, but
we had to put the name of the table into the class-definition,

so this was not flexible enough. Despite Oracle-Connectivity
was OK.

So we ended up with the best (shameless marketing)
enterprise ORM in python-world: SQLAlchemy (http://www.sqlalchemy.org)
Yes, thereâ€™s no official download and the developers
say thereâ€™s only Version 0.9.1 – but itâ€™s pretty close to final 1.0

and it really suits our needs.

The great difference between SQLAlchemy and the rest of the bunch
is that other ORM-Mappers try to fit relational databases by all means
into Object-World. But a relational DB is no misled ObjectStore and
the whole analogy breaks more and more down with
– larger databases

– transactional aspects
– complex datasets/queries over many tables.

SQLAlchemy has the same philosophy as Z3 in
– extreme separation of concerns and
– integration of standard DB Adapters
– high quality of code + easy readability

(anybody said â€œpythonicâ€
– good quality of documentation

SQLAlchemy treats a Class as a Class, a RDB as a RDB, and
(you guessed it, right?) a RDB Table as a
RDB Table and connects them via mappers, which can easily
enriched with some standard methods/queries provided by

the framework.
It makes heavy use of the new dynamics aspects enriching
Python with the last few releases in the 2.x-line.

With SQLAlchemy I could drop the lines of code of my sql-related
stuff somewhere around 20-30%. And I REALLY like the approach
of â€œWriting less codeâ€. Doing all my RDB-Stuff in python

is an extra-Bonus (OK, to be honest – some complex queries
survived, but I think this is only a matter of time, since
they vanish

So youâ€™re not alone in Zope3-Oracle-World. To
cite the glorious A. A. Milne â€œAnd then there were threeâ€
(and maybe even more, if someone answers this post

â€¦ and now for something completely different â€¦

LDAP-Integration

Zope3 is very concise about Authentication via
so-called PAUâ€™s – Pluggable Authentication Utilities.
You could easily plug together Authentication sources
with different Authentication methods – its as easy
as plug your lego-stones together.

Like in RDB-World: First you define an Adapter to your

Datasource – lets call it LDAP-Adapter, right -
which defines and holds the connection to your
external LDAP-Source (http://svn.zope.org/ldapadapter/).
That way, you could even use more than one LDAP-Source
in your Application.

Then you have another clearly defined LDAP-PAS (http://svn.zope.org/ldappas/).

which does the authentication against this Adapter.

And the whole beast (you guessed it again, right?) is a small,
well-defined wrapper around python-ldap. Plus
itâ€™s easy to read and fast to understand (my 2 cents).
Itâ€™s like dejavu all over again

Hint: I just played around with this LDAP-stuff and never
tested that in our production environment,

but I have great confidence from my previous experiences
with Zope3 that it should be working relatively seamless.

â€¦ and now for something completely different â€¦

ZCML

Well, everybody beats on ZCML, since its such an easy
target – â€œHey, itâ€™s XML – thatâ€™s bad. We donâ€™t want
to use XML (for whatever ideological reason), so

Zope3 must be something ill-constructedâ€

If you ask me about my feelings about ZCML, I would
not try to convince you it was made in heaven and tell you
that you are too blind to see the light

But – as often in life: truth lies somewhere in
between the extremes. My 2 cents:

I share your feelings about not direct
debugging ZCML, despite the fact that Zope3.2 brings
very concise error-tracebacks.
The Zope3-Guys are aware of the problems users have with
ZCML. They try REALLY hard to bring as much ZCML back
to python as possible – see
http://www.z3lab.org/sections/blogs/philipp-weitershausen/20051214_zcml-needs-to-do-less
for a thorough discussion from one of the core developers
of Zope3. Looking from Zope3.0 to Zope3.2 (the current release) some
stuff vanished from ZCML, so those guys do their homework
and will do it even more on the upcoming Zope3.3 release.
The best thing at the end (now Z3-Team will really beat me ;-):
You can write up and use
ALL (right, ALL) ZCML-directives right away as python-code, if you donâ€™t
XML.
And yes, this is even documented (call me old-fashioned,
but I read docs first). You may say, that there are such
a huge amount of READMEâ€™s and other .txt-stuff spread
over the whole Zope-Project, that its not easy to get into it.
But Zope3 provides you with a toolscript
called â€œstatic-apidocâ€ which gives you a clear, concise
overview of the whole documentation as a static website.
Now if you look at the README under zope/component or
at the static-apidoc unter â€œComponent Architectureâ€
you find methods like provideAdapter, provideUtility,
which do – surprise, surprise – the equivalent of
ZCML-alternatives. Or look at zope/component/site.py
or zope/configuration
You want to see this in action ? Look at this
2-part-example of a simple Z3-object publishing system
without any piece of ZCML:
Part I: The Zope Component Architecture -
Interfaces, Adaptation, and Duck Typing
http://griddlenoise.blogspot.com/2005/12/zope-component-architecture-interfaces.html
Part II: The Zope Component Architecture -
One Way To Do It All
http://griddlenoise.blogspot.com/2005/12/zope-component-architecture-one-way-to.html
Browsing through the docs, you can find the other
replacements in a straightforward way (or you debug
the xmlconfig-stuff from zope/configuration, which
gives you the corresponding callables for ZCML)
If thatâ€™s too tough and time-consuming – no problem,
ask on the Zope3-Users mailinglist. Those guys are
REALLY helpful to get you into Z3-world.

â€¦ and now for something completely different ..

Kool-Aid and the magic world of interfaces and adapters

Speaking with developers about Zope3 you often hear
that interfaces and adapters are too much magic and
they have to drink so much Kool-Aid to understand them.
I wonâ€™t put this here into a lengthy pro-con-discussion

of these concepts, since Iâ€™m not really a core developer,
but more an application developer/maintainer.

But to tell you my story:

It took me a while to GET the main ideas/principles
behind this stuff – to be honest,
2 days of intensive, dedicated work and 5 litres of H2O.
After one week of working with Zope3 I was more productive than
before. Plus I had learned a lot of new stuff about programming
in a quality way. Yes, dealing with Zope3 has made

me a better programmer – even if I never should do anymore project
with it ;-(

Plus it helped me to get my things done better + faster.

Well, thatâ€™s Kool-Aid I really like !

I wontâ€™t say this world of interfaces and adapters
is the easiest to understand. But again:
There are a good amount of play-around-with-it-tutorials/docs/books
around which take you into Zope3-World. Just give it a try !

Want some examples? Want some simple apps to play around with?

Well, it was never easier than with Zope3 – see yourself

Hereâ€™s some easy stuff which you can work through in less than 1 hour
(well, the last example takes you longer

a) â€œZope3 in 30 minutesâ€ – at
http://zissue.berlios.de/z3/Zope3In30Minutes.html
showing you step by step how to build your first
simple Z3-application to collect your bookmarks on a server

b) Did I mention the magnificent Jeff Shell? His blog griddlenoise.blogspot.com is a rich and really insightful source for getting into Zope3 – plus itâ€™s really fun to read. In his archives you find this funky little thing http://worldcookery.com/files/jeffshell-todo/ where Jeff tells you how to build a Rails-like ToDo-Application in some simple steps. In every step he shows you what to do and why he thinks this implementation is carefully thought out in Zope3 and what is the reason they did it this and no other way.

c) Want a fresh new zope3-site without understandig â€œall the magicâ€
inside? Choose life – choose the z3 project starter

http://www.zope.org/Members/adytumsolutions/z3projectstarter/z3projectstarter_released
Answer a few simple questions and you have a project skeleton
to play around with without deeply understanding all this â€œkool-aidâ€ upfront.

d) Philipp von Weitershausens Website/Book about Zope3 -
http://worldcookery.com

You will find more of this tutorials on Phillips website under
http://worldcookery.com/appetizers.

And this is not the end. New stuff is landing every day
in Zope3-space – like this little gem about events/notifications
and how they help you handle complex application
architectures:

http://remarkablysimple.blogspot.com/

Or you look at www.z3lab.org where you can get a peek

of the Zope3-ECM-Initiative and so on so on â€¦

So, my advice to you: Fire on feedster.com, type in Zope3 â€¦
and youâ€™ll find a lot more of this diamonds.

I made the experience that most of the complaints about
Kool-Aid come from developers who specialized in certain
frameworks/habits and now had difficulties to extend
their mindset since they had to leave known territory.
They struggled with Zope3, found some hurdles and then gave up,
since â€œthereâ€™s so much kool-aidâ€.

I was surprised to find out that absolute newbies to programming
get productive with Zope3 very fast and easily -
maybe since they are not fixed on certain stuff.

I mean, sometimes I donâ€™t get it: People want to write programs
for real complex enterprise scenarios, but at the same
time tell me itâ€™s too hard to spend a few hours to play with some toy examples
and read some docs and play with the marvellous python command prompt
trying to push their brain into a new direction.

Believe me: This whole interface-adapter-pattern definitely helps

you in bigger/complex projects evolving over time.
Remember Fred Brooks Mythical Man-Month, which made so many of us aware
that change of requirements is inherent in any software project -
even if the whole system is in production use.

Zope3 has not ALL, but a lot of REALLY GOOD answers
to this situation every developer faces from time to time

â€¦ and now for something completely different ..

Querying the ZODB from outside applications

Iâ€™m convinced that it should be easy and I know

that the guys in the Bebop-Project did some stuff
in this direction, But Iâ€™m no expert in that,
since relational databases
constitutes more of my work. If you want to
have a profound answer on this – push it
on the Zope3-Users-Mailinglist (sorry, that
wasnâ€™t a sufficient answer, right?). Those
helpful souls there will really show you
the best and easiest way to do it.

â€¦.. and now for something completely different â€¦.

Coming back to your Plone-dissatisfaction

I donâ€™t want to tell you that Plone is bad. Or Plone
sucks. Or stuff like that. For me itâ€™s the right tool
for the projects it was made of – usable portal solutions
for medium size. The same holds for Django which
is also OK, if I want to make a fast RDBMS-UI-App.

The good thing is: For different work tasks there
are different tools in my toolchest. Itâ€™s my

responsibility to choose the right one for
every new project, but it leaves me with a
warm safe feeling that the toolbelt is filled
with such good quality stuff.

And the good news is just around the corner:
Plone and Zope3-World are converging – approaching
each other with every day. Now what does that mean?

Since I worked with plone it was easy to find my
way in Zope3-World. Zope3 tried to learn from

the Zope2 AND the plone lessons and put a lot of
the best breed of Plone (which constituted at the
same time those hard-to-manage architectural overhead)
back to the core of the framework. Well, this
is not totally right, since there is no such thing
as a monolithic core of Zope3-framework – the greatest lesson
learned from the problems with complex Zope2-projects.
Which is the best news of all

Know Archetypes ? There is schema-driven content-types

with form generation (zope.formlib)

Know Skins and Layers ? Use them right away in Zope3

Know Portlets ? Generalized to Viewlets and managed via ViewletManagers.

RessourceRegistries? Now known as RessoureLibrary.

â€¦ and so on â€¦ and so on â€¦

But at the same time the Plone guys push their stuff
more and more towards the proven Z3-technologies -
and by handing over Z3 the framework responsibilities
the Plone community again can concentrate on being

the big shot at their homeground -
to provide you with the â€œMacOS of CMSesâ€
(well at least thatâ€™s what Limi told me

I also appreciate the other web frameworks
in Python world – and Iâ€™m happy to see that
there will be a â€œWebFrameworkâ€-Track during
Europython this year where zopies, djangistas
and turbogearianos and all those funky-stuffistas
will get into fruitful

discussions about solutions. Iâ€™m really
looking forward to this meeting since
we can learn a lot from each other
if we leave our minds open for the NEW.

â€¦. and now for something completely different â€¦.

There is so much more to say about this marvellous
piece of software (like the integration
of other templating languages like meld or clarity
or the integration of standards like Java-like Portlet-Stuff or

WFMC – the Workflow-Coalition – and and and)
but let me come to an end, since itâ€™s really late
and I need some sleep:

I work for 20 years with software and applications.
Zope3 is one of the most professionall, mature
and qualitative outstanding frameworks I saw.

Itâ€™s really fun to work with, if you have a sense for
lasting quality solutions, if you want to be
proud of the stuff you created.

Thanx for your patience + Good night,

Holger @ Germany

PS: If youâ€™ve got any specific question about Zope3,
drop me a not at booradley at web.de. Or visit some
of the links in my rant. Or subscribe
to the Zope3-User-Newsgroups and ask your questions.
This world is really full of possibilities

Holger Froebe February 8, 2006
6:13 pm

Ah, and I forgot – the collaborative development
of my favourite Linux distro (Ubuntu)
is managed by a Zope3-Application -

see http://launchpad.net â€¦

or the shiny Z3-based Schooltool if you want
to manage ressources and calendars â€¦
see http://www.schooltool.org

Zope3 is really smoking

Simple substring counting script in Python

Bogdan — Wed, 21 Jun 2006 17:41:27 +0000

Approximately a month ago I endeavoured to use Python as my main shell-scripting language. At that moment, I was already aware of multiple benefits you get when you use Python for scripting:

source-level cross-platform scripting: your script will run anywhere, where Python compiles; expanding this statement – your script will run anywhere, where there is a C compiler (needed to build Python itself)
high-level language: you can iterate all the lines in a text file with as little as one ‘for’-statement, for example (see the actual example below)
simple/minimalist syntax: no curly braces around blocks of statements, no semicolons after each and every line of code, etc. Python at a glance looks much more understandable, than, for example, Perl.
the power of C in a language-interpreting system
it is interpreted! This gives easyness of debugging: modify, execute, see the trouble – with no compile/link stages
and, despite being interpreted, it is fast!

For the comparison (in speed, memory use, program size) with other computer programming languages, please see the “Computer Language Shootout Benchmarks”. Here I provide the link only to the comparison of Python with Perl and comparison of Python with PHP (which can also be used as shell-scripting language, albeit after some tinkering with settings and stuff)

Below is an example of the 2-minute script in Python, which counts the number of occurrences of some string in a file.

"""Read FILE and count number of occurences of SUBSTR."""
version = 0.01
import sys
def main():
from optparse import OptionParser
opts = OptionParser(usage="%prog [options] FILE SUBSTR",
version="%prog " + str(version),
description="Read FILE and count number of occurences of SUBSTR.")
opts.set_defaults(verbose=False,flush=False)
opts.add_option("-v", "--verbose", action="store_true", dest="verbose", help="Print every line containing substr [default: %default]")
opts.add_option("-f", "--flush", action="store_true", dest="flush", help="When verbose, flush every line [default: %default]")
(options, args) = opts.parse_args()
if len(args) != 2:
print "Two arguments required for correct processing"
opts.print_help()
sys.exit(2)
infile = args[0]
substr = args[1]
lines_count = 0
substr_count = 0
lines_substr_count = 0
if options.verbose and not options.flush:
msg = ""
f = open(infile, 'r')
for line in f:
lines_count += 1
found = line.count(substr)
substr_count += found
if found > 0:
lines_substr_count += 1
if options.verbose and not options.flush:
msg += str(found) + ": " + line
elif options.verbose and options.flush:
print (str(found) + ": " + line).replace("n","")
f.close()
if options.verbose and not options.flush:
print msg
print "Lines read from file: ", str(lines_count)
print "Lines with substring found: ", str(lines_substr_count)
print "Total substrings detected: ", str(substr_count)
return
if __name__ == "__main__": main()