Autarchy of the Private Cave

Hello Jason,

when I read your post, I smiled a little.
Because I was in a similar situation in January/February
2005. Let me introduce: My name is Holger
Froebe, I work for the IT of a university hospital
here in Germany (does this count for enterprise

requirements? I hope so.) – sorry for my crude
english.

………and now for something completely different ……….

We had some web-applications written in plone and
wanted to extend this stuff throughout the company
which failed via some of the reasons
(Integration of Oracle/User management) which you adressed
in your rant. The main goal of this extension project
was to ensure integrity of data stemming from

different sources (mainly Legacy systems, like SAP,
Oracle, stuff from File/FTP server, MySQL, SQLServer -
the whole spectrum

Why did we fail ?
First let me get one point clear: I think Plone has
its strenghts and merits and if you stay close to
the main street of its framework layout and its
original intention (see below), you can get very
satisfactory results – see Oxfam, Ebay-Developer Plattform,

Motorola and a lot of other impressive projects.

But if you look at its history, you see a pattern:
Plone started as a replacement for the User Interface of
the Content Management Framework (CMF) of Zope2. But
over the years more and more architectural
stuff slipped into the original Skin Package – which
suddenly became a framework of its own.
It started as one package – now
you have 13 (or so) Zope packages which constitutes Plone.

And this software stack got bigger with ev’ry release: There
is python, then comes zope, then plone, then put archetypes and
on top of it ArchetypesContentTypes. To add it, all this
stuff has strong inner dependencies.
So with all those dependencies
Plone slowly drifted away from its original goal (getting a more usable
+ visually appealing UI for Zope-CMF) and now does a lot of stuff
which should be done better deep down
in the software stack – may it be a pure python library

or a standard Zope Component/Product.
And that – to my totally personal mind – was the reason we failed
with a complex enterprise scenario.
If you’re interested in this point of view,
I’ll recommend you Chris Withers insightful talk
“Plone rocks my world” – http://www.simplistix.co.uk/presentations

………and now for something completely different ……….

But – What was the solution ?

We tried different approaches, but stayed
closely in well-known Python territory and rounded up
the usual suspects: TurboGears, Django, Zope3, to name
the most prominent.

And the winner was … Zope3 (now its time
to put on my fireproof suit, right

We started to work with it in Spring 2005 -
and we never looked back.

Man, this is such an amazing piece of software !

For me it’s like a piece of art
Everyday I work with it I’m getting more
enthusiastic about it.

Zope3 is build on some values which in my mind
really counts if you want to build an enterprise system:

• Quality
• Dynamics + Extensibility
• Flexibility
• Reuse, reuse and reuse again
• Clear and concise separation of concerns
• Focus on core competencies
  = Avoid the Not-invented-here-syndrom
  • integrate proven solutions from outside your world

Zop3 is not the monolithic big framework like many
other appservers today, but a collection of loosely

coupled pieces where every piece has a clear and defined
responsibility and quality. There are a some
thousand tests to ensure the last one. You can use nearly every
of this little pieces (Zopies call them “components”)
outside Zope. This separation of concerns to the extreme
leads in a fast way to locate, isolate and remove errors -
that really saved some of my days in the last few months.

At the same time the Zope3-Guys build via ruthless refactoring a framework
which brings Zope-World closer to Python standard world

(stuff like WSGI, relational Database connectivity etc. – see
below).

Heck, you can even use Zope-Code for a client
app which doesn’t know anything Persistency or ZODB.
Yes, it’s true: You can write pure desktop applications
using Zope-Code without caring about ZODB – Example:
The CCPublisher2 rewrite of the CreativeCommons-Project,
see http://svn.berlios.de/svnroot/repos/cctools/publisher/trunk

or http://wiki.python.org/moin/PyCon2006/Talks#48
Or you want to use a collaborative tool for groups with
instant + slick visual feeling – try Bebop which strongly
relies on Zope3-Technologies on server AND client side.

…….and now for something completely different …..

The Zope3-Team discussed very thoroughly: What are the
core competencies of Zope and what not ?
For example: Zope2 had its own webserver – ZServer.

But why write and maintain such a beast when there
is something better in Python-World ?
So the Zope3-Guys integrated Twisted and
got best of both worlds.

… and now for something completely different …

Relational Databases and Zope3

You described very well in your rant the problems with Oracle.
This was nearly 100% identical to our experience,
so my smile from the beginning of this post

comes a little bit from the pain I left behind me.

With Zope3 there are two approaches:

a) Use cxOracleDA-adapter (http://svn.zope.org/cxoracleda/)
Looking at this source code I was amazed how easy and
straightforward it is to integrate a well-known and proven
python-Standard DB-Adapter into Zope3-world.
The same pattern again: Take a proven quality piece

of software from standard python world, put on a small wrapper
and – whoa – use it right away in Zope3.

Well, and then you could put sql-expressions into your templates
via sql-expressions (uhm, not really recommended or use
your zsql-scripts from Zope2.

But then we thought there should be a better solution,
which breathes the spirit of Zope3 … which led us to

b) an OR-Mapper-Solution

We tried SQLObject first, since it had a “native” Zope3-Integration
via sqlos. But Oracle connectivity is not one of the main targets
of SQLObject, despite there is a Oracle-SQL-Object-0.6.1-branch
in the repository.

PyDo2 from the Skunkweb-Project looked really promising, but
we had to put the name of the table into the class-definition,

so this was not flexible enough. Despite Oracle-Connectivity
was OK.

So we ended up with the best (shameless marketing)
enterprise ORM in python-world: SQLAlchemy (http://www.sqlalchemy.org)
Yes, there’s no official download and the developers
say there’s only Version 0.9.1 – but it’s pretty close to final 1.0

and it really suits our needs.

The great difference between SQLAlchemy and the rest of the bunch
is that other ORM-Mappers try to fit relational databases by all means
into Object-World. But a relational DB is no misled ObjectStore and
the whole analogy breaks more and more down with
– larger databases

– transactional aspects
– complex datasets/queries over many tables.

SQLAlchemy has the same philosophy as Z3 in
– extreme separation of concerns and
– integration of standard DB Adapters
– high quality of code + easy readability

(anybody said “pythonic”
– good quality of documentation

SQLAlchemy treats a Class as a Class, a RDB as a RDB, and
(you guessed it, right?) a RDB Table as a
RDB Table and connects them via mappers, which can easily
enriched with some standard methods/queries provided by

the framework.
It makes heavy use of the new dynamics aspects enriching
Python with the last few releases in the 2.x-line.

With SQLAlchemy I could drop the lines of code of my sql-related
stuff somewhere around 20-30%. And I REALLY like the approach
of “Writing less code”. Doing all my RDB-Stuff in python

is an extra-Bonus (OK, to be honest – some complex queries
survived, but I think this is only a matter of time, since
they vanish

So you’re not alone in Zope3-Oracle-World. To
cite the glorious A. A. Milne “And then there were three”
(and maybe even more, if someone answers this post

… and now for something completely different …

LDAP-Integration

Zope3 is very concise about Authentication via
so-called PAU’s – Pluggable Authentication Utilities.
You could easily plug together Authentication sources
with different Authentication methods – its as easy
as plug your lego-stones together.

Like in RDB-World: First you define an Adapter to your

Datasource – lets call it LDAP-Adapter, right -
which defines and holds the connection to your
external LDAP-Source (http://svn.zope.org/ldapadapter/).
That way, you could even use more than one LDAP-Source
in your Application.

Then you have another clearly defined LDAP-PAS (http://svn.zope.org/ldappas/).

which does the authentication against this Adapter.

And the whole beast (you guessed it again, right?) is a small,
well-defined wrapper around python-ldap. Plus
it’s easy to read and fast to understand (my 2 cents).
It’s like dejavu all over again

Hint: I just played around with this LDAP-stuff and never
tested that in our production environment,

but I have great confidence from my previous experiences
with Zope3 that it should be working relatively seamless.

… and now for something completely different …

ZCML

Well, everybody beats on ZCML, since its such an easy
target – “Hey, it’s XML – that’s bad. We don’t want
to use XML (for whatever ideological reason), so

Zope3 must be something ill-constructed”

If you ask me about my feelings about ZCML, I would
not try to convince you it was made in heaven and tell you
that you are too blind to see the light

But – as often in life: truth lies somewhere in
between the extremes. My 2 cents:

1. I share your feelings about not direct
   debugging ZCML, despite the fact that Zope3.2 brings
   very concise error-tracebacks.

2. The Zope3-Guys are aware of the problems users have with
   ZCML. They try REALLY hard to bring as much ZCML back
   to python as possible – see

   http://www.z3lab.org/sections/blogs/philipp-weitershausen/20051214_zcml-needs-to-do-less
   for a thorough discussion from one of the core developers
   of Zope3. Looking from Zope3.0 to Zope3.2 (the current release) some
   stuff vanished from ZCML, so those guys do their homework
   and will do it even more on the upcoming Zope3.3 release.

3. The best thing at the end (now Z3-Team will really beat me ;-):
   You can write up and use
   ALL (right, ALL) ZCML-directives right away as python-code, if you don’t
   XML.
   And yes, this is even documented (call me old-fashioned,
   but I read docs first). You may say, that there are such

   a huge amount of README’s and other .txt-stuff spread
   over the whole Zope-Project, that its not easy to get into it.
   But Zope3 provides you with a toolscript
   called “static-apidoc” which gives you a clear, concise
   overview of the whole documentation as a static website.

   Now if you look at the README under zope/component or

   at the static-apidoc unter “Component Architecture”
   you find methods like provideAdapter, provideUtility,
   which do – surprise, surprise – the equivalent of
   ZCML-alternatives. Or look at zope/component/site.py
   or zope/configuration

   You want to see this in action ? Look at this

   2-part-example of a simple Z3-object publishing system
   without any piece of ZCML:

   Part I: The Zope Component Architecture -
   Interfaces, Adaptation, and Duck Typing
   http://griddlenoise.blogspot.com/2005/12/zope-component-architecture-interfaces.html

   Part II: The Zope Component Architecture -
   One Way To Do It All

   http://griddlenoise.blogspot.com/2005/12/zope-component-architecture-one-way-to.html

   Browsing through the docs, you can find the other
   replacements in a straightforward way (or you debug
   the xmlconfig-stuff from zope/configuration, which
   gives you the corresponding callables for ZCML)

   If that’s too tough and time-consuming – no problem,
   ask on the Zope3-Users mailinglist. Those guys are

   REALLY helpful to get you into Z3-world.

… and now for something completely different ..

Kool-Aid and the magic world of interfaces and adapters

Speaking with developers about Zope3 you often hear
that interfaces and adapters are too much magic and
they have to drink so much Kool-Aid to understand them.
I won’t put this here into a lengthy pro-con-discussion

of these concepts, since I’m not really a core developer,
but more an application developer/maintainer.

But to tell you my story:

It took me a while to GET the main ideas/principles
behind this stuff – to be honest,
2 days of intensive, dedicated work and 5 litres of H2O.
After one week of working with Zope3 I was more productive than
before. Plus I had learned a lot of new stuff about programming
in a quality way. Yes, dealing with Zope3 has made

me a better programmer – even if I never should do anymore project
with it ;-(

Plus it helped me to get my things done better + faster.

Well, that’s Kool-Aid I really like !

I wont’t say this world of interfaces and adapters
is the easiest to understand. But again:
There are a good amount of play-around-with-it-tutorials/docs/books
around which take you into Zope3-World. Just give it a try !

Want some examples? Want some simple apps to play around with?

Well, it was never easier than with Zope3 – see yourself

Here’s some easy stuff which you can work through in less than 1 hour
(well, the last example takes you longer

a) “Zope3 in 30 minutes” – at
http://zissue.berlios.de/z3/Zope3In30Minutes.html
showing you step by step how to build your first
simple Z3-application to collect your bookmarks on a server

b) Did I mention the magnificent Jeff Shell? His blog griddlenoise.blogspot.com is a rich and really insightful source for getting into Zope3 – plus it’s really fun to read. In his archives you find this funky little thing http://worldcookery.com/files/jeffshell-todo/ where Jeff tells you how to build a Rails-like ToDo-Application in some simple steps. In every step he shows you what to do and why he thinks this implementation is carefully thought out in Zope3 and what is the reason they did it this and no other way.

c) Want a fresh new zope3-site without understandig “all the magic”
inside? Choose life – choose the z3 project starter

http://www.zope.org/Members/adytumsolutions/z3projectstarter/z3projectstarter_released
Answer a few simple questions and you have a project skeleton
to play around with without deeply understanding all this “kool-aid” upfront.

d) Philipp von Weitershausens Website/Book about Zope3 -
http://worldcookery.com

You will find more of this tutorials on Phillips website under
http://worldcookery.com/appetizers.

And this is not the end. New stuff is landing every day
in Zope3-space – like this little gem about events/notifications
and how they help you handle complex application
architectures:

http://remarkablysimple.blogspot.com/

Or you look at www.z3lab.org where you can get a peek

of the Zope3-ECM-Initiative and so on so on …

So, my advice to you: Fire on feedster.com, type in Zope3 …
and you’ll find a lot more of this diamonds.

I made the experience that most of the complaints about
Kool-Aid come from developers who specialized in certain
frameworks/habits and now had difficulties to extend
their mindset since they had to leave known territory.
They struggled with Zope3, found some hurdles and then gave up,
since “there’s so much kool-aid”.

I was surprised to find out that absolute newbies to programming
get productive with Zope3 very fast and easily -
maybe since they are not fixed on certain stuff.

I mean, sometimes I don’t get it: People want to write programs
for real complex enterprise scenarios, but at the same
time tell me it’s too hard to spend a few hours to play with some toy examples
and read some docs and play with the marvellous python command prompt
trying to push their brain into a new direction.

Believe me: This whole interface-adapter-pattern definitely helps

you in bigger/complex projects evolving over time.
Remember Fred Brooks Mythical Man-Month, which made so many of us aware
that change of requirements is inherent in any software project -
even if the whole system is in production use.

Zope3 has not ALL, but a lot of REALLY GOOD answers
to this situation every developer faces from time to time

… and now for something completely different ..

Querying the ZODB from outside applications

I’m convinced that it should be easy and I know

that the guys in the Bebop-Project did some stuff
in this direction, But I’m no expert in that,
since relational databases
constitutes more of my work. If you want to
have a profound answer on this – push it
on the Zope3-Users-Mailinglist (sorry, that
wasn’t a sufficient answer, right?). Those
helpful souls there will really show you
the best and easiest way to do it.

….. and now for something completely different ….

Coming back to your Plone-dissatisfaction

I don’t want to tell you that Plone is bad. Or Plone
sucks. Or stuff like that. For me it’s the right tool
for the projects it was made of – usable portal solutions
for medium size. The same holds for Django which
is also OK, if I want to make a fast RDBMS-UI-App.

The good thing is: For different work tasks there
are different tools in my toolchest. It’s my

responsibility to choose the right one for
every new project, but it leaves me with a
warm safe feeling that the toolbelt is filled
with such good quality stuff.

And the good news is just around the corner:
Plone and Zope3-World are converging – approaching
each other with every day. Now what does that mean?

Since I worked with plone it was easy to find my
way in Zope3-World. Zope3 tried to learn from

the Zope2 AND the plone lessons and put a lot of
the best breed of Plone (which constituted at the
same time those hard-to-manage architectural overhead)
back to the core of the framework. Well, this
is not totally right, since there is no such thing
as a monolithic core of Zope3-framework – the greatest lesson
learned from the problems with complex Zope2-projects.
Which is the best news of all

Know Archetypes ? There is schema-driven content-types

with form generation (zope.formlib)

Know Skins and Layers ? Use them right away in Zope3

Know Portlets ? Generalized to Viewlets and managed via ViewletManagers.

RessourceRegistries? Now known as RessoureLibrary.

… and so on … and so on …

But at the same time the Plone guys push their stuff
more and more towards the proven Z3-technologies -
and by handing over Z3 the framework responsibilities
the Plone community again can concentrate on being

the big shot at their homeground -
to provide you with the “MacOS of CMSes”
(well at least that’s what Limi told me

I also appreciate the other web frameworks
in Python world – and I’m happy to see that
there will be a “WebFramework”-Track during
Europython this year where zopies, djangistas
and turbogearianos and all those funky-stuffistas
will get into fruitful

discussions about solutions. I’m really
looking forward to this meeting since
we can learn a lot from each other
if we leave our minds open for the NEW.

…. and now for something completely different ….

There is so much more to say about this marvellous
piece of software (like the integration
of other templating languages like meld or clarity
or the integration of standards like Java-like Portlet-Stuff or

WFMC – the Workflow-Coalition – and and and)
but let me come to an end, since it’s really late
and I need some sleep:

I work for 20 years with software and applications.
Zope3 is one of the most professionall, mature
and qualitative outstanding frameworks I saw.

It’s really fun to work with, if you have a sense for
lasting quality solutions, if you want to be
proud of the stuff you created.

Thanx for your patience + Good night,

Holger @ Germany

PS: If you’ve got any specific question about Zope3,
drop me a not at booradley at web.de. Or visit some
of the links in my rant. Or subscribe
to the Zope3-User-Newsgroups and ask your questions.
This world is really full of possibilities

Ah, and I forgot – the collaborative development
of my favourite Linux distro (Ubuntu)
is managed by a Zope3-Application -

see http://launchpad.net …

or the shiny Z3-based Schooltool if you want
to manage ressources and calendars …
see http://www.schooltool.org

Zope3 is really smoking