Comments on ..: hannosch :..: Performance sprint warmup

@Marius: Interesting work!A current Plone 4.0 / Zo...

2007-12-27T22:48:00.000+01:00

@Marius: Interesting work!

A current Plone 4.0 / Zope 2.11 instance needs 10 seconds to start up of which 2.5 seconds are spent parsing ZCML. This is the biggest block. Second is creating Archetypes schemas (lots of deepcopies of nested classes) and then already time spent on pure importing of Python files.

I wonder if the time to execute the ZCML could be speed up by turning it into real Python bytecode instead of using the pickle protocol. We had some good results with this while working on a TAL to Python compiler (some prototype work based on Spitfire - http://code.google.com/p/spitfire/).

Great improvements!How much time is spent parsing ...

2007-12-27T19:45:00.000+01:00

Great improvements!

How much time is spent parsing ZCML in Zope 2?

At a recent Zope 3 sprint Stephan Richter and I attempted to sidestep the ZCML parsing step altogether (using some nasty hacks to pickle the ZCML configuration actions). Our prototype showed a potential ~33% reduction in Zope 3 startup time (from 3 seconds down to 2 seconds on a pretty new Thinkpad).

My interest in this approach waned (I see little difference between 2 and 3 seconds -- both are either "fast enough" or "too slow"), but Stephan Richter was very enthusiastic and wanted to pursue it. You might want to talk to him.

I would also like to see the speedup from switching to a different XML parser. lxml and cElementTree come to mind (one has a reputation of being fast, the other comes with Python 2.5). While fast parsing is slower than no parsing at all, it would be much simpler to implement and maintain than ensuring all the ZCML configuration actions are safe to pickle.

For those interested in it, I found that five:impl...

2007-10-20T16:43:00.000+02:00

For those interested in it, I found that five:implements in combination with fat Zope2 objects takes quite a bit of time. Moving two of those from Five's configure.zcml to the real classes made those actions about 75 times faster...

The lazy page template behavior is now the default in Zope2.

One more bottleneck I found was hidden in MimetypesRegistry where a 750KiB big XML file with mimetype information was parsed on each startup. I changed the code to create and load a binary format now instead, which safes another whole second on Zope startup...

Let's see if there's more :)

You rock! Just imagine the amount of time plone de...

2007-10-17T12:19:00.000+02:00

You rock! Just imagine the amount of time plone developers will save in aggregate.

@Philipp:I've already looked at zc.configuration, ...

2007-10-17T09:07:00.000+02:00

@Philipp:
I've already looked at zc.configuration, but Plone and Five don't load much unneeded zcml. The whole zope.app.* packages aren't loaded for example.

I'll try to port the lazy stuff to zope.pagetemplate as well, ones I made it configurable through a zope.conf option.

The lazy message catalogs and the automatic mo file compilation are on my list for zope.i18n as well...

Speeding up ZCML would be greatly appreciated. I'm...

2007-10-17T06:42:00.000+02:00

Speeding up ZCML would be greatly appreciated. I'm not so sure the problem is with the XML parser, but this can only be answered for sure by careful analysis.

One low-hanging fruit is to go ahead and disable all the ZCML that you don't need using Jim's new package zc.configuration (http://pypi.python.org/pypi/zc.configuration). This was done for Grok at the Neanderthal sprint: http://svn.zope.org/grok/branches/neanderthal-startupspeed/ and produced a rough 10% speed increase by excluding a few trivial files.

Nice catch with the PageTemplates, by the way. I wonder, could you forward-port this to zope.pagetemplate's PageTemplateFile, too?

Bravo, Hanno!

2007-10-17T02:25:00.000+02:00

Bravo, Hanno!