While the Copenhagen Plone performance sprint is still two weeks away, I thought I spent some time on profiling during the Naples sprint at this years Plone conference as well.
I concentrated on profiling Zope startup time when Plone is installed in it. While this doesn't matter that much in production environments it is of quite some importance for us poor developers.
On my one year old MacBook Pro Zope would start in about 16 seconds at first. Using Marius Gedminas great profilehooks.py I identified four big time consumers.
1. Importing modules
Not much can be done about that, though some of the CMFPlone migrations seem to be a good candidate to be excluded from initial loading and being defered to a point when they are actually needed. A couple of hundred imports could be safed here I guess.
2. Compiling and registering translation files
In a Plone environment so far all about 620 po files where read and parsed two to three times. I needed to adjust the mo file compilation code in my python-gettext package a bit and hacked PlacelessTranslationService to use lazy message catalogs instead. Those won't parse all the files on Zope startup but instead once they are first accessed. A typical site will probably never use all the languages and parsing and keeping those in memory was a bad choice.
After this was done I was down at 11 seconds. About a 30% speed increase.
3. Reading and compiling page templates
It turned out that right now we have around 140 purely disk based page template files for various ZMI screens. Until now all of them were always read, parsed and compiled on Zope startup; most of them are never used. So I added some lazy loading code into Zope's PageTemplateFile, made it available as an option and enabled it from CMFPlone. As the CMFPlone code is read after some of the templates are already loaded, we can only skip about 110 of them in this way. I'll add the whole thing as a zope.conf option, which will be parsed early enough in the process to get rid of the remaining ones.
Measuring the startup time again, I'm now down to 8 seconds, another 30%.
4. Parsing ZCML files
Parsing hundreds of ZCML files is currently the remaining big block for startup time. AFAICS this uses expat right now. It might be interesting to experiment with different XML parsers like lxml to see if some performance gain can be achieved here. I'll leave the exercise to someone else (Florian had shown interest in it...).
By spending about half a day of work on the topic I could speed up Zope startup time by an overall 50%. Not too shabby I think :) Lets see if there are some weak spots while loading the Plone site TTW the first time as well...
The downside to this all is that it is only available on Plone and Zope trunk. Maybe we can backport some of these improvements to the Plone 3.x line.