..: hannosch :..

Goodbye Jarn, Hello Mozilla

2011-11-24T20:00:00.000+01:00

tl;dr Starting in January I'll work for the Mozilla Services Team and will be based in Hamburg, Germany.

It's been almost four years since I left my hometown and old career and moved to Norway, doing Plone development full time at Jarn. For a while now I've been looking for a change of scenery. Unfortunately Jarn's plans to establish a presence in Berlin didn't work out, so I was back in small town Norway for the past eight months.

At the same time I've been using Plone for almost ten years and started engaging with the community nine years back. It's been an exciting and wonderful time, but as of late my enthusiasm has faded away. On the other hand I've witnessed an immense community energy at this years living status sprint and the Plone Conference 2011. I'm certain the striving and ever renewing community will move Plone to the next level - it's just going to be without much of my time.

Now I'm looking forward to an excellent new team. And a new opportunity to contribute to Open Source software and making the web a better place. Catch me in Hamburg, San Francisco and maybe in a Mozilla Berlin office soon. Or in #distutils, #pylons or related Python projects :)

So long and thanks for all the Plone,

Hanno

P.S. Alex Limi cannot be blamed for my departure at all - it was entirely Tarek's fault ;)

Plone 4.1 - Is it any faster?

2011-04-10T21:48:00.002+02:00

Plone 4.1 is in its final beta stage, soon to be released as final. So it's once again time to look at its performance improvements.

In Plone 4.0 we have focussed on improving Plone's raw rendering speed but also memory consumption and handling of binary files. These improvements benefit sites of all sizes and flavors.

On the performance side we have focussed on a different area for Plone 4.1: the catalog. The catalog is responsible for all searches, content listings, portlet listings, the section tabs, the sitemap, the reference engine, uuid lookups and many more things.

For small to medium sites with maybe up to 10.000 content items the performance has been fine. But once your sites gets multiples of 10.000 content items up to millions the performance starts to degrade rapidly.

Over the last couple of years various companies and community members have worked on this problem. So far these improvements have only been available as add-on components with varying degrees of maturity. For Plone 4.1 we have taken the stable and proven bits and integrated them back into the core.

The improvements fall into two broad categories: Improving the data stored in indexes and optimizing the internal search algorithms.

On the storage side we have added two new specialized index types: a UUID index and a boolean index. We use them per default in Plone and update your existing indexes during the upgrade. We've also updated almost all existing indexes to store data more efficiently, just store less data or reduce the conflict error potential for concurrent write operations.

On the search side the biggest change is the introduction of a queryplan. All catalog searches are now instrumented to collect runtime information. Based on this data subsequent catalog searches will be executed in an optimized way. The process is fully transparent and needs no configuration. You can inspect the plan and reports about slow catalog queries in the ZMI screens of the catalog tool.

But in what kind of improvement does this result in?

For a small site or a default empty Plone site there's no measurable difference. The good news being that there's no measurable overhead of all these changes.

In order to illustrate the impact I used one large real site and tweaked it to resemble a default Plone 4.0 and 4.1. The actual production site runs various of the mentioned improvements via experimental backports, so I had to take those out to get a realistic comparison. The site in question has about 250,000 content objects with mostly published content. It uses LinguaPlone and has many portlets listing latest news or related content, so most pages do a number of catalog queries. As each catalog query is faster, you get cumulative effects.

In the graph I show the timings for the frontpage, a news archive with 4,000 news items per language, a normal page and it's edit screen.

For pages doing either multiple catalog searches or catalog searches over large amounts of content we get an improvement of 2x up to 4x. For an edit screen that doesn't even show the navigation tree, there's less benefit. But even rendering the top sections and the language selector results in a 1.4x improvement for this site.

I did the testing on my local iMac with 3GHz and 8gb of RAM while the data was coming from a direct file storage. Effectively all operations were done with all data coming from the ZODB cache or from the operating systems disk cache from memory.

In reality the improvements will be larger, as we read far fewer objects from the database and thus avoid talking to the actual database over the network and doing disk reads. You will also likely be able to reduce the size of the ZODB cache without negatively affecting performance. The freed memory can be used to add more Zope instances or to decrease the hardware cost of running the site.

But of course the results for any given production environment will vary a lot. And none of these numbers reflect real world usage, as visitors will get served cached content from a reverse proxy in most cases. With the inclusion of plone.app.caching into Plone 4.1 setting up efficient caching has become much easier.

There's many more improvements in this area that we will do. But with Plone 4.1 you should be able to run a Plone site with multiples of 10.000 of content items without the need for custom development - system administration knowledge to set up the application cluster and database is still a must though.

Personal: Back to the city life

2010-08-31T13:39:00.007+02:00

Two and a half years ago I left my home country Germany to start a new adventure. I made my Plone hobby into a professional career choice and at the same time left to the Norwegian wilderness to work with all the talented people at Jarn.

Much has happened during that time. Our little company started with three employees and will soon have grown to thirteen and a half in multiple locations. We moved into new custom built office premises in 2008 and are now looking into extending those to meet our growing needs.

I had the pleasure of traveling the world, spending four months in Copenhagen and working on interesting and challenging projects during all that time. Plone itself has come a long way too, culminating in the marvelous Plone 4 release yesterday. I had some concern that spending my time on Plone during the workday would make it less interesting to work on it during my free time. But I'm happy this hasn't come true, I'm as interested in contributing to Plone as I was before - even though I shifted my focus more and more to the infrastructure side and am now spending a good deal of my time in Zope 2 and the Zope Toolkit.

But with all those good things, I still didn't feel happy all the way. I had to realize that I was neither made for dark Norwegian winters nor small town life. Coming from a buzzing multi-million city myself, there was something missing.

Luckily Jarn has established an unofficial presence in Berlin, Germany for some time now, so there was an easy way out for me. As of yesterday I moved to Berlin and will continue to work for Jarn from the new location.

Looking back I wouldn't want to miss those years and it's certainly been the right choice for me. Living abroad for a while is a wonderful experience and teaches you many things about yourself, your home country and many more different perspectives onto the world.

And now I'm looking forward to the next years and Plone 5 at the horizon :)

Hanno

P.S. As a Social 2.0 opt-out I have neither Facebook or Twitter accounts. If you are not interested in my personal notes, you don't have to follow posts labelled "personal" on this blog ;)

Zope 2.13 - alpha imminent

2010-06-14T20:08:00.006+02:00

Dear world,

this is your Zope 2 release manager speaking. We had some discussion in the community about the roadmap ahead for the next Zope 2 release culminating in a plan today.

A first alpha release for Zope 2.13 will be released on June 25 - just over 10 days from now. More alpha releases will follow on a 3-4 weeks basis and a first beta will be out in early September. That is as usual without guarantees and subject to changes. A final release is expected later this year.

Zope 2.12 is continuing to be maintained. A Zope 2.12.7 bugfix release was released just yesterday.

Current Zope SVN trunk (which will become 2.13) is compatible with both CMF 2.2 and CMF trunk and the upcoming Plone 4. It is likely that Plone 4.1 will officially support Zope 2.13.

Zope 2.13 will bring a couple new features and has done a large deal of refactoring and house keeping. It currently only supports Python 2.6. Support for Python 2.7 might be added, if there is community volunteers to do the required work. Python 3.x support is out of the scope. Work has begun to port some of the underlying zope packages to Python 3.x, but there's still a lot more to do before this is feasible for Zope 2 itself.

So what's in it?

Zope 2.13 doesn't have any zope.app.* dependencies anymore. This finishes the transition from the hybrid Zope 2 + 3 codebase. Zope 3 itself has been split up into two projects, the underlying Zope Toolkit consisting of foundation libraries and the application server part. The application server part has been renamed BlueBream. Zope 2 only depends and ships with the Zope Toolkit now. Large parts of code inside Zope 2 and specifically Products.Five have been refactored to match this new reality. The goal is to finally remove the Five integration layer and make the Zope Toolkit a normal integral part of Zope 2.

Zope 2.13 comes with native WSGI support. First pioneered in the repoze.zope2 project, this capability finally found its way back into the core and obsoletes the externally managed project. With WSGI Zope 2 can natively talk to a variety of web servers and isn't restricted to its own ZServer anymore. It also opens up new possibilities for writing or reusing middleware in Zope 2 or factoring out capabilities into WSGI endware. It's expected that this new deployment model will over time become the default and the old ZServer implementation will be deprecated. There's no concrete timeline for this yet.

There's an ongoing effort to refactor Zope 2 into more independent modularized distributions. Zope 2.12 has already seen a lot of this, with the use of zope.* packages as individual distributions and the extraction of packages like Acquisition, DateTime or tempstorage to name a few. Zope 2.13 continues this trend and tries to externalize packages containing C extensions. In a not so distant future it should be possible to run a Zope 2 like environment without the ZMI and the corresponding code. This is part of the vision for Zope 2 to evolve into a more modern and maintainable codebase, while preserving a realistic gradual upgrade path for existing installations and projects based on it. The Zope 3 project has shown that a complete rewrite is no viable answer for a large class of existing software.

There's a large number of smaller features and improvements in the underlying libraries and Zope 2 itself. For example Zope 2.13 will ship with the new ZODB 3.10 version, which sports some noticeable performance improvements. The move to the latest version of the Zope Toolkit libraries makes integration of Zope community projects like z3c.form much easier.

After the rather long release periods of Zope 2.12 and 2.13 we expect to switch to a shorter release period of six to nine months for the upcoming 2.14, 2.15 and beyond releases.

Discussion of the Zope 2 core development takes place on the zope-dev mailing list. Feel free to engage, leave comments on this blog or contact me in private.

Your release manager,

Hanno

Zope 2 - Releases

2010-04-06T18:11:00.006+02:00

Hello World,

this is your new Zope 2 release manager speaking. I'm happy to report my first successful Zope 2 release, version 2.12.4. You can find it at the usual place: http://pypi.python.org/pypi/Zope2/2.12.4. More information about the release can be found at http://docs.zope.org/zope2/releases/.

If you are interested in more background information about me becoming release manager, there's a full announcement on the Zope developer mailing list at https://mail.zope.org/pipermail/zope-dev/2010-March/039880.html.

Don't worry, I won't bother you with blog posts about every maintenance release in the future. The next major feature release is still some time off, there's an initial roadmap discussion at https://mail.zope.org/pipermail/zope-dev/2010-March/039915.html.

All the best,

Hanno

Plone 4 - How much faster is it?

2010-01-17T18:33:00.010+01:00

During my work on Plone "trunk" in the last year, I kept posting some fancy graphs showing the speed improvements for rendering pages. Much has happened since I last posted one of these in March 2009.
The newly coined Plone 4 is almost ready for a beta release and some companies have started using the alphas for production projects. Overall it's snappy, fun and stable. But how does it stack up against my experimental work? I got the good old ZPTDebugger out and optimized some more rough edges during the weekend and squeezed out another 10%. See the result for yourself:

Compared to Plone 3.3 Plone 4.0 is about 50% faster, ranging from 30% for edit pages up to 70% for anonymous page views. If you use the Chameleon template engine, you get two to three times the speed compared to 3.3.
As a little experiment I trimmed down the default content, portlets and viewlets in a 4.0 site to match what I had done in my experimental work. So I removed all portlets, hid the section, personal bar, site actions and content history viewlets and measured again. In the Chameleon setup you get the same kind of results as I did back in March. For anonymous page views, we are roughly at 50 requests / second. Removing all those elements might sound somewhat odd, but if you don't need some of these features or use xdv to supply some of these elements as static HTML, you can actually get close to these numbers.
It's good to see my work finally in the mainline and available to all. Now onwards to the next adventure, making Plone easier to use and develop for :-)
P.S. The controlpanel stats are indeed correct, though they look pretty odd. I didn't try to find the cause of the slowdown compared to all other page types.

Sprints and Conferences

2009-08-31T22:02:00.018+02:00

I've been traveling a lot recently and didn't really report about what I've been doing. So here's a short summary of the last months and what is to come.

From Easter until end of June I stayed at a customer site in Copenhagen and helped push the new Plone-based norden.org website into going live. After quite a long time of being in development it was good to get this out to the public. Most of July I spent back in Tønsberg in a new apartment relaxing and enjoying the warm and quiet Norwegian summer.

In early August I visited Bristol again, this time for the Balloon sprint organized by the awesome Team Rubber. Their offices are really nice and strategically placed in a lively neighborhood with lots of pubs around. While I missed out on the actual Balloon festival itself, I got around to help MJ fix some nasty Acquisition and Python 2.5+ 64-bit issues. Otherwise I spent my time on attacking some long standing annoyances in plone.reload cumulating in a 1.0 beta 1 release. plone.reload allows to reload the code and ZCML configuration for a running Plone process based on an idea from Guido for a better reload mechanism. Now it comes with 100% test coverage and supports reloading code inside decorated methods.

With some time at home and regular work, next up was the Repoze "CMS" sprint in Arlington, Virginia. Irene and Chris Shenton together with all the other sprinters made this one of the best social experiences I had on a sprint. By the end of the sprint we had managed to empty a keg of beer and enjoy lots of good food and conversations :-) On the actual work side I participated in all the opinionated discussions we had, but kept to my own little toy project for the code side of things. Still being work-in-progress I got a long way into integrating Babel with Zope. An attempt to replace the dated and custom zope.i18n package we use with the de-facto standard for i18n and l10n in Python web environments. I made this into an experiment to try out Mercurial some more, so the code ended up on bitbucket. The code is far from finished - the message catalog and translation infrastructure is working, but the language negotiation and locale data support is still missing.

Next up I'm going to attend the unofficial Zope conference of the year, better known as the German Zope User Group conference in Munich next week. In my role as Plone 5's release manager I was invited to give a keynote about the state of Plone along side two keynotes from Martin von Löwis and Lennart Regebro. I'll attend the after-conference sprint, hoping to do some work on Distribute and head straight to the second incarnation of the Grok Neanderthal sprint for a couple of days in Cologne. I'll try to spent some time working on the Zope Toolkit - the extraction of more reusable packages from the former Zope 3 project. Ending another two week roadtrip with some well-deserved vacation in my former home town Hamburg.

And then it's only a short time to the annual Plone conference in Budapest, which I'll be attending with my fellows co-workers from Jarn. A company that values their employees and makes all of my travels possible thanks to our 10% manifesto.

At the end of the year I will have stretched myself to being on the road for 150 days, attended six sprints and three conferences including their sprints. I think next year I'll try to stay a bit more in one place. If only the Python, Zope and Plone communities weren't so much fun to hang out with ;-)

Visualizing the persistent structure of a page

2009-05-23T14:03:00.003+02:00

Inspired by David's blog post I took his script and modified it slightly for a different purpose. Instead of showing an entire Plone site, I used it on a single page, with only a simple title and text. Here's what a page actually consists of on the persistent data level:

So what you think of commonly as one page object is indeed at the minimum ten persistent objects. As soon as you add references to it or store certain other information on it, it will grow even more.

If you ever heard complains about what is wrong with the way we store data in Plone, this might make it more obvious. Another thing you need to know about the ZODB is that it is slightly simplified just one big table with a unique identifier (oid) mapped to the pickle data. While conceptually the above object references are important, on the DB level loading any of these objects from the DB is the same operation. If you have a ZEO setup where loading an object incurs some network latency, you'll end up with ten times the latency for loading what seems to be a simple page object.

So if you ever wondered why you need to increase your ZODB cache to numbers in the thousands and Plone needs so much RAM, the above should give you some ideas. Fixing these problems is incredibly hard after they have been introduced, but people like Laurence Rowe keep reminding us of the importance of it. </rant> ;)

Can I make my Plone 3.x faster?

2009-03-05T00:23:00.007+01:00

In my series of showing of fancy graphs, I have gotten repeated complaints about not caring about Plone 3.x enough. Now I did clarify that many of the improvements we make to Plone trunk are too risky or backwards incompatible to do in Plone 3.x. But some of the improvements have made it back into Plone 3.3.

Most importantly of all Malthe Borch is working like crazy to make Chameleon compatible with the next Plone major release numbered 3.3. He certainly deserves any kind of support you can give him. While Chameleon is not going to be part of the official 3.3 release, thanks to our move to zc.buildout you will be able to install it painlessly into your sites. As with the other experimental.* packages announced at various times, this is still not going to be for the casual user. But if you are an experienced Plone integrator those options are worth checking out today.

Plone trunk has seen another couple of improvements pushing it close to the 50 req/s limit, but taking Plone 3.2 as a baseline, the slogan today is: Plone is getting twice as fast with every release:

Plone trunk continues to get faster

2009-03-01T17:12:00.004+01:00

It's been more than two months, since I had a detailed look at the state of Plone trunk performance. This weekend I spent on evaluating what impact the changes we made during that time had on performance and worked a bit more on it.

I made another round of improvements to the way we use CMF actions and converted some of the last Python scripts used in every page to browser view based code. Using as little untrusted code as possible to be part of the page rendering proved once again to be a fruitful endeavor.

My current findings indicate that both the sheer overhead of security checks for untrusted code, but sometimes more importantly the hard to understand and spread out code itself cause the performance problems we have. An example I found today was the language selector viewlet. A nice mixture of code and nesting found in the template, some code in the viewlet class, both backed up by the language tool and querying a language utility in the backend. Each of the pieces looked good in itself, the combination meant that even for a site with a single language a large dictionary of all languages in the world was deep copied, and iterated over twice to determine that the number of languages used in the site was not more than 1. The relevant information here being easily accessible as simple attribute access away on the language tool. Concentrating the control flow from all the different places into the browser view made it far easier to understand what happened and avoid the insane overhead.

Long story short, compared to Plone 3.2 as a baseline, we got up from a factor of being three times faster to about four times faster now.

Zope 2.12 alpha one - it's done!

2009-02-26T22:24:00.003+01:00

Today marked an important milestone in the history of Zope2. With the first alpha release of the next major version of Zope2, it proves once again to be still a serious player in the Python world. Thanks to a focused effort of its active community we have taken Zope2 to the next level.

This release not only introduces support for both Python 2.5 and 2.6, includes the latest versions of the ZODB and the Zope component libraries, both still under heavy development each, but also redefines its installation story: Zope2 is now fully eggified and plays nicely with the standard way of installing Python packages today: setuptools - easy_install Zope2 at will.

If this wouldn't be enough in itself, the last weeks have seen an incredible uptake in updating Zope2's documentation and http://docs.zope.org/zope2/ is shaping up nicely. You can read more about the new release at: http://docs.zope.org/zope2/releases/2.12/.

In order to include the latest versions of the ZODB and the Zope components, a final release is not expected before Summer 2009.

The next major version of Plone is developed based on Zope 2.12 and we are more than happy to have such a strong foundation for our work.

Berlinale Sprint 2009

2009-02-17T22:28:00.006+01:00

I'm back for a couple of days from Jarn's first 10% sprint.

While everybody else was working on cool and exciting things, I happened to be in the wrong IRC channel at the wrong time. As a result Tres Seaver and me started to do something about Chris McDonough wining about old and unmaintained artifacts.

After a couple of days and nights of more work, you can now witness the new Zope2 Book - 2.12 edition. Finally we put an end to the confusing situation of the 2.6 edition being hosted at zope.org versus the 2.7 edition being generously hosted by Chris. The book has come home and is now available in a new and shiny form at http://docs.zope.org/zope2book/ thanks to Jens Vagelpohl.

Most of the time was spent converting the book from its old structured text source into a more modern reStructuredText format. With Sphinx and the design taken from docs.zope.org it was easy enough to produce the nice HTML output you see now.

We also did spent quite some time to update the actual content of the book and it is starting to be a valuable source again. The purpose of the book is still to introduce newcomers to Zope, who arent't full-fledged developers. It won't ever be a good guide on how to develop complex applications with Zope2 today. We weren't able to catch up on five years of history in the matter of days, so don't expect the book to be all new and shiny, but you will neither find ZClasses in it anymore and might be surprised to see links to the Python Package Index.

If you want to contribute to the book, the source material is available in the Zope SVN repository at svn+ssh://svn.zope.org/repos/main/zope2book/trunk. The web content is updated directly from SVN once a day. If you don't have commit rights, you can send patches or corrections to me and I'll gladly apply them for you.

Inspired by those activities Tres and Baiju M have also started to update the Zope Developers Guide and are busy converting it to reStructuredText as we speak.

The future of Zope2 is still looking bright ;)

Update: Baiju finished the ZDG conversion and it's available at http://docs.zope.org/zdg/. The links at http://www.zope.org/Documentation/Books/ have been updated by Jens.

If anyone feels strongly about the design, please get in touch with the Zope Foundation and the people behind new.zope.org. I'll not start tweaking CSS for one book here.

Baarn Plone UI Sprint 2009

2009-02-02T00:59:00.004+01:00

I'm back in Norway after four most productive days spent in Baarn on this years Plone UI sprint. Many thanks to Danny Bloemendaal, Wichert Akkerman, Cornelis Kolbach and Informaat for hosting this event once again! It was wonderful and I'll happily come back next time to try out some more different pancakes ;-)

The work part of the sprint had more presentations and large group discussions than the average Plone development sprint, but these have been both interesting and very productive. I think we got a lot closer to getting a common vocabulary and ideas about how to approach a next generation theming and UI story for Plone.

Luckily we didn't start from scratch, but could base our work on the ideas and quite detailed proposals that have been made in recent month. Especially the Deco and Blocks proposals and the prototype implementations of Rob Gietema from Four Digits have been of tremendous help.

Wichert has already written about how we divided the whole group into five smaller teams. I'll try to give a short summary of what has been worked on by each team.

Content editing

When it comes to content editing one of Plone's distinguishing features is its approach to do real inline WYSIWYG content editing. Most other CMS's have a separate administration interface, that has little to do with the website driven by the system. Plone's integrated approach has been a huge success, but has proven to have various difficulties over the years.

Based on the experience and proposals the team looked into different ways of more clearly separating the administrative UI from the website itself. Examples of this are moving the administrative controls to the edge or most commonly the top of the browser window, lowering the number of controls shown by default and consolidating the controls into a single area.

Another area has been the support for so-called working modes. So far Plone determines what kind of controls the user sees primarily based on the permission set of the user in a given context. The standard view of a page mixes in inline editing controls and directly offers access to even the most uncommon operations with equal priority (think of View, Edit, Rules, History, Sharing but also assigning portlets).

The idea behind working modes is to make it an explicit choice of the user to go into any kind of editing operation. Usually most users will only view information, even if they have the right to edit. The switch to edit mode should be fast, easy and seamless, but nonetheless explicit. Once the user sees the editing controls, he should still see only the most common controls, to make them more effective to use. More uncommon options can be hidden on different pages or using modal dialogs.

We have seen various prototypes of how such separation of concerns could look like and what is technically feasible in modern browsers. One of the things everybody agreed on, is the desperate need for user testing of these approaches, to see what works in the real world for normal users.

Page assembly / Page layout

One of the main ideas behind this work, which is based in large parts on the Deco proposal, is to make it possible for normal users to create rich pages including rich layouts through the standard Plone UI in an intuitive but powerful fashion. It also unifies the editing story for the main content area and any kind of secondary columns, taking away the complexity of entirely different ways of working with these as seen today.

The team gathered around Rob to continue the work on the existing prototypes and incorporate feedback from the other teams into the ongoing work. I'm sure Rob will give us more details in the next days on the progress that has been made.

In the daily wrap-up presentations we have seen a lot of work being done on the modal dialogs, like integration of the actual image browser, examples of tabbed settings in the dialogs and save dialogs including more advanced options like the change note field or the possibility to do workflow changes. The prototype from before the sprint is visible online and lets you have a sneak peak at the kind of changes we are working on.

Control panels

The control panel or global administration UI team gathered around David Convent and part of the GW20E crowd and had time to start implementing many of the ideas and sketches done in the first day. If you followed the SVN commit messages you will have noticed the new plone.app.users package growing over the last days. As seen by the name the main focus has been on the user related settings and general user related data.

The entire range of user dashboard, profile or author page, personal settings, password change forms but also join forms and general member data administration have gotten an overhaul. The goal is to clean up the confusing UI in this area (navigation between the different pages is quite unintuitive today) and provide more flexibility to adjust parts of the pages in question. At the end of the day it should be easy to add for example the country to the member data, make it a required field on the join form and get it into the users data screen.

Backend / Site Theming

The team gathered around Laurence Rowe, Eric Steele and Denys Mishunov and worked and putting the Blocks proposal into action and refine it based on actual experience from designers.

The goal in this area is to redefine the entire theming story for Plone in a way that makes it vastly easier. The focus is on making it possible to theme a Plone site effectively with little knowledge about Plone itself using HTML, CSS and images as the main tools. Knowledge about any kind of Zope component architecture or various forms of METAL macros, viewlets, portlets, skin templates or browser pages should no longer be required.

Towards the end of the sprint we had a quite advanced demo that showed this in action and we were able to see how even a Plone 3 site was themed in this way. While the new approach uses a unified concept of tiles as "possibly dynamic chunks of HTML" instead of all the concepts we had before, bridging code to provide viewlets and portlets into this new model has been written and many more of the older concepts should be easy to bridge into the new model. An evolutionary upgrade path is going to be possible with this, not causing a major break in knowledge and technology investments already being made. Denys captured one of the demos on video and I hope he and Eric will tell the world more about this exciting new story.

Testing AJAX

With all exciting new technology and especially a focus on a much richer client side editor environment, comes the dreaded question on how to do effective testing of AJAX applications. The team gathered around Balazs Ree and Godefroid Chapelle of KSS-fame, who brought extensive knowledge in the area with them. Various different approaches on how to effectively test the real application, doing so in a repeatable and automated way, while making it easy to write tests have been discussed and some of them prototyped.

The tools of choice today are still Selenium and with Selenium RC and its Python bindings a way to instruct a browser from a Python test. Thanks to collective.buildbot and buildout setting up an automated testing infrastructure on multiple platforms using different browsers has become vastly simpler. Progress has been made in various directions, picking up old work from the early KSS days as well as utilizing new infrastructure like WSGI middleware. I hope Jean-Paul Ladage and Balazs Ree can tell more about the concrete achievements being made.

Summary

As you can read there's been work done in various directions and I'm excited about all of them. With the progress being made on the sprint I'm looking forward to see the wider community feedback on these topics. Now that we have gotten actual prototypes to play with and are starting to see concrete implementations of the different pieces this no longer is an abstract exercise.

Profiling with kcachegrind made easier

2009-02-01T13:48:00.007+01:00

During the third day of this years Baarn UI sprint, I couldn't resist to look into profiling Plone's startup speed again. As I've been recently getting used to using kcachegrind for visualizing profiling data, I wanted to make it easier to get the profiling data right in this tool.

When doing profiling on full application requests the WSGI middleware repoze.profile is an excellent tool to use. But for profiling startup speed I needed to get back to profiling an individual function call. profilehooks to the rescue? Sadly not.

In order to convert the profiling data into a format understandable to kcachegrind I needed the data to be in the cProfile format produced by Python 2.5's new profiling library. profilehooks still lives in the Python 2.4 world and has profile and hotshot support.

So I made a new tiny package called profilestats which will use the cProfile module instead. It works on both Python 2.5 and 2.6 and writes out the profiling data directly into a file called cachegrind.out.profilestats for kcachegrind to grok.

On the actual Plone startup speed front there isn't any real news. Parsing and executing ZCML, deepcopying Archetypes schemata and just the sheer number of imports and packages to load are still the slowest parts. Especially the recursive algorithms used in zope.interface and zope.component are still the slowest individual parts by a large offset.

How to protect your Data.fs

2009-01-31T13:06:00.005+01:00

I'm currently enjoying some very productive time at this years Baarn's UI sprint organized by Informaat in the Netherlands. While I'm primarily working on the testing group, where we try to get a full-roundtrip Selenium-based testing story in place, we were looking into ZODB demo storages for different reasons.

As a side effect we came up with the following little snippet:

from ZODB.DemoStorage import DemoStorage
from Zope2.App.startup import getConfiguration

# Read the database configuration from zope.conf
dbtab = getConfiguration().dbtab

# Get the root storage and open it
base_storage = dbtab.getDatabase('/', is_root=1)._storage

# Wrap an in-memory demo storage around our normal storage
Storage = DemoStorage(base=base_storage)

If you put this code into a file called custom_zodb.py in your INSTANCE_HOME, your Zope will start up with its normal file storage, but wrapped in a demo storage.

With this file in place all changes you do will only be written and kept in memory. Once you restart the server, you are back to the state of the Data.fs on your file system.

The next time you want to experiment with a large customer Data.fs, show a demo based on some predefined content, provide a Live-CD or host a demo site, this might come in handy. If you want to feel even more on the save side, opening the underlying storage in read-only mode or ensuring that the operating system level user doesn't have write permissions to the file could be additional measures to take.

Plone trunk versus kcachegrind

2008-12-13T01:51:00.008+01:00

David Glick reminded me a while back of a visualization tool for profiling data called kcachegrind. After trying to optimize Plone trunk based on simple Python profile data for a while, a more advanced analyzing tool was desperately needed.

In order to convert the normal Python profiling output into a format readable by kcachegrind, you need to run a script called pyprof2calltree over it. The publically available version seems to be broken in some ways but David took on the challenge and sent me a fixed version.

After getting familiar with the tool and poking at Plone trunk for a bit, I was able to identify a couple of more places, which I hadn't found so far in my optimization attempts. So at the end of day two of the Plone Performance Sprint 2008 happening in Bristol right now, I got some notable results. Compared to the last results I posted three weeks ago, everything but edit pages have seen another 50% speed increase. Thank you David!

One of the changes was inside ResourceRegistries and can be readily applied in Plone 3.x as well. It saves about 10ms per request in a default site. All other changes still revolve around optimized actions handling. The amount of speed increase you can get by rather simple and straightforward changes still amazes me. Stay tuned for more.

Update:

This morning I found another problem with content type icon expressions being recompiled on every page load. I updated the graph with the new results: We got up from 24 requests/s to 27 requests/s ... I like it :)

Update 2:

I found a bit more places to optimize and took a look at Archetypes edit screens in particular. Updated graph is above, we are now officially three times faster than Plone 3.2 and have reached the magical 30 requests / sec number.

Plone versus Python 3

2008-12-06T20:06:00.002+01:00

Python 3 final has been released. What does this mean for Plone and how could a roadmap for adoption look like?

I peaked into my crystal ball and came up with a couple of statements that reflect my own personal opinion. So here is one possible scenario for the future:

The Plone 3 series will see major releases up until 3.4 or 3.5 in a roughly six to nine month iteration and stretch into 2010.
Plone 3 will see support for Python 2.5 and 2.6 in Plone 3.4 while staying backwards compatible with Python 2.4.
Plone 4 final will be released in early 2010.
Plone 4 will recommend Python 2.6 and might be backward compatible with 2.5.
Plone 4 will see multiple 4.x releases in the same way 3.x did. These will continue throughout 2010 and 2011.
If a Python 2.7 is released during the lifetime of Plone 4.x, support for it will be added.
Plone 5 will run on Python 3 and see a release in 2011 or 2012.

This is only one possible scenario. What do you think?

Plone LOC statistics

2008-11-20T23:36:00.007+01:00

A couple of days ago I removed the source code statistics for Plone from ohloh.net. The way ohloh calculates statistics from Subversion repositories is long broken for any project that copied or moved any of its code base in Subversion. Plone has done so many times in its lifetime and as a result ohloh was showing wildly incorrect statistics.

But fear not - ohloh's lines of code counting tool called ohcount is available for download and local install. Executed on a checkout of our codebase it provides quite accurate results.

A distinction between the Zope, CMF and Plone parts of our stack isn't practical anymore for Plone trunk. In a completely egg-based world, we have cases like plone.portlets depending on zope.app.content, while it is not included in the Zope2 egg anymore. Where should one place Products.ZopeVersionControl, which is pulled in for Products.CMFEditions but written by the Zope community? The number that is comparable over all releases is the total, so here it is:

The main source of the recent decline is the removal of lots of zope.app packages from the Zope2 installation, which were never used nor usable inside Zope2. zope.app.rotterdam, the example theme for the zope.app ZMI replacement is the most famous example of those. But we are also busy at reducing the amount of old code we have been carrying around in our own packages.

I expect to see the total amount of lines of code to decrease further, while we are developing Plone trunk. Yes, that is: We will be able to deliver more features with less lines of code, without cheating on quality, tests or documentation.

News from Plone trunk

2008-11-20T19:09:00.014+01:00

Some people following the developer mailing lists or blogosphere have noticed that Plone trunk has seen quite some interesting changes as of late.

Since I'm the current intermediate release manager for the upcoming Plone 4.0 release, I thought I might share some of those exciting news with you. The framework team for Plone 4.0 is currently being selected and will suggest a final release manager to the Plone Foundation board. I'm available as a candidate, but the decision is yet to be made.

Over the summer we have seen an amazing effort from some of our Google Summer of Code students and their mentor Sidnei who brought us Python 2.5 and 2.6 support for Zope trunk. This has been followed up by Sidnei and Matthew Wilkes and we can run Plone trunk with all Python versions from 2.4 up to 2.6 now.

In other news, we run Plone trunk based on the eggified Zope2 trunk as of last week. Thanks to David Glick for helping with some tough Acquisition related changes. This means we have moved our entire stack towards eggs. As a side effect of that move the amount of code we need to pull in from Zope2 has gotten a lot less. Instead of 105 zope and zope.app packages we only pull in 65 of them anymore. I'll post some current lines of code graphs soon.

The most amazing news has been unveiled by Malthe yesterday. I have been lucky enough to be in Copenhagen and cheer Malthe while he got Plone trunk running entirely based on Chameleon. While he posted some impressive numbers for the Chameleon versus ZPT comparison for Plone trunk, I thought I put this into perspective and compare it with Plone 3.2.

We have a 50% to more than 100% speed increase. And yes we get 16 requests per second out of a completely uncached front-page.

I included the number for the HTML filter control panel as an example of a somewhat complex formlib-based form. This gives you a hint, at what we can expect from a framework like dexterity when it comes to edit screens.

This is just the beginning of what we can do and I expect more performance improvements to come. We have seen a lot of improvements made to the content creation, catalog and search performance during the last year which still need to make it into Plone trunk.

With the next performance sprint in Bristol coming up in December, the future of Plone looks bright to me. And all of this just happens while we redefine the content editor and administration UI for Plone at the same time.

The future of Plone happens now,
Hanno

Project Messerschmidt vs. NKOTB

2008-07-18T15:08:00.020+02:00

There have been rumors for some time, since Alexander Limi gave a most impressive lightning talk at last years Plone conference, about a templating language that sets out to put an end to our search for the perfect one.

It's major feature: Speed, speed and speed. Compared to any other templating system out there, by an order of magnitude faster.

Some time has passed and careful observers have spotted NKOTB at times. During last years performance sprint organized and sponsored by Headnet a critical mass of people met and tried various approaches to put this ingenious piece of art to some use.

Some time passed again and NKOTB got a proper name and a public presence, now to be known under the name spitfire. But when you put too many intelligent people into one room, you won't always get them to push in one direction. So Project Messerschmidt was born. An attempt to mock the same underlying approach of spitfire but make it vastly simpler.

So where are we today?

Spitfire is under active development, but to the current day is not integrated into any Zope environment except for some initial attempts. Spitfire only has support for Cheetah-like syntax today.

What about Project Messerschmidt?

It is nowadays known as z3c.pt and used by projects like Vudo or repoze.bfg. It has a full implementation of the TAL standard including the i18n namespace. While it does make some conscious choices about different default behavior than zope.pagetemplate it is virtually compatible.

The most notable differences are lacking support for METAL, the provider expression and to the current day no Five or Zope2 integration. The later two should be a matter of one focused day of work to finish.

But what about the main objective, speed? There is one established shoot-out test for templating languages called 'bigtable'. The objective is to render an HTML table of ten columns and 1000 rows. It is somewhat like the 'top speed of your car' number. It certainly isn't the only number you are interested in, when buying a car. But it's an easy to compare number, that has some objective nature to it.

So here are the numbers, judge for yourself:

Update: Thanks to a comment by Alexander, I made the chart a bit more impressive :)

Update 2: Since I've seen Donna experimenting with Mako, I added it to the graph.

Plone reloaded

2008-07-18T14:45:00.008+02:00

I've been silently working on a project called plone.reload which has been noticed by some.

After I got positive feedback from Martin Aspeli, Kai Diefenbach and Jon Stahl I felt it might be ready.

Danny Bloemendaal calls it:

The best improvement since the day (automatic) product refreshes died in Zope.

And my Co-Worker Denys Mishunov told me:

We should include it in every project we do.

So what is this project that gets people excited?

Its PyPi page is short and concise without any glamour:

Configuration and code reload without server restarts.

What does this mean for you? It brings back the good old refresh feature, we used to have for Products. Remember the refresh.txt files? Does it work for products alone? No!

This project tries to bring code reloading back into the modern world of Zope 3. Reload code located in browser views? Not a problem. What about that new browser:page you registered? Just reload all your ZCML without having to restart your server. Do you need to mark your packages in any way? No, you don't.

This project is most useful when you do user interface oriented work. You can work on file system based templates just fine in Zope debug mode, but so far modifying underlying browser views, viewlets or portlets always required you to restart. This time is over!

Is this useful as a general reload feature for all code imaginable? No, it's not.

For all application specific logic unit tests are still your best and fastest bet. For content types, register them once, get your GenericSetup profiles up and keep them as stupid as possible. Then plone.reload will help you on getting your user-interface in shape, without spending your time on getting coffee while Plone restarts all day long...

This project is heavily based on the work done by the fine Products.RefreshNG folks and wouldn't been possible without Guido himself contributing to the idea. So if you find it useful, remember the titans on whose shoulders we stand upon.

Hanno

Back to blogging...

2008-07-18T12:29:00.008+02:00

Dear readers,

I have kept silent for a while on this blog. That seems to be a natural tendency for anyone who is busy in the Plone community.

After leaving my home country for the Nordic wilderness five month ago, I have been somewhat overwhelmed with all the new impressions and having a new job. But for a couple of weeks, I have silently started to get back into the community. Besides starting to catch up on my bug wrangler duties, I also started to use my 10% time for less visible work.

Two of my recent projects deserve their own posts, so stay tuned to get an update on 'Codename Messerschmidt' and the project that got me the highest amount of personal 'thank you' mails yet in my open source career...

Cheers,
Hanno

Some more ideas to consider

2008-02-06T16:51:00.001+01:00

After all the awesome blog posts and mailing list posts about Plone's future, I don't have much left to say. I do agree with lots of the points mentioned and try not to reproduce those here.

I'd like to bring up some more minor ideas nonetheless.

1. Make add-ons first class citizens

The number of features that everyone wants to see in Plone is growing and even at this point I think Plone does more than we can guarantee to deliver good quality for.

In my opinion the only way out of the current 'everything should be in the core' meme is to embrace add-ons more and make them first class citizens of the Plone ecosystem. Alex Limi already posted some ideas on how to improve the installation and update story and I wholeheartedly agree with him. We need to provide the entrypoints and metadata for add-ons to integrate seamlessly with a Plone installation. I have some specific ideas on how to do this and plan to rework the add-ons control panel of Plone in the near future.

Obviously this goes hand in hand with all the planned efforts for turning the products section of plone.org into something more usable. The PyPi-like interface goes a long way of making it simpler for developers to upload their packages to plone.org and ratings and reviews give you another metric for judging a packages quality.

2. Plone should be lighter

In my opinion a default Plone installation at this point does too much and it is too hard to turn off some of the unneeded features for a particular site. What I would like to see is for Plone to come with two different flavors of default configuration. One that resembles what we have right now and which is created by default by the installers. I'd call this the 'evaluation version'. The other one I would like to see is the 'minimal version', which configures a Plone site with just the options you need to have for a working system. No content added, no portlet assignments, one simple one-state-workflow, no content rules, staging, versioning, openid or what not.

What I would like to do is to group all the configuration needed for one particular feature into an extension profile and present this as a feature control panel. So you can get versioning with a single click in the control panel or you can add it as a GenericSetup dependency in your policy package for your site and have the same effect.

This allows integrators to reuse the new base profile called minimal version as the base for their policy packages without hopefully having to delete anything from the created site. The minimal version obviously becomes the GenericSetup base profile. As a nice side effect we can get faster test runs by running most of the integration tests that require a full Plone site against the minimal version.

3. More fine grained configuration upgrade procedure

Closely related to the above mentioned, I think we need a better upgrade story for the configuration of both Plone itself as well as add-ons. We have the underlying technology with GenericSetup upgrade profiles available. What remains is to expose this in the user interface, so you know if an installed add-on needs an upgrade.

For Plone itself we need to distinguish between essential upgrade steps and non-essential ones and give the user the choice to apply the non-essential ones according to their needs. For example creating the site mananger in the Plone 3 migration is essential, reordering the CSS files in the registry is something that should be left to the user to decide with a small description of what will happen and why we do it.

4. Optional code

In the same way I'd like to see the Plone configuration to become lighter, we could think about using the same principle for code. Everyone who has ever used a Windows installer for a more complex product, knows their ability to choose from a variety of options during install time. So you can either install only the base product or install some options with it. Later you can go back to the installer and add or remove more of these options. I think this should be doable with a set of more independent packages which we are getting better at and an installer based on buildout.

Why should we think about this at all? Less code is less code to run, to eat your memory and CPU cycles, to contain bugs and security holes and to look at when trying to figure out how to extend the system.

However this has a big risk of resulting in more non-reproducible bugs and more integration errors between different parts, so it needs to be considered carefully. Plone the product as downloaded by the casual user should still come with all the bells and whistles.

5. Unicode!

This is a personal pet peeve of mine. We are already in a good starting position to master the move from a system using encoded strings to a fully Unicode aware application. There are some major next steps we need to take to get there eventually, so one day we can theoretically run on Python 3. Obviously this is not something we need to do in the next year but Python 3 lingers on the horizon and one of the certain things about it, is that we won't be able to use it unless we fix our Unicode story.

For Plone 4.0 I would like to do some changes to Archetypes and PortalTransforms to get better Unicode support. Still on the todo list are external data sources like LDAP, SQL and quite some configuration stored in the ZODB like old Zope2 properties. Maybe more for 5.0 we will need to think about using a Unicode aware publisher, which goes nicely with ditching the Zope2 publisher and going for a repoze based setup. There is more work here and a fine balance to keep between introducing backwards incompatible changes and moving forward.

Personal note: I'm on the move...

2008-02-02T18:57:00.002+01:00

There are some big changes coming up in my personal and professional life which I wanted to share with you.

For a few days I'm officially homeless now. I handed over my apartment to my landlord last week early in the morning. In February I'm staying with friends and hijack some couches. Thanks a lot to my brother for hosting me in the first week!

While I still have to work for one more week in late February for my old employer - a small hospital in the eastern part of Hamburg - I'm already looking forward to my new job opportunity.

But first I'll attend the Plone Strategic Planning summit in San Francisco to help shape the future of my favorite Open Source CMS. I think this is going to be a great opportunity for Plone and hopefully we will see lots of interesting stuff happening on and especially after the Summit.

Finally end of February I will move to a beautiful little sea-side town at the Fjord of Oslo in cold Norway. Amongst other things this little town is world famous for one of the coolest Plone companies called Jarn, home to many dedicated and extremely skilled Plone addicts. It is with great pleasure that I'll have the opportunity to join them starting in March.

So prepare yourself what happens, when I don't only work on this project in my spare hours but can share my experience on a daily basis with lots of other amazing long term core contributers. It is truly an amazing time for me and Plone these days :)

Fewer Zope Server restarts

2008-01-25T01:20:00.001+01:00

In my ongoing attempt to make filesystem based Zope development faster, I had a look at the nice but somewhat scary RefreshNG product.

One of the nice things which I thought about myself is its ability to reload all ZCML and reconfigure the global site manager without a server restart.

I took that idea and made it into its own package plone.reload which has a first release at http://pypi.python.org/pypi/plone.reload/0.1. All you need to do is to call http://server:port/@@zcml_reload while being logged-in as a Manager. You can do that as often as you like :)

Casual testing tells me that it should work in a standard Zope2 or Plone environment, though I only tested it against Zope 2.11 / Plone 4.0.

Feedback and bug reports welcome ;)

Update:

Sorry for the premature announcement. It seems there is some caching going on, which I need to work around. Probably this happens thanks to the new local site manager... Right now changing global GenericSetup steps works fine, but not much more. I'll post an update once I have tested this myself more thoroughly.

Update 2:

I got a new 0.3 release out, which should fix the caching issues. Sadly it does so by invalidating all the ZODB caches, which means you will wait a bit more after hitting the Force-Reload button in your browser. I found that just refreshing the site gave an acceptable wait-time, though.

MailHost redux

2007-12-27T00:59:00.000+01:00

For some time we had quite an interesting and diverse landscape of mail host solutions for Zope and Plone.

With the upcoming release of Zope 2.11 we finally have a nice and feature rich default MailHost implementation in Zope 2 based on the zope.sendmail package. Major kudos to Andreas Jung for doing that work.

Today I have spent some hours to fix the standard mail host used in Plone provided by the SecureMailHost product to become nothing more than a tiny backwards compatibility layer on top of Zope 2's standard MailHost. Hopefully this work will go into Plone 4.0.

At that point we'll have a transaction-safe default mail delivery which you can turn into an asynchronous delivery by nothing more than a switch in the ZMI. And obviously it has SSL/TLS support as well :)

If you are interested in testing this, please get the code from http://svn.plone.org/svn/collective/SecureMailHost/trunk.

Feedback welcome,
Hanno

Confession of a civilian

2007-10-21T10:26:00.000+02:00

Here it is:

I do TTW-customization on almost all of my sites!

Most of them are simple sites that use no custom logic and have only some templates and CSS files customized. I work with people on these projects that don't know Python or anything of the Zope/Plone stack. Explaining them the portal_skins ZMI is easy enough, explaining them shell access, server restarts and theme products is just not productive. And restarting the server for one change would take down the four other simple sites hosted on the same server as well - that's just not a good idea.

One thing I'm not going to do however is doing any kind of TTW development. My simple rule is to never use Python scripts in the customs folder, just customized page templates. For Python code and any kind of logic you need to know Python and some parts of the stack anyways, so the step to file system based code is a lot smaller and the advantages of proper development tools gets bigger.

The only thing I personally have found to be too cumbersome is to do Plone version upgrades with TTW templates. I need good diff tools to compare the changes made to the original template and my customized versions (say main_template), so I usually end up dumping the whole templates to a file system theme product before the upgrade and do the diffing and adjusting work there.

Performance sprint warmup

2007-10-16T23:22:00.000+02:00

While the Copenhagen Plone performance sprint is still two weeks away, I thought I spent some time on profiling during the Naples sprint at this years Plone conference as well.

I concentrated on profiling Zope startup time when Plone is installed in it. While this doesn't matter that much in production environments it is of quite some importance for us poor developers.

On my one year old MacBook Pro Zope would start in about 16 seconds at first. Using Marius Gedminas great profilehooks.py I identified four big time consumers.

1. Importing modules

Not much can be done about that, though some of the CMFPlone migrations seem to be a good candidate to be excluded from initial loading and being defered to a point when they are actually needed. A couple of hundred imports could be safed here I guess.

2. Compiling and registering translation files

In a Plone environment so far all about 620 po files where read and parsed two to three times. I needed to adjust the mo file compilation code in my python-gettext package a bit and hacked PlacelessTranslationService to use lazy message catalogs instead. Those won't parse all the files on Zope startup but instead once they are first accessed. A typical site will probably never use all the languages and parsing and keeping those in memory was a bad choice.

After this was done I was down at 11 seconds. About a 30% speed increase.

3. Reading and compiling page templates

It turned out that right now we have around 140 purely disk based page template files for various ZMI screens. Until now all of them were always read, parsed and compiled on Zope startup; most of them are never used. So I added some lazy loading code into Zope's PageTemplateFile, made it available as an option and enabled it from CMFPlone. As the CMFPlone code is read after some of the templates are already loaded, we can only skip about 110 of them in this way. I'll add the whole thing as a zope.conf option, which will be parsed early enough in the process to get rid of the remaining ones.

Measuring the startup time again, I'm now down to 8 seconds, another 30%.

4. Parsing ZCML files

Parsing hundreds of ZCML files is currently the remaining big block for startup time. AFAICS this uses expat right now. It might be interesting to experiment with different XML parsers like lxml to see if some performance gain can be achieved here. I'll leave the exercise to someone else (Florian had shown interest in it...).

Conclusion

By spending about half a day of work on the topic I could speed up Zope startup time by an overall 50%. Not too shabby I think :) Lets see if there are some weak spots while loading the Plone site TTW the first time as well...

The downside to this all is that it is only available on Plone and Zope trunk. Maybe we can backport some of these improvements to the Plone 3.x line.

Having fun with setuptools

2006-09-22T17:41:00.000+02:00

As already noted yesterday the CheeseShop is getting interesting for the common Zope folks. So I took the opportunity to play around a bit more with i18ndude and setuptools.

The result is a new i18ndude 2.1 release that is available as an egg from the CheeseShop. All you need to do, is to run easy_install i18ndude to get the latest version.

But while reading the setuptools documentation I found yet another useful feature which is called automatic script creation. Now I have developed on the MS Windows platform for quite some years and have always hated to have to write wrapper scripts around Python scripts myself in order to make them callable.

Now thanks to setuptools this step is automatically taken care of and an i18ndude.exe is created automatically if you easy_install i18ndude :)

But I was even more adventurous and tried out the integrated distutils support for building nice installers and found that I only had to type in a single command to get a nice Windows installer in addition to the egg build and have them automatically uploaded to the CheeseShop:

python setup.py register bdist_egg bdist_wininst upload -s

While you have to be on a Windows system to be able to create that installer, this is not harder than doing an tar.gz release.

So make sure to grab the new experimental installer for i18ndude 2.1.1 and report back any issues you find in the bug tracker.

If this turns out to be as easy as it looks like right now, you might find new installers for the various plone.* packages in the future as well. Now a way to register those releases on plone.org from the command line as well would be nice ;)

Cheeseshop gets zopified

2006-09-21T14:40:00.000+02:00

After reading Phillips blog post about the new framework categories for Zope 2 and 3 in the Cheeseshop I got to work and updated the i18ndude registration as well as pre-registered the plone.i18n and plone.app.i18n packages, so right now we have four packages registered for the two frameworks which are all around internationalization :)

For a nice short explanation how-to register a package in the Cheeseshop you should read Daniels blog post on the topic.

For those of you that have already upgraded to Subversion 1.4 you might find that setuptools does not work anymore.

If you see a AttributeError: 'NoneType' object has no attribute 'group' you want to try these steps (stolen from a post of Phillip J. Eby):

easy_install -eb. setuptools==dev06

This will create a setuptools subdirectory in the current directory. Then do:


cd setuptools
python2.4 setup.py install

If you haven't yet upgraded to Subversion 1.4 you can spare yourself this hassle and upgrade setuptools first ;)

DZUG conference sprint 2006

2006-09-17T15:40:00.000+02:00

Last week I have been in St. Augustin at a three days pre-conference sprint before the annual DZUG e.V. conference which extended into the two conference days as well. We had a room in the University of Applied Sciences Bonn-Rhein-Sieg where the conference took place, but thanks to the excellent organization of Nina Schütz all of us five sprinters shared an apartment in Cologne, where we soon discovered a free wireless network that allowed us to work all day long. In the evenings we familiarized ourselves with the local flavor of beer called Kölsch in quite some nice pubs and bars as well ;)

While I originally planned to polish up the last bits of my PLIP's for Plone 3.0 (CMF 2.1 and some i18n features) I had an idea the night before the sprint which I worked on instead.

If you have looked into Plone 2.5 you will have noticed the introduction of a new product called GenericSetup that is used to configure a Plone site. If you are not yet familiar with it, you should really learn about it by reading Rob Miller's excellent tutorial that is available on plone.org.

GenericSetup supports both setting up a site (a persistent Plone portal) from scratch through something called base profiles but also supports a concept of extension profiles that can be used to extend the configuration of a site after it has been created.

Now these extension profiles can be used for installation of add-on products that need to register new content types, workflows, actions, create tools or anything else an add-on product would like to do. But right now there is one essential thing missing for this kind of additional configuration: Uninstall support.

Right now we use CMFQuickInstallerTool to provide us with an easy way of installing add-ons and it has integrated support for recording the changes each individual product makes at install time, on which it's uninstall feature is based on. While this recording feature is far from perfect it handles at least 90% of the use-cases and has proven to be working quite well.

Now the most obvious thing is: Why don't we teach CMFQuickInstallerTool how to install add-on products based on extension profiles rather than through an external method?

After skimming through the code I realized that this is far easier than I would have imagined. So I spent the most part of the sprint working on this and had it fully functional at the end, including some nice test coverage. Right now you can find the code on a branch in the collective. It is tested with Plone 2.5 and Zope 2.9 only but should work for any supported combination of Plone 2.5 and up.

Please try it out and report back any issues you find. You just need to replace the CMFQuickInstallerTool with my branch, there are no additional dependencies. If everything goes as planned we might have this included in Plone 3.0 and eventually even in Plone 2.5.2+

As a personal note I might add that I begin to fall in love with sprint driven development. I think I begin to understand those developers out there that would like to see their life being a never-ending-sprint ;)

Sprintingly yours,
Hanno

The summer is over

2006-08-22T22:25:00.000+02:00

Dependent on where you are located on this planet, the summer season may still go on for some weeks or is yet some month away, but at least Google's Summer of Code 2006 is over.

I had the privilege to be one of three students, who where allowed to work on projects for the Plone Foundation. A big thanks to Google for making this happen and to all the people in the Plone community for supporting the idea and me specifically. A special thanks goes to my mentor Alec Mitchell, who has given me all the support I wanted.

Now it is time to look back at the last few months and see how much progress I made and how many of my goals I actually reached. For those of you not familiar with my project, the short form was: Reimplement as much as possible of the existing i18n/l10n infrastructure found in the core Plone project based on Zope3.

To be honest I did not achieve as many of my goals than I had hoped for myself. But the good news is that while some of the work is still to be done, on the one hand I will continue to work on these and on the other hand that quite a lot of my time has gone into creating or refining more infrastructure and process for developing with Zope3 in the context of Plone, which will hopefully benefit all of you.

But here are some concrete things I have been working on:

1. Finish the move to use Zope3 Messages to internationalize Python code.

This has been started in Plone 2.5 as detailed in PLIP 108 and now has been extended to the world of Archetypes. On the current Plone 3.0 bundle you will find that both Archetypes and all of ATContentTypes use Messages now to mark any textual data which needs translation as part of the UI. For an example on how this looks like you can view the event schema definition in ATCT. The two most prominent benefits of this are the ability to automatically extract all these messages from your product to build the Gettext translation templates and the ability to define these messages in your products own translation domain. Both of these should make it considerably easier to internationalize your products resulting in more products being available in multiple languages.

2. Sanitizing the lists of available languages and countries

So far we had three different implementations in AT, Plone and PloneLanguageTool used for various language lists and they where heavily out of sync. I reimplemented these as Zope3 utilities as part of my plone.i18n product and changed all the various places to use these centralized lists instead. Of course there are some different utilities for meta data languages or content languages now and you have the ability to customize these lists easily by adding local persistent variants of these utilities as implemented in plone.app.i18n. If you don't understand what I mean by that: Don't worry - there'll be a nice simple UI for you, which just allows you to select the languages you want to be available. After all this is still Plone - nice defaults and ease of use ;)

3. New reusable and extensible text normalization

When creating new items in Plone the technical id which ends up being used in the URL is usually autogenerated from the title of your object. As the URL is currently still limited to pure ASCII characters, you will have to normalize non-ASCII chars in some way to an ASCII equivalent for all languages except English. So far there was a hardcoded list of character mappings in CMFPlone itself, which you had no chance to enhance or customize easily. Now the whole process is based on Zope3 utilities as well allowing for easy customization and re-use. You can find the code in the plone.i18n package as well. As an added bonus the title normalization is now language dependent, so you can have meaningful mappings regarding your language and the normalization process is not slowed done by having to look at mappings for the whole Unicode character range at once.

4. Making our nice country flags available as Zope3 resources

If you have installed PloneLanguageTool so far or have a multi-language site based on LinguaPlone you will have noticed some nice icons that use country flags to symbolize the languages you want to read content in or translate to. These are now part of plone.i18n as well and thus available in any Zope3 based project rather than being stuck in a Plone specific product.

5. Lots of bug fixes...

Developing new things while there are still critical bugs for the most recent version didn't make much sense to me, so I tried to kill all serious internationalization related bugs for Plone 2.5 and some more as I was on it ;)

And now some notes on the infrastructure side

6. New ZopeSkel templates to quickstart your Plone product

As the Plone community is currently adopting Zope3 development practices, one of the new things you have to learn about, is writing your products not as Zope Products but as regular Python packages. These days this involves providing some metadata to enable them to be packaged as eggs. As this is quite some boilerplate to remember Daniel Nouri has created a new product called ZopeSkel based on Ian Bickings PasteScript to generate product skeletons. After some extensive discussions I have added some nice templates to quickstart products in the plone and plone.app namespaces.

7. New GenericSetup support for local components

As I already blogged about some days ago, followed up by Martin Aspeli's blog entry about local adapters and their use in plone.portlets I have some fresh news from this front. The whole export/import handlers are now part of their own project called GSLocalAddons and I have extended the handlers quite a bit. They are now able to create registrations for utilities and adapters and as a special mode, they can even create utility registration from your already existing content in the ZODB. So making your CMF tools available as utilities shouldn't be any harder as to write two lines of XML now.

8. CMF 2.1 support for Plone 3.0

Only partially related to my SoC project has been my work on CMF 2.1 for Plone. At least this adds one of the last missing pieces to the i18n-puzzle by using Zope3 Messages for things like action titles and some more CMF specific things. I had stopped working on this to concentrate my efforts on the more vital parts of my SoC project, but now that this is over, I can start working on it again.

What remains to be done?

As you can read in my project description there are still some features currently available in PloneLanguageTool and PlacelessTranslationService not yet available in Zope3. After I cleaned up my current projects, I will start working on these again, so we might be able to retire both of these products in Plone 3.5 in the next year. So stay tuned, as the journey to a better world of internationalization isn't over yet :)

We are getting local...

2006-08-16T09:56:00.000+02:00

As you may have noticed Plone developers are currently pushing hard to use new Zope3 technologies to bring sanity again to the world of Plone code.

So far we have been using interfaces and browser views in Plone 2.5, but for Plone 3.0 we are going to rely on Zope 2.10 which has a far easier local component story. So you can expect to see not only global utilities and adapters but also their local counterparts.

But how do we actually create those in our persistent CMF-based portals? Well, why not do the obvious and use our existing configuration setup called GenericSetup to create them for us?

Imagine a simple file called sitemanager.xml with the following code:

< ?xml version="1.0"? >
< utilities >
< utility
 interface="plone.app.i18n.locales.interfaces.ICountries"
 class="plone.app.i18n.locales.countries.Countries"
 />
< /utilities >

Wouldn't it be nice if you could use something like it? Well you can, I made the first prototype as part of my Google Summer of Code project and it's working just fine.

You can take a sneak peak at the code at: http://dev.plone.org/collective/browser/GSLocalAddons/trunk/exportimport/sitemanager.py. Once this has a decent amount of tests, I would consider this a valuable addition to GenericSetup itself.

P.S. I should note that I stole some ideas from Lennart Regebro's work on CPSSharedCalendar, thx mate ;)

Switching to Mac OS X

2006-07-22T16:54:00.000+02:00

So far I have been quite a satisfied user of Windows XP, having to deal with the whole Windows platform as part of my day job anyways. Put as of late I found myself in a position where I was constantly hitting some edges I did not like about it. For some time I was able to get myself going by using more and more helpful Unix tools, which were ported natively to the Windows platform by some friendly folks.

But in the last couple of weeks I grew even more impatient especially about the visual look and responsiveness of the graphical user interface. My first reaction has been to get myself a copy of the new Windows Vista and evaluate if I would eventually be able to hold myself back and wait on the next Windows version that promised to get a new breath-taking look and feel. But after some days of extensive testing I had to finally realize that Windows Vista would merely be able to catch up with the graphical features of all the other OS'es available right now, but would by no means be able to show any innovation or even take the lead for some month. Considering that I would have to wait for the next Windows version after Vista probably for four years or more, it just wasn't an option.

After looking out for contenders I got myself copies of Ubuntu Linux and for some old habit a copy of OpenSuse and tried both KDE and Gnome again. While both gave me a lot of flexibility and have improved a lot since I last looked at them a couple of years ago I still wasn't satisfied. My main problem with them was that I would have to spent weeks to configure them in a way that I would find them usable and fun to work with. This showed me once more why I like Plone so much - sensible defaults and not too many options on the standard config screen, while still retaining the flexibility of being able to customize every part of it.

Now as I needed to get myself a notebook I did the one obvious thing after all. Being a core Plone developer I had seen the majority of friendly folks at conferences and sprints using some hard- and software of a certain fruit company. Now thanks to my current Google Summer of Code project I had the money to buy myself one of these fruit boxes myself :)

As a nice add-on bonus I got myself a notebook with a native English keyboard so I would not have to dislocate my fingers anymore while typing some code which included some of those rare characters like brackets, which are at insane positions at the standard German keyboard.

Thanks to some nice how-to from philiKON I was able to get those few umlauts I needed on occasion working with ease. For all those lazy folks out there you can get my keylayout file for U.S. (German) and a nice icon for it. I hope you like the icon Philipp ;)

Now I have to get back to coding as the CMF 2.1 integration bundle for Plone 3 still needs some love...

ArchGenXML finally supports i18ndude 2.0

2005-11-19T18:53:00.000+01:00

Luckily I have meet jensens today on IRC and after some discussion we finally got ArchGenXML to support the new i18ndude 2.0.

So be sure to grab the latest version of ArchGenXML from SVN trunk or wait on the next official 1.4 release. As always you can get the latest i18ndude release from http://plone.org/products/i18ndude/.

Please note that i18ndude 1.0 won't work anymore, so be sure to upgrade simultaneously ;)

What am I doing?

2005-11-16T15:55:00.000+01:00

I'm an active core developer of the Plone™ Open Source Content Management System. My main focus is currently on internationalization. In this role I'm maintaining the i18ndude, i18ntestcase and PlacelessTranslationService products and co-leading the i18n team responsible for the translation of Plone.

Currently I'm working hard on getting Plone to use Zope3's i18n features like MessageID's. See my PLIP 108 and PLIP 111 for details.

Some things I have on my todo list:

Write a how-to on internationalization of your AT-based product using the capabilities of Plone 2.2 (MessageID's)

Rewrite the localization part of the translation_service of Plone using Zope3's locales support, removing the date/time handling from the po files

Deprecate the current i18n approach of AT and replace it with a MessageID-based approach. Get ArchGenXML to support it (and the new i18ndude)

Rewrite PloneLanguageTool using Zope3 technologies and incorporate the language negotiating which is part of PTS into it

Get rid of PTS as fast as we can ;)

Support launchpad/rosetta to translate Plone or write a PloneI18NCenter product which does some of this stuff

Implement support for number dependend messages in Zope3

Make the world a better place...

If you want to help on any of this please let me know. Ideas on what has to be improved in i18n/l10n are most welcome.