Ticket #10776 (closed PLIP: fixed)

Opened 5 years ago

Last modified 4 years ago

Update to Zope 2.13

Reported by: hannosch Owned by: hannosch
Priority: minor Milestone: 4.1
Component: General Version:
Keywords: Cc: plip-advisories@…

Description (last modified by hannosch) (diff)

Proposer: Hanno Schlichting
Seconder: None as yet

Motivation

Zope 2.13 is the next stable release of Zope 2. It comes with a new and faster version of the ZODB, support for Python 2.7, built-in WSGI support, updated Zope Toolkit packages and a much faster ZCatalog. It also makes all of zope.app.* optional and has a number of internal cleanups. You can read more about it at  http://docs.zope.org/zope2/releases/2.13/WHATSNEW.html

In the past Plone has had a very conservative update policy towards upgrading Zope 2 releases, which has frequently led to a mismatch between the Zope distributions Plone shipped with and those developers wanted to use. This is again the case for Plone 4.0 and developers relying on for example z3c.form.

Assumptions

With Zope 2.12 we have gotten a fully eggified Zope release including all Zope Toolkit distributions. This has made it much easier to selectively upgrade individual distributions. In practice this currently does not work. In the refactoring of Zope 3 into the Zope Toolkit there's been a massive amount of interdependent changes to all distributions and a bulk of new distributions. Upgrading individual distributions without getting the whole has become impossible, without risking subtle and sometimes critical breakages in unexpected places. There's many distributions were even using new supposedly bug fix releases isn't possible [see  http://download.zope.org/Zope2/index/2.12.11/versions.cfg for a list].

Zope 2.13 solves this issue by updating to a real and stable Zope Toolkit release. This release is also used by both BlueBream and Grok and thus has a higher chance of being kept stable and maintained over a longer period. Sharing the same version set with Grok also makes it possible to use Grok technologies inside Zope 2 / Plone in a sane way. z3c.form is another example of a library that needs more recent versions of Zope Toolkit libraries than those provided in Zope 2.12.

While Zope 2.13 does a leap forward in the provided Zope libraries, virtually all of them are 100% backwards compatible in their API's. A demonstration of this is that both Plone 4.0 and CMF 2.2 work with Zope 2.13 without any more modifications (some modifications where required in the test runner, which interacts with many internals).

The classic Zope 2 code itself is being factored out into multiple distributions in the 2.13 release. Going forward this trend will continue. This will allow us to both quickly pull in new bug fix releases of things like DateTime or Products.MailHost and with Zope 2.14 and going forward to drop unused parts of Zope 2 from Plone. For example ExternalMethods and PythonScripts will be entirely optional in 2.14. five.formlib integration and Products.ZSQLMethods are already optional in 2.13.

While Zope 2.13 provides native WSGI functionality, it is outside of this PLIP to make use of this in any way. There's a large number of open questions around WSGI and the changes it requires to instance creation and setup, which warrant their own PLIP. A good number of those will also have an impact on the Zope 2 codebase itself, which is going to be already past a beta release by the time this PLIP gets started and thus will have to wait for Zope 2.14. I expect to see more community experimentation being done with WSGI once the capability is there. We might see a PLIP for Plone 4.2 to see those experimentations being solidified into good practice, standards and documentation.

Proposal & Implementation

A PLIP config is available at  https://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.1/plips/plip10776-zope213.cfg.

While Zope 2.13 itself is still changing, there might be more minor changes required to the Plone codebase. Those are to be provided in PLIP branches, when they aren't backwards compatible and truly minor in scope. It will also help to review all remaining use of zope.app packages in Plone itself and use their zope.* replacements instead. In order to keep backwards compatibility with add-ons, Plone should extend the zopeapp.cfg configuration file, which provides version pins for all zope.app.* distributions.

Deliverables

See Proposal above.

Risks

While the changes to the Zope Toolkit packages should be all backwards compatible, there's a chance that some of them might cause problems. Generally most of the changes to zope.app.* packages where in code and configuration neither used nor usable inside Zope 2 and Plone. Naturally these can cause no trouble.

Some of the changes in Zope 2 itself might cause problems with tests and their interactions with internals. As Plone 4.0 already broke many of these without fixing them, I consider this to be acceptable. I'm assuming that we will see a PLIP for plone.app.testing to make it into 4.1, which should provide a clean new base for tests. PloneTestCase based tests and the various sorts of layer setups we have seen, are just too broken to be fixed in a reasonable way.

Zope 2.13 is currently in late alpha with a beta scheduled for early September. The code is already stable and could be released as a final almost at any point. It's therefor unlikely that a delayed Zope 2.13 release could delay a 4.1 release.

Participants

Hanno Schlichting <hannosch>, Zope 2 release manager

Progress

The PLIP was merged on November 17, 2010.

Change History

comment:1 Changed 5 years ago by hannosch

  • Description modified (diff)

comment:2 Changed 5 years ago by hannosch

  • Description modified (diff)

comment:3 Changed 5 years ago by hannosch

  • Description modified (diff)

comment:4 Changed 5 years ago by hannosch

  • Status changed from new to assigned

comment:5 Changed 5 years ago by hannosch

  • Description modified (diff)

comment:6 follow-up: ↓ 7 Changed 5 years ago by ldr

Would it make sense to merge this now so other PLIP implementors use it as their base?

comment:7 in reply to: ↑ 6 Changed 5 years ago by hannosch

Replying to ldr:

Would it make sense to merge this now so other PLIP implementors use it as their base?

We could certainly do that. I think I got all the tests passing again, so it should be stable enough.

comment:8 Changed 5 years ago by esteele

Your PLIP has been accepted for consideration for Plone 4.1.

Framework Team voting on this PLIP was: Alec +1 Craig +1 Elizabeth +1 Laurence +1 Martijn +1 Matthew +1 Rob +1 Ross +1

The initial implementation deadline for your PLIP is October 1st, 2010. The Framework Team would like this PLIP to be completed as soon as possible since it's a change to the underlying infrastructure. Announce its readiness here once your implementation is ready for review.

comment:9 Changed 5 years ago by hannosch

  • Description modified (diff)

comment:10 follow-up: ↓ 11 Changed 5 years ago by jonstahl

Question: what if any impact will this have on existing add-on products or customizations that work with 4.0? (If the answer is other than "none" there might need to be some docs written for the upgrade guide. ;-)

comment:11 in reply to: ↑ 10 Changed 5 years ago by hannosch

Replying to jonstahl:

Question: what if any impact will this have on existing add-on products or customizations that work with 4.0? (If the answer is other than "none" there might need to be some docs written for the upgrade guide. ;-)

Hopefully there is next to no impact. Some very advanced add-ons doing tricky Zope Toolkit stuff could be affected, but other than testing them I don't know how to tell. As I said in the PLIP text, the changes are almost all in internals of the Zope Toolkit and have little to no impact on existing APIs.

comment:12 Changed 5 years ago by cah190

  • Cc plip-advisories@… added

comment:13 Changed 5 years ago by cah190

(In [40358]) Review for PLIP 10776. Refs #10776.

comment:14 Changed 5 years ago by eleddy

(In [40359]) zope 2.13 plip review. refs #10776

comment:15 follow-up: ↓ 19 Changed 5 years ago by davisagli

It concerns me a bit that the update to Zope 2.13 appears to require an update to CMFCore 2.3 (currently unreleased), which deprecates some stuff that was available in 2.2. That doesn't seem good for a Plone .x release.

comment:16 Changed 5 years ago by alecm

I also saw issues when attempting to downgrade my Zope. Such problems are to be expected, but it will be tremendously important to document this fact and provide a stronger than usual recommendation that Data.fs needs to be backed up before upgrading.

Regarding CMFCore deprecations: my understanding is that removal of previously deprecated features is generally acceptable in a .x release, though not in a .x.x release, and that new deprecations are generally acceptable at any release level as long as the feature is not prematurely removed. We'll certainly need to discuss the impact of these CMFCore changes though.

David, is there a summary of the deprecations and backwards incompatible changes in CMFCore somewhere?

comment:17 Changed 5 years ago by davisagli

Alec, thanks for the reminders regarding our deprecation policy. I took a closer look at the changes on CMFCore trunk and none of them are a concern, given that policy (or even particularly a concern without it).

comment:18 Changed 5 years ago by esteele

FWIW, I'm only seeing test failures in plone.app.blob and wicked. The GenericSetup failures have been cleared in 4.0 (which 4.1 currently uses as a versions base).

comment:19 in reply to: ↑ 15 Changed 5 years ago by hannosch

Replying to davisagli:

It concerns me a bit that the update to Zope 2.13 appears to require an update to CMFCore 2.3 (currently unreleased), which deprecates some stuff that was available in 2.2. That doesn't seem good for a Plone .x release.

What makes you think it requires CMFCore trunk?

The buildout has an autocheckout for CMFCore, which uses the 2.2 branch as per the sources.cfg in the 4.0 coredev buildout. That's simply because we require the yet unreleased CMFCore 2.2.3 (scheduled for a release after the bugday tomorrow).

comment:20 Changed 5 years ago by hannosch

Issues that came up: The DateTime pin to an older version is not specific to this PLIP. It's required on Plone 4.0.x as well to deal with GenericSetup / DateTime / CMFEditions issues.

Regarding the missing image/file types, I have a feeling that David might have found and fixed the cause for that yesterday.

I'll try to update and test the buildout again this week and sent you a note once I fixed the issues.

comment:21 Changed 4 years ago by hannosch

With the current version of the plip config all tests pass (requires ZODB > 3.10.0).

comment:22 Changed 4 years ago by hannosch

Zope 2.13.0 final is released. I think all issues have been addressed and this should be ready for merging.

comment:23 Changed 4 years ago by hannosch

  • Status changed from assigned to closed
  • Resolution set to fixed
  • Description modified (diff)

comment:24 Changed 3 years ago by davisagli

  • Component changed from Infrastructure to General
Note: See TracTickets for help on using tickets.