Ticket #13283 (confirmed PLIP)

Opened 18 months ago

Last modified 10 months ago

Merge some plone.app.* packages into the Products.CMFPlone distribution

Reported by: davisagli Owned by: davisagli
Priority: major Milestone: 5.0
Component: Backend (Python) Version:
Keywords: packaging Cc:

Description (last modified by witsch) (diff)

Proposer: David Glick
Seconder: Andi Zeidler

Motivation

Plone's core currently includes 38 packages in the plone.app namespace. In many cases this separation is unnecessary and slows development of Plone core. In some cases there are good reasons for the separate packages, but we have identified 18 packages that have little benefit to being packaged separately (they are interdependent and always required in Plone) and just cause unnecessary work for the release manager. We propose merging their repositories into the Products.CMFPlone repository (including history) and always releasing these packages as part of the Products.CMFPlone distribution.

Assumptions

  • Package namespaces must not change.
  • Package history must be retained.
  • There are other dependencies of CMFPlone for which we should maybe do this, but we are focusing only on the plone.app.* ones for now.

Proposal & Implementation

Repositories can be merged using the subtree merge strategy described at  http://nuclearsquid.com/writings/subtree-merging-and-you/

This merges both the history of development (appearing as a separate git branch + a merge) and the distribution of the packages (i.e. they are located in the same egg).

Git submodules were considered, but would still be a pain for both the release manager and other users of the coredev buildout.

Deliverables

We plan to merge the following packages into Products.CMFPlone:

  • plone.app.content
  • plone.app.contentlisting
  • plone.app.contentmenu
  • plone.app.contentrules
  • plone.app.controlpanel
  • plone.app.customerize
  • plone.app.i18n
  • plone.app.layout
  • plone.app.portlets
  • plone.app.querystring
  • plone.app.registry
  • plone.app.workflow
  • plone.app.vocabularies
  • plone.app.viewletmanager
  • plone.app.search
  • plone.app.uuid
  • plone.app.users
  • plone.app.z3cform

These plone.app packages are Archetypes-specific and not included in the proposal:

  • plone.app.folder
  • plone.app.blob
  • plone.app.collection
  • plone.app.imaging

These plone.app packages are optional features and not included in the proposal:

  • plone.app.caching
  • plone.app.dexterity
  • plone.app.discussion
  • plone.app.form (at least, hopefully it is optional soon)
  • plone.app.iterate
  • plone.app.linkintegrity
  • plone.app.locales
  • plone.app.textfield
  • plone.app.theming
  • plone.app.openid
  • plone.app.redirector
  • plone.app.upgrade

These packages are javascript that sometimes needs to be updated to newer versions, so are not included in the proposal:

  • plone.app.jquery
  • plone.app.jquerytools

Risks

Sometimes individual packages need to be upgraded beyond the "official" version for a release of Plone in order to fix a bug. This can still be done in an emergency, by releasing the package that needs to be upgraded in its own egg, and making sure it gets positioned in sys.path to take precedence over the package in the CMFPlone egg. But this should be necessary less often, because the simplified Plone distribution will be easier to release more frequently. Also, we've chosen packages for this proposal that are fairly stable.

Completing this PLIP makes it a bit harder to port fixes between master and the maintenance branch of the packages in question. It is still possible by treating the old package as a special remote and doing a subtree merge, but it is a bit more complicated.

In some cases add-ons will declare dependencies on distributions for these packages that are no longer necessary, since the code is in the Products.CMFPlone distribution. This can be mitigated by making new empty releases of the distributions that are being deprecated, so that it is harmless if they are accidentally included.

Participants

David Glick & Andi Zeidler.

Progress

So far we have conducted some experimental merges which can be checked out using dependency-merges.cfg in 4.3 coredev.

Change History

comment:1 Changed 18 months ago by witsch

  • Description modified (diff)

fix buildout config name

comment:2 Changed 18 months ago by ldr

Generally +many with the proviso that I'd favour vertical integration over horizontal integration, something like:

  • Core page rendering and content management UI.
  • User management - I'd love a lot of this stuff to be optional in the future - many sites authenticate with apache and have no concept of user registration.
  • Portlets - merge plone.portlets and plone.app.portlets together.
  • Configuration registry - merge plone.registry and plone.app.registry together.
  • z3c.form related stuff - plone.app.z3cform feels like it should end up merged with plone.z3cform / plone.autoform / plone.supermodel at some point in the future
  • archetypes suff
  • dexterity stuff
  • plone.app.contentrules feels like a separate functional thing that would ideally be optional.

I'm not quite sure what to think of these two yet:

  • plone.app.querystring feels like it should end somewhere else but not sure where.
  • plone.customerize is really cmf level thing.

comment:3 follow-ups: ↓ 5 ↓ 8 Changed 18 months ago by hannosch

+1. I'd also merge some Products into CMFPlone, for example:

  • CMFQuickInstallerTool
  • PasswordResetTool
  • PlacelessTranslationService
  • PloneLanguageTool
  • PlonePAS
  • statusmessages

comment:4 follow-up: ↓ 13 Changed 18 months ago by gotcha

Can the following sentence be expanded ? "In many cases this separation is unnecessary and slows development of Plone core."

It feels a bit like an axiom. Not sure I want to swallow it like this.

I would even take a different positiont : is it clear that we cannot do the opposite, push some pieces that are currently in CMFPlone into those packages?

We might be able to get rid of some internal dependencies by following a track similar to what was done for Zope 3/ ZTK.

I cannot help but have the feeling that we will create a bigger spaghetti bowl where it will be harder to cleanup some dependencies that arose from bad initial designs.

comment:5 in reply to: ↑ 3 Changed 18 months ago by esteele

Replying to davisagli:

Plone's core currently includes 38 packages in the plone.app namespace. In many cases this separation is unnecessary and slows development of Plone core. In some cases there are good reasons for the separate packages, but we have identified 18 packages that have little benefit to being packaged separately (they are interdependent and always required in Plone) and just cause unnecessary work for the release manager. We propose merging their repositories into the Products.CMFPlone repository (including history) and always releasing these packages as part of the Products.CMFPlone distribution.

Yes, please! Merging just those 18 packages would cut out about 50% of the time I spend making Plone releases.

And a big +1 to those Hanno mentioned as well. PasswordResetTool in particular could be pawned off on the cpy PLIP (#10359) implementers. I believe that and the AT packages were the only core packages outside of CMFPlone containing CMFFormController scripts.

comment:6 follow-up: ↓ 14 Changed 18 months ago by gotcha

Eric,

Can you tell us which part of the release process eats most of your time ?

This way, we might come up with improvements to that process. This would be useful to all of us as we all need to make releases for our projects.

comment:7 Changed 18 months ago by kleist

  • Status changed from new to confirmed
  • Component changed from Unknown to Backend (Python)

comment:8 in reply to: ↑ 3 Changed 17 months ago by amleczko

+1

Replying to hannosch:

+1. I'd also merge some Products into CMFPlone, for example:

  • CMFQuickInstallerTool
  • PasswordResetTool
  • PlacelessTranslationService
  • PloneLanguageTool
  • PlonePAS
  • statusmessages

and +1 for hannosch proposition

comment:9 Changed 17 months ago by thet

I want to pick up Robert Niederreiters idea of using a plone.app namespace package and moving all of the merging candidates to it.

The reason is:

  • CMFPlone is already a huge package and shouldn't get much bigger.
  • We would keep the refactoring direction of moving code out of Product.* (including CMF*) namespaces in general into the nicer (cosmetical, though) plone.app namespace
  • The implementation of this approach could very easily be done, since no plone.app package exists yet. Turning it into a namespace package keeps it extensible.
  • External plone.app.* addons also would not need any change (as with the original approach).

No setuptools magic is needed to provide a second namespace (Products.CMFPlone and plone.app.*).

comment:10 Changed 17 months ago by davisagli

I would be fine with calling the combined distribution plone.app instead of Products.CMFPlone -- but I think Products.CMFPlone should then be bundled as part of it. (Since Products.CMFPlone is usually heavily interdependent with changes in packages like plone.app.layout and plone.app.upgrade.)

comment:11 Changed 17 months ago by rossp

+1 for merging the plone.app packages listed into CMFPLone, +1 to hanno's suggestion about also merging other Products.* packages.

comment:12 Changed 17 months ago by davisagli

  • Milestone changed from 4.4 to 5.0

Approved for 5.0. So we have some time to refine our plan here. Next step is for me to incorporate the feedback on which packages to include.

comment:13 in reply to: ↑ 4 Changed 17 months ago by gotcha

Replying to gotcha:

Can the following sentence be expanded ? "In many cases this separation is unnecessary and slows development of Plone core."

It feels a bit like an axiom. Not sure I want to swallow it like this.

I would even take a different positiont : is it clear that we cannot do the opposite, push some pieces that are currently in CMFPlone into those packages?

We might be able to get rid of some internal dependencies by following a track similar to what was done for Zope 3/ ZTK.

I cannot help but have the feeling that we will create a bigger spaghetti bowl where it will be harder to cleanup some dependencies that arose from bad initial designs.

I did not receive any answer to this comment.

I have the feeling that the only motivation is to reduce time spent at releasing packages. I think we should rather improve the tools we use to release (which would also help us all in our project work).

comment:14 in reply to: ↑ 6 Changed 17 months ago by gotcha

Replying to gotcha:

Eric,

Can you tell us which part of the release process eats most of your time ?

This way, we might come up with improvements to that process. This would be useful to all of us as we all need to make releases for our projects.

Is this question useless ? I'd really appreciate an answer.

comment:15 Changed 17 months ago by gotcha

In  http://article.gmane.org/gmane.comp.web.zope.plone.devel/31202

I stated

1) IMO, working with small packages has another goal than just reuse : small packages are a way of making fixes available in a more modular way.

What I mean is that if I need a fix for an issue, I am happy to only test and upgrade the new release of a small package (or few packages) that are included in the fix rather than having to wait for the next release of the whole system to upgrade my whole system.

(Yes, I live in real world and my projects are not perfect, there are many times where my systems are not in a state where I can "just" upgrade to next release.)

As a consequence, small packages are a better way to get contribution of fixes in Plone core rather than having teams build patches that are not maintained publicly.

IOW, thinking of small packages only in term of reuse is not taking in account other advantages which are not technical but rather tied to organizational aspects of our distributed community.

I think that this PLIP works against this idea (that was approved at least by Laurence, Martin and Lennart)

I understand very well that the release manager wants his problem solved. But I think this PLIP is not the right answer.

comment:16 Changed 17 months ago by esteele

Here's how many packages (not counting Zope) we've been tracking through each release since moving to our current structure:

Plone 3.285
Plone 3.395
Plone 4.0117
Plone 4.1149
Plone 4.2165
Plone 4.3203

Honestly, that should be my entire argument right there, but I'll continue anyway.

Sure, my personal lusting after this is to reduce release time. A single package release takes me very little time, but each additional package builds on that exponentially. One branch of a package may span multiple minor releases of Plone, each needing a check to make sure it hasn't broken compatibility with the older ones. Does this package have actual code changes or just PEP8 changes? Oh, wait, this one was listed as only having test fixes in checkouts.cfg, but now it's got a bug fix, nobody moved it though. Now that I've made it through the list, somebody's made changes to plone.app.layout and plonetheme.sunburst and those will need another new release.

We're building a lot of process and infrastructure around this need to have such fine-grained control over code updates. It's rare that I'm asked to cut a new release of a package in between Plone releases. If someone needs something sooner, they can create a private release for themselves (which we need to do a better job of documenting) in the interim. If I can reduce the time it takes for me to cut Plone releases, that means there can be less time between Plone releases, which seems like a win for everyone.

For everyone else, we get the potential for fixes not spanning 2-3 packages (most of the culprits being the ones suggested by this PLIP). That makes it easier for new developers to do and it makes it easier to review and merge pull requests.

Look, I'll cut releases whether we do this or not, but I'm an idiot that way. If you ever want someone else to step into this role, they're going to need some sanity applied. Otherwise, we're Java and nobody wants that.

comment:17 Changed 16 months ago by aclark

+1000. Just noticed this. I've already done a bit of experimental combining of packages that resulted in failure ( https://github.com/aclark4life/plock). So I would heavily be in favor of seeing this happen in some form and if anyone needs any help please let me know. My main motivation is: Zope2 runs on Heroku ( http://zope2.herokuapp.com/), Plone does not (easily) due to the large number of packages.

comment:18 Changed 10 months ago by thet

While I share gotcha's objection regarding smaller packagages can lead into faster releases, I also see that Plone's huge number of packages is not really something to be proud of - even more, because very few of them are interchangeable with others, using the Zope Component Architecture.

So for my sake, let's reduce them.

Besides of this, I'd love to see a reduction of interdependencies between modules in general. My favorite example is the navigation portlet in plone.app.portlets, which itself depends on plone.app.layout and Products.CMFPlone to build up it's NavTreeStrategy and NavTreeQueryBuilder adapters. I don't go into detail, find out yourself and you'll see you need plenty of time to understand this. But thats a different PLIP.

comment:19 Changed 10 months ago by esteele

The Framework Team has decided to move on to Plone 5. Updating milestones accordingly.

Note: See TracTickets for help on using tickets.