Ticket #11773 (closed PLIP: fixed)

Opened 3 years ago

Last modified 22 months ago

Ship with Dexterity

Reported by: optilude Owned by:
Priority: major Milestone: 4.3
Component: General Version: 4.3
Keywords: Cc:

Description (last modified by davisagli) (diff)

Description

Proposer: Martin Aspeli Seconder: David Glick

Motivation

Dexterity is widely considered the 'next generation' content type framework for Plone. It has a few advantages over Archetypes:

  • A TTW types story. This is an important feature for many users, and a barrier to the adoption of Plone among non-developers and light integrators - crucially, it is also a feature most of our competitors have.
  • Has comprehensive and up-to-date documentation.
  • Less boilerplate, and so faster development.
  • Uses modern ZTK components like zope.schema, z3c.form, etc.

Dexterity integrates well with existing Zope, CMF and Plone infrastructure, including other Archetypes-based content: the lower common denominator here is the CMF types system and the ZTK.

Dexterity is used in production by many individuals and companies, and usually becomes the preferred framework for those who have used it, fitting more naturally into their workflow. Anecdotal evidence also suggests Dexterity is easier to learn for new users, presumably for the same reasons.

Dexterity fits into various bits of the future Plone roadmap, notably as the core of the Deco layout system, and on its own to provide a way for non-developers and integrators to model their business requirements as types with custom schemata.

Previously, there has been an assumption that Deco, Diazo and Dexterity would all land together in a grand "Plone 5" release, where Plone core would no longer rely on Archetypes for its core types.

However, the amount of interdependencies and level of risk in this vision means it is very difficult to reach, falling into the very "big release" trap we are trying to avoid with the improved PLIP process.

By shipping with Dexterity in Plone 4, we will:

  • Give Plone a TTW types story.
  • Make it easier for developers to get started with Dexterity, removing the need to install a separate KGS
  • Encourage an ecosystem of useful, reusable "behaviours" that give power to integrators and less advanced developers
  • Provide a stepping stone towards the "Plone 5" feature set and the basis for further evolution

You can read more about Dexterity's philosophy here:  http://code.google.com/p/dexterity/

Assumptions

N/A

Proposal & Implementation

  • Release Dexterity 1.0 as-is. [DONE!]
  • Remove Plone 3 BBB support [DONE!]
  • Create plone.autoform, plone.supermodel and plone.directives.form branches that remove the martian dependency for defining form hints in schemata, and refactor plone.app.dexterity to have no dependency on martian or five.grok [DONE!]
  • Make the standard "related items" field based on UUIDs (plone.uuid) instead of plone.app.relationfield. This is less powerful, but simpler and does not rely on intids and weak references, making it more robust for content export/import and edge cases not well handled until Plone grows parent pointers. [Update: The existing zc.relation-based related items behavior has been moved to plone.app.relationfield, and plone.app.dexterity no longer depends on zc.relation support.]
  • Release 2.0 versions of the various packages [Update: branches for plone.dexterity and plone.app.dexterity now exist.]
  • Include these in KGS

Deliverables

  • New package releases
  • KGS updates

Dexterity is a conglomeration of many packages, most of which are generic and not tied to Dexterity itself. A brief description of each follows.

It should be noted that most of these packages are very small, on the principle of "one package, one function".

Low-level dependencies

These packages were largely created for Dexterity, but are generic, relying only on the ZTK and (in some cases) z3c.form.

plone.alterego
Provides support for dynamic modules. This is used to support TTW schemata.
plone.synchronize
A decorator for synchronising access to methods across threads. Used as part of a schema caching optimisation.
plone.behavior
Adds the concept of a "behaviour", which is like a reusable adapter that can be turned on and off per-type or per-instance. Many of the standard features, like locking, versioning, staging, etc are implemented as behaviours, which are enabled per-type in the FTI or TTW editor.
plone.rfc822
Marshalling and unmarshalling of objects described by zope.schema to/from an RFC822 representation. This is used in Dexterity's WebDAV support.
plone.supermodel
Defines an XML syntax that represents a zope.interface/zope.schema model. This iis the basis for TTW types definitions, and may be used by tools (think ArchGenXML) to allow round-trip editing of schemata. (Already in Plone core as of Plone 4.1)
plone.autoform
Allows z3c.form forms to be configured based on hints stored in tagged values in schemata. The tagged values can be added as directives in the interface 'class' body or in a plone.supermodel schema. (Already in Plone core as of Plone 4.1)
plone.schemaeditor
Support for through-the-web editing of zope.interface/zope.schema schemata. This is used by the TTW types editor.

Core packages

These are the core of the Dexterity framework.

plone.dexterity
The core types system, which depends on CMF only and is theoretically reusable outside Plone. This adds a custom FTI and base classes for folderish and non-folderish content types. The FTI holds a reference to a schema, which can be defined as a zope.interface schema, in a filesystem XML file, or in a TTW XML file.
plone.app.dexterity
Plone integration, including the control panel with the full TTW editor and standard behaviours for Dublin Core and other "standard" fields.

Fields and widgets

These packages provide support for field types beyond zope.schema and widgets beyond z3c.form's standard widgets.

plone.namedfile
Blob and non-blob file and image fields.
plone.formwidget.namedfile
File/image upload widget for plone.namedfile
collective.z3cform.datetimewidget
Date/time widget.
plone.app.textfield
Rich text field with support for portal_transforms, "safe HTML" and WYSIWYG editors.

Convention-over-configuration

These packages provide support for martian-style convention-over-configuration.

five.grok
Basic ZCA configuration. Essentially a wrapper around grokcore.component, grokcore.view, grokcore.security, grokcore.viewlet, grokcore.site, grokcore.formlib and grokcore.annotation (also included).
plone.directives.form
Convention-over-configuration for plone.autoform and z3c.form views. (Presently also used for plone.autoform schema hints, but this is being divorced from martian and moved into plone.autoform itself).
plone.directives.dexterity
Convention-over-configuration and useful base classes for registering new content classes (which are rarely needed in Dexterity) and custom Dexterity add/edit forms.

Behaviours

These provide higher level, Plone-specific integration on an opt-in basis, available in the TTW editor as well as through GenericSetup.

plone.app.referenceablebehavior
Allows Archetypes ReferenceFields to reference Dexterity content.
plone.app.stagingbehavior
Support for plone.app.iterate.
plone.app.versioningbehavior
Support for CMFEditions.

Risks

This section attempts to provide answers to some of the risks and concerns that have been raised about Dexterity in the past.

Codebase growth and long-term maintenance burden

(aka the "nothing in Plone core uses this yet" argument)

This is to a certain extent an inevitable risk of an evolutionary approach. We feel Dexterity is proven, widely used and stable enough (it's over 3 years old) for the risk of it being a "dead" technology to be low. Furthermore, there is patently a lack of innovation around the Archetypes framework, and little appetite for major refactoring to make it more coherent with other ZTK and Plone technologies.

It is worth nothing that Dexterity contains a lot less code than Archetypes, and is generally well-tested and well-documented. Multiple contributors who have contributed to its maintenance over the past year, beyond the ones responsible for this PLIP.

Users invested in Archetypes will be worried

We need to clearly communicate that Archetypes will remain a viable option for the foreseeable future.

Martian based configuration (aka "Grok-style" configuration) may be confusing

UPDATE: We now have branches that make Grok-style configuration an optional extra.

The Dexterity documentation suggests using convention-over-configuration based on Martian (the same library used by Grok) for tasks such as registering views, forms or event handlers. Some have expressed reservations about this for two reasons: Potential confusion with the Grok framework, and potential confusion given multiple ways to register components.

The first point is best addressed with documentation, although it would be feasible to create a shim around five.grok if desired to hide the "grok" name. (Note that five.grok is largely maintained by the Silva and Grok communities, with some support from the Dexterity developers). However, this must be balanced against the desire to be a good citizen in the Zope world and not reinvent the wheel.

On the second point, this is ultimately a choice that can be left up to developers. At the end of the day, ZCML and Martian-based convention-over-configuration are just two ways to execute zope.configuration actions and manipulate the component registry.

Regardless, we suggest that for the foreseeable future, Plone core (including plone.app.dexterity) uses only ZCML configuration, leaving martian configuration-over-convention as an integrator choice. It follows that plone.directives.form, plone.directives.dexterity and five.grok should be included in the KGS.

Note: Presently, martian is used in the configuration of form schema hints, which are the primary way to do things like make fields in forms permission-dependent or change widgets. However, there is a branch that removes this dependency, making martian-based configuration entirely optional and focused mainly on views, forms and other component concepts. The intention is to complete and merge this as part of the implementation of this PLIP.

zc.relation and five.intid

UPDATE: Dexterity no longer includes plone.app.relationfield or a relation-based related items field by default.

The "related items" behaviour in plone.app.dexterity and the standard way to do relation fields (plone.app.relationfield) is based on zc.relation, which in turn relies on zope.intid via five.intid. This is a powerful reference abstraction, made easy to use through a sensible API in z3c.relationfield/plone.app.relationfield.

However, the Zope 2 integration is somewhat brittle and will probably remain so until we have parent pointers for all content, as it relies on paths and traversal to re-establish an acquisition chain. Furthermore, relying on intids makes it harder to move content between sites or databases.

To address this, we propose to ship with a simple referenceable behaviour based only on plone.uuid and portal_catalog. The related items behavior and field would still be available by installing plone.app.relationfield, but not included by default.

Lack of feature parity with Archetypes

The current list of missing features can be found in the Dexterity issue tracker ( http://code.google.com/p/dexterity/issues/list). The most important ones are:

  • No support for multi-lingual content. The current plan is to make LinguaPlone more Archetypes-agnostic and implement optional Dexterity support for this.
  • No support for Link Integrity. A blueprint exists for a behaviour adding support for this, but it has not been built yet.

Note that there are behaviours for staging and versioning, which we would propose to include in the KGS, though they may need some more testing and could be made optional/future inclusions if not ready.

Participants

  • Martin Aspeli
  • David Glick
  • Laurence Rowe
  • Steve McMahon (particularly docs)

Progress

  • Dexterity 1.0 is released.
  • Dexterity trunk has been cleaned up to remove Plone 3 support.
  • Dexterity trunk no longer depends on plone.app.relationfield and the various packages needed to support zc.relation-style relations.
  • Dexterity trunk no longer requires including five.grok and martian (there is an optional extra to do so if desired).

Change History

comment:1 Changed 3 years ago by optilude

  • Description modified (diff)

comment:2 Changed 3 years ago by optilude

  • Description modified (diff)

comment:3 Changed 3 years ago by optilude

  • Description modified (diff)

comment:4 Changed 3 years ago by optilude

  • Description modified (diff)

comment:5 Changed 3 years ago by optilude

  • Description modified (diff)

comment:6 Changed 3 years ago by optilude

  • Description modified (diff)

comment:7 Changed 3 years ago by optilude

  • Description modified (diff)

comment:8 Changed 3 years ago by optilude

  • Description modified (diff)

comment:9 Changed 3 years ago by hannosch

This sounds like a really great PLIP. I especially like the proposed approach for grok and intids. Can we have this in the core tomorrow ;) Excited!

comment:10 Changed 3 years ago by smcmahon

  • Description modified (diff)

comment:11 Changed 3 years ago by optilude

  • Milestone set to 4.2

Proposing this for 4.2

comment:12 Changed 3 years ago by davisagli

  • Description modified (diff)

Some updates:

  • Dexterity 1.0 is released.
  • Dexterity trunk has been cleaned up to remove Plone 3 support.
  • Dexterity trunk no longer depends on plone.app.relationfield and the various packages needed to support zc.relation-style relations.

comment:13 follow-up: ↓ 14 Changed 3 years ago by optilude

(In [50390]) Spin the plugins out into plone.app.themingplugins. Refs #11773

comment:14 in reply to: ↑ 13 Changed 3 years ago by optilude

Replying to optilude:

(In [50390]) Spin the plugins out into plone.app.themingplugins. Refs #11773

D'oh, wrong ref. :) Meant #11880

comment:15 Changed 3 years ago by davisagli

  • Description modified (diff)

A big update: Dexterity trunk no longer depends on five.grok and its dependencies. The schema directives have all been reimplemented to work without grok. Some less essential features (such as grok-style registration of form views) can still only be used if the new "grok" extra is enabled.

I would love to have the framework team begin reviewing this.

comment:16 Changed 3 years ago by ldr

FWT approved Dexterity for 4.2 at todays meeting and reviewers have been assigned.

Concerns were raised over mixing unicode and encoded strings in the catalogue for values indexed with KeyWordIndex or ValueIndex - we need to check that all metadata field accessors are returning encoded strings for compatibility with Archetypes.

comment:17 Changed 3 years ago by rossp

I wasn't aware that there was no PLIP *.cfg for dexterity in plone-coredev. I don't think this should be reviewed outside of coredev. So I'm waiting to review until there's a config in coredev.

comment:18 Changed 3 years ago by davisagli

(In [50582]) start of a .cfg for the Dexterity plip (refs #11773). still need to investigate why a few tests are not running, looks like some test dependencies are not declared correctly. the tests pass fine in the official Dexterity buildout.

comment:19 follow-up: ↓ 20 Changed 3 years ago by davisagli

(In [50588]) include test dependency fixes. buildout is now usable. the new packages' tests can be run with bin/test-dexterity. refs #11773

comment:20 in reply to: ↑ 19 ; follow-up: ↓ 21 Changed 3 years ago by rossp

Replying to davisagli:

(In [50588]) include test dependency fixes. buildout is now usable. the new packages' tests can be run with bin/test-dexterity. refs #11773

Rock! Have the appropriate packages been added to alltests? Also, why the separate test script? If the PLIP is "ship with" then shouldn't the dexterity tests be just like the others?

comment:21 in reply to: ↑ 20 Changed 3 years ago by davisagli

Replying to rossp:

Replying to davisagli:

(In [50588]) include test dependency fixes. buildout is now usable. the new packages' tests can be run with bin/test-dexterity. refs #11773

Rock! Have the appropriate packages been added to alltests? Also, why the separate test script? If the PLIP is "ship with" then shouldn't the dexterity tests be just like the others?

I also added them to bin/test so they can be run like the others (and which should cause them to be included in alltests). But I thought it would be convenient to be able to run just the new stuff separately as well.

comment:22 follow-up: ↓ 23 Changed 3 years ago by robgietema

(In [51359]) Added review. Refs #11773

comment:23 in reply to: ↑ 22 Changed 3 years ago by ldr

Replying to robgietema:

(In [51359]) Added review. Refs #11773

Creating filesystem based content types is a lot easier then using Archetypes with a lot less boilerplate. The models approach using the xml files is nice because of the export ability in the TTW editor. Some settings are not possible with just the models xml so it seems the python based schemate are preferred. Converting the model xml to python based interfaces is not hard but can be a lot of work with big schemata.

Which features are not available in the model xml? The xml model is generated from the field's interface, so there should be feature parity between the two. Form, security and validators (which all work based on interface annotations) all have xml equivalents.

comment:24 Changed 3 years ago by robgietema

We had problems with ObjPathSourceBinder, here's an example of python version:

somerelation = RelationChoice(
                   title=u"Some relation", 
                   source=ObjPathSourceBinder(object_provides=ISomeInterface.__identifier__),
                   required=False,
               )

comment:25 follow-up: ↓ 26 Changed 3 years ago by eleddy

(In [51371]) move review to correct branch, add some clarity, refs #11773

comment:26 in reply to: ↑ 25 Changed 3 years ago by davisagli

One response to Rob's review. For most of the items I agreed, and have created tickets in the Dexterity tracker at  http://code.google.com/p/dexterity/issues

The control panel is called "Dexterity Content Types". Since this is the only control panel for content types it should be either called just "Content Types" or "TTW Content Types".

It is not the only control panel for types. Plone already has a Types control panel. It would be a possibility to merge the two, but that may be difficult if we only want to add Dexterity to the Plone KGS and not install it by default (which is what this PLIP proposes).

comment:27 Changed 3 years ago by davisagli

Responses to Liz's review. (If I skipped something here, it means I agreed and added a ticket for it to the Dexterity tracker.)

I'm sure this is obvious but just in case, needs to be installed by default

This PLIP proposed adding Dexterity to the KGS but not installing it by default, so that is intentional. (I would envision making plone.app.dexterity a dependency of the Plone egg, so that it would be easily installable if you install Plone, but still possible to install Products.CMFPlone without getting it.)

There is no obvious way to edit these types. It may not be possible, but at minimum shouldn't I be able to update the description? Bonus for type name. There is no edit tab like default content types in plone which was kinda missing in my end user perspective. Well, their is but it is the fields tab, which is weird. I think this paradigm should be tightened up. For example, add an edit tab where the description can be updated, then change the fields tab to have a title of "Fields"

This is one of those "forgot about it but totally intended to do that" things. I'll ticket it.

[a bunch of good feedback on the rich text field]

Most of these are consequences of the fact that the field edit forms are autogenerated from the field schema, and we haven't actually spent a lot of time on UX for these. Thanks for the specific suggestions; I've ticketed them.

The default mime type descriptions are head exploding. They need descriptions and some implications should be addressed. I'm not even sure why a description has a mime type unless its just for filtering out html? Also, setting to None with text doesn't make sense at all. I think a better paradigm is "leave blank". That goes double for rich text (default value field).

Well, leaving it blank in the form is how you set it to None. We'll improve the field names/descriptions, which were not written with the TTW editor in mind.

The difference between text and text line is no obvious until you add them. I've seen these differentiated by "text line" and "paragraph text" or something like that. I wish we could actually have a description next to each on of these controls, maybe a la radio.

I'm sure we can manage descriptions or images or whatever else we want, but I would really appreciate input from someone on the UI team on how to make this easier to get right.

Editing on a field has javascript confirmation to leave the page. Hm.

Can you clarify where you were encountering this?

Max length validation is wrong. It just cuts things off. As an end user, I type something and then on save its gone. There is no counter or warning or anything. In most cases I assume the end user won't be the person who created the type and won't have this inside knowledge.

This is arguably an issue in z3c.form, which is already part of Plone. But an excellent point.

Preview of a type must include automatically generated stuff like title and description. These are pretty common first things to create. Also, people need to know that these things can be turned on and off.

You mean that it should show fields that are inserted by behaviors. I agree, but I'm not sure how to do this and also keep it possible to sort the fields (fields from behaviors cannot be sorted arbitrarily, but are be inserted relative to the main fields). Suggestions welcome.

When I edit a type, all of my objects of that type became unindexed.

Really? That's pretty surprising. What sort of change did you make to the type?

Read only doesn't make sense for some fields. How would a multiple choice widget be read only?

This controls whether the field is shown when editing an instance of the content type, or not.

It does; it's just not enabled by default. Will be fixed soon.

Behaviors has several bugs. Selecting all exclude from navigation, and or next previous navigation causes stack dump. Choosing basic metadata and dublin core doubles the title and description. These are all when everything else is selected.

Dublin core includes 4 of the other behaviors (basic, categorization, effective range, ownership), so you don't want to enable them if Dublin Core is enabled. There's an open ticket to represent this hierarchy in the behavior control panel somehow. This is another place I need UI help.

I'm going to stop here until some of this stuff is discussed. I don't know if this is the intent of the plip and want to make sure I'm not looking at the UI as an end all.

The UI is where Dexterity needs the most work, and I really appreciate the feedback. It will be very useful to have a list of specific issues to knock off.

Maybe related but likely not: ...(snip)... AttributeError: contentValues

Not related; it looks related to Hanno's catalog changes in 4.1. Please add a ticket to the Plone tracker and make sure Hanno sees it.

At some point PIL wasn't required for startup/testing. Looks like this reintroduces that dependency, which is a bummer.

I got mixed feedback on not requiring PIL (people *want* things to break obviously if it's missing in production), so I didn't make the import in plone.namedfile conditional.

"Vocabulary or source providing values" is a bad description. Is this comma seperated? line separated? Can I point to some code somewhere? I suggest actually looking at google forms for better way to do this. And are we still using that multi choice in and out thing? Why? Can't this just be checkboxes? This widget makes me so sad.

It's an accident that this shows up at all yet. At some point this will let you select from a list of vocabularies that have been made available.

I would love to see someone replace the default z3c.form widget for multiple selection with the dropdown list of checkboxes used by plone.app.collection. If that's done in plone.z3cform as the default widget for this type of field, Dexterity will get it automatically.

comment:28 Changed 3 years ago by rossp

  • Milestone changed from 4.2 to 4.3

FWT has decided to postpone this PLIP to 4.3 to give us some time to work through the bugs the wider user base will encounter.

comment:29 Changed 3 years ago by rossp

The FWT has wrapped up work on 4.2 and can start work on 4.3 whenever we have PLIPs to review. So can you as proposers or implementers please check in on your PLIPs and let us know what the status is and when we can expect issues to be addressed and implementations complete so we can review them for merge in 4.3.

comment:30 Changed 2 years ago by davisagli

Many of the items noted in the reviews for 4.2 have been worked on, and I have just updated the PLIP buildout so it can be reviewed again for 4.3. (Run bin/buildout -c plips/plip11773-dexterity.cfg in the 4.2 coredev buildout. You'll likely have more success with a fresh clone since some repositories were moved to git.)

As reminders:

  • This PLIP is only to make Dexterity available, not to make it installed by default. Compare PLIP #12344 which wants to introduce a Dexterity-based equivalent of ATContentTypes and let users select which content type system to use during site creation.
  • New dependencies (but only when plone.app.dexterity is installed) are collective.z3cform.datetimewidget, plone.alterego, plone.app.dexterity, plone.app.textfield, plone.behavior, plone.dexterity, plone.formwidget.namedfile, plone.namedfile, plone.rfc822, plone.schemaeditor, plone.synchronize, rwproperty, and z3c.blobfile. mocker and plone.mocktestcase are test-only dependencies.
  • You can run bin/test-dexterity to run the tests for the new packages only, or the normal bin/alltests to run all tests.

Also, a new request: While a goal of this PLIP is to make it possible to use Dexterity without requiring five.grok as a dependency, grok-style configuration is the most established way of using Dexterity, so it would be convenient to include five.grok and its dependencies in the Plone KGS so that also extending a good-py KGS is no longer necessary. This would also make maintaining a working KGS easier for me, since I could tune it for each Plone release series rather than depending on a complex good-py configuration to generate the correct adjustments for each major Plone release. Is the FWT is okay with including five.grok and its dependencies in the KGS (but *not* installed unless a developer includes it or plone.directives.dexterity in their setup.py)?

comment:31 Changed 2 years ago by eleddy

No one really wants grok included but since all the documentation relies on it we can't really say no. Go for it

comment:32 Changed 2 years ago by eleddy

I was going to review this plip again but then I realized that it doesn't matter and not much will change from my original review besides the already filed tickets. We must ship Dexterity to attract the attention span of more than just the current maintainers. The core product is solid and we can get the warts as they come.

My only complaint at this point is that the repo be moved into the plone namespace on github. I don't know what you guys want to do about multiple tickets in multiple places but as long as they are both maintained... go for it.

+1 for inclusion.

comment:33 Changed 22 months ago by davisagli

  • Component changed from Infrastructure to General

comment:33 Changed 22 months ago by kleist

  • Status changed from new to confirmed
  • Version set to 4.1

comment:34 Changed 22 months ago by davisagli

  • Status changed from confirmed to closed
  • Version changed from 4.1 to 4.3
  • Resolution set to fixed

This PLIP has been merged. The Plone 4.2 and 4.3 KGS's include the versions needed for Dexterity. Dexterity still needs to be selected while adding a Plone site to enable it.

Note: See TracTickets for help on using tickets.