Member fullnames are not handled as Unicode internally

[miohtama]

I got the following traceback when using the latest Plone 2.5-zope-2.9 SVN bundle (revision 9594, CMFPlone 2.5-beta2 SVN/UNRLEASED). Looks like Plone login doesn't handle some Unicode aspects correctly and fails when language contains non-ascii characters. My Firefox 1.5 autonegotiated language to Finnish. I didn't investigate this further.

The site instance is clean Plone 2.5 site created via ZMI. I didn't try other actions which might have triggered the same bug. I am not sure if it's login related at all, but I couldn't add any content for the site without being able to log in first.

Workaround is to install PloneLanguageTool and force site language to English (disable autonegotiation).

------
2006-04-23T22:08:17 ERROR Zope.SiteErrorLog http://localhost:8080/unstable/login_form
Traceback (most recent call last):
  File "F:\usability\unstable\Zope\lib\python\ZPublisher\Publish.py", line 115, in publish
    request, bind=1)
  File "F:\usability\unstable\Zope\lib\python\ZPublisher\mapply.py", line 88, in mapply
    if debug is not None: return debug(object,args,context)
  File "F:\usability\unstable\Zope\lib\python\ZPublisher\Publish.py", line 41, in call_object
    result=apply(object,args) # Type s<cr> to step into published object.
  File "F:\usability\unstable\Data\Products\CMFFormController\FSControllerPageTemplate.py", line 96, in __call__
    return self._call(FSControllerPageTemplate.inheritedAttribute('__call__'), *args, **kwargs)
  File "F:\usability\unstable\Data\Products\CMFFormController\BaseControllerPageTemplate.py", line 39, in _call
    return self.getNext(controller_state, REQUEST)
  File "F:\usability\unstable\Data\Products\CMFFormController\ControllerBase.py", line 243, in getNext
    return next_action.getAction()(controller_state)
  File "F:\usability\unstable\Data\Products\CMFFormController\Actions\TraverseTo.py", line 36, in __call__
    REQUEST, bind=1)
  File "F:\usability\unstable\Zope\lib\python\ZPublisher\mapply.py", line 88, in mapply
    if debug is not None: return debug(object,args,context)
  File "F:\usability\unstable\Zope\lib\python\ZPublisher\Publish.py", line 41, in call_object
    result=apply(object,args) # Type s<cr> to step into published object.
  File "F:\usability\unstable\Data\Products\CMFFormController\FSControllerPythonScript.py", line 107, in __call__
    result = FSControllerPythonScript.inheritedAttribute('__call__')(self, *args, **kwargs)
  File "F:\usability\unstable\Data\Products\CMFFormController\Script.py", line 141, in __call__
    return BaseFSPythonScript.__call__(self, *args, **kw)
  File "F:\usability\unstable\Data\Products\CMFCore\FSPythonScript.py", line 108, in __call__
    return Script.__call__(self, *args, **kw)
  File "F:\usability\unstable\Zope\lib\python\Shared\DC\Scripts\Bindings.py", line 311, in __call__
    return self._bindAndExec(args, kw, None)
  File "F:\usability\unstable\Zope\lib\python\Shared\DC\Scripts\Bindings.py", line 348, in _bindAndExec
    return self._exec(bound_data, args, kw)
  File "F:\usability\unstable\Data\Products\CMFCore\FSPythonScript.py", line 164, in _exec
    result = f(*args, **kw)
  File "Script (Python)", line 28, in logged_in
  File "F:\usability\unstable\Data\Products\PlonePAS\tools\membership.py", line 337, in createMemberarea
    default = '')
  File "F:\usability\unstable\Data\Products\CMFPlone\TranslationServiceTool.py", line 35, in utranslate
    return utranslate(*args, **kw)
  File "F:\usability\unstable\Data\Products\CMFPlone\i18nl10n.py", line 29, in utranslate
    return service.translate(*args, **kw)
  File "F:\usability\unstable\Zope\lib\python\Products\Five\i18n.py", line 50, in translate
    target_language=target_language, default=default)
  File "F:\usability\unstable\Data\Products\PlacelessTranslationService\PlacelessTranslationService.py", line 94, in tra
nslate
    return service.translate(domain, msgid, mapping, context, target_language, default)
  File "F:\usability\unstable\Data\Products\PlacelessTranslationService\PlacelessTranslationService.py", line 553, in tr
anslate
    return self.interpolate(text, mapping)
  File "F:\usability\unstable\Data\Products\PlacelessTranslationService\PlacelessTranslationService.py", line 592, in in
terpolate
    text = text.replace(string, value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 22: ordinal not in range(128)

[hannosch]

The problem here is that the member fullname is not handled as Unicode internally but mixed with a status message which is of course Unicode. When the fullname contains non-ascii characters you could get a UnicodeDecodeError. The actual error is now circumvented, but you won't see the member name in these cases.

[svincic]

A related issue that may have solved/circumvented this is #5499

[rocky]

Going to make this a little more technical based on discussion between myself and hannosch. Essentially what's happening is the z3 message objects have a superclass of type unicode (essentially making them unicode objects). And when those unicode strings have variable markers in them and some code inserts a variable that is not of type unicode or type string with ascii encoding then we see UnicodeDecodeErrors. This is not just a problem with the full name property of a member but this just happens to be the first place this issue is demonstrated.

As of this writing, the portal status messages that were causing this problem with full name have been modified to not display full name side-stepping the issue for the time being.

[wichert]

Postponing this for now: it needs some more thought and we have a workaround in place for now.