Unicode error in calendar portlet

[jonathan_lewis]

On a brand new Plone site (Plone 2.5, Zope 2.9.3, Python 2.4.3), using Japanese browsers (Mac Safari, Windows IE), I added an event that will take place later in the current month.

The event displayed OK in public draft mode. When I clicked "publish" I got the following error.

When publishing an event that will occur next month, the error occured when I clicked the "next month" arrow in the calendar portlet.

Site Error

An error was encountered while publishing this resource.

UnicodeDecodeError

Sorry, a site error occurred. Traceback (innermost last):

Module ZPublisher.Publish, line 194, in publish_module_standard Module Products.PlacelessTranslationService.PatchStringIO, line 34, in new_publish Module ZPublisher.Publish, line 146, in publish Module Zope2.App.startup, line 222, in zpublisher_exception_hook Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Shared.DC.Scripts.Bindings, line 311, in call Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.CMFCore.FSPageTemplate, line 195, in _exec Module Products.CMFCore.FSPageTemplate, line 134, in pt_render Module Products.PageTemplates.PageTemplate, line 104, in pt_render <FSPageTemplate at /test1/event_view used for /test1/testevent1> Module TAL.TALInterpreter, line 238, in call Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 749, in do_useMacro Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 780, in do_defineSlot Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 715, in do_condition Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 780, in do_defineSlot Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 749, in do_useMacro Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 715, in do_condition Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 691, in do_loop_tal Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 715, in do_condition Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 749, in do_useMacro Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 455, in do_optTag_tal Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 691, in do_loop_tal Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 691, in do_loop_tal Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 715, in do_condition Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 457, in do_optTag_tal Module TAL.TALInterpreter, line 442, in do_optTag Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 715, in do_condition Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 455, in do_optTag_tal Module TAL.TALInterpreter, line 437, in no_tag Module TAL.TALInterpreter, line 281, in interpret Module TAL.TALInterpreter, line 339, in do_startTag Module TAL.TALInterpreter, line 405, in attrAction_tal Module Products.PageTemplates.TALES, line 227, in evaluateText Module Products.PageTemplates.TALES, line 221, in evaluate URL: file:CMFPlone/skins/plone_portlets/portlet_calendar.pt Line 97, Column 25 Expression: <PythonExpr '\n'.join([toLocalizedTime(cur_date)]+[getEventString(e) for e in dayeventslist?])> Names: {'container': <PloneSite at /test1>,

'context': <ATEvent at /test1/testevent1>, 'default': <Products.PageTemplates.TALES.Default instance at 0x2457c60>, 'here': <ATEvent at /test1/testevent1>, 'loop': <Products.PageTemplates.TALES.SafeMapping object at 0x350e260>, 'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0x244c210>, 'nothing': None, 'options': {'args': ()}, 'repeat': <Products.PageTemplates.TALES.SafeMapping object at 0x350e260>, 'request': <HTTPRequest, URL= http://127.0.0.1:8080/test1/testevent1/event_view>, 'root': <Application at >, 'template': <FSPageTemplate at /test1/event_view used for /test1/testevent1>, 'traverse_subpath': [], 'user': <PropertiedUser 'jonathan'>}

Module Products.PageTemplates.ZRPythonExpr, line 47, in call traceback_info: '\n'.join([toLocalizedTime(cur_date)]+[getEventString(e) for e in dayeventslist?]) Module Python expression "'\n'.join([toLocalizedTime(cur_date)]+[getEventString(e) for e in dayeventslist?])", line 1, in <expression> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 4: ordinal not in range(128) (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'ascii' codec can't decode byte 0xe5 in position 4: ordinal not in range(128)) Troubleshooting Suggestions

The URL may be incorrect. The parameters passed to this resource may be incorrect. A resource that this resource relies on may be encountering an error. For more detailed information about the error, please refer to error log.

If the error persists please contact the site maintainer. Thank you for your patience.

[sparcd]

This is not just a problem with Japanese. I have a client that the same problem occured to. To side-step the problem I changed:

getEventString(e) for e in day['eventslist']

to

str(e) for e in day['eventslist']

[hannosch]

Ok, the problem here is a nasty one. The toLocalizedTime script is supposed to return Unicode and uses the utranslate methods of the translation_service tool for this. Due to some premature optimization the whole utranslate methods do not return Unicode in most cases anymore but encoded strings. Only if they contain Zope3 Messages and are thus handled by the FiveTranslationsService Unicode is returned.

To fix this I will need to come up with some evil hacks which I won't do unless we have some tests for Unicode handling as I'm pretty certain to break something at some different place otherwise :( At least as I tried to circumvent the problem by some simple means I broke things at other places, but without test failures and only some in-browser testing this is an insane approach. I'll start to write Unicode tests first...

[hannosch]

(In [10384]) Added some workaround code for the general misbehaving utranslate and uLocalizedTime methods. They should return Unicode now again, but right now they only work for the single supported site encoding of utf-8. This fixes #5609.