UnicodeDecodeError on livesearch_reply

[lecler]

After upgrade from plone 2.5 to plone 2.5.rc1, livesearch does not work.

Plone site language : french.

Traceback (innermost last):

Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Products.CMFCore.FSPythonScript, line 108, in call Module Shared.DC.Scripts.Bindings, line 311, in call Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.CMFCore.FSPythonScript, line 164, in _exec Module None, line 119, in livesearch_reply

• <FSPythonScript at /intranet/livesearch_reply used for /intranet/irisa>
• Line 119

Module RestrictedPython.PrintCollector, line 23, in call

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 68: ordinal not in range(128)

[hannosch]

This was fixed on the Plone 2.5 branch already and thus the fix will be included in Plone 2.5.2.

[pfurman]

You mean this branch  https://svn.plone.org/svn/plone/CMFPlone/branches/2.5/skins/plone_scripts/livesearch_reply.py ?

If so, there is still a problem when description contains unicode characters exactly in the MAX_DESCRIPTION position. That's because of display_description will contain only half of this character.

The easiest way to reproduce this error is to add long enough description, containing only 'non ascii' characters, and then trying to find such item by live search.

I repaired it by replacing

110: display_description = result.Description

with

110: display_description = result.Description.decode('utf-8','replace')

however I'm not sure if it's the best sollution. Please take a look at that, eventually correct and add to livesearch_reply.py

[kirck]

I've got the same problem after upgrading to 2.5.1 (from 2.5). Language of the site is German. What is strange is, that the word: "termine" arouses the error as there are no Umlaute in that word. Other words work...

[hannosch]

(In [11398]) Fixed another UnicodeDecodeError in livesearch_reply. This fixes #5828. Also added a general safe_unicode method to utils.py, which can be used to convert any Unicode, 'utf-8' or 'ascii' encoded string to Unicode.

[aprilmay]

I tried your modification and it works well.

But it broke the Ploneboard product: it's impossible to add a new forum. I haven't investigated that much, but i guess it's because some translation is missing in Ploneboard.

Here is the error log:

{{{...

Module Products.PageTemplates.TALES, line 221, in evaluate

• URL: file:Archetypes/skins/archetypes/edit_macros.pt
• Line 56, Column 8
• Expression: <PythonExpr here.translate(default)>
• Names:

{'container': <PloneSite at /acommongood>,

'context': <Ploneboard at /acommongood/portal_factory/Ploneboard/ploneboard.2006-12-10.7630382831 used for /acommongood>, 'default': <Products.PageTemplates.TALES.Default instance at 0xb71f3a2c>, 'here': <Ploneboard at /acommongood/portal_factory/Ploneboard/ploneboard.2006-12-10.7630382831 used for /acommongood>, 'loop': <Products.PageTemplates.TALES.SafeMapping object at 0xb35004cc>, 'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0xb729112c>, 'nothing': None, 'options': {'args': (),

'state': <Products.CMFFormController.ControllerState.ControllerState object at 0xb351c54c>},

'repeat': <Products.PageTemplates.TALES.SafeMapping object at 0xb35004cc>, 'request': <HTTPRequest, URL= http://www.acommongood.dnsalias.org/portal_factory/Ploneboard/ploneboard.2006-12-10.7630382831/base_edit>, 'root': <Application at >, 'template': <FSControllerPageTemplate at /acommongood/base_edit used for /acommongood/portal_factory/Ploneboard/ploneboard.2006-12-10.7630382831>, 'traverse_subpath': [], 'user': <PloneUser 'webmaster'>}

Module Products.PageTemplates.ZRPythonExpr, line 47, in call

• traceback_info: here.translate(default)

Module Python expression "here.translate(default)", line 1, in <expression> Module Products.CMFCore.FSPythonScript, line 108, in call Module Shared.DC.Scripts.Bindings, line 311, in call Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.CMFCore.FSPythonScript, line 164, in _exec Module None, line 14, in translate

• <FSPythonScript at /acommongood/translate used for /acommongood/portal_factory/Ploneboard/ploneboard.2006-12-10.7630382831>
• Line 14

Module Products.CMFPlone.TranslationServiceTool, line 35, in utranslate Module Products.CMFPlone.i18nl10n, line 35, in utranslate Module Products.CMFPlone.utils, line 463, in safe_unicode

TypeError: coercing to Unicode: need string or buffer, NoneType found }}}

So i guess Ploneboard is not the only impacted product, so this error might be somewhat important.

I fixed it by adding the corresponding except clause in safe_unicode (line 464 of utils.py)

    except TypeError:
        value = None

There might be a better solution but this works and is scertainly harmless.

[hannosch]

The safe_unicode method has been fixed to handle empty messages and those consisting of numbers. I assume your problem will be fixed by this as well. Please try it again once Plone 2.5.2 is released.

[hannosch]

Milestone 2.5.x deleted

[hannosch]

(In [11398]) Fixed another UnicodeDecodeError in livesearch_reply. This fixes #5828. Also added a general safe_unicode method to utils.py, which can be used to convert any Unicode, 'utf-8' or 'ascii' encoded string to Unicode.