translate.py is unusable

[ajung01]

translate.py seems to return translations in the encoding of the underlaying .po file. This makes a proper usage impossible if the site encoding is different from the encoding of the .po file. The method should either return translations as unicode or a least as string+encoding.

[dn]

Tried it out:

folder_contents.pt has from line 108:

<input class="standalone"

type="submit" name="folder_factories:method" value="Add New Item" tabindex="" tal:define="ttype python: num_types==1 and here.translate(types[0].Title())" tal:condition="python: num_types" tal:attributes="tabindex tabindex/next|nothing;

value python:ttype and here.translate('Add New ${type}', {'type': ttype}) or

here.translate('label_add_new_item', default='Add New Item')" />

My site encoding is UTF-8. The page comes out in UTF-8 correctly. plone-de.po is encoded in iso-8859-1. Still, "Neuen Artikel hinzufügen" (Add New Item in german) renders correctly.

So I cannot reproduce.

[ajung01]

I request to re-open this issue again since it is reproducable:

• a script in PCNG calls:

msg = context.translate(msgid='changed_saved', default='Changes saved', domain='plonecollectorng')

print repr(msg) return printed

• the output is '0xc4nderungen wurden gespeichert'

• This the an iso-8859-1 encoded string and *not* UTF8 (which is the encoding of the Plone site).

So this is a bug! If you look at the implementation of the translation_wrapper method you will see that the method calls PTS.translate() instead of PTS.utranslate() which returned the string in the encoding of the .po file and there is *no* conversion to the site encoding which is completely imposssible using PTS.translate() since it does not expose the encoding of the .po file.

[vinsci]

Reopened as requested.

[dn]

If you look at the implementation of PTS.translate(), you will see that at first it gets the translated string through a function called getMessage() in line 507 (CVS HEAD).

From the definition of getMessage() in GettextMessageCatalog:

def getMessage(catalog, id, orig_text=None):

"""get message from catalog

returns the message according to the id 'id' from the catalog 'catalog' or raises a KeyError if no translation was found. The return type is always unicode """

I would consider it a major flaw if this function were to return the message in the encoding of the PO file. (That would be totally useless, really. However, translate() is contrary to what you say *not* totally useless as all i18n:translates in ZPTs use it to get their translations.)

Later in PTS.translate() there is a call to """text = context.RESPONSE._encode_unicode(text)""". The comment says "# ask HTTPResponse to encode it for us".

The implementation of _encode_unicode(text) in ZPublisher/HTTPResponse.py:

def _encode_unicode(self,body,

charset_re=re.compile(r'text/[0-9a-z]+\s*;\s*' +

r'charset=([-_0-9a-z]+' + r')(?:(?:\s*;)|\Z)', re.IGNORECASE)):

# Encode the Unicode data as requested if self.headers.has_key('content-type'):

match = charset_re.match(self.headerscontent-type?) if match:

encoding = match.group(1) return body.encode(encoding)

# Use the default character encoding return body.encode('latin1','replace')

That lets me think that it's actually very feasible to use translate() and not utranslate(). Further, how would you explain that it obviously works in folder_contents.pt? Can you try it out, too, and tell me if the button comes out right or wrong? (See my first follow-up.)

I think what you could be seeing is the hard-coded fallback in _encode_unicode. Are you sending a proper request?

[ajung01]

Using PTS.translate() is completely wrong. The code in translate_wrapper() seems to be build for PTS and Localizer. If PTS is installed utranslate() should be called and the result should be converted to the site-encoding of the Plone site. Making one could deprecate Localizer support for Plone 2.1 in favour of PTS.

[dn]

Could you explain why the way translate() chooses the encoding (through HTTPResonpse.py as explained earlier) is wrong? Or under which circumstances it does not work? I don't see why it should be our responsibilty to care of the encoding if there's code that already handles it. Can you tell me in which way that particular code inside HTTPResponse.py is flawed?

I'm sorry if I'm being ignorant...

[limi]

If Andreas can't explain this, we should close this bug.

[longsleep]

issue #3972 marked as duplicate of this bug

see #3972 for a description of the problem

[longsleep]

this is important. The translate wrapper needs to be fixed using utranslate (so type unicode is returned) and all Localizer stuff in there needs to be removed as that wont ever work with localizer.

[longsleep]

marked #3966 as duplicate of this bug

[Anonymous User]

See PLIP 98 -  http://plone.org/products/plone/roadmap/98

[hannosch]

PLIP 98 implemented and merged.