TYPO3 Core

Assigning the bug to our charset hero Masi.

I am not sure if I understand everything from the bug, but this is what I just experienced in a different setup involving JS escape() and IE 5.5 (with security patches on Win98):

Some IEs (I cannot believe the newer ones do that) convert UTF-8 strings into iso-8859-1/windows-1252 before performing the actual escaping.

This will also break popup wizards in the BE! What's more: any code that relies on JS TBE_EDITOR_rawurlencode() in the BE will get wrong results.

So what needs to be done is, check out which versions of IE are affected. Decide if we should introduce a work-around for this versions. If we need one, then we devise one.

More info:

The MS docs say that Unicode chars shall be returned as %uxxxx. Unfortunately I cannot test right now on a newer IE. But obviously any decoding routine of Typo3 must be aware of this format.

PS: %FC is iso-8859-1 for German umlaut ü, so the implict conversion is obvious.

I modified getUpdateJS for extension sr_feuser_register:
if (window.decodeURIComponent) { unesc = decodeURIComponent('".rawurlencode($convValue)."') } else { unesc = unescape('".rawurlencode($Nvalue)."') };

where $convValue is the $Nvalue converted to utf-8.

The form now works correctly with utf-8 in Mozilla 1.5 and IE6 and in any other encoding if utf-8 conversion is available (TYPO3 3.6+ or mbstring or utf8_encode).

edited on: 11.03.05 08:01

Ok., as far as we know now there are to solutions to solve it, both require browser sniffing. In one case we determine if escape() is "old style" (charset transparent) or "new style" (Unicode), in the other case we test for the availability of encodeURIComponent().

As this behaviour affects more code than just getUpdateJS() I vote to wait for a general solution.

I replaced the "rawurlencode" call by a call to "addslashes" this should be sufficient as the string which gets encoded is quoted. Every character is allowed in the string except the quotation mark.

This problem also occured to me with Mozilla Firefox 0.9?

I just found this comment in class language:
<cite>
Converts the input string to a JavaScript function returning the same string, but charset-safe.
Used for confirm and alert boxes where we must make sure that any string content does not break the script AND want to make sure the charset is preserved.
Originally I used the JS function unescape() in combination with PHP function rawurlencode() in order to pass strings in a safe way. This could still be done for iso-8859-1 charsets but now I have applied the same method here for all charsets.
</cite>
It describes function JScharCode($str) which makes use of t3lib_cs::utf8_to_numberarray(). So it seems Masi already has found a solution but doesn't remember ;-)

I adapted JScharCode() for use in class tslib_cObj and it works for me. The attached file is a patch file for version 3.6.1. Perhaps Masi can port this patch for 3.8.0?

Hi,

in JScharCode() at this line: $GLOBALS['TSFE']->siteCharset;

What is siteCharset? Didn´t foud any other point in source. Now metaCharset?

I´m also think that we must modify the Line:

$validateForm=' onsubmit="return validateForm(\''.$formname.'\',\''.implode(',',$fieldlist).'\',\''.rawurlencode($conf['goodMess']).'\',\''.rawurlencode($conf['badMess']).'\',\''.rawurlencode($conf['emailMess']).'\')"';

to be now:

$validateForm=' onsubmit="return validateForm(\''.$formname.'\',\''.implode(',',$fieldlist).'\','.$this->JScharCode($conf['goodMess']).','.$this->JScharCode($conf['badMess']).','.$this->JScharCode($conf['emailMess']).')"';

to support Validation Messages.

The patch I provided is for version 3.6.1 (as mentioned).

In class tslib_fe (which is instantiated in TYPO3 as $GLOBALS['TSFE'] normally) the variable siteCharset was replaced by renderCharset and metaCharset (see TSRef [1]) in version 3.7.0. The right substitution for siteCharset would be renderCharset for newer versions². So if someone wants to use the patch on 3.7 or 3.8 "siteCharset" has to be changed to "renderCharset" in line 37 of the patch file.

IMHO Masi should provide an equivalent function in t3lib_cs, that could be used in BE and FE. Of course it's possible to apply my patch to other parts of the core (as suggested by haustier). But that's just a workaround for urgent problems and not a solution to the underlying issue.

There are many places in the core where it's necessary to have a function at hand that provides valid UTF-8 text. tslib_cObj is IMHO not the right place for such a function, as it's not directly concerning content as such. There's already at least one other bug report (#14699 [3]), that applies to this issue.

[1] http://typo3.org/documentation/document-library/doc_core_tsref/quot_CONFIG_quot/
[2] the split took place in revision 1.33 of class.tslib_fe.php
[3] http://bugs.typo3.org/view.php?id=1030

Hi Masi,

could you please take a look at this issue again? IMHO it needs a general solution, as JS functionality is used in TYPO3 everywhere. I think you're better in on that charset and multilanguage stuff than anyone else.

Thanx
Thorsten

Extension is for Typo3.8.0. It fixes also Problems with these kind of strings in Form Validation Output.

Feel free to try.

T3X_htutf8js-0_0_0.t3x contains absoultely no code whatsoever.

Anyway, I think the best solution would be to drop the whole encoding stuff completely. The second best is to use it only for iso-8859-1.

The rational is that modern browsers don't need the encoding (which is untrue for really old browsers of the bad old days) and it's now broken (since IE 5.5 and Mozilla 1.3) for all charsets except iso-8859-1. Old browsers didn't have support for UTF-8, so there's little loss.

Sorry.

I uploaded the false t3x. New one is now there and all files should be in there.

Please let me now when you find some problems.

Thanks

Does it solve utf-8 character problem in JSMENU?? I install the extension and find that all chinese with utf-8 in JSMENU still broken.

Gideon

No it doesn´t solve this problem. But if you look into class.ux_tslib_content.php, there is a Function "JScharCode".
At typo3.8 there is in class.tslib_menu.php at line 2710:

$title=rawurlencode($data['title']);

this must replaced with a call like:

$title=JScharCode($data['title']);

and line 2722:

$codeLines.="\n".$var.$count."=".$menuName.".add(".$parent.",".$prev.",0,'".$title."','".$GLOBALS['TSFE']->baseUrlWrap($url)."','".$target."');";

like this:
$codeLines.="\n".$var.$count."=".$menuName.".add(".$parent.",".$prev.",0,".$title.",'".$GLOBALS['TSFE']->baseUrlWrap($url)."','".$target."');";

i think.

No guarantee. If it works, please post it here.

If I replace $title=rawurlencode($data['title']); with $title=JScharCode($data['title']);. It shows me a blank page.

and

what's the different between

$codeLines.="\n".$var.$count."=".$menuName.".add(".$parent.",".$prev.",0,'".$title."','".$GLOBALS['TSFE']->baseUrlWrap($url)."','".$target."');";
$codeLines.="\n".$var.$count."=".$menuName.".add(".$parent.",".$prev.",0,".$title.",'".$GLOBALS['TSFE']->baseUrlWrap($url)."','".$target."');";

Gideon

I have added a new Version of the Extension which also Handle utf-8 chars in JSMENU.

Greetings to gideonso :-)

$codeLines question:

if you look at the part :

... 0,'".$title."','".$GLOBALS['TSFE'] ...

there is the difference. Only two ' must deleted.

It works!! It works!! It works!!!

Horray!! Horray!! Horray!!

Gideon

OK.

I already posted this note but nobody seemd to mention it. So again as I want to solve this problem now as it is one of the Gremlins which we shall fix during our bugfixing session.

Could anybody enligthen me what the sense of:

$codeLines.="\n".$menuName.".defTopTitle[".$count."] = unescape('".rawurlencode($levelConf['firstLabel'])."');";

Obviously someone hasn't read a HTML or XHTML DTD (Document Type Definition). Else it would be clear that CDATA sections don't need to get escaped ! And everything inside a <script> tag is CDATA by default (no matter wheter you note the CDATA opening and closing tags)

Of course the content of an "onClick" event isn't CDATA but PCDATA ... This means HTML entities get escaped (ü becomes ü). So you would have to escape the & character also when you use a php-var insided an onclick event.

So in my opinion all those rawurlencode and unescape things are unnecessary and
a simple
addcslashes($str '\'')
would do it. Just escape all ' if you use them as start and end delimiter for your JS strings. No other characters need to get escaped.

Maybe someone tought generating entities is required in the whole HTML file (but in fact it is not required for CDATA type)

At least this is true for modern browsers (IE, FF, Safari, Opera). Maybe some older (4.x Netscape :) complains about it.

I attach a .diff to this bug (utf8_JS.patch). Could somebody with access to an older browser test it ?

Bernhard, your too young :-)

Old implementations of Javascript had all kind of problems with non-ASCII characters. But you're right, all this stuff is not needed for todays browsers.

Hi Bernd,

your patch didn´t work for me. I´m using Firefox 1.0.7.
When i look into the sourcecode (html) there are no & entities, there are only utf-8 codes.

Well ... I explained it. There is no need for entities !

Masi already brought it to the point : entities in javascript strings are required for pre 4.x browsers (Do you still know the Turning-Wheel Logo of Netscape 3.x ??? :)

You can only check if it works by inserting a FE-User Registration Form (or a Newsletter Subscription form) and fill it out not completly. You will get to the same page again with error messages. If then special characters in the fields are garbled it doesn't work. If "Ü" stays at it is it works fine.

You can also test it with a JSMENU and page-titles having special characters or japanese or something like that.

Hi Bernd,

here is a piece of source code:

<----snip----->
onsubmit="return validateForm('a7867b54b21bdc5f71467146bd2208bbf','Nachname,Nachname%2A,Vorname,Vorname%2A,Strasse,Strasse%2A,PLZ_und_Ort,PLZ%2FOrt%2A,eMail,eMail%2A','','Sie%20haben%20nicht%20alle%20ben%C3%B6tigten%20Felder%20ausgef%C3%BCllt.%20Bitte%20%C3%BCberpr%C3%BCfen%20Sie%20Ihre%20Eingaben.','')">
<---snap----->

So it looks before AND after your patch.
Both didn´t work for me.

Testet with a mailform and not filled required fields.

Did i made a mistake?

The actual patch:
utf8_JS_2005-10-22.patch
fixes following problems:

Mailform goodMess, badMess, emailMess with utf-8 entities. (No bug #)
FE-User admin/Direct Mail subscription utf-8 entities in values (Bug #14264)
JSMENU with utf8 (Bug #14264)
Editpanel confirm dialogs with utf8 (Bug #14264)

please try it out and write me back if you had a negative or positive experience:
kraftb@kraftb.at

If this works fine and no problems come up it will get added to the core and be contained in T3 4.0.0

Great work Bernd,

it works fine in our projects.

Thanks

Hi,
it would be great if you could check that your patch as well fixes bug 1270, I think it might be the same issue but I am not sure. Greets, Sebastian

With Bernhard's solution, with a field of type textarea containing a linebreak, I get a javascript error "unterminated string literal" in updateForm. If the field does not contain a linebreak, I do not get the error.

I suggest replacing line
$value = addcslashes($value, '\'');
in function t3lib_div::quoteJSvalue
by line
$value = addcslashes($value, '\''.chr(10).chr(13));

I tried (on 3.8.1) Bernhards patch utf8_JS_2005-10-26.patch, which works fine together with the small change suggested by Stanislas in comment 3773.

Great!

I just uploaded a new patch that incorporates the change suggested by Stanislas in comment 3773 with Bernhards patch.

Hello,

I uploaded a patch "utf8_JS_2005-12-13.patch" which seems to be earlier than Karstens but it incorporates The suggestion from Stanislas from 08-11-2005 (ID (0003773))

It probably also fixes the problem currently active on the dev list (I made this change for Michael Stucki and he reported it to work):
http://lists.netfielders.de/pipermail/typo3-dev/2006-January/014832.html

And was already sent to the core list and approved by Kasper (:
http://lists.netfielders.de/pipermail/typo3-team-core/2005-December/000806.html

I will put it into CVS now.