TYPO3 Core

please add relations to #8264 and #8578

forgot to say:
all this does not happen, if you use forceCharset = utf-8 for the Backend!

This is my idea which could be implemented in class.typo3ajax.php

any comments welcome!

Hey Tobi,

good stuff. However, I just thought of something else... can it also be related to the Prototype AJAX.Options.charset variable, which is set to UTF-8 by default? (see http://www.prototypejs.org/api/ajax/options)

Anyway, the approach makes sense, however I always dislike introducing another $charset variable (I worked on this UTF-8 by default, where you have DB charset, Content Charset, Client Charset, Backend Charset)...

Anyway, the solution makes sense, we should run it by Masi however. Could you send him an email?

Another solution would be: do the switching within Javascript

1. Have a global "TYPO3BackendCharset = ..." variable in backend.php

2. Add this to common.js, to make encodeUIRComponent do whatever it should depending on the charset

var classicEncURIcomp = encodeURIcomponent;
function encodeURIcomponent(str) {
if (TYPO3BackendCharset == 'utf8') {
return classicEncURIcomp(str);
} else {
return escape(str);
}
}

Also, for both options, we need to document this in the source code why we do such complicated things. :)

Hej Benni,

thanks for your comment.

i didn't know "AJAX.Options.charset" yet. i'll have a look on it.

Your solution to replace the "encodeURIcomponent" by "escape" if the Charset is not "utf-8" would fail.
i had the same idea to solve #6812, but there i a problem with "escape".

"escape" does not encode "+". Unfortunately a "+" will become a " " (space) in urlEncoding. This caused #8264

I my option (after a short discussion in the core-list with Dmitry and Masi) the best solution would be to handle the encoding on server-side.

i will prepare a patch and then get in touch with Masi for further discussion.

As talked with you over PM, I agree. However (to fix the problem where it initiates and to nag in this client side solution), we could create a customized encodeURIcomponent for ourselves, doing strreplace on the string ourselves, if it's a different charset. I somehow feel we will need the charset stuff a bit more on the client side at some point.

Patch added.
i'll post it in the core list soon

Posts in Ajax.requests (prototype) are always UTF-8 encoded, but they are expected to be encoded like the backend is (iso-8859-1 by default).
This caused special chars to get mixed up (happes in shortcut menu and in the t3editor)

Solution:
    Recode/convert all GET and POST values for every Ajax.Request where the content-type charset does not match with the backend charset
    this is handled by class.typo3ajax.php to ensure it works correct for every ajax-call which uses the new ajax.php

Note:
    For the t3editor i introduce an "workaround" for this issue earlier. There i tried to to solve it on client side (javascript) by replacing "encodeURIcomponent" by "escape". Unfortunately "escape" does not encode "+" correct. see #18897. This workaround should be removed. See my next post.

Note II:
    This issue is related for all requests which use "ajax.php/class.typo3ajax.php" in the backend.
    Most of them do not send any content, but just request some code from the server. So they had no problem at all.
    Any ajax request, which sends content (e.g. input-values, names) to the server, would have the same problem and thus should be tested.

What/Howto test:
    shortcut menu: see bugtracker #18897
    t3editor: depends on another patch: see #18897 (next post)
    any other case which uses ajax.php/class.typo3ajax.php

some testcases:
    Use default charset (iso-8859-1) for the backend (by not setting forceCharset)
    Use UTF-8 for the backend (set forceCharset to utf-8)
    Use any other charset
    Use special chars (ä ü ö) and a plus sign "+" in the shortcut title
    ...