TYPO3 Core

FYI

Will someone fix this before 4.0?

I don't think so...

Hi Ernesto! What do you suggest to solve the problem?
Is the character only generated in the RTE or also in plain textareas?

@Michael Stucki,

The problem is in all text-fields. Masi made some comments about it in t.p.content-rendering, but it looks that the solution is somehow more complex.

In the meantime I've made http://typo3.org/extensions/repository/search/cron_latin1cleaner/, which fixes that specific problem by extending t3lib_parsehtml::HTMLcleaner to cleaning that specific chars if metaCharset is "iso-8859-1". But that might not be the best place to do that, because we might want to have the entities stored in the database already (or not?), so a better place needs to be found.

I have just tried to validate a page where some texts were paste from msword.
I got this message from the validator : "non SGML character number 146".
This character correspond to the htmlentity "’".
It should exist a way in the core to replace character which are not HTML valid (0 to 31 inclusive and 127 to 159 inclusive), by their corresponding HTML entities.
I look around the code but i really don't know where to patch for this.

I think the priority should be upgraded because pasting from msword is really a common way of working for editors, and conformity is becoming the rule.

@Fronzes:

you might as well change the character encoding of your website to "windows-1252" and all will work without any further change. Your site will validate even with those SGML-characters, because they are supported in this charset. All clients that I know will be able to handle that. And your clients will still be able to paste their Word-Text into TYPO3, because their original encoding is also "windows-1252".

windows-1252 = latin-1 + chars#14188-159

See: http://en.wikipedia.org/wiki/Windows-1252

Cheers,
Ernesto

or better yet: change the charset of your website to utf-8. :)

IE will happily paste the range 128-159 plainly and everything else as HTML entity (wether you like it or not). So if you want strict content you must get rid of the fake ASCII range in TCEmain. For plain text fields you must remove or transliterate the characters, for HTML fields you may use entities.

To do this t3lib_cs could be extended to hold (in a file and loaded on demand?) the necessary mappings for the iso-8859-* charsets and the corresponding windows charsets.

@Martin Kutschker,

If my Windows Charset is Windows-1250 (Eastern European) and my HTML-Page is ISO-8859-1 (Western European), what will IE paste into text/rte fields? I thought IE will already "convert" stuff to the charset of the current Webpage on FORMs submit, and the only problem is the 128-159 range when Windows-1252 IE submits to ISO-8859-1 page, because those characters will be submitted as BYTES (instead of entities) and TYPO3 won't do anything about it.

I think TYPO3 has no way of knowing the original OS charset used to paste content into text-fields, so I guess we have no use of a mapping between Windows charsets and ISO-charsets. On a ISO-8859-1 site, we just have to make sure the 128-159 chars aren't stored, as they are illegal in this charset, which means convert them to entities (in html-fields) or "transliterate" them in text-fields.

Do you know any other constelation where such illegal bytes in a ISO-charset can be pasted by Windows OS? I haven't heard of any, but maybe there are.

AFAIK, IE honours the charset of the HTML page. That is, translates all characters in the paste buffer (whatever charset as this is handled by the OS) not in the charset of the page to entities. The one expection is that it treats iso-8859-1 as windows-1252, which the mentioned result of byte values in the illegal range.

I don't know if IE treats iso-8859-2 as windows-1252 in that regards.

Anyway we don't need to know the orginal charset of the paste buffer. If our charset is iso-8859-1 than the illegal byte range will contain windows-1252 characters. We have to test if this is true for other iso/windows charset pairs (eg iso-8859-2/windows-1252).

@Martin Kutschker, exactly.

we already have a "mapping" table in TYPO3, using the "script_to_charset_unix/windows" and matching families:

west_european : iso-8859-1 : windows-1252
estonian : iso-8859-1 : windows-1257
east_european : iso-8859-2 : windows-1250
baltic : iso-8859-4 : windows-1257
cyrillic : iso-8859-5 : windows-1251
arabic : iso-8859-6 : windows-1256
greek : iso-8859-7 : windows-1253
hebrew : iso-8859-8 : windows-1255
turkish : iso-8859-9 : windows-1254
thai : iso-8859-11 : cp874
lithuanian : iso-8859-13 : windows-1257
chinese : gb2312 : gb2312
japanese : euc-jp : shift_jis
korean : euc-kr : cp949
simpl_chinese : gb2312 : gb2312
trad_chinese : big5 : big5
vietnamese : - : windows-1258
unicode : utf-8

In my little researches, we only have this "problem" with Latin-1, which is a "pure" subset of W-1252. All other ISO vs Windows combinations are disjunct, so we cannot simply "fill the gaps" of illegal ISO characters with entities or translitarations from the Windows character sets.

E.g. Latin-2 is not a subset of W-1250, so a W-1250 browser should do conversions to Latin-2 anyway, else many characters will be simply wrong. I cannot imagine, that this is the case.

So we should decide:

1) what kind of "fix" do we need, when trying to store illegal ISO-8859-1 chars?

a) should we convert those characters to entities?
b) should we transliterate those characters?

2) when will we apply the "fix"?

a) At the point TYPO3 will save the data into the database (tcemain)?
b) Or just when getting backend form submission with inappropriate data (alt_doc.php)?

This is the "translit" done by iconv (from windows-1252 -> iso-8859-1):

128: Euro sign: EUR
130: baseline single quote: ,
131: florin: ?
132: baseline double quote: ,,
133: ellipsis: ...
134: dagger: +
135: double dagger: ?
136: circumflex accent: ^
137: permile: ?
138: S Hacek: S
139: left single guillemet: <
140: OE ligature: OE
142: : Z
145: left single quote: '
146: right single quote: '
147: left double quote: "
148: right double quote: "
149: bullet: o
150: endash:
151: emdash: -
152: tilde accent: ~
153: trademark ligature: (TM)
154: s Hacek: s
155: right single guillemet: >
156: oe ligature: oe
158:: z
159: Y Dieresis: Y

I doubt that this is what the users will want to see in their text-fields.

As of translating the illegal characters to entities, this is probably not what is expected in pure text-fields, which might go through htmlspecialchars before rendering (e.g. page titles in menus), and which would produce &#xxx; strings in browser output.

I guess we have yet another "simple" alternative: change the default charset from iso-8859-1 to windows-1252 in tslib_fe. Might freak out some Unix-guys, but I am sure any Unix-Client can handle that correctly by now and at least we will have valid pages with the characters that the user entered (€ = EURO).

BTW even mantis has windows-1252 as charset, which is probably why the € sign shows correctly. :)

Another "simple" alternative would be to translate illegal iso-8859-1 to corresponding entities only at render time, and keeping the iso-8859-1 charset. Which is more or less what my extension cron_latin1cleaner does.

@ernesto, I'm aware that only windows-1252 is a real superset of a iso codepage. Still we should test what IE really does in such situations.

As for transliteration: of course this is not what the user expects, but if you want to stay clean, you have to make a sacrifice. Or simply set forceCharset to windows-1252.

If you want to do the entity conversion during rendering is the question, when to do it (TS rendering, etc). But the easist and IMHO only practicable solution is to use the page postprocessing (what you do already).

But I fail to see the real problem. Who really cares so about those characters? And why do those not use one of the practical solutions available. 99.99% of the users don't care and why should be TYPO3 burdened with just another conversion.