TYPO3 Core

locallang.xml are converted into native charset before cache into typo3temp, even the xml is stored in unicode.

And if the page is render in UTF-8 or any other charset.
It will do 2nd convertion form native charset to utf-8, and do the 3rd convertion to the renderCharset.

As a result.
It clause datalost if the default charset of the locale do not contains the character. Also, there is a huge performance hit when you using alot of localized label in the page.

Solution suggested:
All locallang labels should cache in RENDER CHARSET.
locallang*.php will be cached like an locallang*.xml (If the php does not contains logic)

Because it can print the label directly from cache. The csConv can skip loading the charset mapping files. It will also save some memory.

In my test,
read EXT:chc_forum/pi/locallang.php and convert all labels into UTF8.

Typo3 takes 550ms if not cached.
Typo3 takes less then 10ms if cached(I still do include the source php and md5 the $LOCALLANG for doing checksum).
Typo3 takes less then 2ms if I convert it into xml by extdeveval.

--
Easten language is a problem, they are usualy have 3 charset encoding in current use.
e.g.
ch(Simpified Chinese) is using GB2312 in typo3 by default.
But, some character only exists in GBK (a newer version) charset
The result clause the final display may not same as expected from XML.

Also, there is another question. Typo3 default hk(Tradinal Chinese) is using Big5.
  But big5 is never a standardized by any offical organization at all.
  I wonder which version of big5 typo3 is using.
  FYI: http://my.opera.com/community/forums/topic.dml?id=104187
  I suggest typo3 update it to BIG5-HKSCS, it is already widely supported.

PS.
If you ask me for patch, I need some time to do that.
I have to extract that logic from my local version, it is mixed with my others customization.

(issue imported from #M2731)