TYPO3 Core

could you test the attached patch?

there is an error with your t3d-file. I also tested with 4.1 and was not able to read the file

Well, if the file is broken, the bug is in the t3d export. I suggest fixing the export instead of supporting the import of broken files.

The export was made with the latest revision (3444) from the 4.2 branch, exporting the page tree starting at the site root.

I exported and imported t3d in 4.2 without any problems.

Hi David,
thanks for sending me the export file. It seems like the file is corrupted. But the interesting thing is: Why?
The first file part is defined to have a length of 6065 bytes, but really has a length of 8734 bytes. The data is compressed using zlib, thus I don't see a relationship to unicode/multibyte characters. I'm going to do some more tests on this issue.
What is the exact PHP version and your zlib/bzip wrappers you're using?

Here's some information from the phpinfo() page. What do you mean by wrappers?

PHP Version 5.2.4

ZLib Support enabled
Stream Wrapper support compress.zlib://
Stream Filter support zlib.inflate, zlib.deflate
Compiled Version 1.2.3
Linked Version 1.2.3

Directive Local Value Master Value
zlib.output_compression Off Off
zlib.output_compression_level -1 -1
zlib.output_handler no value no value

Do you have mbstring.func_overload enabled?
http://de2.php.net/manual/de/ref.mbstring.php#mbstring.overload

I guess so, since it delivers exactly the data in your export file:
strlen($data): 8733
mb_strlen($data, 'utf-8'): 6065

The attached patch does not fix or revert the PHP setting, but shows a warning in the install tool and adds a warning entry if a file seems to be corrupted in tx_impexp.

Oliver, thank you very much tracking this down. The mbstring.overload setting is indeed enabled on the server.

The proper fix for this is not to rely on that the length of a string is it's byte count because this is not true for unicode character sets. (Supporting them is one goal of Typo 4.2, isn't it?)

I suggest introducing a byte_count function using mb_strlen($string, '8bit') to determine the byte count of a string. (When the mbstring extension is not available it's of course safe to fallback to strlen.) What you want is not the length of the string but the byte count!

The mbstring.overload setting is very important for UTF-8 setups as it makes sure, that things like input validation continue to work. (Otherwise length checks for input fields will fail mysteriously when special characters are entered.)

Hi,
yes, supporting UTF-8 is a goal which does not mean to enforce it. mbstring is not enabled in PHP5 by default - there are also some distributions that require to install php5-mbstring explicitly as module.
Thus, changing all occurrences of "strlen()" in the TYPO3 Core to "mb_strlen()" is a no-go. But, if a developer wants his extension to be working also in unicode environments, he could e.g. use the following in his classes (front-ent as example):

if (function_exists('mb_strlen')) {
$length = mb_strlen($string, $GLOBALS['TSFE']->renderCharset);
} else {
$length = strlen($string);
}

We could put those hybrid-functions to t3lib_div which will determine whether to use single-/multi-bytes - but that's all.

I'm running a lot of UTF-8 sites without any problems and without the need of setting "mbstring.func_overload".

Thus, changing all occurrences of "strlen()" in the TYPO3 Core to "mb_strlen()" is a no-go.

I fully agree, that's exactly what the overload setting is for. Setting it requieres no further porting to utf8-enable an application.

The only special case is, when you need to cout the bytes a string uses (which is rarely needed, probably only in the importer).

I suggest introduce a byte_count function:

function byte_count($string) {
if (function_exists('mb_strlen')) {
// string may be encode with a multibyte character set
return mb_strlen($string, '8bit');
} else {
// string is probably 8bit encoded
return strlen($string);
}
}

Wherever you want the length of string you use strlen (which may have been overloaded to support your multibyte character encoding) and when you want the byte count you use byte_count. No furter references to any mb_ functions needed.

This works for all kind of setups, regardless of the installed extensions or used character sets.

As said before: The only thing that is needed, is to realize that the length of a string may not be it's byte count and thus treat it differently.

The overload setting is needed for the following case:

A username may only be 8 characters long.
When someone enters 8 characters of which some a special characters, the UTF-8 representation will be more than 8 bytes and thus rejected.
With the overload setting the string length will be determined correctly and it works as expected.

did you looked at t3lib_cs::strlen ?

David, of which username are you talking of? Neither BE nor FE users are restricted to 8 characters/bytes. But both are restricted to ASCII.

The comment was not about Typo itself but custom PHP code.

As the use of the mbstring overload setting is discouraged officially, this bug can be closed.

I suggest applied the above patch to warn when this setting is enabled.

Resolved, no change required as suggested by reporter.