TYPO3 Core

Dimitri Koenig wrote:

tx_indexedsearch.php->markupSWpartsOfString($str) removes unnecessary whitespaces at the beginning:
[...]

But sometimes this produces ugly replacement characters (U+FFFD/65533).

Any solution?

i changed this to UTF-8 within a XClass-Extension


$str = str_replace(' ',' ',t3lib_parsehtml::bidir_htmlspecialchars($str,-1));
$str = preg_replace('/[\s][\s]+/u',' ',$str);

and ...

$parts = preg_split('/'.$regExString.'/iu', ' '.$str.' ', 20000, PREG_SPLIT_DELIM_CAPTURE);


what works for me. Might be a good idea to change this charset specific...

But there where a couple other issues with indexing or output of UTF-8 especially in chinese and '&nbsp' in general, e.g. cropping of chinese search result outputs strange (question mark) characters. the solution was allowing cropping only at whitespaces, not within words...

I changed in t3lib_cs::entities_to_utf8

some lines to UTF-8:


$param2 = (is_numeric(ENT_HTML5))?ENT_HTML5:ENT_QUOTES;
$entities = get_html_translation_table(HTML_ENTITIES,$param2,'UTF-8');


and in function t3lib_cs::crop:

@if ($i === FALSE) { // $len outside actual string length
            return $string;
        } else {
            if ($len > 0) {
                if (strlen($string{$i})) {
                    $string = substr($string, 0, $i);
                    $lastWhiteSpace = strrpos($string, " ");
                    //t3lib_utility_Debug::debugInPopUpWindow(array("last"=>$lastWhiteSpace,"length"=>strlen($string),"string"=>$string));
                    if($lastWhiteSpace){
                        $string = substr($string, 0, $lastWhiteSpace);
                    }
                    return $string . $crop;;

}
            } else {
                if (strlen($string{$i - 1})) {
                    $string =  substr($string, $i);
                    $firstWhiteSpace = strpos($string, " ");
                    //t3lib_utility_Debug::debugInPopUpWindow(array("first"=>$firstWhiteSpace,"length"=>strlen($string),"string"=>$string));
                    if($firstWhiteSpace){
                        $string = substr($string, $firstWhiteSpace);
                    }
                    return $crop . $string;
                }
            }@

I have teste the issue with all kinds of utf characters ( from http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt as well as the one mentioned in the ticket) and couldn't reproduce.

It might be the case, that this bug was fixed with PHP upgrade to 5.4. In 5.4 they changed the default PHP charset to utf-8.
Also functions like htmlentities, html_entity_decode, htmlspecialchars etc are now utf-8 aware by default.

5.4 has just reached end of life, so we can safely close this ticket.

Please let me know if the issue is not solved for you with modern PHP version.