Core

Dimitri Koenig wrote:

tx_indexedsearch.php->markupSWpartsOfString($str) removes unnecessary whitespaces at the beginning: [...]

But sometimes this produces ugly replacement characters (U+FFFD/65533).

Any solution?

i changed this to UTF-8 within a XClass-Extension


$str = str_replace(' ',' ',t3lib_parsehtml::bidir_htmlspecialchars($str,-1));
$str = preg_replace('/[\s][\s]+/u',' ',$str);

and ...

$parts = preg_split('/'.$regExString.'/iu', ' '.$str.' ', 20000, PREG_SPLIT_DELIM_CAPTURE);


what works for me. Might be a good idea to change this charset specific...

But there where a couple other issues with indexing or output of UTF-8 especially in chinese and '&nbsp' in general, e.g. cropping of chinese search result outputs strange (question mark) characters. the solution was allowing cropping only at whitespaces, not within words...

I changed in t3lib_cs::entities_to_utf8

some lines to UTF-8:


$param2 = (is_numeric(ENT_HTML5))?ENT_HTML5:ENT_QUOTES;
$entities = get_html_translation_table(HTML_ENTITIES,$param2,'UTF-8');


and in function t3lib_cs::crop:

@if ($i === FALSE) { // $len outside actual string length
            return $string;
        } else {
            if ($len > 0) {
                if (strlen($string{$i})) {
                    $string = substr($string, 0, $i);
                    $lastWhiteSpace = strrpos($string, " ");
                    //t3lib_utility_Debug::debugInPopUpWindow(array("last"=>$lastWhiteSpace,"length"=>strlen($string),"string"=>$string));
                    if($lastWhiteSpace){
                        $string = substr($string, 0, $lastWhiteSpace);
                    }
                    return $string . $crop;;

}
            } else {
                if (strlen($string{$i - 1})) {
                    $string =  substr($string, $i);
                    $firstWhiteSpace = strpos($string, " ");
                    //t3lib_utility_Debug::debugInPopUpWindow(array("first"=>$firstWhiteSpace,"length"=>strlen($string),"string"=>$string));
                    if($firstWhiteSpace){
                        $string = substr($string, $firstWhiteSpace);
                    }
                    return $crop . $string;
                }
            }@