Bug #16178
closedUTF-8: Broken special characters on cached pages
0%
Description
I believe this is closely related to ##0003303, but not exactly the same bug. I'm using Typo3 with UTF-8. In the frontend uncached pages are displayed fine, but as soon as pages are cached, some pages are displayed with '?' instead of the umlaut. Strangely this doesn't happen with all pages, just with two or three of around 20. When I clear the FE cache, all pages are displayed fine. As soon as the pages are cached, some of them are displayed wrong again. This error occurs on different pages after each time clearing the cache.
I'm using these options:
Typoscript:
config.renderCharset = utf-8
config.metaCharset = utf-8
config.additionalHeaders = Content-Type:text/html;charset=utf-8
Typo3ConfVars:
[forceCharset] = utf-8
[multiplyDBfieldSize] = 2
[SYS][setDBinit] = SET NAMES utf8 SET CHARACTER SET utf8
[FE][tidy] = 0
MySQL database:
MySQL version: 5.0.21
MySQL charset: UTF-8 Unicode (utf8)
MySQL connection collation: utf8_general_ci
PHP version: 4.4.2
(issue imported from #M3547)
Files
Updated by Martin Kutschker about 18 years ago
Should not matter, but your TS is overcomplete. config.renderCharset = utf-8 is all you need. metaCharset defaults to renderCharset and the HTTP header is also set automatically.
Updated by Valery Romanchev about 18 years ago
I have the same problem on TYPO3 4.0. MySQL 4.1
I get it after implementing
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8'.chr(10).'SET
CHARACTER SET utf8';
see http://bugs.typo3.org/view.php?id=1262
I have default charset latin in my.cnf
Database is in utf8
Important:
1) when pconnect is disable in localconf.php this problem appear all the time.
2) I get this problem one time when do update of guestbook record in TYPO3 List module. So it look like MySQL+PHP update problem.
Now I am trying to solve this by changes in my.cnf
I comment out log-bin and disable query cache -
by now site works OK.
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
- Default to using old password format for compatibility with mysql 3.x
- clients (those using the mysqlclient10 compatibility package).
old_passwords=1
default-character-set = latin1
- log-bin
query_cache_limit = 2M # default was 1M - query_cache_size = 64M # default was 0
query_cache_size = 0 # default was 0
query_cache_type = 0 # was 1
table_cache = 256 # default was 64
key_buffer_size = 64M # default was 8M
max_allowed_packet = 16M
max_connections = 300
[mysql.server]
user=mysql
basedir=/var/lib
[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Updated by Valery Romanchev about 18 years ago
Unfortunately disable of log-bin and query cache does not 100% solve the problem. Yesterday site works fine (after clear cache, after a lot of insert/update operations etc)
But just now I get home page with "????????" only insted of all russian letters.
The reload of the page solve the problem... But this not stable behavior is even worse then ablosutly not working!
On the same server I have many 4.0 sites without "SET NAMES utf8.... " - all site works fine every time (of cource without proper sorting and seach, because of no utf8_general_ci connection collation)
Updated by Valery Romanchev about 18 years ago
Now I use:
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;';
As I understand from http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html
we need only this.
Site works now.
Finally I use:
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET character_set_connection = utf8;';
it look like minimum settings - and it works for me.
Please test this, if you can.
Updated by Jan Wulff about 18 years ago
I'm using
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET character_set_connection = utf8;';
now and so far everything is working again. I supposed that it had something to do with the database, because the error seemingly occured at random.
Thanks for the info, great work.
P.S.: I'm a bit overcautious with Typoscript when searching for a bug, but I appreciate the info, Martin.
Updated by Valery Romanchev about 18 years ago
I get random problems with update with
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET character_set_connection = utf8;';
(sometimes I get ???? after update content)
So I turn back to
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;';
(and I do not see this errors with update)
I also check bugs at http://bugs.mysql.com/
(for "SET NAMES utf8" ) I find something, but not exactly this
So the problem is not easy.
Updated by Jan Wulff about 18 years ago
Sorry, then I have misunderstood you. I haven't had any new problems so far, but I will keep an eye on the website.
Do you mean this mysql bug report?
http://bugs.mysql.com/bug.php?id=19637
At first we should find out, if the problem occurs when the cache is written or when it is read, but I can only take a look at it, when and if the problem occurs again.
Updated by Alain Sch¤fer about 18 years ago
check if you have HTML tidy enabled in the Installer Tool.
I had a similar problem and $TYPO3_CONF_VARS['FE']['tidy'] = '0'
solved it. Or you might check the tidy documentation how to
run it correctly with uf8.
Updated by Dmitry Dulepov about 18 years ago
[multiplyDBfieldSize] = 2 is NOT needed here. It is only of your database is latin1 but content is utf-8.
Updated by Ries van Twisk almost 18 years ago
I had the same problem but it seems like the problem is a bit faugue in ANY CASE,
Here are my two cents.
Inded I had teh same problem... DB is set to utf-8 collation and encoding, setDBInit and all the stuff... Page generation was ok, but cached pages shows teh little ??
To solve it in my case I change the field HTML in the table cache_paches from blob to longtext with the correct collation (utf_8_general_ci, butter is to use utf8_unicode_ci, but this DB was in utf8_general:ci already).
For me this makes sence, ANY text field should really be in the correct collation and not in a binary (blob) format. What might happen in hour case is that mysql is going to do a latin1->utf8 encoding because the blob is handled as binary. Thus showing the?? because it's encoding again.
When storing teh data in the right encoding from the beginning we are 100% sure that mysql is not going to do a extra encoding.
thanks,
Ries
www.rvantwisk.nl
Updated by Martin Kutschker almost 18 years ago
Ries, please have a look at the current SVN code (trunk). AFAIR all BLOBs have been changed to TEXT fields.
Updated by Ries van Twisk almost 18 years ago
Hi martin,
I hardly ever look at trunk....
But I am glad to hear that all blob fields are finally changed to appropriate fields.
thanks,
keep up the good work,
Ries
Same issue is posted here : http://bugs.typo3.org/view.php?id=0003303,
Updated by Patrick Broens over 17 years ago
There are still BLOBs in system extensions:
./typo3/sysext/cms/ext_tables-index.sql
./typo3/sysext/cms/ext_tables.sql
./typo3/sysext/cms/ext_tables_static+adt.sql
./typo3/sysext/dbal/ext_tables.sql
./typo3/sysext/sys_action/ext_tables.sql
./typo3/sysext/indexed_search/ext_tables.sql
./typo3/sysext/impexp/ext_tables.sql
./typo3/sysext/tsconfig_help/ext_tables.sql
./typo3/sysext/tsconfig_help/ext_tables_static+adt.sql
I've already posted it to the core list to solve this. It's a pitty this didn't get into version 4.1
I ran into the same problems described in this bug and changing the BLOB to TEXT solved this issue
Updated by Kirill Klimov about 17 years ago
I could confirm the same problem (? appearing instead of text for some of cached pages randomly) on TYPO3 4.1, MySQL 4.1 with
$TYPO3_CONF_VARS["BE"]["forceCharset"] = 'utf-8';
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8'.chr(10).'SET CHARACTER SET utf8';
setting
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;';
has fixed the problem (at least I can't reproduce it anymore for quite some time).
Thanks to Valery Romanchev for the hint.
Updated by Ries van Twisk about 17 years ago
@Kirill :
Try to set it like this: $TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;'.chr(10).'SET CHARACTER SET utf8';
As far as I know these are SQl statements and the need to be separated by a ;
I don't know much about internals of setDBinit, may be the get split by chr(10) and executed seperatly. Or that they are send to mysql as one SQL statement.
If they are send as one statement. Would mysql not prevent this because I remember php's mysql client is buggy and can only execute one statement at a time?
Properly teh guru's can answer....
Updated by Marc Wöhlken over 16 years ago
We have different TYPO3 installations running on one dev-server.
The strange behavior of random pages being displayed with garbled characters appeared under very special circumstances:
- You have at least on installation using UTF-8
- You have at least one installation using another char encoding (e.g. iso-8559-1)
- You use the same(!) db-user and password to connect to MySQL
This setup resulted in random pages being garbled within the iso-8559-1 installation. UTF-8 was always fine. I suppose this was caused by a shared MySQL connection.
Solution: Use different db-users to separate the db connections.
Updated by Martin Kutschker over 16 years ago
Marc, another solution is to disable persistent connections. But of course this might be a performacen penalty.
Updated by David Bruchmann over 16 years ago
The Expression
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;'.chr(10).'SET CHARACTER SET utf8';
causes a MySQL-Error becaus it seems to be wrong!!!
I'm not sure about the delimiter, but chr(10) is not required I think.
I don't know if the rest of the SQL-statement should be written with "," or with ";" as delimiter.
The second statement I haven't found at all in the mysql-manual, instead I found the two statements with the same sense:
1) CHARACTER SET utf8
2) CHARSET utf8
As I understood the manual the right line has to be:
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8; CHARACTER SET utf8';
or
$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8; CHARSET utf8';
I don't know who head postet the wrong statement at first, but it can be found in many forums and nobody has checked it's syntax.
Nevertheless I have problems to save data in the backend and get the mentioned Error-message. Obviously it's hard to find anyone who can give a clear hint, but I'll check Martin's one.
Updated by David Bruchmann over 16 years ago
After some Experiments I found out that my Syntax
'SET NAMES utf8; CHARACTER SET utf8';
from above has to be right because it produces another result than
'SET NAMES utf8; SET CHARACTER SET utf8';
Latter produces the same result as omitting the last statement. The result from the first one isn't correct but it's accepted by MySQL, so the result from the second one only is right because the last statement is wrong and ignored by MySQL.
So, Valery's hint only to use 'SET NAMES utf8;' seems to be the best one.
Updated by Christian Wolff about 14 years ago
its still repoducable i have to currently 2 sites with this problem.
on different servers both seem to randomly change some cached pages
i use typo3 4.3.3
Updated by Ralle Büchnitz about 14 years ago
I can confirm this too.
I use Typo3 4.3.3 and have updated db to utf-8 and set all params to utf-8 as well.
Then randomly the pages are displayed correct and at another time there are problems with german-umlauts. It seems to be a caching problem, then after clearing all caches and "force reload all" from browser everything is fine.
For me helped the solution with 'SET NAMES utf8;
SQL-Server-Details: MySQL 5.1.44 on external Server
READ: http://wiki.typo3.org/index.php/UTF-8_support
// If you read the whole article, you will also recognize that some options mentioned above are not necessary.
Updated by Christian Wolff about 14 years ago
i solved the Problem by converting, the field HTML in the table cache_pages to text instead of mediumblob it semese that sometimes there goes somthing wrong with the UTF-8 in the blob.
after changeing the field type everything works fine.
@Chrstopher im pretty shure that i had tried SET NAMES utf8; and it din't change anything.
Updated by Ralle Büchnitz about 14 years ago
I tested it on my local server and solved nearly all problems with
SET NAMES utf8;
-----
After clearing all caches, deleting all temp_cached* files and logging out from Backend. The frontend looks fine! After logging out, typo3-frontend should open a new connection with the new dbinit-param. After this you can relogin and the backend looks good, too. - If you still have problems with specialchars - simply overwrite them / or at least some for testing purposes. Always clear cache after updating the wrong displayed signs.
-----
After relogin to backend and reloading the frontend some little problems remain viewing tt_news content. But this could be a problem of tt_news.
Updated by Steffen Gebert over 12 years ago
- Category deleted (
Communication) - Status changed from New to Closed
- Target version deleted (
0)