TYPO3 Core

I can't reproduce this on my test system yet.

Did you activate the t3editor extension in the ExtMng, or did you see a simple textarea?
Do you edit the "whole template" record or just the Setup or Constants?

perhaps you can give us a step by step-way to reproduce this issue.

regards
tobias

Only the Setup area get trimmed.

No t3editor is not yet activated.

Step by step-way comming from a 4.1.6 typo3 installation:
- upload new source files
- change symlink
- activate install tool
- go to backend and perform db update
- raise functional level to 4.2
- clear all caches

=> go to the website and get typo3 error message

- go to BE and open template - find out that it was trimmed at line 55
- go the List Module and klick any page -> get the mysql error

In case of the templates I'm using a sysfolder to store them - so there is just a "Link Template" in the root of the website.

Could I change the severity and reproducibility tag of this bug? Seems that I was that hungry while posting this bug that I missed this settings...

The same here.
I upgraded from 4.1.6 to 4.2 without installation problems.
after the "empy cache", all templates were cut till line 23, even the extension templates!

What character encoding do you use inside the database?
Could it be, that the TypoScript contains umlauts ("ö", "ä", etc.)?
What do you get if you look with phpMyAdmin to the table "sys_template". The "Template Setup" part is stored in the field "config"? Can you see all data there?

Many fields of type BLOB have been changed to TEXT during the development of TYPO3 4.2. TEXT fields have a character set, if this is wrong, it might result in the behaviour you mentioned.

To track down the mysql_fetch_row() warning, please go to the Install Tool, select "All Configuration" and enable the setting for "sqlDebug". Next time something went wrong on database queries, there'll be a debug information which shows the query, the error message of the DBMS and a functional PHP backtrace.

character encoding of the database is as recomended: utf8_general_ci

The debug output of the SQL query already helped me to get rid of the errors. They were not related to the typoscript problem. (The tt_news extension - which is no longer installed on the system caused the error. After installing the latest version and uninstalling it again the error disapered - seems that delete extension don't mean delete???)

Arg ok, there is exactly one Umlaut in my typoscript code (needed to hard code one value into it) and thats the place where the code gets trimmed. After looking into the database I could confirm, that the code is really trimmed. So the answer to your question would be no, I couln't see all data there.

Seems to be the same problem, that I reported here:
http://bugs.typo3.org/view.php?id=8262

Additionally, check your database using Install tool. I have seen the case when database field length was not enough to store whole TS.

Are you using iconv as CS handling tool? Or maybe Mysql uses it. Iconv has the unfortunate habit of stopping the processing if it finds a byte code that isn't supported by the character set.

I can reproduce this on my test system yet.

when come "ü" cut after "ü" and "ü"

4.1.x -> 4.2
utf8
php5

Anything which I should check for you on my installation?

K. I have the same problem.

I am upgrading from 3.8 to 4.2. I noticed, that after a DB compare etc. and everything done, my FE breaks after the first umlaut. I then checked the DB and found that the data is still there and stored correctly. Notice that I haven't done anything in the BE yet (not even logged in).

I unfortunately cannot remember in the 4.2 development phase why this would happen.

oh, and yeah.

I haven't set any "forceCharset" or similar. Just a plain installation.

In case anybody was interested in the solution. Olly and I tracked it down.

In my case, I was porting over a 3.8 installation with a mysql 4.0 using latin1 coding. Then I exported it and imported the latin1 data into a new utf8 database.

Once I did the upgrading in the DB analyzer in the install tool, it stopped working. We found out that it is related to the change of "Blob" to "Text". If you leave this out in the updater, your Installation still works.

But this is not how we solved the issue... I then dropped my DB and created it again, this time with latin1 charset. Then I imported the DB dump (latin1 to latin1) and did the classic approach, with no setDBinit or forceCharset in the TYPO3. It all worked as expected.

Still, not everybody has the possibility to do this like me, so we need to find an alternative for 4.2.1 so people don't loose their FE when updating.

Benjamin, Michael Stucki has a guide how to migrate from latin1 (or any other charset) to utf8. Anyway, the main point is that you must change the BLOBs to TEXT columns before you dump the latin1 as utf8.

Alternatively you can dump everything as latin1 (as-is) but without the table definitions. The resulting file can be convert on the CLI with iconv (use windows-1252 NOT iso-8859-1!). If you create the empty DB with the definitions from 4.2 you can import the converted sql file right away.

Note 1: Both ways work ok for the BLOB columns that contain text data. Problems will arise with columns that contain serialized PHP arrays (they break) and columns with XML data already in utf-8 (double encoding will occur).

Note 2: setDBinit is necessery when the default charset for the server and the client is not what you need (see Mysql docs).

Hi
I have force charset set to utf8, database collation is set to utf-8, field constans and cofig are HTML columns in database, I have also SET NAME utf8 in db init.
My ts template is also trimmed after first occurence of polish character.
I have migrated from 4.1.3 version of typo3 with same settings as above.

TS is trimmed ONLY when using ts editor.

Bug still exists with T3 4.2.1

I don't think so.
I've made some tests on two diffrent website and all seem to work ok
This is my test Typoscript

10 = TEXT
10.value = ?ó

20 = TEXT
20.value (

?ó

)

30 = TEXT
30.value = test

I just ran into the same problem. When updating from 4.1.6 to 4.2.1.

All templates containing german umlaute in comments were destroyed, when the sys_template table is altered from blob to text.

This is a really serious bug. It completely breaks the website.

There should at least be a warning in the update wizard.

Hallo
gibt es schon eine Lösung?
Gruß
Bernd

UPDATE:

sorry folks I hope I did not create any work.
the error was on my site, the directory and DB dumps I was given where corrupt.

all systems up and running!

rgds,
GUIDO

<-- S O L V E D -->
i ran into the same problem today when I wanted to make a full backup from an online development installation and install it on my mac locally.

TYPO3 4.2.1
database online is charset and collation utf8_unicode_ci
I downloaded all directories and dumped the DB with bash mysqldump

on my mac local I created a new DB, again charset and collation utf8_unicode_ci
imported the DB with bash mysql command
modified in localconf.php db/user/pwd

I can login to the backend and even access frontend

unfortunately the baseurl has to be modified to point to myMac and not online domain
after saving the error occured on frontend

tried complete installatio/move again and just cleared cache ... bang
The page is not configured! [type= 0][]

the guy who setup the page has loads of umlaute in the templates
already in online instalation
$TYPO3_CONF_VARS['SYS']['UTF8filesystem'] = '1';
$TYPO3_CONF_VARS['BE']['forceCharset'] = 'utf-8';
where set from the beginning.

the problem is, that this is online on a development server and needs to be moved soon to the production server of the cutomer. as I cannot install it on myMac locally I am pretty sure it won't work on liveserver too.

please help me
rgds
GUIDO

i just got the same issue when updating from 4.1.3 to 4.2.5. my database is already utf8_general_ci but the content of the BLOBs seems to be originaly latin1 encoded.

the solution for me was to convert the BLOBs manually (NOT using the Database Analyser's COMPARE) so that they are not truncated.

i did it in three steps:
-- rename original field
ALTER TABLE `sys_template` CHANGE `config` `zzz_deleted_config` BLOB NOT NULL;
-- add new column
ALTER TABLE `sys_template` ADD `config` TEXT NOT NULL AFTER `zzz_deleted_config`;
-- copy original value to new column using utf8
UPDATE `sys_template` SET `config` = CONVERT USING utf8);

you have to do it for every single BLOB that the T3 Database Analyser wants to ALTER. i my case 23 columns.

after you did that empty the cache and check if everything's fine. now you're free to run the T3 Database Analyser and drop the the zzz_deleted_* columns.

i attached the SQL for all the columns i had to convert (maybe more or less for you?) in my database - maybe it will be helpful for some of you. before you try that make a backup of your database and check if that solution works for your system.

cheers!

ALTER TABLE `pages` CHANGE `media` `zzz_deleted_media` BLOB NOT NULL;
ALTER TABLE `pages` ADD `media` TEXT NOT NULL AFTER `zzz_deleted_media`;
UPDATE `pages` SET `media` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `config` `zzz_deleted_config` BLOB NOT NULL;
ALTER TABLE `sys_template` ADD `config` TEXT NOT NULL AFTER `zzz_deleted_config`;
UPDATE `sys_template` SET `config` = CONVERT USING utf8);

ALTER TABLE `fe_groups` CHANGE `subgroup` `zzz_deleted_subgroup` TINYBLOB NOT NULL ;
ALTER TABLE `fe_groups` ADD `subgroup` TINYTEXT NOT NULL AFTER `zzz_deleted_subgroup`;
UPDATE `fe_groups` SET `subgroup` = CONVERT USING utf8);

ALTER TABLE `fe_groups` CHANGE `TSconfig` `zzz_deleted_TSconfig` BLOB NOT NULL ;
ALTER TABLE `fe_groups` ADD `TSconfig` TEXT NOT NULL AFTER `zzz_deleted_TSconfig`;
UPDATE `fe_groups` SET `TSconfig` = CONVERT USING utf8);

ALTER TABLE `fe_users` CHANGE `TSconfig` `zzz_deleted_TSconfig` BLOB NOT NULL ;
ALTER TABLE `fe_users` ADD `TSconfig` TEXT NOT NULL AFTER `zzz_deleted_TSconfig`;
UPDATE `fe_users` SET `TSconfig` = CONVERT USING utf8);

ALTER TABLE `fe_users` CHANGE `usergroup` `zzz_deleted_usergroup` TINYBLOB NOT NULL ;
ALTER TABLE `fe_users` ADD `usergroup` TINYTEXT NOT NULL AFTER `zzz_deleted_usergroup`;
UPDATE `fe_users` SET `usergroup` = CONVERT USING utf8);

ALTER TABLE `fe_users` CHANGE `image` `zzz_deleted_image` TINYBLOB NOT NULL ;
ALTER TABLE `fe_users` ADD `image` TINYTEXT NOT NULL AFTER `zzz_deleted_image`;
UPDATE `fe_users` SET `image` = CONVERT USING utf8);

ALTER TABLE `pages_language_overlay` CHANGE `media` `zzz_deleted_media` TINYBLOB NOT NULL ;
ALTER TABLE `pages_language_overlay` ADD `media` TINYTEXT NOT NULL AFTER `zzz_deleted_media`;
UPDATE `pages_language_overlay` SET `media` = CONVERT USING utf8);

ALTER TABLE `static_template` CHANGE `include_static` `zzz_deleted_include_static` TINYBLOB NOT NULL ;
ALTER TABLE `static_template` ADD `include_static` TINYTEXT NOT NULL AFTER `zzz_deleted_include_static`;
UPDATE `static_template` SET `include_static` = CONVERT USING utf8);

ALTER TABLE `static_template` CHANGE `constants` `zzz_deleted_constants` BLOB NOT NULL ;
ALTER TABLE `static_template` ADD `constants` TEXT NOT NULL AFTER `zzz_deleted_constants`;
UPDATE `static_template` SET `constants` = CONVERT USING utf8);

ALTER TABLE `static_template` CHANGE `config` `zzz_deleted_config` BLOB NOT NULL ;
ALTER TABLE `static_template` ADD `config` TEXT NOT NULL AFTER `zzz_deleted_config`;
UPDATE `static_template` SET `config` = CONVERT USING utf8);

ALTER TABLE `static_template` CHANGE `editorcfg` `zzz_deleted_editorcfg` BLOB NOT NULL ;
ALTER TABLE `static_template` ADD `editorcfg` TEXT NOT NULL AFTER `zzz_deleted_editorcfg`;
UPDATE `static_template` SET `editorcfg` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `include_static` `zzz_deleted_include_static` TINYBLOB NOT NULL ;
ALTER TABLE `sys_template` ADD `include_static` TINYTEXT NOT NULL AFTER `zzz_deleted_include_static`;
UPDATE `sys_template` SET `include_static` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `include_static_file` `zzz_deleted_include_static_file` BLOB NOT NULL ;
ALTER TABLE `sys_template` ADD `include_static_file` TEXT NOT NULL AFTER `zzz_deleted_include_static_file`;
UPDATE `sys_template` SET `include_static_file` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `constants` `zzz_deleted_constants` BLOB NOT NULL ;
ALTER TABLE `sys_template` ADD `constants` TEXT NOT NULL AFTER `zzz_deleted_constants`;
UPDATE `sys_template` SET `constants` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `editorcfg` `zzz_deleted_editorcfg` BLOB NOT NULL ;
ALTER TABLE `sys_template` ADD `editorcfg` TEXT NOT NULL AFTER `zzz_deleted_editorcfg`;
UPDATE `sys_template` SET `editorcfg` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `resources` `zzz_deleted_resources` BLOB NOT NULL ;
ALTER TABLE `sys_template` ADD `resources` TEXT NOT NULL AFTER `zzz_deleted_resources`;
UPDATE `sys_template` SET `resources` = CONVERT USING utf8);

ALTER TABLE `sys_template` CHANGE `basedOn` `zzz_deleted_basedOn` TINYBLOB NOT NULL ;
ALTER TABLE `sys_template` ADD `basedOn` TINYTEXT NOT NULL AFTER `zzz_deleted_basedOn`;
UPDATE `sys_template` SET `basedOn` = CONVERT USING utf8);

ALTER TABLE `tt_content` CHANGE `image` `zzz_deleted_image` BLOB NOT NULL ;
ALTER TABLE `tt_content` ADD `image` TEXT NOT NULL AFTER `zzz_deleted_image`;
UPDATE `tt_content` SET `image` = CONVERT USING utf8);

ALTER TABLE `tt_content` CHANGE `media` `zzz_deleted_media` BLOB NOT NULL ;
ALTER TABLE `tt_content` ADD `media` TEXT NOT NULL AFTER `zzz_deleted_media`;
UPDATE `tt_content` SET `media` = CONVERT USING utf8);

ALTER TABLE `tt_content` CHANGE `records` `zzz_deleted_records` BLOB NOT NULL ;
ALTER TABLE `tt_content` ADD `records` TEXT NOT NULL AFTER `zzz_deleted_records`;
UPDATE `tt_content` SET `records` = CONVERT USING utf8);

ALTER TABLE `tt_content` CHANGE `pages` `zzz_deleted_pages` TINYBLOB NOT NULL ;
ALTER TABLE `tt_content` ADD `pages` TINYTEXT NOT NULL AFTER `zzz_deleted_pages`;
UPDATE `tt_content` SET `pages` = CONVERT USING utf8);

ALTER TABLE `tt_content` CHANGE `multimedia` `zzz_deleted_multimedia` TINYBLOB NOT NULL ;
ALTER TABLE `tt_content` ADD `multimedia` TINYTEXT NOT NULL AFTER `zzz_deleted_multimedia`;
UPDATE `tt_content` SET `multimedia` = CONVERT USING utf8);

As I had the same problem I was comming along to this thread.
After having now performed some steps, I would conclud that the problem is actually with the update to 4.2.x

The template is destroyed due to the cutoff of any text following any German Umlaut in the constants and setup area.

After several failures I have finally loaded a backup into a test environment (DB and source) and perfomed a test os the site. Everything worked fine; the template was complete (constants and setup).
So the loaded data in the DB which is stored in a BLOB must have been ok (couldn't check the BLOB content)

After having performed the update from 4.0.2 to 4.2.6 the page failed. As in any other tests before the template(s) where cut off at the first German Umlaut. This leads to an incomplete/damaged template leading to a failure in the generation of the page.

The problem is in the update of the DB where the BLOB is converted to TEXT.

Added:
The above solution works, when all BLOBs are converted manually according the above example BEFORE the Update Wizzard is executed. Run the Update Wizzard before to find our which Fields are effected.

I am having the same problem with update from 4.1.13 to 4.2.11, too.

I used a different approach that seems to work up to now:
ALTER TABLE fe_groups CHANGE TSconfig TSconfig text CHARACTER SET latin1;
ALTER TABLE fe_groups CHANGE TSconfig TSconfig text CHARACTER SET utf8;

The test DB I use also has more fields that were shown in the database analyzer to be changed from blob:
cache_pages.HTML
sys_template.description
sys_template.sitetitle
sys_template.title

The problem still exists in 4.3.x. I tried to update an old 4.1.12 installation to 4.3.2 and it truncated my TypoScript templates at the first german umlaut. On another installation, I updated from 4.2.10 to 4.3.2 and it broke.

There has to be a better solution than to manually convert every single blob field... and there should be some warning in the docs, I think.

I had the same issue! While saving large typoscript on a fresh install of TYPO3 4.2.2 in my root-page I got the error while saving.

The solution of Larsen helped me to fix it using phpMyAdmin:
------------
ALTER TABLE fe_groups CHANGE TSconfig TSconfig text CHARACTER SET latin1;
ALTER TABLE fe_groups CHANGE TSconfig TSconfig text CHARACTER SET utf8;
------------

This did it. Thanks a lot.

As pointed out, this problem occurs when you somehow store data with a charset different from the charset of your database. Then MySQL cuts the data at the first char not fitting into this particular charset.