Project

General

Profile

Actions

Bug #96057

closed

Typo3 does not handle UTF-characters from U+10000 and higher correctly

Added by Thomas Sam Wittich over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
Backend User Interface
Start date:
2021-11-23
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
10
PHP Version:
7.4
Tags:
Backend, UTF-8, Database
Complexity:
medium
Is Regression:
No
Sprint Focus:

Description

While checking out what's the problem with embedding Twitter Posts with emoticons in text, i found our that typo3 cannot handle UTF-characters correctly when saving them to the database:

  • Saving to the database throws error: 1: These fields of record <nnn> in table "tt_content" have not been saved correctly: bodytext! The values might have changed due to type casting of the database.
  • The database, table and column-charset are all UTF-8, General CI.
  • The Typo3-UTF-8-Settings for database connection and Locale are UTF-8 as well.
  • Typo3 cannot write to the database; when i insert the characters directly with SQL into the database, the frontend shows up correctly, while opening the record in the backend only shows up an "?" instead of the proper character; so Typo3 frontend can handle it, Typo3 backend not.
  • Typo3 has no problem with UTF-8-Character up to U+FFFF. These can be saved and worked with in the backend as well. The trouble starts with code pages from U+1000 and above. There is the split where Typo3 backend can handle UTF-8 (or is this UFT-16 then?!? Don't know that part).
  • Database analyzer is fine in this case - the columns have proper type (text oder mediumtext). Referring here to completely different cases with the same error, where sometimes this was an solution.

Please consider investigation into it and solve it.

Thank you,
Thomas

Actions #1

Updated by Georg Ringer over 2 years ago

  • Status changed from New to Needs Feedback

thanks for reporting the issue! Do you use a proper utf8mb4 connection? See https://docs.typo3.org/c/typo3/cms-core/master/en-us/Changelog/9.5/Feature-80398-Utf8mb4OnMysqlByDefaultForNewInstances.html for details

Actions #2

Updated by Thomas Sam Wittich over 2 years ago

Hi Georg,
thank you for that hint. It solved the problem. Had no utf8-mb4 charset, but just utf8.

Actions #3

Updated by Wouter Wolters over 2 years ago

  • Status changed from Needs Feedback to Closed

Suggestion from Georg resolved the issue for the reporter. Closing as solved.

Actions

Also available in: Atom PDF