Bug #93680
closed
wrong handling of utf8 of fal filenames when importing youtube videos
0%
Description
TYPO3 9.5.24
UTF8 Filesystem in Installtool active
de_DE-UTF-8 systemLocale in Installtool
Server Max OS X
The filename of the video contains an emoji:
Module filelist, add external url results in this response:
Added media
20_Unsere_Erfolgsstory__{emoticon - redmine cannot save it}.youtube
The file is on the disk with exactly this name - Mac OS X is able to manage this characters. But TYPO3 shows the wrong filenames in the file module:
20_Unsere_Erfolgsstory__ - the rest is missing.
So the systems and TYPO3 manages the download and the saving of the file correctly but the identifier in sys_file is wrong.
Updated by Guido Schmechel over 3 years ago
Test in v11:
- File is saved correctly, emoji replaced by "_"
- typo3/sysext/core/Classes/Resource/OnlineMedia/Helpers/AbstractOEmbedHelper.php throws an exception at the insert with "Incorrect string value".
Updated by Martin Kutschker over 3 years ago
Is the DB set to use utf8_mb4?
With the older mb3 none of the emojis can be saved. The reason is that mb3 does NOT cover the complete set of Unicode codepoints.
Updated by Riccardo De Contardi over 3 years ago
- Status changed from New to Needs Feedback
Updated by Peter Linzenkirchner over 3 years ago
- the video can be loaded but not played
- it is impossible to delete the broken video file in the file module of typo3
You can check it with this video: https://www.youtube.com/watch?v=FsJiVC2nBX8
Updated by Riccardo De Contardi about 2 years ago
I tried to reproduce the problem on the following:
- latest TYPO3 master 12.0.0-dev
- TYPO3 11.5.16
- TYPO3 10.4.32
Under the following conditions:
- I use TYPO3 on MACOS with MAMP
- The database collation is utf8mb4_general_ci
Test procedure¶
I created a new "text and media" content element and used "add media by URL"; The result is that the video is correctly shown on frontend
Moreover, editing the "Video Metadata" > "Title", the field shows the emoji
I checked Filelist > user_upload
- on the file list the video has been saved as 20_Jahre_ITC_GmbH_-_Unsere_Erfolgsstory______.youtube
- if I edit it, the title filed shows the emoji (*)
Then I
- removed the video from the content element (remove the sys_file_reference)
- Filelist > user_upload I tried to delete the video "file" and it has been succesfully deleted
(*) (this actually happened on the latest TYPO3 master because EXT:filemetadata was installed)
Is this sufficient or a different test is necessary?
Updated by Stefan Froemken almost 2 years ago
This is not a TYPO3 problem, this is a configuration problem.
First of all you have to set all DB tables and columns to utf8mb4_unicode_ci. That's needed to allow storing values like icons and smileys.
Second: You have to set DB charset in your LocalConfiguration to: utf8mb4. That's needed to transfer the real MB4 value (smiley) to the database. In many TYPO3 installations this value is still "utf8" which removes the 4th byte for smileys while query.
Third: If possible I prefer enabling SYS UTF8filesystem to 1
With that configuration the problem is gone.
IMO that issue can be closed now.
Updated by Riccardo De Contardi 7 months ago
- Status changed from Needs Feedback to Closed
There has been no feedback since a long time; plus, considering what Stefan Froemken (thank you) wrote, I assume that it is safe to close this issue for now.
If you think that this is the wrong decision or there is still work to be done please open a new issue with a reference to this one.
Sorry for the very late reply.
Thank you.