TYPO3 Core

Hi Stephan,

Tnx for your research on this. This is indeed a problem. One I run into last week my self when a college uploaded some files through ftp.

Would be great if you could provide a patch for this to fix it and cover it with tests.

If you need some help with pushing a patch to our review system please let me know.

Gr. Frans

My current solution is a dirty hack without any tests. I'm a bit busy with other stuff, so I can't offer you something really useful for now; but approximately end of may I could take a look at this. I think i'll need some advices then, as I'm not yet familiar with the Gerrit-Flow. And I didn't sign the CLA yet.

Maybe someone else starts or even solves this tasks before end of may or I'll do so then …

Cheers,
Stephan

Hi Stephan,

could you share your "dirty hack" with us? It could be a good starting point for someone to create a patch.

gr. Frans

Well, I prepare my “dirty hack” to make it shareable. At least it try. Due to implementation specific issues, I'm not going to share it as it is, because I'm sure this hack raises more questions than it would solve. I'll describe some details as soon as I have something online, so please read my next posting if you are curious about this decision. And if someone really insists, because he/she needs to solve some (related) troubles right now or the like, you should contact me directly, and ask for the hack.

You can watch the implementation progress in our TYPO3.CMS fork's branch “feature-57695-fal-unicode-normalization” at github ( use this link if you prefer an up-to-date diff/comparison ). I strongly appreciate any feedback and will accept any helping pull-requests.

Btw.: Not a single line of code from the implementation above has ever been really tested or even run (as my original dirty hack is implemented in python and involves manipulating the “sys_file”-table from outside TYPO3). I'll mark the first working upcoming commit with “[ready for testing]”, as soon as it is “ready for testing” … :-)

Cheers,
Stephan

P.S.: I still expect (=hope) to be ready at the end of may 2014.

So, for now I'm stuck at the decision which strategy to implement first. Here are my current candidates:

1. Implement a converter for sys_file-entries and local filepaths respectively that forces all files to be in a certain unicode-normalization form
   - Pro: Avoids duplicate files or file-reindexing completely.
   - Con: Onetime overhead after moving TYPO3 installations between differently normalized filesystems.
   - Con: File-identifier based relations might break and must be reconstructed by some clever algorithm.
   - Con: Mixed environments (like caching-proxys or external CDNs in front of Mac OS X-based TYPO3-Servers) might cause new troubles ?
   - Hint: This is what my dirty hack currently partially solves - with python's excellent unicode support.
   - Hint: This strategy may result in a scheduler-task or an extbase-command.
   - My conclusion: I don't like this strategy, as I'm (we are ?) not able to predict all consequences, especially for the changing file-identifier.
2. Leave all filepaths as they are and force a particular unicode-normalization form just for the calculation or validation of the file-identifier.
   - Pro: Clever, smart and sounds simple. We just need to identify the best place(s) for this strategy to implement.
   - Pro: Should allow the mixed environments I mentioned above.
   - Con: If we fail to implement this strategy right, we (still) have the chance of duplicate files or file-reindexing, due to identical files which are just distinguishable by their file path's unicode-normalization. Especially Linux-based environments which usually don't care about unicode-normalization in file paths, might 'cause troubles.
   - My conclusion: This is what I'd like to go for - but not without any constructive feedback. I expect this path to consume some more time to implement, compared to the other strategy. At least writing proper initial unit tests cause headaches. :-)

• Python and ruby both have excellent unicode-support for a while now (at least longer than any PHP-based solution I know - sorry for this dumb assumption). Do they solve this issue and if yes, then how ?
• All Java-based frameworks and projects (Alfresco, Solr, Eclipse, Spring, Play) I evaluated suffer from this bug or don't tackle this problem - although java has the same unicode-normalization support as PHP has the programmers seem not to be aware of unicode-normalization. (Write once and run everywhere has always been a lie, though … ;-)
• I remember that samba once suffered from a (possibly similar) unicode-normalization bug, which turned out to be a missing unicode-normalization feature. Maybe we can learn from their solution ? (more research needed)
• We should not implement a solution that assumes that the whole filesystem has unicode-normalization-awareness. We should deal with the unicode-normalization like the FAL does it with case-sensitivity for filepaths. FAL checks each file-storage for being case-sensitive or not.
  - If we would check each filepath or at least its underlying folder for being unicode-normalization-aware, we could finally compose a (local) filesystem from (real) file mounts (not those from FAL) with different normalizations. This differs from the case-sensitivity-check, I mentioned above …
  - If we would check each file-storage for being unicode-normalization-aware, we could finally compose a filesystem from FAL's file mounts with different normalizations. This is quite similar to the case-sensitivity-check, I mentioned above …
  - The current setting $GLOBALS['TYPO3_CONF_VARS']['SYS']['UTF8filesystem'] is not enough to reflect/configure this kind of algorithm.

Does anybody have something to add to my thoughts ? Any strategy I overlooked ? Any obvious mistake I made ?

Hint: my two previous comments have been updated several times. So just for the case you're reading the emails only, it is advisable to read the online versions as well.

Stephan, thanks for your work on this - I'm keen on having a look into your Github changes. I guess others will do this as well. So let's collect the feedback here in this issue before creating a change set for the upstream repository.

Just for the record - here a (generated) list of open TODOs I'd still like to check. It might help someone to given suggestions or the like:

1. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/backend/Classes/Controller/BackendController.php#L233
2. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/backend/Classes/View/ThumbnailView.php#L373
3. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/CharsetConverter.php#L698
4. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/CharsetConverter.php#L748
5. Keep UnicodeNormalizer::filter method in sync with \Patchwork\Utf8\Bootup::filterString() in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/UnicodeNormalizer.php#L219
6. Really use workaround for https://bugs.php.net/65732, means to enforce unix-linebreaks, everytime and everywhere ! in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/UnicodeNormalizer.php#L222
7. Patchwork-UTF8 implementation handles cp1252 as a fallback too, but we don't do so. Is it ok for to fallback to plain utf8_encode ?!? in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/UnicodeNormalizer.php#L236
8. Use UnicodeNormalizer::filterInputArrays method during core-bootstrap ? If yes, then avoid double-encoding by TSFE->convPOSTCharset ! in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/UnicodeNormalizer.php#L275
9. keep method in sync with \Patchwork\Utf8\Bootup::filterRequestUri() in /TYPO3.CMS/typo3/sysext/core/Classes/Charset/UnicodeNormalizer.php#L466
10. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/core/Classes/Http/AjaxRequestHandler.php#L77
11. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/core/Classes/Imaging/GraphicalFunctions.php#L2662
12. Figure out if we need/want unicode-normalization as well … DONE Not really needed ! in /TYPO3.CMS/typo3/sysext/core/Classes/Localization/Parser/LocallangArrayParser.php#L148
13. Figure out if we need/want unicode-normalization as well … DONE Not really needed ! in /TYPO3.CMS/typo3/sysext/core/Classes/Localization/Parser/LocallangArrayParser.php#L155
14. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/core/Classes/Resource/Processing/LocalImageProcessor.php#127
15. Test all unicode-normalization forms, this seems to cover NFC only, see http://forge.typo3.org/issues/57695 in /TYPO3.CMS/typo3/sysext/core/Tests/Unit/Resource/Driver/LocalDriverTest.php#L1284
16. check if “list(…) = $parts;” is really not need below in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4709
17. Using SYS[unicodeNormalization] as fallback in FE would make sence, or not ? in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4862
18. Different normalizations for SYS (inkl. CLI+BE) and FE could introduce new troubles ?!? in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4863
19. Using SYS[unicodeNormalizeInputs] as fallback in FE would make sence, or not ? in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4875
20. Different lists of input-array normalization for SYS (inkl. CLI+BE) and FE could introduce new troubles ?!? in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4876
21. Implement SYS[unicodeNormalizeInputs] in TYPO3 bootstrap in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4877
22. Is Unicode::normalizeInputArrays too weak … and the better approach below ? in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4945
23. Why do we only convert $_POST charset ?!? Why not $_FILES, $_ENV, $_GET, $_POST, $_COOKIE, $_SERVER & $_REQUEST ? in /TYPO3.CMS/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L4956
24. Figure out if we need/want unicode-normalization as well … DONE implemented in $basicFileFunctions above in /TYPO3.CMS/typo3/sysext/frontend/Classes/Imaging/GifBuilder.php#L705
25. Figure out if we need/want unicode-normalization as well … in /TYPO3.CMS/typo3/sysext/scheduler/Classes/Scheduler.php#L432

Good Night,
Stephan

Hey,

just stumbled upon this issue and read through the description and all the comments. As I took a look at the charset converter just a few days ago, I wanted to comment on your last proposal: How about extracting the charset converter and your solution to one separate package for charset / unicode handling with PHP? I did some research and there is currently nothing really fitting on the market so such a package would provide a real value. I know that this ticket is a bit older, however if you are still interested in working on this kind of package, feel free to contact me (either here or directly on Slack or via email).

public static Normalizer::normalize(string $string, int $form = Normalizer::FORM_C): string|false

→ thus, we're implicitly using Normalizer::FORM_C (NFC)