TYPO3 Core

Unfortunately the transliteration is currently language independent. The vowels with umlauts are transliterated according to German custom. Which is fine for the huge German user base.

As a workaround you may change Translit.txt and and delete all files in typo3temp/cs.

Yes it still exists in 4.5-4.7. Haven't tried 6, but I suspect nothing has been done there either.

Still present in 6.2.11 I guess, the file /typo3/sysext/core/Resources/Private/Charsets/unidata/Translit.txt does not contain a "Scandinavian" section.

What about adding an option (in Install tool?) where to specify a custom file instead of modifying the original in the core?

Riccardo De Contardi wrote:

Still present in 6.2.11 I guess, the file /typo3/sysext/core/Resources/Private/Charsets/unidata/Translit.txt does not contain a "Scandinavian" section.

What about adding an option (in Install tool?) where to specify a custom file instead of modifying the original in the core?

This would be very handy. Now we have to manually update the Translit.txt file every time after updating TYPO3 source. As many of the developers are from central Europe, they don't understand the issue. And I think it's not an issue strictly to scandinavians, as the transliterations should really be language specific, not system wide.

Hoping this to be part of 7 LTS

This still seems to be the case. For a client this is a big issue in regards to speaking URLs that have (in their eyes) wrong letters in them.
The solution as provided earlier by Jan Helke seems like a nice solution, using the Install Tool to override the default transliteration files.

Back then the reasoning to not accept this was to rework the CharsetConverter completely, but now all trough version 4.x, 6.2LTS, 7.6LTS and now 8.x this has not been addressed.

Is it possible to reconsider the given solution?

Hey Bart,

maybe we can use a PHP library to do the work now that we cleaned up CharsetConverter big time, do you know any good places to look for?

We have now tested the new 9 LTS with Site Handling and Url Routing. Unfortunately the issue is still not completely resolved, but we are getting near!

Let me go through the issue by example:

- Editor creates a page in Finnish language, titled "Ääni" (that is "Sound" in Finnish)
- In the Page properties, TYPO3 shows generated URL Segment "/aeaeni"
- This is incorrect for Finnish (and Swedish, Danish, etc) language, so the editor must manually override the URL Segment as "/aani"

For pages, manually correcting all URL segments is manageable, but somehow irritating. However, if the URL contains content from other records such as News or an integrated Product database, there may not even be an opportunity to input correct URL segments and all the generated slugs are translittered wrong.

I have tracked the URL generation down to sysext/core/Classes/DataHandling/SlugHelper.php where the function sanitize is used to make slugs URL compatible. This function uses CharsetConverter->specCharsToASCII to convert extended letters to ASCII characters. No locale or language parameters are provided, and the CharsetConverter uses a hardcoded path to Resources/Private/Charsets/unidata/Translit.txt where the character replacements are defined. This file is only compatible with German way of spelling umlaut characters.

Do you have ideas how this could be improved so that we could somehow provide conversion tables for languages other than German? Essentially we still have to edit Resources/Private/Charsets/unidata/Translit.txt manually after updating TYPO3 and remember to clear typo3temp cache after that.

Some research notes:

- PHP has a Transliterator in the `intl` extension, but we'd need to provide custom rules there, too.
- Our Symfony `intl` polyfill does not polyfill the Transliterator meaning we'd have to introduce `intl` as hard dependency
- There seems to be no decent PHP transliteration solution covering our use cases out of the box

May be possible:
- Provide the possibility to register own translit.txt file in localconf to allow integrators to use custom transliterations

Susanne Moog wrote:

Some research notes:

- PHP has a Transliterator in the `intl` extension, but we'd need to provide custom rules there, too.
- Our Symfony `intl` polyfill does not polyfill the Transliterator meaning we'd have to introduce `intl` as hard dependency
- There seems to be no decent PHP transliteration solution covering our use cases out of the box

May be possible:
- Provide the possibility to register own translit.txt file in localconf to allow integrators to use custom transliterations

Just a short notice that symfony/string, which has been released just a couple of days ago, does a really great job, normalizing and converting string, respecting locales properly:

(new AsciiSlugger('de'))->slug('Näe ja koe')->toString(); // Naee-ja-koe
(new AsciiSlugger('dk'))->slug('Näe ja koe')->toString(); // Nae-ja-koe

We should evaluate how much own logic we actually need if using this library.
Could replace our current slugger and parts of the CharsetConverter (maybe even all of it).

Today IMHO all the transliteration is not needed at all. All the world is happy with Unicode.

Martin Kutschker wrote in #note-26:

Today IMHO all the transliteration is not needed at all. All the world is happy with Unicode.

That is a great solution, if we can use Unicode in URLs.
Modern browsers seem to support it, but to be sure the URLs should still probably be percent-encoded?

Just for the sake of completeness, transliteration doesn't only apply to slugs and URLs in TYPO3. It also applies to file names, where UTF-8 encoding may still not be working.

After some research, it looks like modern browsers will seamlessly handle percent-encoded URLs for more thorough l10n.

Symfony supported this as of version 6.1 in 2022: https://symfony.com/blog/new-in-symfony-6-1-improved-routing-requirements-and-utf-8-parameters

Just look around Wikipedia and you'll see this in action. This doc provides some examples: https://developers.google.com/search/docs/crawling-indexing/url-structure

Paul Hansen wrote in #note-29:

After some research, it looks like modern browsers will seamlessly handle percent-encoded URLs for more thorough l10n.

Yes that is absolutely true. At the same time, it doesn't remove the need for transliteration — especially where typing is involved. "That accented letter isn't on my keyboard" or "how do I type 漢?"

Mathias

Patch set 2 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 3 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 4 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 5 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 6 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 7 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 8 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 9 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 10 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 11 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 12 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574

Patch set 13 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/85574